The following article was submitted by Pat Wilcox ahead of her session at the Uplifting Voices: Conversations on Trauma & Recovery conference. This second part covers the methods and purpose of research on trauma-informed care. A part of "The Measuring Trauma-Informed Care Series" created by the Traumatic Stress Institute (TSI), this article has been abridged and separated to fit our blog. You can read part I, Measuring Trauma-Informed Care Across Sectors: What & Why, here.
***
Below are questions that you need to answer when constructing a TIC program evaluation. It is critically important to think through these questions in advance so that you get the information you want and do not waste valuable time and money. Program evaluation always takes more time and effort than expected, but advanced planning can make a huge difference.
It is critical to answer this question at the outset of the evaluation.
There are many possible purposes for TIC evaluations. Here are some of the most common.
We want to take a baseline measure of TIC to see where we are right now so that we can decide if and how we need to intervene.
We want to determine if we are ready for a TIC initiative so that we can either get started now or create the conditions necessary for successful TIC change.
We want to determine if and how our TIC intervention (TIC training, policy revision, trauma screening, etc.) is having an impact so that we can fine-tune our actions.
We want to determine if our TIC initiative made the people we serve better off so that we can determine if we are fulfilling our mission.
We want to assess where providers in our system are regarding TIC implementation so that we can determine how to help them become more trauma-informed.
We want to determine if our TIC change efforts are sustained over time so that we can decide if we need to recommit to TIC implementation.
Especially for organizations and systems that do NOT have a lot of experience with program evaluation and data, this is an important question to answer upfront. It does not make sense to spend time and energy collecting data when you do not know what to do with it once you have it. Organizations get excited about collecting data but underestimate the skill needed to use the data effectively and the time it takes to do so.
If you lack data expertise or the human resources to do program evaluation, there are now automated online TIC measures (ARTIC Scale, TICOMETER) that do the work for you. Or consider securing the services of a research consultant, local doctoral student, or professor to help you plan the program evaluation and manage the data.
There are three types of outcomes you can measure. Outcomes that examine:
To what extent your organization or system is trauma-informed.
Whether being trauma-informed impacts the quality of your organization or system’s services.
Whether the quality of the services makes people served better off.
Again, showing a change in the extent to which your system is trauma-informed is much easier than showing that people receiving services are better off. Our ideal, of course, is to show that people served are better off, but currently there is little empirical evidence that TIC system change can produce these types of outcomes. This does not necessarily mean that TIC does not positively impact people served. It may just be difficult to capture the change via measurement.
TIC experts believe that the intensity of the TIC intervention likely determines the type of change you can show empirically. Only a multi-pronged, intensive, sustained intervention within a healthy system is likely to show that people served are better off.
Therefore, it is wise to choose outcomes partly based on the intensity of your intervention.
Decisions about from whom or where to collect data (and other issues) may require approval through an Institutional Review Board (IRB) depending on your organization and the nature of your evaluation. Sometimes, program evaluation is regarded as different from research and does not require IRB approval. Check whether your organization has an IRB and what its requirements are.
If you are collecting data via a survey tool, you need to decide who you want to complete the survey. Is the survey for students, clients, parents, or staff? In an ideal world, you would want to give the survey to everyone in the school, organization, or system. But this can be expensive and time-consuming.
An alternative is to survey a random sample and work hard to get everyone in your sample to complete the survey. For example, if you have 200 staff in your organization, you can randomly select 75 to take the survey to save yourself work.
Remember that if you are giving a survey to children, you generally will need to get permission from their parent or guardian, and this adds another time-consuming step.
If you are collecting system data or metrics (school suspensions, restraints, staff turnover, permanency rates), it is easiest to choose metrics that you already collect rather than starting to collect an entirely new metric. Also, be certain that the metric you choose is a meaningful indicator of change and that it is, in fact, in need of change. For example, it is difficult to show change in staff turnover if staff turnover is already low.
Below is a table listing examples of commonly used target outcomes by category type and how these outcomes can be measured.
It is important to use psychometrically validated tools when possible, assuming that they fit the purpose of your evaluation. Using validated tools increases the likelihood that your results are, in fact, valid and not distorted by the tool being used.
Getting a high response rate to surveys is very important in program evaluation – and it can be very difficult. You can only be certain of your findings if your response rate is good. It is easier to get a good response rate if you have a captive audience (i.e., do a survey at a training, in a staff meeting, at a pre-arranged event aimed at giving the survey). If you don’t do it this way, you can still get a high response rate -- it just takes more work. Using an online survey can help to get higher response rates.
Think carefully about strategies you can use to ensure people complete the survey. These include prioritizing which survey tool(s) you use and time it takes to complete them at a minimum; getting leadership buy-in and having leaders urge participation; using incentives such as a raffle, gift cards, even a small monetary payment to drive participation; setting aside work time to complete the survey or get coverage to free staff; sending frequent reminders; begging and badgering.
If you are using a survey, you need to decide how many times and when you want to administer it to people. If you can, it is always good to do a baseline measure, ideally before any TIC intervention or any new intervention starts. Then, you want to decide when to give the survey after the intervention has started (i.e., every three months, every six months, at the end of each school year, after TIC training and three months later).
The closer in time you collect data to when the TIC intervention happens, the more likely you are to see change and to know if that intervention produced the change. Outcomes will likely show more change, for example, right after a TIC training happens rather than several months later because knowledge and attitudes tend to erode without intentional reinforcement. On the other hand, an increase in knowledge or attitudes may only be truly meaningful to you if you can show that they are maintained over time.
If you are tracking system data and metrics, you should ideally compile the data at regular intervals (i.e., monthly, bi-monthly, quarterly).
Here are some examples:
An organizational TIC task force that wants to monitor and make decisions about the change process.
Staff members who want to know that the TIC initiative is having an impact.
Leadership, boards of directors, and boards of education that want to show that the resources spent on TIC paid off.
Funders that want to show that their funding made a difference.
Referral bodies that want to know that TIC change efforts are working.
Larger service systems that want to show that schools or organizations have met requirements for a TIC mandate.
Reporting your findings to stakeholders is a critical part of program evaluation. Whether your results are what you hoped for or not, reporting results promotes learning in the system. Reports should be simple and visual so that anyone can understand them. A summary report or a PowerPoint presentation are excellent ways to present the findings. The ARTIC Scale online provides reports for you that you can share with stakeholders. Be certain to leave time for discussion of how stakeholders understand the findings and the implications for further embedding TIC in the system.
Measuring TIC is essential to understand your progress and challenges, to present your changes to the world, and to target further change efforts. Furthermore, it is essential to the trauma-informed care field to be able to demonstrate the efficacy of trauma-informed care.
Substance Abuse and Mental Health Services Administration (2014). SAMHSA’s Concept of Trauma and Guidance for a Trauma-Informed Approach. Rockville, MD: Substance Abuse and Mental Health Services Administration.
This blog post was excerpted from The Measuring Trauma-Informed Care Series created by the Traumatic Stress Institute (TSI), a global leader in TIC implementation and measurement. The primary author is Steven Brown, PsyDTSI, with Tulane University, created the Attitudes Related to Trauma-Informed Care (ARTIC) Scale, one of the most widely used validated measures of TIC. The ARTIC Scale measures professional and para-professional attitudes toward TIC.