Demystifying the Data Systems: Measuring What Matters

For the last several years, the Harvard Kennedy School’s Government Performance Lab (GPL) has been helping jurisdictions implement and scale alternative approaches to public safety through its Safety and Justice portfolio. So far, the team has provided technical assistance to 35 jurisdictions across the country with the goal of reducing harmful resident-system interactions, such as arrests or incarceration. 

As the GPL helps build this new dimension to emergency response, staff identified a lack of data guidance as one of the biggest issues. As a new field that combines many different systems, departments, and agencies, defining, finding, sharing, and tracking data is uniquely difficult.  

Thanks to a new GPL toolkit Measuring What Matters, written by Marianna Yamamoto, Gabriela Solis Torres, and Nya Anthony, cities have shared standards and language for data collection. This is incredibly important, since alternative response programs exist to safely divert 911 calls away from law enforcement and prevent unnecessary trips to the emergency room, especially for non-violent and behavioral health calls. Jurisdictions are testing out these new programs in the hopes that they may reduce “unnecessary criminal justice system involvement” while creating “more equitable outcomes for communities of color, people with serious mental health disorders, and others disproportionately impacted” by the criminal justice system. But without reliable data, these teams can't validate their work and, as city budgets tighten, that reliable data matters more than ever. As Solis Torres said, "it's really difficult for an elected official to support a program" that can't show its impact. 

Now, for the first time, the Measuring What Matters toolkit allows city leaders to develop clear, reliable, and accurate metrics for their alternative response programs, and, ultimately, strengthen and expand the field. 

Why Jurisdictions Need this Tool 

There are Data and Process Problems 

A data or technology issue can often reveal underlying structural and process issues. For Solis Torres, this quickly surfaced in conversations with municipal leaders; one example is that many existing systems don’t have data fields to collect information about alternative responses. To address this, alternative or co-response teams create their own metrics and data systems that don’t necessarily connect on the back end or align with existing data definitions. This reveals a fundamental problem: jurisdictions with alternative response programs haven’t fully integrated their teams into the broader emergency response ecosystem. 

Thanks to the Measuring What Matters Data Dictionary, jurisdictions now have access to 56 data fields across four data sources essential for effective data collection. By developing shared definitions for each data source and field, suggestions for common locations, and explanations for how the fields interact, jurisdictions can address the data, structure, and process issues together.  

While attempting to manage the issues above, many teams have developed their own shorthand or workarounds. Solis Torres explained that best practices might not always be achievable for all jurisdictions, so the tool provides a few different options for places that aren’t able to change their tech systems or upskill 911 responders. "We wanted to give program leaders different ideas for how to collect some of the trickiest data," she said. "We know some data can be challenging to gather, but everyone should be trying to do their best." 

For some, this means making technical changes to add data fields, while for others, it’s entering a hashtag into the notes to create a searchable term; the GPL team supports jurisdictions using whichever one they can commit to.    

Tracking Systems Focus on Incidents, not People 

Another interesting point Solis Torres made was around the ideological differences between systems. When a call comes into a 911 call center, the telecommunicator uses the computer-aided dispatch (CAD) system to track the incident. However, the alternative response team’s records management system (RMS) typically tracks the person.  

This encapsulates one of the central tensions between the traditional and alternative response programs and how they’re evaluated. As Solis Torres explained, Police and Fire program managers must answer questions about solving or responding to incidents, while alternative response managers are asked ‘how are you connecting people to services?’ or ‘what are the outcomes for the people you help?’.  

This mismatch is a practical problem when it comes to actually linking the two systems together. Every 911 call gets a unique CAD incident ID, automatically generated the moment it enters the system. But unless the alternative response team also logs that same ID in their own RMS, there's no way to connect the story of the call with the story of the response. When asked what the most common data problem she encounters is, Solis Torres pointed to exactly this: "when folks are building their systems, they don't bring across the incident IDs." Without it, she said, "it can get really hard to connect the dots." 

Answering both questions is important, but again, the current system fails to capture or evaluate the nuances of alternative response. The new toolkit recommends that alternative responders track incident dispositions as closely as they can to the other emergency responders for better and easier comparison; then, after tracking it in that language with the same ID, they can use an “encounter outcomes” field to add service metrics and details.  

While that does add additional work for the alternative responders, they’re already used to putting together reports and many teams build dedicated report-writing time into their shifts. And since most teams are dispatched in pairs, one person can type and the other can focus on talking with the person in need. As jurisdictions integrate these data fields into their technology and processes, the GPL recommends thinking through streamlined data entry – for example, using check boxes or drop-down forms to quickly and consistently track outcomes.    

Data Alignment is Retrofitted 

One of the clearest examples of this problem is what Solis Torres described as a common blind spot. "We see sometimes that teams will report all the calls that their team went on," she said, "but not how they received the call." This is an important distinction to track, because alternative responders can be dispatched directly by the 911 call center, or called in later by law enforcement after determining a situation doesn't require an officer.  

Tracking how the team is sent out matters for two reasons. One, it indicates a potential mismatch between what call centers think is appropriate for police and what police think is appropriate for alternative responders. And two, since these programs exist to divert calls away from law enforcement in the first place, a secondary response might not count as a successful diversion.  

It's a distinction unique to alternative response, and some programs try to untangle this after the fact by comparing arrival timestamps between responding units on scene. However, that can be even more difficult in places where alternative response teams are run by contracted nonprofits, or where call centers themselves are outsourced. In those cases, Solis Torres said getting the data right starts with what a jurisdiction asks for at the procurement stage. For example, Los Angeles contracts with three separate service providers to run its program, but because Los Angeles set clear expectations in the RFP, all three use the same data system and collect the same metrics. 

Solis Torres hopes program managers can use the new toolkit to do this. "They can take this tool and say ‘here are the metrics I want,’" she said. Building that alignment in from the start, rather than retrofitting it later, is often the difference between a program that can demonstrate its impact and one that's left trying to reconstruct metrics. 

Call Centers are Brought in Too Late 

Finally, when asked about data sharing, Solis Torres flagged that a common mistake is bringing in the 911 call centers too late. While a pivotal aspect of any emergency response, alternative response groups often develop through advocacy or at the request of an elected official, so the 911 call takers and dispatchers are not included from the start.  

“One of the things that we always do, right when we're starting a project, is talk to 911,” said Solis Torres, “We need to get them up to speed because 911 is such an important player.” Jurisdictions need to retrain staff on the new alternative response option, update protocols, and introduce new data collection. The toolkit helps explain the work while establishing the shared definitions and flagging when some of that data is already collected.  

According to Solis Torres, “the earlier you can bring them on the better,” since many people underestimate what this will take. “I think a lot of times people simplify the role of 911,” she said, “they think nothing really changes for the staff, but oftentimes they’re providing a lot of support…without additional funding.”  

It's worth noting what this toolkit doesn't cover. As Solis Torres explained, it’s deliberately scoped to the crisis response encounter itself, not including follow-up care, Medicaid or insurance requirements, or longer-term outcome tracking. For teams looking to measure what happens after that initial encounter, those pieces would require their own set of standards. 

Looking Ahead 

The diversion rates for these teams are still quite low compared to 911 responses, especially since they often have very limited staff and hours when compared to 24/7 fire and police responses, although that number is growing with the increase in alt/community responder programs. According to research from the Law Enforcement Action Partnership, these programs “are now responding to more than 230,000 calls per year, with zero fatalities or serious injuries” nationwide. This is why it’s so important for alternative response programs to work early and directly with 911 call takers and carefully track outcomes to prove their worth.   

“In terms of professionalization of a field and long-term development, being able to have some shared standards and shared definitions is important,” said Anthony, “just like fire departments have national standards for uniforms and response times, alternative response programs have to start thinking about these things too.”