• October 29, 2020
• COVID-19

As COVID-19 was rapidly sweeping the United States in late February and early March, cities were in the extraordinary and unprecedented position of needing to protect their residents' health and safety without comprehensive federal guidance or consistent epidemiological information about the new disease. With official information changing abruptly and misinformation about the novel coronavirus spreading almost as quickly as the virus itself, city officials knew they needed to have timely and effective messaging to slow the spread of the disease in their communities.

Confronted with the immediacy of the challenge at hand, how could city officials ascertain if their messaging was truly making an impact on individual behavior? Luckily, What Works Cities (WWC) expert partner the Behavioral Insights Team (BIT), a group dedicated to applying behavioral insights to inform policy and improve public services, was able to partner with 12 cities to test behaviorally-driven messages around COVID-19 health and safety to find the most effective and memorable messaging. The cities in the study not only learned about effective pandemic messaging, but also 1) how to apply behavioral insights to improve all types of municipal communications strategies, and 2) the value of rigorous evaluation.

“Mayors really felt like evidence based communications were a matter of life and death,” said BIT Senior Advisor Michael Kaemingk. BIT’s WWC team, led by Carolina Toth & Kelsey Gohn, were handling increasing requests about how to drive behavior in an evidence-based way. BIT turned to the WWC network, social media, and the team’s prior partnerships to find 12 cities that were interested in running a rapid messaging trial. To start, the BIT team had to learn each city’s individual goals: some wanted to emphasize mask mandates while others focused on stay at home orders, social distancing, or contact tracing. “Once we knew that goal, then we drew on the behavioral science literature to develop different message frames and content that we could test,” said Kaemingk.\

BIT also supported the cities by charting out the study details, including drawing samples, determining representative populations, and randomly assigning participants to each communication group. In the end, BIT helped the cities run 14 randomized control trials to help city staff identify effective messages, based on how well participants could understand and recall the messages, and how they plan to adjust their behavior (aka “behavioral intentions”) based on the messages. The trials ran from March 31 to May 26 and surveyed over 36,000 participants.

All the studies were run on BIT’s Predictiv platform, which meant that the team could recruit from across a broad variety of demographics. Kaemingk acknowledged that BIT usually tries to run experiments “in the field” with real outcome measures and random observations, but since these trials were done online, they could only report on behavioral intentions, instead of observing actual behavior changes. While the team knows measuring actual behavior would be best, given the urgency and nature of the challenge, utilizing an online platform that measures behavioral intentions meant that they could complete the trials faster.

“One of the main takeaways was the importance of actually testing the messages,” said Kaemingk. “We found there were some messages that worked in one city were not always the best performers in another. Or there were messages that cities expected to perform best that didn’t.” Gohn outlined one instance where a city expected an “appeal to duty” message to perform well because of the high concentration of military personnel; however, that wasn’t the best performing message for the area. Determining this through testing allowed cities to adjust their framing(s) and rethink their communication strategies to be more effective and impactful.

This same situation happened in New Orleans, Louisiana. “Going through this pandemic has underscored the importance of taking a behaviorally informed approach to messaging,” said Melissa Schigoda, Director of Performance and Accountability for New Orleans. “It’s really highlighted that different demographic groups sometimes respond differently to the messages.” Schigoda spearheaded the BIT messaging work in New Orleans, where they chose to test text messages instead of posters. The city already has the NOLA Ready system, which auto calls, texts, and emails residents information about hurricanes and floods; they are now using it to share information about COVID-19 cases and testing locations.

After Hurricane Katrina, which devastated the city in 2005, many residents expressed pride in their city with the slogan “New Orleans, Proud to Swim Home,” which appeared on bumper stickers and t-shirts. Schigoda and her team came up with the COVID-19 twist on that slogan, with a “New Orleans, Proud to Stay Home” message, which they assumed would be memorable and effective for local residents. Unfortunately, despite the clever play on the phrase, this message did not test well and didn’t seem to impact people’s willingness to shelter in place. Therefore, the city went with other, more effective messages, such as B (Duty).

In Kansas City, Missouri, the DataKC team was used to running randomized trials. However, as the pandemic strained the city’s workforce and stretched budgets, the data team knew they needed additional help to quickly produce effective safety messaging. Bo McCall, a Senior Performance Analyst at DataKC, reached out to BIT and collaborated on creating posters to use in the Predictiv trials. Similarly to New Orleans, Kansas City’s simpler messages worked best, not necessarily the most “clever” or designed ones.

For Chris Hernandez, Kansas City’s Communications Director, this trial reinforced his position on stripping away technical and government jargon. He is making sure that the simple and direct style is applied to all the city’s COVID-19 messaging, in order to maintain consistency and effectiveness. Hernandez says the city recently redesigned its COVID webpage, which had become a complicated and difficult to navigate subpage. The redesign uses the principle findings from the trials, even if the guidance around certain precautions has shifted. As Hernandez explained, “some of the specific messages have changed, and were changing even as we got the results of the study, but the idea of being direct and very clear and very simple in the messaging is something that endures — we can still use those as lessons.”

This is a perspective that Kaemingk hopes all cities will come away with. The data gathered through this kind of testing helps cities find what methods of communication are most effective for their populations, regardless of what the exact words are. He shared these thoughts as final takeaways for cities that want to improve their messaging, and in turn improve the health and safety of their residents.

“Even though there are relative best performers, we were able to pull out some overarching lessons that I think would be useful to any city or government developing these sorts of communications,” Kaemingk said. These tips are:

1. Reduce cognitive load: keep messages simple and use plain English to improve recall and understanding.
2. Put the key action at the top and make it extremely clear what people should do.
3. Shorten the messages: less text meaningfully impacted how well people understood and recalled the message.
4. Use images and graphics to help convey the information.
5. Rigorously test messages to determine what works.

Both New Orleans, LA and Kansas City, MO have achieved What Works Cities Certification at the silver and gold level, respectively. What Works Cities Certification is the national standard of excellence for data-driven government— helping cities benchmark their progress and develop a roadmap for using data and evidence to deliver results for residents.