Moving the Needle with Better Metrics and Smaller Experiments

Most cities operate on assumptions that were handed down years ago and consider frontline workers as executors of top-down decisions. In this episode, San Francisco's former Chief of Strategy and Performance Jessica MacLeod shows what happens when you treat them as problem-solvers with lived expertise and new ideas. Add visible systems mapping, permission to experiment, and data that has narrative, and suddenly government can move at the speed of actual change.

In this episode, you'll learn:

• Why frontline stories unlock patterns dashboards miss—and make data actually meaningful
• How mapping hidden assumptions lets you test what actually works versus what you've always done
• How brighter lights reduced drug complaints 54% in one park, then scaled across the city
• Why small experiments at every level create change makers faster than top-down mandates
• Why empowering staff to solve problems (not interrogating them on numbers) unlocks actual performance change

Listen here, or wherever you get your podcasts. The following is a transcript of the conversation. 

Stephen Goldsmith:

Thank you and welcome back. This is Steve Goldsmith from the Bloomberg Center for Cities at Harvard University. I'm joined today by Jessica MacLeod, who recently, kind of recently, was appointed by Mayor Daniel Lurie as San Francisco's, get ready for this title, Chief of Strategy and Performance. Jessica spent a lot of years across government and in civic tech and was the co-founder of the US Digital Response during the pandemic. Welcome, Jessica.

Jessica MacLeod:

Great to be with you. Thanks for having me.

Stephen Goldsmith:

Thank you. Well, you've got a lot of background and most of it is really relevant to the current job. So talk us through a little bit what your background was and how you think that relates to your current job.

Jessica MacLeod:

Sure. Like you mentioned, I've been in the government technology, civic tech space for about the last 18 years, both in the public sector side - I was the director of digital services and technology for the city of San Rafael - and then also on the private sector side, running customer success for several different government technology companies. And as you mentioned, had the real honor of being one of the several co-founders of the US Digital Response back in 2020 when COVID-19 hit. So that was a great experience during a huge crisis, but to see the whole community come together and be part of that effort.

Stephen Goldsmith:

Well, you have an enormous amount of tech talent in that city administration, maybe more than at least almost any other city. So in that constellation of people with important titles, what is Chief of Strategy and Performance? Those two are not necessarily always together.

Jessica MacLeod:

This is a really interesting role. It's all about helping our city get really concrete about what the top priorities are, how we define what success looks like and how we create an evidence-based plan to achieve it in creative new ways. The second piece of it is building a culture and operating norm to get teams who don't normally come together to solve problems, doing so, collaborating, experimenting, using technology, using new innovation techniques to try things, double checking their assumptions and getting more data driven about how they make progress against our citywide goals.

Stephen Goldsmith:

How do you think about your job as it relates to other folks who are experienced in technology and have other responsibilities? I'm thinking about, I was Deputy Mayor for Mike Bloomberg and DoIT reported to me and it was a service agency for the other agencies, but also was the protocol setting agency, right? It was a rule enforcer and a service agency. So what is your role, your agency's role with respect to servicing, providing, working with the other departments?

Jessica MacLeod:

Well, in our Mayor's Office, we have an interesting structure. It's a very large city. So even though we're only 49 square miles, we have over 32,000 employees. We have over 52 departments across the city. We're the only city and county in California. And so we have a unique structure in the Mayor's Office where we have a chief's structure. So when I came on, there were already four other Chiefs in place, all overseeing specific portfolios of work in a policy vertical such as Health and Human Services or Public Safety. And my role was brought on to really be the horizontal layer to help us achieve goals across all of those different verticals. So make sure that we are operating as effective as possible and we're coordinating well with our partners across the city to deliver in the best way that we possibly can.

So a lot of that also looks like working closely with our central technology teams, whether that's our Department of Technology, our Digital Services Team, our Data SF team, and our Emerging Technologies team. So it's been great to get exposure to all of these different ways of working and also try to create some consistency and some uniformity to it.

Stephen Goldsmith:

You and I have spent a lot of time talking about Stat programs, but as you think about your job, the performance measures or the causes of better performance may be scattered across agencies, not just in an agency. And the more intractable the problem, think homelessness or the unhoused, will have multiple drivers. So how do you define performance across multiple agencies in a way that can make an impact?

Jessica MacLeod:

Well, just to say, when I came on board, we already had citywide scorecards that our controller's office publishes. So lots of great metrics that are already out there and available to the public. Several of our departments have strategic plans. They publish their own goals and objectives. But to your point, most of the most complex challenges that we face cannot be solved within one single department. They require multiple different departments coming together to look at different aspects of the work and identify what their piece is in helping to achieve a shared objective and then coordinating that's going to allow them to do that hand in hand with other folks that they may not be used to collaborating with.

So when I joined, we reviewed and refined the Mayor's priorities and set specific objectives around those priorities and then worked with each chief to name specific strategies that we would have to develop to achieve those goals, whether that's in public safety, reducing retail theft or reducing the presence of our public drug activity, and then would gather leaders from all different departments, whether that's our neighborhood street teams, our public health leaders, our public works managers that are out in the community, all of the staff that would touch those issues to create these cross-departmental teams.

And those groups are really built around one specific named condition in the city with a clear owner and a clear way to measure progress. So the questions that would guide the work of those teams are really simple. It's "what are we trying to change or improve? Who owns that? Who owns what aspects of it? What is the evidence telling us about how we're making progress or not making progress? What can we learn about the problem so we understand it more deeply? And what do we need to change this week to get things moving in the right direction?"

So we put some specific or some really basic dashboards in place for internal use, mostly starting with the open data that we already had broadly available so that data sharing wasn't a concern or a barrier for us, and then just began working with departments to help them align their own internal strategies to achieving those goals as well. So what are the tactics they're deploying? How do they think about making sure that their work is more aligned with what their partners are doing across the city?

Stephen Goldsmith:

So if you're involved in performance measurement, then I would assume you're also involved in performance improvement, right?

Jessica MacLeod:

Yes.

Stephen Goldsmith:

So let's think about: what if the answer that drives performance involves causation? How do you use your AI tools across five different agencies to figure out the cause of a drainage problem, the cause of a pothole problem, the cause of a fill-in-the-blank? How are you thinking about performance improvement as it relates to causation and analytics?

Jessica MacLeod:

We think about this a lot. So it's one thing to come in and build a bunch of frameworks and help people set goals. And it's another thing to actually give them the tools to understand what the right path is to making progress against those goals. And we didn't want to wait years to get to that point. So we've been building the plane while flying it, and we've really started with the groups that have data that update the most frequently because that gives us the tightest feedback loop around where interventions are making a difference or not.

For example, we have one group that's focused on setting objectives to improve safety in specific neighborhoods of our city where the sense of resident safety has really declined. We see increases in 311 calls around public drug use and what's tagged as sort of homelessness related concerns or encampments. We see 911 calls related to property theft and things like that. So those are the areas where we really wanted to learn more about what's going on there and determine what interventions are going to make the biggest difference to help those numbers go back in the right direction.

And when those groups had gotten together previously, again, these are maybe six or seven different departments sending representatives that are close to the work on the ground, hyper-local management staff and line staff that are dedicated to a specific neighborhood, police captains rather than the police chief, that sort of thing. They know the neighborhoods really well and they deeply understand the local context. And so a lot of the work that they were doing ran on anecdotes. "This is what we saw last night. This is what we saw last week. Here were the biggest issues and here's what we're going to do about it next week."

But we couldn't really say with any certainty in the early days that the things that we were doing were moving the needle in a sustainable way. It would look better the next week, but maybe things would decline again. So interventions like foot patrols and outreach visits and park cleanups were tracked, but they were really scattered in spreadsheets and emails and we could confirm that work had been done, but we couldn't quite tie that causality to it, that this is the positive impact we made. So what we've done is we've looked at those specific call types that we see in those communities and we bring that dashboard up in these meetings and we ask people who walk those streets every day to share their lived experience; "What do these numbers mean to you in real life? Who are these folks that you're seeing? Are they the same people that you see week over week? Are the people that we're seeing calling in the morning commuters? Are they people taking their kids to school? What's the nature of the challenge?"

And then when we look at where there's an opportunity to run an intervention, let's say for example, in one of our neighborhood safety groups, we had a particular park where drug use was concentrated in the overnight hours and our Rec and Parks Department were brilliant. They just increased the luminosity of some of the lights that they put in the parks and they made them brighter. And we were able to track that the drug related complaints in that specific park during the peak hours dropped by 54%. And weeks later, we took that same intervention that worked in that park and we scaled it to one of our BART plazas where we're seeing concerns overnight as well. So we're able to actually scale what works and learn and test and try new things.

We had another similar situation in a different neighborhood in the Mission District where we increased cleanings of specific parks. We dedicated a park ranger presence, added a couple of new security cameras, and we saw drug related 911 calls go down by 65% and homelessness related concerns go down by 71%. Overall, I think my philosophy on how do you get to causality and how do you help drive true performance is you create this culture of curiosity and testing and learning. From a human perspective, nobody performs better when they feel like someone's looking over their shoulder and evaluating their work, especially if the right answer isn't clear. But the antidote to that fearfulness is a culture of curiosity, feeling like you can safely ask good questions, you can try new things, and we can run these interventions on really short cycles and see what the results look like in the data.

Stephen Goldsmith:

What a wonderful answer. Culture of curiosity is what we've been evangelizing about. We've been looking at a couple places about how democratized or decentralized access to data through natural language inquiries, layered data, generative AI questions like GeoAI questions, "show me this, show me that," involve and unveil more curiosity across the organization. How far away do you think you are from the place where expertise will be dramatically decentralized with respect to curiosity?

Jessica MacLeod:

Yeah, it's such a great question. It's something we are really pushing for, of course, with the right safeguards in place, but I want to give a huge shout-out to my Assistant Chief Zoya Khan, who's been helping us bring in AI tools into our performance work and has worked with great folks, including a software engineer on our innovation team, Aaron Hans, and they've been working closely with our Assistant Chief for Public Safety on how to pull and analyze real-time public safety data across multiple police districts and compare conditions to prior periods, identify where things are improving and where they're not. So we have maps where you can see where displacement is happening, for example, after an intervention, which was something that was just anecdotal before and really hard to track. So where are certain neighborhoods heating up? Where are they cooling off? How can we start to see patterns across that work and get ahead of it a little bit more?

They're also able to surface hotspot patterns that just aren't visible in aggregate numbers that you would see in a typical Power BI dashboard in a Stat program. So for example, that flat district-wide encampment total masked the rising conditions in three different neighborhoods, and we are able to actually point to displacement rather than resolution. So rather than saying we've solved the problem, we can see that the problem is moving, we can see where we need to go next and we can better understand what the needs of those folks are that are moving into different parts of the city.

Lastly, the one thing I'm really excited about is not just democratizing access to data, but democratizing the skills to ask really good questions of data because that's one of the harder things to learn, I think, that really stumps people when they get into performance meetings. It's one thing to look at patterns and trends and see hotspots, but how do we pre-build data-driven questions into each meeting agenda so that the time that everyone's spending in the room is spent on making good decisions and gleaning really interesting insights? We're doing all of that today and we're using AI for it.

Stephen Goldsmith:

These are such fun answers. You could show the way for the country. All right, let's get down to the important stuff. Tell us about how data makes trash pickup better.

Jessica MacLeod:

I love this story. This is one of the first things that my Assistant Chief and I worked on when we first started. We had not even really built out the full infrastructure around performance management yet, but we wanted to demonstrate an example of what using data to help make more evidence-based decisions in a resource constrained environment looks like. Because oftentimes we'd go into these conversations with different departments and they would say, "We are so underwater in terms of putting all the resources out there. We're spending the money that we've got. We don't have any additional resources. We have to figure out how to make this problem go away, but we simply can't throw anything more at it."

And so, one of the biggest priorities when Mayor Lurie came in was to ensure that our residents have clean streets to walk on and take their kids to school on and commute through. It's just so fundamental for our city. And we had really gotten a reputation during COVID for our streets being in pretty bad shape. Things have improved quite a lot over the last couple of years, and our Public Works Department, huge shout out to our Public Works Director, Carla Short, who's an amazing leader and has done an incredible job in putting resources out there and keeping our streets clean every single day. But there were still challenges that she and her team were facing that we needed to find a new approach to.

So we started structuring the problem in a new way. What was every possible root cause with no overlap of why there was trash on our streets in the morning? Before we can test anything, we had to know what we were testing. We had to figure out what question we wanted to ask and what levers we were going to have access to pulling on this issue. And so when we though about why our streets were unclean, it was either people aren't disposing of trash properly or the trash just isn't staying where it should. And this is where the data initially pointed us. We have about 5,000 city trash cans across San Francisco. So either the cans are in the wrong spot based on where people need them or they're the wrong types of cans or they're just not being prioritized for pickup in the most impactful way and serviced in the most optimal way. And this is really what 311 showed us.

We didn't wait for super perfect data. We started with historical data of the number of overflows across all cans so we could see which ones were our "highest offenders." And then we looked at where there are daily, weekly, seasonal kind of patterns and overflows so we can understand the nature of the challenge. And then we really looked at sort of this 80/20 principle, what are the handful of cans that drive the biggest, most visible problem? And visible was really key because what we did that I think was most interesting was brought in a new layer of data, which is foot traffic. So we could try to solve the problem citywide, but really solving the most visible cans that impact people's experience walking to work, walking to school every day, navigating what I consider to be the most beautiful city in the world and having to step over trash on the street. Those are the ones we wanted to start with.

And so like I said, there's 5,000 public trash cans, 3,000 of them get a daily first pass. 2,000 of them get a second or third pass, but only with 12 trucks. And so it's a very limited number of trucks to take on that huge number of remaining cans. We looked at about 108 targeted stops based on the foot traffic data and we prioritized those and we're able to see, I believe it was 70 or 71% reduction in overflows and overflow complaints primarily in the Tenderloin, Lower Nob Hill and Civic Center within two weeks.

Stephen Goldsmith:

And the results emanate from observations, not sensors necessarily, not cameras, but observations and reports?

Jessica MacLeod:

So we have had sensors on our cans historically. They're all going through a big replacement and upgrade right now. So we have historical sensor data. We also have been looking primarily at 311 reports and thanks to photos with a number of our 311 reports, we can have definitive proof that those cans have stayed clean.

Stephen Goldsmith:

And does your AI feature read those photos?

Jessica MacLeod:

We had a great partner in one of our civic tech residents who has built a tool called Solve SF, and it's a great reporting tool that sits on top of our 311 data. And he has AI capabilities that allow you to analyze the size of the garbage sprawl around the cans, the nature of the trash, whether it's illegal dumping of furniture versus loose trash that's likely spilled from the street and how widespread it is and all of that, as well as when those reports are taking place, you can sort of see the different patterns and trends around it.

Stephen Goldsmith:

So we've talked obviously about technology. That's the purpose of our podcast and purpose of your conversation. Some of it deals with technology itself, but a lot of it deals with the culture of getting people to use new technologies in different ways. So how do you think about cultural change? I know the Mayor talks about it and that was one of his goals, but how do you play a part in that?

Jessica MacLeod:

Yeah, it's my favorite part. I think everything we're trying to do starts with humans and starts with the cultural piece, and it's no different with performance or with technology. On the performance side, I was really upfront when I first started presenting to our department heads and all of our city staff that our entire first year was going to be about helping us have clarity on our goals, helping us find new creative ways to work together, creating that culture of inquiry and curiosity so that we could learn what works and then focus on the accountability piece once we know what to do about the problems that we want to solve. So from a cultural perspective on the performance side, that's been the priority.

On the technology side, technology touches everything that we do. So if you're looking to drive results across any priority, you start digging into why something is off track and you're pointed back to a lack of just-in-time information, a lack of information sharing, burdensome workflows that eat up valuable staff time. There's a technology component to every challenge and there's also very likely a technology component or aspect to many of the solutions. The pace at which software is changing also creates a unique opportunity for us to introduce new approaches. Historically, technology has meant we need to go through a very long cycle of procurement and then adopting something that's really large and expensive. And right now we can experiment in a much more lightweight way.

Our fantastic procurement team is doing all kinds of interesting, innovative things around this to help the city move more quickly at experimenting and trying new tools in a super low risk environment. And we love to see that. I think that we are the epicenter of technology, the epicenter of AI. I mean, San Francisco, it's all born here, so we want to support our neighbors and friends and we also want to be on the cutting edge.

Stephen Goldsmith:

Since you're on the cutting edge, give us two or three lessons that other cities should adopt that allow them to benefit from your role and your work.

Jessica MacLeod:

That's a great question. I think the first one is to build bridges, and that really looks like not just connecting different departments to one another, but government is so hierarchical and hearing from folks on the front lines and having their stories and their experience about the day-to-day as someone who is working behind a counter, a service counter, someone who is a corridor worker picking up trash on the street or an outreach worker or speaking with folks that are unhoused and struggling with drug addiction on the street, those sorts of things, those stories really enrich a lot of the policy decisions that we make and they bring a lot of color to the data that we look at. And so I think bringing in different voices, creating some narrative around the data that you look at so that they're more than just numbers, but you can really start to see patterns and opportunities to try new creative solutions, I think is a great starting point.

Mapping systems is also really important. I'm a huge fan of seeing things in terms of complex systems. When you're able to make assumptions, like sort of unspoken assumptions, known and make systems visible, a lot of these things are already in place. We do a lot of work around how do you craft a theory of change, sort of a logical framework that helps us get from what we do today to achieving our mission. There are so many assumptions there that folks are already operating under based off of what has been handed down over the years, but making that known, making it visible, and then putting numbers around it really helps us test out, does this still hold true? Should we be trying something else? Does the system we have in place really set us up to achieve this goal based on how it's functioning today or not and being transparent about that.

Lastly, empowering as many people as you can to experiment with solving problems within their own domain and within their own control. It's really tempting to want to make big structural changes. And at the same time, there are small things, whether it's one team is going to experiment with their working hours and moving it to two hours earlier and see what impact that has or removing a couple of steps in a process to create something more simple. Those are all things that create an entire ecosystem of change makers across your organization so that it's not just concentrated at the top.

Stephen Goldsmith:

If your technology is as good as your enthusiasm, San Francisco is in great shape. This is Stephen Goldsmith, professor at the Bloomberg Center at Harvard with Jessica MacLeod, one of the country's not just leading technologists, but leading performance managers on how to use technology to change the quality of life. Thank you so much for being with us today, Jessica.

Jessica MacLeod:

Thanks for having me, Steve. Great to see you.