The ThinkND Podcast
The ThinkND Podcast
Evidence Matters, Part 12: From Line Items to Impact
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Episode Topic: From Line Items to Impact
Discover how local government leaders move beyond ad hoc evaluations to embed research, data, and evidence into everyday budget decisions. Listen in to gain practical insights on building internal capacity, fostering cross-departmental partnerships, and creating the political will needed to make evidence-informed budgeting a lasting reality.
Featured Speakers:
- Ricardo Basurto Davila, County of San Diego
- Ryan Flynn, The Lab @ DC
- Jen Tolentino, Results for America
- Sam Quinney, Tobin Center for Economic Policy
Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/22a2e7.
This podcast is a part of the ThinkND Series titled Evidence Matters.
Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.
- Learn more about ThinkND and register for upcoming live events at think.nd.edu.
- Join our LinkedIn community for updates, episode clips, and more.
Welcome
1A big welcome to everybody to our Evidence Matters series. I'm Carrie Chihock, and I've spent my career in public service, um, both at the federal and the local government levels, uh, using data and evidence to advance equitable community outcomes. Um, I'm really excited to be here with all of you. It would be great if, um, those of you in the audience could give us a little sense of who you are. Pop into the chat, um, your name, organization, and what's bringing you to the conversation today. Um, we'd love to know a little bit more about what's, what's on your mind. Um, and I'm here with my co-host, Vincent Kwan. Vincent, do you wanna pop on and say hello?
Speaker 2Sure thing. Very excited for today's conversation. I'm Vincent Kwan, and I'm the Director of Strategic Partnerships and Innovation at J-PAL North America. And for background, J-PAL North America is a research and policy center based at MIT that focuses on evidence-based policymaking and using rigorous research to identify and ultimately lift up, uh, proven solutions for economic opportunity. So very excited for today's conversation.
1Yeah, great to be here with you, Vincent. And, uh, you know, Evidence Matters is our collaborative learning series that's bringing together researchers, governments, and community perspectives on what it really takes to promote rigorous learning to advance equitable outcomes. And, uh, I was just thinking, um, as I was getting ready for this, Vincent, it's our fifth year in Evidence Matters. Can you believe it?
Speaker 2It's hard to believe. Time really flies. I still remember when you and the Wilson Sheehan Lab for Economic Opportunities, LEO, launched this webinar series back during the pandemic many years ago. And J-PAL North America has always been such a champion of the series. In fact, it's one of the reasons why we had, uh, announced you as our inaugural government evidence champion back in, in twenty twenty-three, and it's really exciting to see how far Evidence Matters has come in the past five years.
1Yeah. It's amazing. I can't believe how, how much time has flown. And both Leo at Notre Dame and I were so excited to bring J-PAL on as a formal partner in this series. Um, we all bring complementary perspectives, uh, to this and a common mission of really speeding up learning across places, so sharing what's actually working to improve economic mobility in our communities.
Speaker 2And I think it's those complementary perspectives that really define what Evidence Matters is all about. We're not focused on the headline results of individual research studies, but rather digging into those in-between spaces, what, uh, we often call the messy real-world work of building and sustaining government research partnerships to build high-quality evidence that decision-makers ultimately can use to change practice and improve outcomes for the residents that they are hoping to serve.
1Yeah, and it can actually really be messy sometimes. And I think that, um, you know, our topic today is a great illustration of that. There are few things messier than government budget processes, um, and there are also few things that have a greater, uh, possibility of actually producing results. So that's what we're gonna talk about today.
Speaker 2And that's why I'm so excited for today's conversation about evidence and budgeting. It's more than just bringing evidence forward into budget decisions, but rather I think the budget process can be a mechanism to build an evidence mindset, a culture of learning within local government.
1Yeah, I couldn't agree with you more, Vincent. And today we have an amazing panel of people who are going to share their experience and reflections on how they go about doing just that, building that culture of learning through using evidence in the budget process. So let's go ahead and introduce them, shall we?
Meet the Panel
Speaker 2All right, let's start off with Sam. Um, Sam Quinney, welcome to the conversation. Can you say hello and introduce yourself?
Speaker 3Sure. Uh, great to be here. My name's Sam Quinney. I am the director of the Yale for Connecticut Initiative at the Tobin Center for Economic Policy. Uh, and prior to this, uh, I spent about a decade, uh, working on evidence and evidence-based budgeting, among other things, in the lab at DC, in the Washington DC
1government. It's so great to have you here, Sam. Um, Sam is one of the founding members of the lab at DC, and the lab is an applied, uh, research and design team that's actually part of DC government, and that goes back to twenty sixteen. And those of us who are interested in evidence-informed policymaking really look to the lab as a leading, um, best-in-class example. And they've produced not only a bunch of rigorous studies, um, that have helped to improve the lives of people in DC, they've also made government processes like forms, budgeting more accessible and using evidence in, in all of that. And since I'm singing the lab's praises, I also want to introduce Ryan, who's worked all around the world actually, um, bringing a community perspective into, uh, into, into civic processes. So Ryan, do you want to introduce yourself?
Speaker 4Hi, everyone. I'm Ryan Flynn. Pleasure to be here. Thank you. Um, I am a civic design researcher with the lab at DC, which is an evaluation and design shop within the government of the District of Columbia. Um, and for the past three years, I've co-led our effort to bring evidence to bear on city d- um, budget decision-making processes
Speaker 2Right. I'm very excited to hear, Ryan, about your experience and perspective as a civic designer, so definitely looking forward to digging in there more during today's conversation. Let's, uh, travel to the other side of the country now. On the other coast of the United States, we have Ricardo. Ricardo, do you wanna briefly introduce yourself?
Speaker 5Absolutely. Thank you. Hi, I'm Ricardo Basurto Avila. I'm the Chief Evaluation Officer for the County of San Diego, uh, where I lead the Office of Evaluation, Performance and Analytics. It's a bit of a mouthful, so we just call it OEPA. And if you hear me throughout the answering questions refer to OEPA, that's, uh, that's what it is. Thank you for the invitation.
1Yeah. Great to have you, Ricardo. Um, and OEPA was formed just a few years ago in twenty twenty-one, I think. And under Ricardo's leadership, San Diego County has really become a place that we're all looking to as doing some of the most exciting work in evidence-informed policymaking. So I think, um, those of you in the audience are really gonna appreciate hearing Ricardo's experience, which is more recent, about putting evidence into the budgeting process alongside the lab at DC, which has, um, several years more history doing this. Uh, so now let's turn to our esteemed moderator for today's conversation. Jen, do you wanna say hello and introduce yourself?
Speaker 6Thanks, Carrie. Hi, everyone. I'm Jen Tolentino. I'm based out of Los Angeles. I'm the Director of Local Practice at Results for America, and prior to this, I've been with Results for America for about five years. Um, before that, I was the Director of Innovation for the LA Mayor's Office.
Speaker 2So Jen brings not only experience working within local government in LA, but most recently has been leading cohorts of local government at RFA's Evidence and Budgeting Sprint. So we're really excited, Jen, for you to lead today's conversation.
1Yeah. You're the perfect person to help walk us through this, Jen. And, um, I'm sure some of you have been, uh, engaged in some of the work that Results for America has been doing. Uh, we'd like everybody to participate by putting your questions into the chat, share your reflections and your question. Jen's gonna moderate a conversation for us over the next forty minutes or so, and then we'll have a little bit of time for some Q&A. At the end, we'll collect your questions through the chat, try to answer some of them i-in the chat, and then, um, hopefully get to some of your questions live towards the end. So with that, I'm gonna turn it over to Jen to kick us off
Speaker 6Great. Thanks so much. Um, I'm excited to dig into this with all of you. So we have three really interesting and useful perspectives represented here: Sam, who helped launch this work in DC, Ryan, who is carrying it forward and making it work for departments now, and Ricardo, who is still in the early stages of building this into San Diego County's budget process. So what I'm hoping we can explore together is not just why evidence matters, but what it actually takes to make it part of the way government works. How do you get started without making this too big or too burdensome? How do you build trust and buy-in across departments? How do you use evidence to support better budget decisions while still leaving room for innovation, learning, and equity? So I'll start with a few quick-- just one quick grounding question for each of the panelists. And as Carrie mentioned, we will have time at the end for audience questions. Please feel free to add those questions into the chat as well, so we can kind of have a queue going, and we'll get started. So the very first kind of quick question for everybody, um, you are all coming at this from different stages of the work. Um, from your perspective, what is the budget challenge this work is helping to address? So maybe I will first pass it over to looking at who looks most ready. Ryan, you smiled. Let's go with you, Ryan
Speaker 4You caught me before my mute button was ready. Um, sure. Well, um, as you said, we've been, we've been working on this for several years. Um, we've done a lot to standardize how agencies submit evidence for consideration and budget-making decisions, um, and then how that's processed. Uh, and that solves a, a couple related problems, if you'll give me the, uh, the liberty. Um, the first is practically and immediately, it lets us, um, process evidence at scale, um, to make decisions across historically, you know, hundreds of proposals in the course of a buz-budget cycle. Um, but just as importantly, uh, we've been increasingly focused on designing processes that, um, help build the muscle for evidence-based decision-making in the budget and beyond, um, through creating expectation that evidence be brought to the table, um, and supporting agencies, um, regardless of where they are in terms of their own practice, uh, to try to find and, and, uh, propose the evidence that, uh, decision-makers n-need to make evidence-informed decisions, um, whether it's the budget or other program decisions.
Speaker 6Great. Thank you. Um, Ricardo, what budget challenge is this work helping address?
Speaker 5Uh, is the, sort of the typical economic problem, right? The limited resources. We have, uh, a number of needs, uh, important. Uh, San Diego, like many other places in the country, we have challenges with homelessness. Uh, justice system is, uh, also a very important part of our, our budget, et cetera. And, you know, everybody would agree all of these are important, uh, issues to address and to solve. Um, but we need to figure out not only how we prioritize, uh, but also, um, how do we make sure that we are spending our money on things that actually yield the results that we, that we want. And it's very difficult without a explicit approach, expl-- an explicit framework to try to tease out, you know, what actually has intended impacts from what doesn't, uh, to, to, to do that prioritization. Uh, so, um, you know, uh, everybody's invested in the things that they do, and sometimes it's difficult to be objective and, uh, in, in making those decisions if, if you don't have a structured approach to, uh, to, to make the decisions. So anyway, that's, that's the challenge that we're trying to address to this.
Speaker 6Yeah, that makes perfect sense that this brings order and data and evidence into that decision-making process and enables you to really center resident outcomes. It's great. Sam, what do you think?
Speaker 3I think I'd build on what Ricardo and Ryan, uh, said. It is-- Uh, the key challenge is folks that are making budget decisions at every level of government, you don't have a crystal ball. Uh, you don't know what's gonna happen in the next year. You don't know what, uh, how well something's necessarily going to play out, what other challenges are gonna come up. Uh, but when you incorporate more evidence into budgeting, it gives you a better chance to kind of narrow in on the scope of, of what may happen, be more prepared, uh, for things that come along. Uh, but also by being, uh, really certain that of the things that you can do, you're gonna have the best path to impact, as Ricardo was saying
Speaker 6Amazing. Okay. All right. So it makes sense. There is a very clear challenge, and there is some aspect of a solution that I think we're all circling
Building and Sustaining the Work
Speaker 6around. And so Sam, I'm gonna pass it to you first to kind of maybe kick us start, get started. Um, when you were first getting this work off the ground in DC, how did you think about building buy-in across agencies and budget leaders, and how did you avoid making, sort of using evidence not just another compliance exercise?
Speaker 3Yeah, absolutely. For a little bit of context, and, and Carrie shared this, um, a bit in her intro, but the lab at DC started in DC government in twenty sixteen, twenty seventeen, depending on when you count. Uh, so about ten years ago, uh, at an important time in the evidence-based policy movement and a time where there was a lot of different things being tried. And I would say where the Washington DC government started and in the administration of Mayor Bowser was really, uh, kind of a curiosity about how evidence could be used, uh, and an idea of, okay, we have this new team, let's see how this could play out, and how could you make our budget decisions a little bit better or a lot better? Uh, and so, uh, there's some things that I would share from the beginning of it in terms of building buy-in and making things authentic is gonna be a combination of things that we did at the time and also that we learned over the years, uh, because a lot of it has to be a learning process, and you're not gonna get it perfectly from the start. But what it boils down to is when you're thinking about evidence and asking agencies to justify with evidence what they're asking for in the budget process is to show that it's going to be used, show that the evidence that they provide is gonna be used. Uh, and so I think I have, uh, one slide, um, uh, that I'll briefly flash up here. Uh, this is a budget cycle, roughly for Washington DC. There's a lot of places on here where evidence could be used. Um, but a key part of making it seem, uh, making it authentically useful to agencies and to the, and to the government, uh, is picking an area to start and somewhere where you and your role actually can make something happen and something different. And so these are all of the parts of the budget process in Washington DC, preparation from agencies, formulation in the executive branch, a legislative process, the execution of it, the assessment of it. You could put evidence in all of these areas, but pick somewhere to start. And we knew that because we were based in the mayor's office, uh, and reporting to the budget director, it was in this area where you see the little graph next to it, that we had a role where we could authentically do something with evidence and show that it could be useful. Uh, and so you can take that slide down, uh, for that part of it. And so that's a, a big part of it is showing that it's going to be used and making sure that it's being used fairly. Uh, I think anybody who's been in government, who's been in an agency, who's been in different parts of it, is very familiar with compliance activities and also gotcha activities. Nobody wants to be hearing from the auditor. I didn't see any auditors in the chat so far, but maybe, um, a very important role to play. And so it's really important to make sure that, uh, if you're asking for evidence, you're trying to have people bring it into the process that it's gonna be used, but it's gonna be used fairly Anybody who's worked in, uh, social science research or, or program evaluation, uh, will know that if you look at the things that we could be confident are going to work, it's about this big, and I don't have enough room, uh, on my screen to show the breadth of things that state and local governments do, but it's, it's this-- it's beyond this much. And so it'd be really, um, thinking about how we can treat evidence as one part of that full budget process, a process that includes laws, it includes politics, it includes feasibility, it includes state capacity, and finding a role for it and making sure that it's, it's one part of it. And one of the ways that we did that in Washington, D.C. from the start, uh, is to really allow it to be less about accountability and less about kind of gotcha moments, and more about how can evidence make an agency's case stronger. So you wanna, uh, invest in this, this program, you're making the case for why we need to, for example, invest in a different model to address homelessness. Well, if there's evidence supporting that model, that's a bonus. That's an extra point in your favor on top of all of the other things. And Ryan may talk a little bit more about how that plays out. Uh, and then the final thing that I would say is that it's important to make it conversational. Uh, the thing that people hate about compliance activities or kind of government black boxes is you put your information in, and then either you hear nothing, uh, or you just hear an unjustified decision afterwards. And so what we tried to do, especially over the years, is make evidence a conversation, uh, between agencies and us and other parts of the bud-- of, um, the budget team and the mayor's office about, okay, what are you trying to do? What evidence is there? What evidence might you use in the process? And that makes it build that auth-authentic relationship between what we are trying to do, which as we started talking about in this session, is we're trying to get to the best outcomes possible.
Speaker 6Fantastic. That's so helpful to kind of really orient how you framed it, the ongoing relationship-building aspect of, like, how do you get the buy-in? What does it look like? How do you show that it's actually meaningful? I think that, like, feedback loop, um, of continuous improvement, too, is really important. Um, so now I'm curious, Ricardo, um, as we've discussed, you all are just starting this work. Um, and so I'd be interested to learn kind of how are you thinking about building those relationships, trusts, and sort of what are those first manageable steps that you're thinking about? Like, where is your decision point that you're maybe kind of approaching this work from?
Speaker 5Yeah, that, um, that relationship building is a crucial step, and especially for us, and I'll explain in a minute why. Um, I think if you ask anyone who's in the audience right now, and pretty much anyone in the world, I mean, they would agree, yeah, of course, government should spend the money on things that work. Uh, so it sounds like it should be easy for this to happen, right? I mean, it's like, well, everybody should be doing it. But it's, uh, it- it's not that simple, and part of the reason is that finance folks and the evidence folks or evaluation, whatever you call us, we o- operate in different worlds. We speak different languages. So the, you know, you, you talk to the finance people, and they talk about general purpose revenue and encumbrances, and, uh, you know, for ex- I, I, I hadn't heard about encumbrances until I had this job, and I, I had probably been a director of my department for a whole year before I even heard the word encumbrances. I had no idea what they were talking about. I don't know how many in the audience actually know what that means. Um, and on the other hand, we speak about randomized control trials, statistical significance, theory of change. Um, and so it's not that easy to, to communicate and come to actually just even agree that we want the same. Um, now in some places, and I believe that's the case in DC, evaluators work side by sides, by side by, with finance, with budget management. Um, so that makes it a little easier to, to sort of start heading in that direction, right? To interact, uh, realize, "Hey, we should be working together." It's not the case in San Diego, right? Our chief financial office, uh, is separate from MOEPA, and so building that relationship was, you know, was really a critical first step for us. Um, just a random, uh, a random thing that, uh, uh, happened yesterday. We had a, we actually had a budget hearing yesterday, a public, uh, hearing yesterday, um, led by our board of supervisors, and in the public comment, somebody mentioned No romance without finance. And, uh, I, I had no idea what he was talking about, but it clicked with me. And, uh, and I thought, I need to mention that tomorrow at the presentation because, um, I think at, at, at the end of the day we realized there's no way that OEFA can take this on, on our own. I mean, the, then suddenly make the county invest into, in things that work if we don't involve the bu- budget folks. And so now how do we do that? And last year, Results for America actually gave us the perfect opportunity to do that. They organized a five-week, um, sprint, uh, called County Budgeting for What Works, if I remember correctly. Um, and that gave us the excuse to approach finance and tell them like, "Look, the, we want to, to invest on things that actually work, uh, but it's tough to get started. Uh, why don't we apply to this sprint and, uh, participate, and it'll give us an, a, a way to, to start thinking about this together?" They were interested. They joined us. We applied, we got in. Um, and it gave us, obviously it was the, uh, the perfect excuse to approach them, right, for the initial approach, but it also gave us sort of a structured environment where we could craft our goals together, agree on, well, this makes sense, this makes sense, this makes sense. Help everybody come to terms with this is what we mean by evidence and budgeting, right? This is, this is what it means, and this is what it doesn't mean. It helped us engage with other county governments who were also having the same questions, same ideas, same interests, and so we heard what everybody had wanted to do. And most importantly, it gave us the momentum to start working on it and continue working it, right? For five weeks we were talking about this. Um, and then to continue working, which, uh, you know, to date, we're still, uh, we're still building. Like I said, we're getting started, but we're, we're actively building towards how we're gonna figure this out. Um, I should say, just to close the, the question, is that building a relationship with finance was only the first step of this, right? I mean, in terms of involvement in the budget cycle, all departments in the county are part of it. You have department directors, you have the program managers, you know, people who figure out this, uh, this is what we want to do, this is the new program that we want to create, this is how we want to address this problem. And so eventually all of them need to understand what we're trying to do and need to embrace it, right? And Sam mentioned earlier, you know, first step is for them to actually believe that the evidence, if we're asking them to provide the evidence, that evidence is gonna be used. If they don't believe that it's gonna be used, and they just see it as a checkbox, that they, that is just part of their, uh, uh, of what they have to do, then it is likely, it is less likely they're gonna embrace it. So that's, that's our next step in terms of building those relationships and the buy-in, is to make sure the folks understand what we're trying to do and that they believe that we're onto something that can be good for the county.
Speaker 6Great. Thank you. Yeah, it's so interesting to see sort of, Sam, where you kicked it off and where it is now, which I'm excited to bring Ryan into the conversation, and Ricardo, sort of like you using that sprint as an organizing moment to sort of build that bridge with the budget office. Um, and full disclosure, did not plant that RFA plug, but very grateful for it. Thank you. Um, so Ryan, um, turning to you, you've continued the momentum that was built, um, now going on 10 years, and I mean, how are you able to sustain this work, right? There's the initial momentum of like, "Okay, we're gonna do this. We're doing it well," but what does it mean to kind of continuously have this refined? You have a human-centered design background, so very curious about how you're, you know, focused on improving this ongoingly with departments, and what have you learned about the process being more workable, and are there any, like, tools or templates or rubrics that, you know, any of the other government folks on this call might be able to, to steal or replicate?
Speaker 4Sure. Well, I mean, j-first of all, a big part of it is that delivering on what Sam and Ricardo were talking about, you know, being good, honest brokers and partners, um, being useful to decision-makers, um, being good conversation partners with agencies, um, as they seek to develop an evidence-based proposal for, you know, what, what they're experts in, um, and what they wanna do to shape, uh, the services to citizens. Um, and then another, you know, another big piece of it is, um, standardization and process. It allows our team, um, to continue to do all of those things. Um, with a-- I know we're blessed with a larger team than many governments have that's focused solely on evidence generation, but, um, nonetheless, it, you know, there's more that we would love to do, and so we need to focus on being efficient. Um, and so just, just a little bit of background on, like, how we function. We sit centrally. Um, as Sam said, we come in r-relatively early in the budget cycle, um, when the m-the budget that the mayor will propose, um, is still pretty fluid. Um, and we serve as a, a hub of sorts, a conversation partner, if you will. Um, and so agencies will propose, uh, what they wanna do, uh, with the marginal dollar. Every agency starts with a budget. Um, they, they or a budget envelope, they, they budget to that. Um, and then they're asked: What would you do if you had a little bit more revenue, um, a little bit more money to play with? In most years. Um, this year, our economy and our revenue are tight enough that this didn't play out government-wide the way that it usually does, and the way that I'll explain. But, um, agencies are asked to propose, okay, what would you do with the marginal dollar? Um, and that's where we focus on those proposals because that's where, um, to Ricardo's point, the, the sort of thinking at the margin or that we could do this or that, and we wanna bend it towards the, um, best possible impact for resident is most stark, um, and, and where evidence, in our view, can really play a, a, an important role in helping guiding decision-makers. Um, and so standardization, um, through a couple of tools, which you see on the bottom, uh, helps us make it as easy as possible for agen-agencies to make an evidence-based case, um, and help budget writers process that, um, at scale. You know, in a normal year, outside of last year, you know, there'd be over two hundred proposals that need to be considered. Um, and so I can show just a little bit about how we've approached this. I know that there are many other folks with lots of experience, um, doing similar things from around the country, friends and colleagues, and I'd love to see what they think on the chat. But, um, I can talk a little bit through our, our process. Um, and it, and it hinges around two forms. Um, the first one is the budget proposal form, we call, you know, in, in glorious government speak form two. Um, Uh, which is filled out by every agency when they want to make a proposal for something to do with a marginal dollar. Um, and some of that are things that they wanna do something new or they want to expand on a program they think is promising, and evidence is required to make that type of proposal. Now, there are other types too, um, you know, restoring one-time funding that fell away or something like, you know, procuring a piece of software, there is a business case. Evidence is optional. Um, we read all of them and consider all of them. Uh, and, uh, we give each, uh, each proposal an, an evidence star rating, which I will show in a minute, but if you stay here for just a second.
DC's Evidence Process
Speaker 4Um, we've put a lot of thought into trying to, like, make it as easy as possible for agencies to submit the same information, um, and, and to, uh, submit the type of ev-evidence that, that really moves the needle. And so we ask four questions, which are-- you may not be able to read, but are on the second page here. So first of all, what's the outcome? What are you trying to move? Um, the second is what evidence supports your conviction that, you know, within your, within your ability, this is the best way to approach moving that outcome? Um, the third is how is your proposal similar to what the evidence says works? Um, and the third is, you know, how is it different? Recognizing that, you know, you, you can't just drop something from one place and put it into another without consideration of context. Um, and so on the basis of that, uh, we give every proposal a star rating. Um, and if you go to the next slide, I can just talk through a little bit about what that looks like. Um, and so Uh, this may look familiar to some folks. It's not our invention. Um, it comes from the federal government. Um, IES and the Treasury adopted that for the state and local recovery funds that were, uh, part of the COVID recovery package. Um, and so these star ratings are largely process-based, largely based on how the evidence was generated. Um, give budget writers a sense of, like, what type of causal claims you can make. If you're getting four or three stars, um, with a reasonable set of assumptions, you can say, okay, intervention X, um, had a part in causing outcome Y. Um, but we, we don't stop there, um, to, to Sam's point that, um, there's a lot, um, uh, there's a lot that happens outside the gold standard for evidence that's important to decision-makers as they balance a variety of considerations. Um, and we also want to encourage agencies to, um, to bring the best possible information because, you know, something is better than nothing, um, from the decision-making perspective and also from, you know, building the muscle for, uh, evidence-informed policymaking. And for us, it's also an invitation to a conversation. Um, if there isn't strong evidence, um, and for many good ideas, there is not yet, um, maybe it's an opportunity for us to work together to build that evidence base. Um, when we're not doing the work on budget, um, the lab at DC, you know, runs evaluations and works with agencies to build, build evidence. And so this is an entry point, um, for those types of conversations as well. Um I'll say just, just-- I've been talking a lot about like format and process, and that's all important, but, um, conversations are also important. And we also recognize that the Form Two that we, um, that we put forward is in many ways, um, our best opportunity to communicate to agencies about the importance of evidence. Um, and so, you know, I bring a design lens to this. I think about the experience of forms and, and processes. Um, and one of the things we've been working on as a team over the past couple years is to try to reach back even before we formally consider things, you know, in-into the pro-- um, the point where agencies are figuring out what they wanna propose, um, to bring evidence to, um, to support their use of evidence there by really clear instructions, um, really getting feedback, um, on not just what the rating is, but why, um, and offering help, uh, to people at the formulation process, uh, to, uh, that, you know, to, to have us help them w-write it so that, um, people don't feel like they need to go it alone or figure it out themselves. Um, I'd love to talk a little bit more about that if the conversation allows, but let me, let me pause there.
Speaker 6That's great. Thank you, Ryan, for sharing that, and it's very helpful to sort of see both, like, the, the form request of how you're asking for this information, but then also paired with the, like, how are you using it to guide decision-making? So I think, like, seeing the usefulness of the information that departments are providing is really, like, at that valuable, like, conversation piece. So thank you for sharing
Trade-offs and Defining Evidence
Speaker 6that. Um, so one thing that came up when you were presenting, Ryan, that I heard was, you know, this process goes for that sort of marginal extra dollar. Like, if you're asking for new funds, like, what are you doing and how are you using evidence to make those decisions? Sam, um, you've often talked about budgeting as a question of trade-offs, like how every dollar spent in one place is a dollar that's not spent somewhere else. And so I'm just curious sort of how evidence can help leaders in this fiscal moment where most governments are not, you know, having very many conversations about, like, additional dollars and where they might put it, but how might that be used to, um, make these choices more cleanly?
Speaker 3Absolutely. The, um, it's kind of strange to be in, uh, at least, uh, the last budget cycle I was in, in the Washington, DC government was probably the hardest that we had been through. Uh, but there was never a budget cycle in any part of, uh, even the, the quote unquote, like, good years, uh, where everyone was like, "Oh, yeah, this is an easy one. Don't worry about it. We got money coming out of the walls." Um, and so it's kind of funny how it becomes all relative, uh, of what a hard budget year is. And, uh, and this is, uh, and but it's certainly not funny the, the kind of economic constraints that, uh, the DC government is put under. And so that really, uh, draws into focus the current times and all of the costs facing both individual people and, uh, and their governments, the importance of, as Ryan mentioned, using that marginal dollar better. Uh, and I think people who haven't been part of a, a government, um, budgeting process really, uh, think of the top line, and they're often given the top line of, you know, four hundred million dollars for education, seven hundred million for police or whatever that balance is. And what is often lost, uh, and I thought Ryan did a good job of reinforcing this, is that most of these things, unless you're going to undertake some of the things that have happened, for example, in the federal government, a lot of these budgets are really durable. Um, there's things that have to be done by law. There's things that are politically untouchable. Uh, there are things that are obligated over time. Governments have to pay their own debt, uh, for anything they did even decades ago. They sometimes they need to pay settlements for things that they did many decades ago. Uh, and so really, uh, as Ryan put it, they're really talking about, like, somewhere in this, like, three to maybe ten percent range within a budget, and it's thinking about that extra money goes fast. How can you do the most with that dollar? And I think it's really important that, um That, uh, people think of it that as the key place for evidence to play a role. And so I think that when you're thinking about, okay, how should we spend this one dollar? Uh, that's where we get into these decisions of trade-offs. And somebody put in the chat a question about how do you think about things across different contexts and research areas or populations? And here's how I tend to think of it and, and how I think the, the lab's budget process has played out and, and really more philosophically our role in it. Um, first, uh, it's to not generally think of things like, okay, uh, well, we should spend more money on education and less money on health. Um, those are the types of things that I think there's probably not a huge role for evidence to play, and those are mostly political decisions or values decisions. Not really sure what the evidence would say comparatively. Like that, that's not necessarily the best use of it. But thinking about within the outcomes that a government wants to achieve, how can you do the best thing with that dollar? And that's where I think evidence has a really important role to play. And, and this is my, uh-- I hope this, this works for people. This is my like, um, uh, this is the map I-- this is the graph I have in my head. Don't make me put numbers to it because it starts to fall apart really quickly. Um, but when we're thinking about budget investments, we-- I picture this graph that's on the-- it'll be, I guess, the left side of the screen, uh, where we have the true impact of what we're going to spend money on. Some-- A homelessness program, an education program, uh, something new for the police force. Uh, and that top line is what is actually going to be generated by the, that spending. It's the true impact of it, of the program. What is it going to do for people? But traditionally, and at the status quo, we operate with this gap, which is that lower line. It's what we know. We have this gap between what we know about what is going to happen with investments and that true impact. And what we're trying to do with evidence and budgeting is to really just kind of drag that lower line up, bring these things closer together by using more evidence so that marginal dollar is going to be spent better to achieve the outcome that we are thinking of. And to get to the, uh, the question that someone had asked is, I think it's really important to think of the opportunity cost largely within outcomes or within agencies. So thinking about, okay, we can agree that we want to improve, say, homelessness, uh, prevention for people. How can we best use our homelessness prevention dollars? How can we best use our education dollars? And evidence and bringing evidence into the process gets us closer to knowing, okay, if we do this, we're going to get that. Uh, we will never know that perfectly, but the idea here is to get closer to it.
Speaker 6Great. Thank you for sharing what is in your brain, and we will not apply numbers to it, but it's a helpful visualization of, like, where are we going and how might we get to that better decision-making through evidence. So thank you. So back to Ricardo, who again, you're getting started. Um, it'd be curious, like part of this conversation has been like getting everybody to agree that evidence matters, right? I think that narrative building, how we're gonna use it. Part of that too is like sort of defining evidence in a way that's useful for budget decisions, especially when programs may not have formal evaluations or perfect data. I think somebody also put in the chat a question around like, how are we talking about a logic model supported by data? I think that question is probably more for the DC looking at the evidence ratings that they use. But Ricardo, I'd be curious, you know, how is San Diego County starting to think about defining evidence? Um, and if you have a response to whether or not a logic model shows up in that, I'd be welcome it, but we can also hold that for later
Speaker 5Sure. Um, you know, having a clear definition of evidence is, uh, is, is a critical part of the process of doing this. Um, because it gives everybody in the organization, and in this case organization, I mean, the County of San Diego or 40-something departments, uh, it gives us a standard language that we can always refer back to. If we don't do this, um, then suddenly we're dealing with everything is evidence-based, and that, that's a problem that I see, you know, uh, time and time again, that, uh, evidence-based became a buzzword. And there's always somebody who finds a way to say this is evidence-based because we have metrics, right? Or because we did a pre/post analysis, or we did a customer satisfaction survey. And so, um, and, and the other part of this, and it's related to that, that everybody will say, "My program is evidence-based," is there's an emotional investment in, uh, you know, departments, staff that have been implementing a program that truly believe in their program, that when we started talking to our finance folks about it, they, they explicitly mentioned that. You know, is there a way that we can sort of remove that emotional component, uh, to think more objectively about is this working, yes or no, right? It's you want it to work, but how do we know that it works? So building that definition of evidence that we're gonna use throughout the, the organization, it makes it easier for everybody to be talking about the same thing. Um, another plug for Results for America. During the sprint last year, then you guys shared with us resources that you've put together that made it much easier to start thinking about this, right? I mean, you have this document with, I think, maybe 20 different definitions of evidence, um, from federal, state, and local government agencies. Um, and you have this cool one-pager, um, like a step-by-step guide on how do you start crafting your own, um, definition of evidence, right, for where you are. And that was cool for us because it helped us realize we don't have to use anybody else's. It's not like there's the best definition, right? Uh, no, no, we don't need to use the one from this federal agency or from the state of, uh, you know, Minnesota or North Carolina. It's ju- the, the, th- there's no best that we have... We can try to craft the one that makes sense for us. And, uh, I have a, a couple of slides that are, be useful to, to, uh, to insert here because, uh, as we started talking about this, I don't want to take credit for this. I mean, that's really the team that participated in the sprint. They figure, why don't we craft a def- definition of evidence that matches the way we, OFA, want to work with county departments? And you mentioned a logic model. Um, somebody else in the chat, uh, asked if, uh, evidence is only really about a, you know, ev- evaluative, I guess imp- I think they meant the impact evaluation methods. Uh, and for us, actually, that's what we wanted to move away from because the way OFA tries to work with our departments is- All the way through the life cycle of a program, right? We don't necessarily want to come in at the end and tell them, "Ha ha, your program doesn't work," right? That, uh, if we can actually work from them from the start when they're thinking about designing a program, and we can help by like, "Look, these are best practices for program design. Let's build a theory of change for program. Uh, and then let's think about, you know, how to make sure that implementation is going well, and it makes sense for the context of the County of San Diego." Um, and then of course, at the end, we, we do want to make sure that we have reasonable, um, you know, reasonable for evidence that the program actually works. But so we think of evidence like a tree, right? It, it all starts, the strong roots, the foundation of evidence is having a good rationale. Why do we believe that this program is likely to work, right? Not just that it makes sense in our head in a one sentence, but actually how do we map, uh, those mechanisms through which a program is going to achieve its intended results. And then once you have that rationale, then let's look at the implementation, right? We are in San Diego, this is California, there's a policy context, there's a, a social economic context that we gotta make sure that the implementation of the program makes sense for us. And then, of course, we have the impact evidence, right? I mean, this is actually... We see that it works. Well, using the appropriate evaluation approaches, um, we can get evidence that it, that it works. So that's how we would like to work with the department. That's how, uh, our team came up with the idea of, you know, the, the evidence tree, uh, that reflects how we work with them. But in addition to this, and if we can go to the next slide, we also think that thinking of evidence this way, it also makes it a little easier for us to come up with recommendations for budgeting purposes, right? The obvious example is a program that has, you know, a strong rationale, implementation evidence, and impact evidence. Then of course, then, you know, it makes it easy to, to, to, to make a recommendation like, "Yeah, if they want more money, they probably should get it." Um, but then, you know, there are other, other combinations. Uh, and maybe I'll just go to the, I'll go to the bottom where we have a program where we have strong implementation evidence, we have strong impact evidence, but now they're asking for money to maybe scale up the program to a new community for a new population, additional services. Then that's when we'll say, "Well, hold on. Why don't we develop a strong rationale for that scaling up," you know, good theory of change, just to make sure that it actually makes sense, that what you intend to happen with that expansion actually is gonna make sense before we go all in, even though we already have some impact evidence. So again, different combinations of these different types of evidence can help us figure out a little better what recommendations to make, um, uh, on the budgeting side. And, um, again, I'm, I'm really excited that we, that we're thinking about it this way. Uh, this is still Early in our process, we're not even yet, you know, to where, where this is, how do we insert this in the, in a form? How do we get the evidence? How do we get departments? But I, I think we're off to a good start that makes a lot of sense for us. And I, and I think the key message for everybody else who's attending, and it's interesting, how can we do this, is again, one size doesn't fit all. The, the, I'm not saying everybody should do what we're doing and everybody should have their own tree, but it's more about what makes sense in your context, right? And the, and, and to, especially to generate the buy-in among everybody else who has to, uh, who has to agree. Uh, and I know, you know, Results for America was great in providing us with resources to get to this point, so I'm sure they can also be for others.
Speaker 6Great. Thank you. And I appreciate the, like, localization both of, like, within your local context, how do you apply this evidence-based model? Also, how do you define evidence for what is gonna be most useful for your particular government? Um, so I have one final question before we'll kind of go into some of the audience questions. I apologize, we're a bit tight on time, but I really wanna, um, go back to Ryan quickly. One really powerful part of DC's approach is that evidence and equity are both visible in your process, and so can you just, like, quickly tell us, like, how that has helped decision-makers see that, like, fuller picture of all of those program trade-offs?
Speaker 4Y-yes, absolutely. And so, um, budgets are policy documents. There are historic and systemic inequities, um, in the provision of government services and in budgets, and, um, DC is committed to trying to identify and work against those. We have an Office of Racial Equity that sits, um, right next to us and right next to the budget folks. Um, I've got a slide that can show you a little bit about, um, their input as well. But, um, uh, since twenty twenty-three, they've also looked at all the same proposals we have and given a similar four-star score. Um, the focus is obviously a little bit different, um, but it's focused on, first of all, alignment to district, uh, and, and agency racial equity goals, um, with the idea of being aware of and dismantling systemic inequities by race and, and indeed by other groups. Um, uh, what data do you bring to bear on that? Not necessarily evaluation information, but what data do you know about the problem and, and the unequal distribution of opportunity and access? How is that disaggregated? Um, what consultation have you done with community, um, and particularly communities that are most marginalized or most impacted by the, um, uh, by what's being proposed, and then a consideration of unintended trade-offs. Um, and one of the things that I really love about this is it puts, uh, consultation with community, uh, right next to other considerations as valid data. Um, and then you can see this is a-- this is, um, on the screen, uh, a bit of the format that budget decision-makers have before them when they're considering, uh, trade-offs and, and, and which enhancements they seek to fund. Um, and, um, because evidence and-- of what works matters, it's there, um, on the slide, and also indeed this slide is a source of discussion, so in the discussion. Um, and racial equity and it-- and, and our, our assessment of awareness and potential impact on racial equity is, it's right there too. Um, now this isn't, this isn't a full answer, and, and I think Jacqueline in the chat pointed out that there are, you know, there, there are inequities in what we even know about, um, uh, what we can know about a program as well. Um, we do our best internally to push against that. Um, the racial equity assessment is also a good check, um, because it considers explicitly, you know, wh-whose voice is not being heard, uh, who has been understudied, um, and are we pushing against that? And so this is one attempt to formalize that, recognizing that, um, it's a step and, and, you know, not, not in itself a solution.
Speaker 6Great. Thank you all so much for sharing that perspective, sharing some of the templates, definitions, process that you all are already laying out. Um, there's so many different examples of what this looks like in state, local, federal government. There's only one federal, federal agencies, but it all really depends on sort of what is the goal and then how are they leveraging
Q&A and Closing
Speaker 6evidence. So we have five minutes for audience questions. Um, I've been trying to scroll back to sort of see when they came in, um, and would love folks to come off mute and ask them. So the first one I'll just go to, um, Gemma. If you-- if your question is still, like, relevant, you wanna bring it up, please come off mute and ask
1May- maybe Jen, I can, uh, this is Carrie, uh, um- Go for it step in. Like I was looking at the questions in the chat, and a lot of them are around-- And I think, um, we addressed this a little bit, particularly, um, in your last comment, Ryan. But a lot of the questions are around the different types of evidence there are, generalizability of evidence, um, trying to understand where, you know, there is evidence and there isn't evidence, and how do we... Are, are we just prioritizing things that happen to have an evidence base because they have been studied? Like, that kinda sums up a whole bunch of the questions to me. And, um, I think maybe having the, the panelists address that a little bit more would be helpful. I think you all addressed that, um, in your answers previously to a certain extent, but there's-- I mean, we could spend an entire, you know, three hours talking about this easily. Does anybody wanna take that?
Speaker 3I can, I can start. And I think, um, the, the biggest part is, is to not treat evidence as the be-all and en-end-all because all of those things that go into what is studied and what isn't, and also to be humble and circumspect about what we know and, and why we know it, uh, about different programs. And so I think a big part of it is making sure that there's room for innovation in big and small ways in the process. And so there can be very-- In the rating system that Ryan, um, shared, there can be extremely compelling, um, proposals that only get one star, uh, in it because nobody studied it, or it's being very well informed by, uh, the reality in Washington, D.C. on the ground that an agency is, is operating on. Uh, and to make sure that the processes allow for that and have space to hold up those things. And so one very concrete way that that can happen in the budget meetings that I was a part of is literally when we have a turn to speak, to be able to hold up those examples and say, "This is one star, uh, but this is extremely compelling for these reasons." And I think that allows some of that bridging of, of what, um, what is being worked on and what isn't, and doesn't, um, doesn't put too strong a weight on what has been studied
Speaker 6All right. Thank you, Carrie, for, like, synthesizing that. There's so many great questions in the chat. Is there anybody else who would be-- We probably have room for one more question if anybody from the audience wants to come off mute and just sort of ask what they're interested to know. I'm sure there's a lot of burning questions
Speaker 2Um, maybe I actually can come off mute then. Uh, this is Vincent. Um, uh, this relates a little bit to the previous question that Carrie helped to synthesize, but knowing that evidence often doesn't come from your specific context, or it might not be focused on that particular population, or it may be related to a similar style of intervention but for a different policy area, how do you all think about generalizability of evidence from different contexts, different populations, different policy areas as part of the budgeting process?
Speaker 3Give the one-sentence response and then want, uh, allow Ryan to build on that, which is anything that helps, anything that gets closer to the truth. Um, and so there's gonna be things that are exactly right, uh, but that's rare. Uh, so anything that gets us closer to understanding the impact. That was three sentences
Speaker 4Um, I'll, I'll add, uh, there's a, there's a healthy balance of, uh, humility involved too, and just acknowledging where the differences are, um, so that's available to decision-makers. We don't see our role as, um, being the arbiters of what is or is not a good decision because the evidence is there or isn't. Um, we see ourselves as being advisors and, um, and weighing in honestly on what you can know and, and what you can't know, at least not yet. Um, and doing what we can, um, through evaluation or through community engagement or through service design techniques, um, to build the information base that's available to decision-makers.
Speaker 5Um, I almost feel like, uh, I wish this was a, um, contest show where I could phone a friend, uh, because I'm sure Catarina, who's actually in the audience, would have a better answer than I than I have for this. But, uh, I, I, I know that, uh, for our evaluation team, generalizability and taking into account the local context, uh, not just taking at face value what has been found in other places is, is important, and that's actually reflected in our, uh, definition of evidence. Um, you know, what I know is, like, like Sam said, you know, anything that helps is good. But also if at least we have concerns that something that's worked somewhere may or may not work here, then that can guide our approach to implementation and making sure that we're tracking, uh, the information that will allow us to make that assessment, right? Okay, in what context, with what characteristics either were the, the services that are being provided, the providers themselves, or the population that is being served, what, what are the factors that actually drive success, uh, and how does that relate to, to the county? So, um, again, uh, uh, it is definitely part of how we're thinking about this. Just I may not have the best answer here
Speaker 6Still a great answer, Ricardo. Thank you. Um, so just to wrap up, thank you all to the panelists. I'm so grateful for you sharing your expertise. My key takeaway here was that, you know, evidence matters. It really matters to bring in how you define that to all of the folks in government that are going to be making decisions. It's an ongoing conversation. This is not meant to tell us what is right, wrong, should be invested in or not. It is meant to be a tool to help guide decision-making, to narrow the gap between what we know and what we really wanna be achieving. So thank you all for really sharing that, and I will pass it over to Carrie and Vincent
1Great. Thanks, Jen. And I think we have a slide of resources for folks who want to dig in a little bit deeper. Of course, we had, you know, limited time here, and we could, uh, spend a lot of time on this conversation. We will send this out to all of you, so you can dig in. I'll just note that, um, Results for America is holding another, um, Budgeting for What Works sprint coming up for counties, I believe. Is that right, Jen? It's for counties?
Speaker 6It is for counties, but we actually are a-- I think we're gonna be opening it up for cities as well, if y'all are interested.
1Right.
Speaker 6So I'll, I'll send, I'll, I'll-- Let me know. We'll, we'll-
1Yeah
Speaker 6adjust
1it. Super. Um, so take a look. And, you know, Vincent, I just wanted to reflect that, um, I think, you know, what I really came away from this is how it's really about creating that deeper culture of learning or a culture of deeper learning, however you wanna put it. Um, it's not about, you know, it's about evidence-informed policymaking and how we can use evidence to move towards better decisions, not using evidence as, um, you know, kind of a grade for whether something passes the bar or not. Did you have a reflection you wanted to share too?
Speaker 2Yes, I couldn't agree more with that, Carrie. I think oftentimes people worry about evidence and evaluation because they see it as a thumbs up or a thumbs down. But I think-
Speaker 7Yeah you
Speaker 2summarized it perfectly, where evidence should be a conversation starter to help us really dig deeper on ultimately how do we improve outcomes given that we do have limited resources to spend. And I think that communication around how evidence should be used, how it can be useful, aligning on key definitions of what we even mean when we say the word evidence in budgeting or evidence for decision-making is really instrumental to that process of, uh, learning and that culture of learning that we're really hoping to encourage.
1Yeah. And I think how that def- definition doesn't preclude other ways of learning about things, right? But it's helping us to think about how we really engage in a deeper process and move things forward to a way that improves outcomes for our communities. So thank you everyone who attended. It was great to have you here. I hope you'll come to more of our events in the future. And, uh, thank you to Jen, Ryan, Ricardo, Sam, uh, for sharing your, your insights and your humble expertise. Really appreciate it.
Speaker 7Thank you, everyone. See you next time.