The ThinkND Podcast
The ThinkND Podcast
Evidence Matters, Part 9: Preparing for Evaluation
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Episode Topic: Preparing for Evaluation
King County Metro and Notre Dame’s LEO lab are re-engineering social service delivery, codifying racial equity through a transformative researcher-practitioner model. This isn’t just data—it’s a co-creative learning pipeline designed for systemic change. Explore how rigorous evidence-building is redefining the value proposition of civic interventions.
Featured Speakers:
- Betina Jean-Louis, Ph.D., Arc of Evidence LLC
- David Phillips, University of Notre Dame
- Jacqueline Kelley-Cogdell, LEO
- Sara Sisco, Hopelink
- Krisda Chaiyachati, University of Pennsylvania
- Maria Jimenez-Zepeda, King County Metro
- Christina McHugh, King County
Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/698ac2.
This podcast is a part of the ThinkND Series titled Evidence Matters.
Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.
- Learn more about ThinkND and register for upcoming live events at think.nd.edu.
- Join our LinkedIn community for updates, episode clips, and more.
Welcome to Evidence Matters
Speaker 10I'm so delighted, to be here with, a great group of my partners who have been working on evidence building in King County and on behalf of King County Metro and the Lab for Economic Opportunities at Notre Dame. I wanna give you all a huge welcome. Um, this is the second in our Evidence Matters series of virtual events, and it's so great to see, so many friends and, and colleagues here. So thanks for joining us. I'm Kerry Chi Hawk. I lead learning and impact strategies and partnerships at King County Metro and, I'm really passionate about advancing racial equity through bringing together our community partners, our research partners, and our staff together. Little Heart for you all. you know, to work on, being, driven by our values and data and evidence informed. And that's why I'm so excited about the, topic of this conversation today on building great researcher practitioner partnerships, king County Metro and, uh, Leo. That's how we refer to the Lab for economic Opportunities. We have a really deep partnership and you're gonna hear more about that today, of course. And we've been working together on this, series of events, which I'm excited about. And I wanna give a shout out to Jenny Garris, who's my planning partner for this at, at Leo. She's Leo's dissemination associate, and though Jenny's kind of behind the scenes today, she's very much been in, you know, front and center in, in planning this, getting it off the. So our intent with this Evidence Matters series is really threefold. We're looking to build skills and knowledge for our staff at King County, but for others, in this field, to focus on equitable impact and community outcomes. We wanna celebrate some of the successes that we've had in building our evidence-based practice, and we wanna share challenges too, and we really wanna learn from you and build kind of a community, of practice here. So we'll continue to have, some of these events about every couple of months and, um, really look forward to being in dialogue with many of you around that. We're, you'll see that a lot of our content is around, transit interventions because we've been really focusing on that in King County, but. You know, I think a lot of what the content, and for this event, particularly today, it really, should be of interest to many different social sector organizations. So glad to have all of you here and seeing like a range of people from transit, from our transit agency, other transit agencies and and other folks. here we're really fortunate at King County to be working with Neo across a big range of, evaluation projects. And our relationship really dates back to 2017. And because of that we've got projects that are at all stages of the evaluation cycle. Some of our projects are really in the conceptual design phase while we've got other projects where we have results and we're working together on, putting those in use in our decision making. So on our agenda today, we're gonna start with a moderated panel conversation. We're gonna highlight three projects at King County, where we're in the early stages of, evaluation, and we wanna use those projects. Not to talk about the projects in particular, but to really use those projects to talk about researcher practitioner partnerships, draw out some lessons about what makes great partnerships and bust some myths about that too. And you'll see on our agenda and as we go through the discussion that, we're focusing on these early stage projects. So our first project transportation navigators is in instilling kind of conceptual design phase. Our second project on the subsidized annual transit pass has a very well developed evaluation plan. And in our third project, we really jumped in quickly, built a project and we're already in data analysis. So I wanna point out that, you know, partnerships, great researcher, practitioner partnerships have to extend beyond, you know, kind of the evaluation itself into really understanding the results, disseminating the results. And so stay tuned. We are going to have a part two where we look at some of our projects that are in later stages and talk about, great, great researcher practitioner partnerships with those projects too. Your participation is a lot of what motivates me in this, having this event, and it's really important to us. So we're gonna plan to, um, reserve plenty of time at the end to have conversation, hear your questions, and have discussion. So as you listen to the moderated panel, go ahead and pop your questions into the chat or write them down for yourself. And then when we get to the question and answer period, we'll ask those questions. Or if your question seems like something quick to answer and really relevant to the discussion at hand, we'll just pick that up and, and answer it within the panel itself. So I wanna move to, uh, introducing a couple of, people. Um, so first, David Phillips from the Lab for Economic Opportunities at Notre Dame. David and I have been working on several research projects together for about four years now, and David's really involved in all three of the projects that are gonna be under discussion today. So I wanted to just call him out at the outset and give him a chance to just say a couple words.
Speaker 2Sure. Thanks Terry. And, and yeah, thanks so much, uh, for the chance to joining this conversation too. like I said, I'm press. I've involved in this partnership. Yeah. It seems, I guess it's almost five years now, uh, that we've been working together and it's, uh, king County, been a wonderful partner to work with, with us at for many reasons. Obviously we have shared commitment evidence, um, and, and doing so in a, that is, helping both people who are doing active, service and creating these programs, but them, particularly for, uh, the folks who were involved in those programs that, that would inform these programs to make them better and to better outcomes in peoples lives. I think even more than that, I think a lot of what makes this partnership great has been, um, the shared commitment of how we go about these partnerships and how we go about, creating these evaluations to have that process also be a good thing for the people involved in it. And so really excited today to share about more of the nuts and bolts of like, yeah, how do we, how have we thought about building these partnerships? What has worked? What have we learned? What, what could we do better? And so, thanks so much for the chance to participate.
Speaker 10Great to have you here, David. And then, uh, I also am so delighted to introduce to you all Betina Jean. when I was planning this event, I knew like right away I wanted to somehow have Betina involved. she has really extensive experience leading partnerships with researchers and the community, through a lot through her role as the, research director at the Harlem Children's Zone. But I've gotten to know Bettina through work that we've done together over this last year where she's been a senior advisor with Project Evident and we've been working with Project Evident, to explore some strategic evidence planning. And Bettina's just been such a wonderful partner in that and really, you know, provided a lot of wisdom and guidance to us. So I've just really appreciated getting to know her. I also let you know that, A lot of, my thinking about, what makes for great researcher, practitioner partnerships is going to be published in a book that pro project evidence has been working on with, the Brookings Institution. It's coming out early next year, so it'll be good timing. with the next part two of, of, of this conversation we're having and it's part of project Evidence, next Generation of Evidence campaign, which is really trying to center the practitioner in the lead of, of evidence building. So, excited to have that relationship with Project excited that Betina is here to moderate and lead our panel discussion. And with that, I'm gonna turn it over to her.
Project 1: Transportation Navigators
Speaker 3Thank you. Hello everybody. I'm really excited to be here. We have a really full and, engaging and interesting, uh, set of speakers who are gonna talk about some great projects that are really going to show what can be done when trustful relationships have been established between, organizations that are providing services and organizations that are helping to, uh, do research and evaluation. That really helps to understand what the implementation and outcomes of those services are. So having been inside an organization, an internal evaluator, who is the bridge between external evaluators and the practitioners, I really have seen firsthand, uh, how we do have those relationships when you have. Researchers who understand the values of the organization and when people, uh, work together to tease apart what's important, uh, how it can really, um, have amazing dividends for organizations and for community members. It's not about organizations. It's ultimately about the people that we all serve and wanna make life their lives better. So, having said that, I'm gonna jump into the conversation because there is a lot, to talk about. And the first project is gonna be about transportation navigators, as you have heard. And I'm just gonna start by asking Jacqueline to introduce herself and to tell us a little bit about the project.
Speaker 10Hi Betina. I'm happy to be here today.
Speaker 4yeah. My name is Jacqueline. I'm a research associate at Leo at here at Notre Dame. yeah, so Transportation navigators is a partnership between Hope Link and King County Metro and, The idea for the program basically stemmed, stemmed from this idea that there are non-cost barriers to public transportation. So, these kind of barriers could be like, you need to know how to plan for a trip. You need to know the timing of buses and trains. Um, you have to have knowledge of where the routes go. You have to feel safe. barriers like this. and we can imagine that these barriers could be exacerbated for people who, are not, are from another country who maybe don't speak English, who have a disability, who maybe have kids. so the program pairs, clients with, community transportation navigator, somebody who is their mentor to help them overcome these barriers, to help them plan around these, these barriers. This, navigator is somebody who's from their own community. So if I'm a Somali immigrant, I would get paired with somebody who's also a Somali immigrant, um, who would know the cultural context I come from, who would better understand, these barriers because they have lived through them themselves. Um, so from a research perspective, our question is really, does this, mentoring service, improve not only short term outcomes such as transit use and mobility, but also longer term outcomes such as employment, health, um, mental and physical, and other wellbeing outcomes? right now the project is in the planning stages, as Carrie mentioned earlier. Um, so Hope Link and King County Metro are kind of working through what exactly this program is gonna look like. It's not set in stone yet. And, the goal is to launch a program, um, next year in February. So that's kind of a, a summary of where Transportation Navigators is, is at right now.
Speaker 3So Jacqueline, so we're gonna turn to David and, ask him to introduce and to tell us about how King County got involved with Leo in this project. so what interested you about this particular study?
Speaker 2Hi, uh, yes. So this, particular project I think grew outta an existing partnership that we had among, Leo and, and Skin and each Metro Transit and, and some other organizations. I think as an academic, I come to this and sort of there are bigger questions about our society that are important, right? So, our cities tend to have lines drawn on the map where people are sorted out by race and ethnicity and, and socioeconomic status in other ways, right? And so the idea that those issues are closely tied to how people get around right, are sort of fundamental to understanding. States, racial equity, poverty states, and, this particular collaboration with King initially, specifically on share barriers, and we have some earlier work, one of these looking at what happens when people have access to free public transit. Um, and one of the interesting things we found there was that, while that encourages transit use quite a bit and might have other outcomes on people's lives, those effects are perhaps a bit more muted for folks who have less access to transit. Uh, and so this idea of like, let's think about non barriers and these other barriers that might be going on, going on, like the ones that Jacqueline mentioned. we should learn a little bit more about how important those could be and what sort of, programs could help people surmount these, these additional barriers other than the, the price tag of getting on the bus.
Speaker 3So we're gonna turn to, to Sarah. Now, this particular soup, has a lot of cooks, and just wanted to, to hear a little bit about Hopelink and, the contribution and, um, engagement with the project.
Speaker 5Thanks Medina. Hi everybody. Good afternoon. My name is Sarah Cisco. I'm the program manager for education and outreach on the Hopelink Mobility Management Team. for those of you that don't know Hopelink, we're a community action agency in King County. we focus on getting folks, stable and, uh, the ability to exit poverty. Our mission is a community free, pardon me, a community free of poverty, and we really see transportation as a huge part of that. Um, and the goal of my team is to really illustrate to the community that transportation is a resource to access accessing services and not a barrier to getting around their community. So we're super excited to be involved in this project because this is. A big piece of our core values as a mobility management program, really ensuring that folks have access to the services they need. And that doesn't just mean transportation. That means transportation to these services, that they can reduce costs, that they have support in their native language, and that they have folks that they can work with that really understand where they're coming from. so we were super excited. Um, we've run some smaller pilots of similar type programs. We run similar type programs, and done a lot of learning over the last couple years and been excited to partner. It's going on almost two years with Leo and King County Metro for this program.
Speaker 3So thanks for telling us about transportation navigators. Um, it's, it's an exemplar of fairly more rare than should be the case of a project where, the evaluation was, uh, was introduced in the beginning. and I would like to hear from your perspective, how do you think that was advantageous? So what did it bring that, the research component was part of the process from so early on?
Speaker 5Yeah, it's, it's really made it, um, it's made it very interesting to think about it from that perspective of, of what comes first. And I, I know David, I think you're gonna jump in here as well. and I think for us it's allowed us to think about what outcomes are we trying to achieve and how can we shape the program so that we understand what outcomes are. Are we, are we looking for health outcomes? Are we looking for financial outcomes? One thing we're always looking for is independence and confidence, social interaction, and what the health benefits are of those types of things. and it also allows us to, to take a critical eye of how we're setting up these programs so that we can really understand what we're trying to get at. and if we, we, the way we can propose it in the way that we think about these programs also allows us to engage in a different set of buy-in, from our partners and end users. If, if they can see that we're creating, we're attempting to create systemic change, that they're gonna have more access to help us create change, we can also build that buy-in. so David, I think you probably have something to add to that as well.
Speaker 2Sure. Yeah. And I, I think from a researcher's point of view, the benefits of getting involved early are similar, right? So that, as Sarah alluded to, and one thing that getting involved early does is it brings to the forefront. What is the point of this program and what is it trying to do? I think that the, the temptation as a researcher, right, can be to focus on what is feasible, and to, and to start with that and then go from there right up. What is it that is possible? What is it that's easy for us to measure? And I think one thing that getting involved early on does is it, it nudges the researcher like me to think about, okay, what is the point of this program? What is it designed to do? And then eventually at some point you have to think about feasibility and what we can actually do practically in terms of evaluation. But let's start from the point of view of what, what the goal of this thing is. I, I think that's one thing that's, that, that really comes in from showing up early on. The other one that comes to my mind is that it, it also allows us to think about how to, um, how to make sure that evaluation and program design work together rather than working against each other. Um, if, if a evaluation is gonna involve some activities around data collection or, or what have you, by making sure that those can be built into the program in the most seamless way, if you have some more time that can allow that to, to work together as opposed to being sort of more in constant.
Speaker 6I thought you wanted to say something.
Speaker 4Yeah, I was just gonna say, um, from a Leo perspective, we, we work with a lot of different organizations and so we have a lot of projects that are, are, are brought, we're brought in at a later stage. And so, um, I think bringing in, like bringing in the research early helps a lot with buy-in, in terms of like from the full organization like standpoint, because it means that something doesn't have to change in terms of the way that you're providing service. it's already built in from the very beginning and, and, um, I think that buy-in is like a, a big part of, of, uh, research success. And so, building it in early means nothing has to really change. and, and that is always easier for us as humans.
Speaker 3Excellent. Thank you. Um, interesting that you're talking about, because I would like to ask Sarah about how you've been engaging the community and how you think that that has had an impact on design of the program and on the evaluation.
Speaker 5Yeah, I think that that's, I mean, that's a big piece of what we bring to this, to this program is our desire to put the needs of our communities first. Like I said, we're, we're a nonprofit. so our goal is, our mission is to engage with the community. and in this process what we've done is really talk to the community about what they wanna see outta a program like this. This past spring, we conducted, focus groups with, two nonprofits, or two local community based organizations, the Filipino Community Center, as well as the Somali fa Family Safety Task Force. Um, and, and talked to a variety of groups, folks from those different groups. We focused on youth, we focused on seniors, and we also focused on, in particular, women with children trying to access transit and trying to understand what a program like this would do. Community transportation navigators would do for their confidence, do for their ability to ride transit, to get around safely. Um, and it goes back to what I said earlier on those outcomes, whether it's financial outcome, health outcome, or just in general social interaction and reducing social isolation. We know we've all been through a rough year with social isolation and, and being away. It's even more evident for our seniors, seniors who don't speak English. And so what we've really tried to do is bring our knowledge of working with the community into this program so that we're engaging them in every step of the way. So we did those focus groups and surveys this past spring. We're in as, as Jacqueline said in the beginning, we're still in the design phase. and now we'll be going back to the community and testing our assumptions on what we think. This program will mean and what the research is gonna show us. And we'll be testing that with the community so that they can also help us shape and co-create this program. I can have a lot of opinions on how this program should run from my experience of doing community engagement. We can, we can think about it from a transit perspective, but what we really need to know is how do we shape this program so it's the most effective for the community to use. So that's really the avenue that we bring in is, is connection to the community and engagement with the community. And we'll be engaging them every step of the way, pre-launch, through evaluation so that we can understand, are we doing this right? Do we need to take a step back? and having folks like David and Jacqueline as a part of this program allows us to shape those evaluation questions so that we can both get feedback from the community to shape the program, but also see what those outcomes are.
Project 2: Subsidized Annual Transit Pass
Speaker 3Excellent to hear about the authentic engagement that you all are involved with, um, where you're not just, you know, making a nod to hear from the community, but really engaging with them. So we're gonna switch our focus now to talk about the next project. thank you all. we, so this, I guess previously we talked about an idolized version of, the work where everything starts together. Now we're gonna talk about another project and I'm ask Maria to introduce herself and to tell us about, king County subsidized annual Transit Pass. Hi
Speaker 7everyone, I am Maria Heman Zepeda. My pronouns are she her, and I'm a project manager at Keen County Metro with Orca Reduce Fairs team. Um, so I do some data analysis and project management for a lot of the projects involving our reduced fair programs here at Metro. It's a little bit the subsidized annual pass. We actually, Metro launched the subsidized annual pass in October of 2020, um, in partnership with Sound Transit Public Health, DSHS Catholic Community Services, with the intention to improve transportation mobility. Uh, so the subsidized annual pass eliminates the cost of transit as a barrier by covering the full cost of transit on metro and on transit services for up to 12 months. And so with the subsidized annual pass, we're really using this as an opportunity to learn how providing access to fullest, fully subsidized transit for those with the lowest incomes, impacts their transit use, mobility, employment, as well as the health and wellbeing of them. And so right now we've actually had over 6,000 people enroll in the subsidized I know pass. Um, we've completed the evaluation design and I'm really excited that we are just about to launch our baseline surveys.
Speaker 3That sounds great, Kat. I know you can't wait to jump into those data. So, Krista, we're gonna switch over to you. you're probably the most unusual member of our panel today. your medical doctor, your, health researcher. and so could you introduce yourself and tell us why is that you were interested in working with a transportation authority? Tell us about the project.
Speaker 8Yeah, so, um, you know, uh, I'm clinically trained as a primary care doc and I, that's probably where the narrative starts is, uh, I'm on, my patients are on the receiving end, transportation services. And so there's lots of stories I'm, I could happily share about how our patients really have always commented on how they struggle to get to their doctor's offices or be able to reach, uh, those facilities. And so I've always been, extremely interested in thinking about the connection between. How people travel and, and are transported, uh, really between home or the community, uh, to medical care, whether that's my office or any other medical service. Um, and so, I, I got lucky because the Robert Wood Johnson Foundation had reached out to me to say, Guha, there's this really great group of folks I've in Pacific Northwest really, uh, interested in asking this question. Would you, would you mind connecting with them? And so, uh, over the last, I believe almost two years now, we've been in connection to really kind of develop these thought processes and, and work in this arena. I also say I think is an incredible opportunity as somebody who does, academic research also to think about, a really impactful evaluation. You know, it, uh, you can't really beat, uh, being able to study something in the quote unquote the real world as it's being implemented, really to see, uh, hard outcomes here about what impact, uh, transportation services are free, transportation services here might have on people's health wellbeing, and that so many other outcomes that are intimately, interact with both health and wellbeings too.
Speaker 3Krista, we're gonna turn back to David, who my touchstone now. So with all of the project, and, uh, I just wanna hear from your perspective, why it was helpful to work with a health sciences researcher. What it,
Speaker 2yeah, so I, I love these interdisciplinary collaborations where people are bringing their biggest strengths to the team, and I think this is a really good example of it, right? So I think, some of it that is just about sort of expertise and different, so, on this particular project, right, we're really interested, as I mentioned before, and how, access to transit and greater mobility affects people's health. I, as, uh, an economist who's interested in, transportation and, racial equity and many other things, but don't have a lot of background in health, don't, I don't know what, what we should measure, right? Like what are the, it was like, okay, to go from hearing from people who running the program to. Questions should in who has greater expertise in who know more about very practical things. Helpful. but I think at a deeper level and where started, I think there's also a lens that the, that health reach in research in particular probably does better than a lot of other disciplines. And that having this, having folks who have specific training, both in academic social science research and in being clinicians side by side is something that's quite common in medicine that's very uncommon in other disciplines. And so you have, you know, maybe oddballs like me or some of the folks associated with Leo who have moved into this space, uh, of trying to be at that intersection of academic research and practitioners, but in a lot of ways. I think that the, one of the things that the health professionals and health science research has done really well is integrating those more fully and how people are trained. And so just the lens that Kris brings to our team of thinking about things through the eyes of its patients and showing up to our joint calls and scrubs, you know, in between, you know, clinic sessions and things like that, it's, is a really beneficial thing to our team that, that's a lot.
Speaker 3Thank you. Maria. Can you tell us from the perspective of somebody who is implementing, an initiative because it, it's all about bringing what has been learned from the research and the evaluation back to the work. Uh, why, how it's been helpful to have, team members, researchers from, different backgrounds, with different experiences involved in the work.
Speaker 7Yeah, and first I actually just wanna build off of what point that, David mentioned. He mentioned how interdisciplinary this subsidizing pass evaluation is, but I don't know if folks really understand quite the extent is, so first we have. And so we're kind of experts in our internal data and systems and transit. but then we also have the LEO folks who are experts on kind of this research and how to think of those technological or like other logistics to make sure that we're actually be able to capture that, the effects that we're seeing are caused by this intervention of the subsidized and not due to a lot of other reasons. And then we have Krista who, like David mentioned, is bringing all of his expertise of both being a practitioner and a medical researcher that's really helping us bring in what is pretty, I think, new for transit is being able to make that leap between is transit impacting not just mobility and economics, but like their health and wellbeing. that is pretty, leaning edge research that hasn't been done a lot before. but then we also have PRR who's a, like a marketing consulting firm who's bringing their expertise on how to reach out to customers, how to, in doing this evaluation work. What are some ways to make that we're reaching people in a way that they will be responsive to, that is attended to their needs and different considerations. And then we also have folks such as, you know, public health and, um, DSHS that can speak to more of that customer experience of like, what is it, what does it look like when someone is enrolling in this pass or interacting with these systems? What does it look like for them to be asked some of these questions? and so what's been really striking to me for this program in that kind of super collaborative approach is how much each of us being able to bring in our own knowledge and our own kind of lens of seeing the world makes this evaluation work be more holistic. Instead of looking at strictly from like, here's this transit lens, or here's this economic lens, we're able to make sure that the evaluation is capturing more of the impacts of like, how does this impact someone as a whole? What are we learning? And then also, we'll also I'm sure, learn what are we missing? And that can also spur other evaluation work. And another thing I wanna add is that I think when we first started trying to prepare for the evaluation, we're doing it as we were trying to launch the program. And so there was some elements of, oh, these are things that we should incorporate that will be helpful later down the road for evaluation. What we also then learned that there, we didn't need to try to rush it. and so we actually started, like I mentioned, we started the subsidized annual pass over a year ago. And only now are we starting some of that evaluation component and looking back at it now, I'm really glad that we did wait and take this time because I think it helped us to really learn more about the different frames and perspectives that everyone's bringing. and to be able to make it a fuller and richer program, that we'll learn a lot more from.
Speaker 3Thank you. I'm glad that you said that actually, because, um, you know, we talked about the first project and I, I labeled it the idolized version, but I certainly did not need to imply that with what you all are doing, that, even if the evaluation starts after the program itself has started, that there isn't a great deal to be learned. Um, and can, it can't be a fabulous project in and of itself. I'm curious though about, for those who are starting, the evaluation after the project has started, whether there are specific things that you have learned or, um, guidelines that you can provide to, to help them to be more successful.
Speaker 8Well, I, I, maybe I'll, I'll, I'll go this time. You know, I, I have, uh, tremendously, as Maria's highlighted here, tremendously appreciated the multidisciplinary aspects to it. And I think that that can totally hold true in the planning phases for an evaluation or an evaluation of stuff, of data that already is in existence. And I think part of that is just bringing different perspectives and really understanding kind of, what data elements make most sense to any given stakeholder that you're functionally trying to engage with or present with. So I think that that feels to me, uh, still, uh, tremendously critical or in a universal theme. I think the other thing that I made me think about, I think as you're asking the question, which is I think, uh, data still matters here, right? Which is like, how do we kind of connect all these data points and data elements together, because that, that would be reflective of the multidisciplinary, multi, multiple different stakeholder, uh, perspectives to get, get. A common goal.
Speaker 7Yeah. And I think I'll just emphasize that even though we weren't able to start the valuation until after the program launched, the fact that we were still thinking about it before it launched definitely helped create, make, make sure that some of the framework, and some of the, the kind of foundations that would be important for the valuation were still in place. And that even still, it wasn't too late, for some of the specifics afterwards.
Speaker 3Thank you. I'll just ask one last question. I dunno if you have anything, uh, any of you have anything to share about what else is important to, bring about effective, successful, research or practitioner partnerships like you guys covered quite a bit, in your comments?
Speaker 8I'll also, there's like a little secret sauce that we may maybe don't talk about all the time, which just, I think the collaborative nature of everyone and. I've generally found on this project, there's a, a real level playing field. Uh, I don't know if it's like the impact of Zoom where everyone's just like takes up equal real estate space on each, on the screens as we're we're talking. But I think in a lot of ways it allows, uh, everyone's voice, I hope, uh, to be heard or feel like they're heard. And, and I think that really is, uh, just to one hopes. It's a really beautiful project at the end. Because, because of that, a level playing field,
Speaker 2I think it's worth calling out too folks, and was mentioned, Robert, who been supporting this project, I think did, they did a good job, I think, of connecting people who would be supportive of that type of collaboration, right? In bringing us together so that, to do something that's fairly complicated as a big team and it's very interdisciplinary and, and could be unwieldy with o other context, right? And, and, and bring folks together who'd be happy, contributing to that, that big, and then getting some support to, to help give some infrastructure and to, to make that work. So I think calling them out also is important because.
Project 3 Teaser: Emergency Rental Assistance Evaluation
Speaker 7And then I wanna add, I think actually building a little bit on what the previous panelist just shared on like, the importance of, um, collaborating with community or getting that buy-in. One thing that I think really ultimately shapes the way we designed or were thinking about evaluation was at first we thought we would be able to do the evaluation to have those baseline surveys at the time that people were enrolling in the subsidized annual pass. And from a kind of evaluation, logistics standpoint, you know, it sounded like a great idea. We're like, yes, this will make it easier. We're capturing people right at the beginning. We won't have to then follow up with them later. but when really important feedback then that we got, again, from our partners that were more involved, in directly in interacting with customers and knowing what it looks like for them to be enrolled in the subsidized pass was, Hey, actually that's not gonna be, beneficial for you or for our clients. Sometimes they might be at the time of enrolling, might be just after they finish answering a whole series of questions of are they eligible for these cash benefit programs? So their mental capacity is going to be kind of limited or also that, you know, the capacity on the the staff is also gonna be a lot, it's a lot to ask for them, for the customers and for the enrollers. And it was, just a perspective we hadn't thought about. And I think in, in realizing that we didn't have to have those two intertwined as we originally thought, and also allowed us to, you know, take more time to think about what is a way that, um, we could still make sure we're capturing the information that we want to in order to learn, um, but be able to make sure that it wasn't, overly burdensome on our customers either.
Speaker 3Thank you for sharing that. so unless, uh, anybody else has anything to say, I think we're gonna move on to our third project. So, um. I, I think all the projects are timely and important. I think that, something that distinguishes this particular one is, how the, how COVID, how of the moment it's in terms of people having to switch really quickly to provide emergency, supports to people that they might not have needed to provide. And I know a lot of organizations who have done that and who have not necessarily, uh, brought the research and evaluation along to be able to understand what the, impact of, those, uh, shifts in focus, and resources, uh, really need. So very excited to talk about this particular project, that has been able to quickly gear up to do evaluation work. so Christina could. Introduce yourself and could you talk with us about King's King County's emergency Rental Assistance intervention?
Speaker 9Absolutely. thanks Betina. and first just thank you Carrie for inviting me, and, uh, thank you David for being the kind of connection point between all of us here and bringing us together. Um, it's, nice to see you all. I'm the housing and Homelessness Evaluation manager for King County DCHS. So I lead a team of evaluators who focus on, performance measurement and evaluation of homeless housing programs, affordable housing programs, and increasingly COVID response in the last year. Um, we've been working with Leo and David in particular for several years now. we started with a randomized control trial of a youth and family prevention program. and so, uh, that, that relationship is really key to getting this project off the ground. Um, an evaluation of our eviction prevention and rental assistance program, which began, last year. And this is primarily a COVID response initiative. Um, the eviction prevention piece is legal aid, and then the rental assistance, piece provides, uh, you know, pays rent on behalf of eligible clients who, um, are at risk of eviction because of COVID-19, uh, impacts. there were three funding streams for this. I won't bore you with the details of that, but basically to say that with each COVID relief bill, more funding came. And so this is all focused on, the first COVID relief bill with$37 million in 2020. there's a 35 million coming through the county in 2021 and hundred 77 more million that's not obligated yet, but likely to come from us. So this is a huge amount of money, um, an unprecedented banks startup this quickly. the whole country is feeling the urgency on this. this money is coming through every community. uh, and in, so in some ways localities are kind of little labs of policy right now. We're all trying to do this new thing, uh, in different flavors in our communities. it's a huge effort to do it this fast to meet all the reporting requirements be equitable and how we distribute it and get the money out the door. And we're all racing the clock with patchworks of local eviction moratoria. and so we, uh, have seen extensions of that, but we're always kind of, kind of pushed to the next deadline. and I think in the broader context of this, I'll just add that the pandemic has pushed boundaries of what local governments can do. Um, aid is coming in huge amounts. Red tape is kind of being cut and it's created a lot of opportunities for natural experiments, which I think is a huge factor in our ability to do this quickly. and so I'll talk a little bit more about that. But, um, I think this is just, we saw kind of a huge opportunity to be able to test something novel at scale that could kind of inform. What, homelessness prevention and rental assistance look like into the future.
Speaker 3Thank you for sharing that, Christina. so I'm gonna turn to David again. and David, I'm sure now, the audience is getting a sense of how deep the partnership they county and Leo, so many good projects. So how did that relationship factor fit into your ability to stuck in quickly on this evaluation?
Speaker 2Yeah, I think it was really fundamental. So Christina and her team at, at, at King County Hs. And, and working together, on a prior project related to homelessness prevention for multiple years when COVID hit and all of this started happening. And so I think that did provide the context in which both sides should come to this and, and say, yeah, this is a partnership that's gonna be useful on both sides. I think some of the important elements there of having that track record. Some of it are tangible things that, right, it's hard to do some research activities quickly, right? It's hard to deal with the legal side of data sharing. It's hard to, understand quickly what administrative records, like what, given that in a short timeline you're not gonna be able to do a lot of new data collection. Understanding what data already exists and not having to sort of sort that out really quickly, um, on the fly ride back. so I think those, some of those like kind of hard quote unquote hard things of, of, of understanding what a context is and what's possible was there, but I think probably even more important that is maybe the soft side of things, right? Of just having had that relationship and trust that goes both ways, right? Both from team county looking at us to Leo and our, okay, these are, these are our partners that we can trust our data with and we can trust, uh, to think carefully about. Of this program to ask question, to know King context. And we can look at them and also know, yeah, they're, they're, a group of people with incredible expertise, right? They're gonna be running this, uh, they're gonna, and we know that they're gonna run it in a way that's, uh, where they're gonna, be a reliable partner. We can understand how those things function so that we can on the fly really quickly, not, you know, have to get involved in every little step along the way, which you can't, when things are so intense. And so, and Christina's team being pulled lots of different directions, just trying to get money out the door to people who really, really need it, right? That we can focus at some, on some of the higher level things, in the, because we have that foundation of having trusted each other work together on, on prior projects. So I think that is sort of a hard side of it, but in some ways like the, the soft side of, of trust in knowing that, okay, this is a partner I work well with, um, is a big part of being able to respond to something that happened so rapidly, program.
Speaker 3Thank you, David. So I'm actually gonna ask a question that, that anybody could respond to. so Erin Hagen had asked the question about the dynamics of forming these relationships. You know, I make it sound like Shangla, but you know, people have divergent priorities and perspectives. and so, you know, how do you, get to that place where, you know, all everybody's aligned and things are moving, uh, forward as they shut?
Speaker 2I mean, one thing that I would say is that one, one inevitable part of it is just sort of sticking with it and, and listening really well and working well together because they're right. There's gonna be differences at the beginning and there's gonna be misunderstanding. Meaning that you have to sort of understand, okay, how, how can I understand the perspective that someone else is coming with? So, like on the previous project that we had with folks at, at ATS and Homelessness Prevention, but one of the conversations that happened there, we started with, okay. What's measure, outcomes only, using, whether this person shows up back using homelessness programs again as an outcome. And early on in the design process, there was a lot of pushback from folks who were providing these services to saying, okay, that's, that's not a full picture of housing stability. When you get to the end of this and get the data, there's gonna be other aspects of housing stability that are important. And then you fast forward a couple years and that turns out to be true, right? And, and so I think it's really important that that, right, that's early on, having both, having a context in which people are invited to share their thoughts on what's being planned in terms of evaluation. And then there's a listening year to say, okay, that might make our life a little bit more difficult on the researcher side to figure out how to do this, but, but to make sure that we're listening so that, that, that, that environment, where people are able to have conversation and be able to say what they actually think about things and hear and have research design and evaluation design respond to that. So that. Actually measuring the right thing. I mean, to me that's, there's no substitute for just that process of conversation and communication. It doesn't mean we always did it right, but it means we, it gives us a better shot
Speaker 3as long as you keep trying.
Speaker 10Anybody
Speaker 3else?
Speaker 10Well, Tina, I might, suggest, uh, that, that Maria chime in here too. She's had some really great perspective on, what it means to be a researcher that I'd love to have her communicate to the group.
Why Evaluate During a Crisis: Rapid Learning, Prevention, and a Built-In Lottery
Speaker 7Yeah. So earlier I think Matina, you mentioned something about, being the bridge between researchers and practitioners, and it reminded me of how, like that's the, the role that I've been seeing myself, except I had phrased it as, you know, capital that I myself as like a lower r researcher working with these capital. And then realizing that that's not actually the, the way that I should be perceiving it. That everyone I've been working with is a researcher in their own way, um, and is bringing in their own expertise and that we at least need to be, be doing a better job of being able to honor the different types of research expertise that everyone is bringing in to each of these different projects.
Speaker 3Thank you. That's really helpful. It resonates. so Christina, I wanted to, to go back to you, to ask you, so given the, the urgency of this project, you know, where everything was happening so quickly, and the need was so obvious, why was it important to make sure to include an evaluation?
Speaker 9Um, thanks. Yeah. Um. I think one is, um, so we knew this was just the first of more to come. Um, so, uh, at the time we were implementing, you know, we knew more COVID relief would come through. This ended up being kind of phase one of three. each, each phase had sort of slightly different requirements and it was very clear in the way that this sort of turned out that the country and treasury at the, uh, national level who was sort of adminis during the funds we're all learning together and sort of adapting the requirements along the way. So I think there was just a huge opportunity here to try to learn as much as we could to inform those subsequent iterations of Epra and the COVID response setting at least. and, and we, and that's exactly what we did. We ran our survey, David and team, in, uh, winter of last year. It was pretty quick turnaround, maybe six weeks or so. and David was kind enough to share some of the data back with us and we immediately started, um, mining the customer feedback from some of the qualitative components. Brainstorming, tangible changes to 2.0 based on that feedback. so, so that was a big piece of it. I think more broadly speaking though, it was an opportunity to make the case for what prevention could look like post pandemic, locally and nationally. I think, we get a lot of attention locally on homelessness response. but you know, once you're experiencing homelessness, every other system has failed you and it's much harder to solve your, or solve your homelessness crisis than it would be to prevent it in the first place. But homelessness prevention is a tricky thing to study because it's hard to come up with that counterfactual. It's hard to measure what would've happened if you did not provide this. And, um, so here we kind of had this nice natural experiment. We had inherent scarcity in the tenant piece of the programs. We had to have a lottery built in to the service delivery model. And that lottery was managed and designed by my team, so we didn't have to kind of deal with the tricky ethical piece. Of randomization because it was just a inherently a, a scarcity issue with the resource. and so I think it, it's a, it's a nice opportunity in a real world setting to test a homelessness prevention program and also to look at what the non housing outcomes were. And so a lot of our, data collection involved other kind of, ancillary benefits of, of the initiative. Um,
Speaker 3thank you, David. Turning to you again. Um, I wanted to just explore a little bit the, the difference in terms of researchers, uh, evaluators often wanting to have time so that they can get to the right answers and policy makers needing to address things as quickly as possible. Um, and so that's a place where those sort of divergence in priorities and perspectives. and could you tell us about how that factors into partnerships?
Audience Q&A
Speaker 2Yeah, definitely. I think you're right, right? It's our different constituencies, right? My academic and policy can make making community that wants, you know, answer whatever question you're gonna answer, but make sure you absolutely have the right answer, right? And maybe a policy making community that's like, this is the question you must answer and then do the best you can. Right? It's, or maybe the caricatures of the two, right? and I think in this particular context, right, it's particularly stark because like Christina said, the next version of the program is being built just a few months later, right? And, and so I think it was incumbent on all of us to make sure that we're coming together at the beginning of this and everybody is setting clear expectations, like, okay, here, here's how this can be informative and useful locally, and not just sort of an extractive thing that's pulling out data and informing other people in other places. And, and here's also how it can be a useful thing given that this is a national conversation that's happening other places. Here's how, what we're learning at King County King. And so that involve like, and so there's a commitment upfront doing both of those things well, and then that requires little specific commitments, right? It requires, like Christine said, a commitment from us as researchers to say, okay, we're gonna add a few questions to this survey that we're funding with, you know, outside research money or whatever. But we add a few questions to this survey that are solely there as process questions to help inform the next enes program, right? And that's part of us being bought into things that are local. And then there's parts of from Christina's team that's bought into this, the fact that this program can help inform other people in other places as well to say, yeah. Right now, folks on her team are helping work us work with, uh, the administrative data side of things so that we can compare results of what's happening in Seattle to what's happening in some other cities. And, and the whole country can learn about how best to run these type of programs. And so I think right there, there's sort of real practical commitments upfront that, that make for, you know, time for everybody that, that, that people are gonna put into this thing to make sure that we can do both those things and do them well instead of just doing one and leaving the other one, shorthanded. It, it takes a lot of planning upfront, I think, and commitment upfront to make sure that we sort of listen to each other, we set those expectations and then you have a partnership where everybody's excited about. We're gonna get outta
Speaker 3okay. Thank you. So we're, we're gonna turn to audience questions. but before we do, I just wanted to give all of you an opportunity to share with us anything that you might have forgotten to say. I know I always have my talking points and there's one or two things that I thought, oh, this is so important, but I didn't mention it. So if that is the case, um, please go ahead and, and share. Alright. Yeah. Oh, that's great. Okay. So, um, let's just jump into some questions then. so actually, going back to the, the first presentation and we were talking about, uh, how great it to start everything, from the beginning and I wonder there are, Any cons to that? So, uh, were there, you know, any places where you wish that, things hadn't started, uh, concurrently.
Speaker 2Alright. Patina, you're asking for the, the research that the research and the,
Speaker 3program, and the program started together and I, I find it as the ideal. but I, I wonder if there are, you know, any, negatives to that?
Speaker 2I mean, there's, there's trade offs, I think. Right? Earlier we spoke about some of the advantages of getting involved early on, um, right. Some of the advantages of starting later is that program design is sort of more set and more clear. What. The thing is thing that's gonna be, and it's easier to then at that point you can kind of design around it, but the, yeah, the disadvantage of of being in that situation of where things are set is that then there might be research activities that are would be good for what the ideal of what we wanna learn, but then would cause some cost and some, some friction for folks who are actually running the program. And then, then you have to make hard trade offs between, okay, are we gonna learn something that's a little bit less than what we wanted to learn? Or are we gonna, have something that's gonna take a little bit more time for people who are operating this thing? Right? And then those real trade offs between what are we gonna learn and, and how hard is it gonna be to do this thing or can get a little bit star once things are are, are in process. I'd think that's the, in my experience, that's been the times we've run into disadvantages starting a little bit later.
Speaker 5And if I can just add to that, I think David makes a lot of good points. I think the other challenge that we see kind of from that community perspective is I don't wanna be part of a research project and so. I don't, you know, it's great that you wanna ask these questions, but gonna do with this information. Um, and so that was something that we really took into account with our first phase, with the community transportation before we even moved into design, is how are we being respectful of the community's engagement in this project? So they don't feel like we're doing research to be frank on them. We're doing research with them and that they'll see outcomes. And so the one thing that we're really carrying through this is before we, we approach new partners about engaging in our program, we're going back to our original partners and making sure that, that they have the right, not the right, but they also have the ability to continue participating in this, that they get to see those outcomes and be part of those outcomes because, we definitely got some of that feedback early on. Well, I don't wanna be a part of a research project or I don't want my clients to just have research done on them. And then also always protecting their privacy. And so kind of balancing our need to gather information with the privacy of the client. So I just, I wanted to throw that in there as well. as, as a, not a, not a negative by any means, but a challenge that we face, in working with the community.
Speaker 3Thank you, Sarah. I'm curious from everybody else whether there are challenges that you'd like to, to speak to as well, because it's always helpful to our audience, uh, to hear about, uh, what others have struggled with in any lessons that they might have learned.
Speaker 8I'll make a comment that, uh, I think we kind of alluded to earlier, which is about the speed of findings or how fast you get to our results. And I, um, I'll say I, I hear, I heard that tension. I think in our collaboration from a policy making standpoint and the timing of. That information, kind of falling in at the right time when decisions are being made. Uh, we also hear it a lot on the healthcare delivery side of like, there are some things that, research would just slow you down'cause you're like, you need to make a decision tomorrow, that kind or yesterday sometimes. and so I think it is, it is kind of, attention here is to choose the right kind of projects, collaborators tempo that kind of makes sense for perhaps some, some longer term strategic vision or some, some way to inform those processes. And at the same time, I think you can, I always, I'll say you can actually incorporate researchers to help think about kind of the yesterday questions of the tomorrow questions. It doesn't have to necessarily lead to say, publication or some kind of product there, but I think there, I hope, I think there's some value in thinking about how to kind of, take the information that you have and, and make the best kind of evaluation or, um, or get to the closest answer you can, to make that that more immediate decision too.
Speaker 6Oh, Carrie, are you raising my hand?
Speaker 10Yeah, I'm, I thought I would just add in a little perspective on this point in the conversation here, which is, you know, I think as decision makers, and this really points to a lot of what Christina was talking about, we're really used to dealing with imperfect information. And so, you know, the fact that we can, we can work with researchers and the, the results that we're getting may not kind of reach the researcher standards of what the quote unquote right answer is. But for us, it's like better imperfect information than what we had before. And so, you know, I'm always interested in trying to get to the best, imperfect information that we can. And really try to understand what those imperfections are so that we're kind of making decisions with our eyes wide open. yeah, I'd I'd also say that, you know, we we're really trying to be clear about the decisions that we have in front of us and making sure that we're tying the research work towards, towards those decisions. And that we never view one study as like the definitive answer, right? Even if we go through a whole process of, you know, a peer reviewed publication and these are the results and we feel really confident in those, there's always more for us to learn underneath that. And so we really have to view, any project as an evidence building project that we're gonna add to, over time. And that's why I think, you know, these longer term, collaborations are, are so important because. We, we basically have an, we have an endless pipeline of, of research questions that we wanna answer. And, a, you know, a great team of people that we can work with over time to, to answer those.
Making Partnerships Work at Scale
Speaker 3And every program, every project can benefit from improvement, right? So while at the har, while I was at the Harlem Children's Zone, you know, if 95% of the pre-K students were dean school ready at the end of the year, I was always focused on that 5% and what to do to get them to where they need to be. And everybody would say, Hey, give us credit for the 95%. I'd say, yes I do. but how about the rest? How can we do even better? and it's not just about one and done, but about continuous improvement. and so I think that these partnerships are also tremendously helpful for that. So one thing that I, I'm always curious about is, you know, already this seems like there's a cast of thousands, for each of these projects, but I know that there's always more people who are involved in preparing the organization, preparing the container organizations so that, uh, there can be success. So I wonder if anybody would wanna speak to the work that was done. you know, the process of, getting buy-in, getting engagement, and making sure that the work moves beyond your little, your, your team cluster.
Speaker 9We can try to add something to that. I'm gonna too my team's horn a little bit and my broader team as well. and we're part of a performance measurement and evaluation unit with DC Hs. And so that unit has grown threefold in as many years. And has been investing heavily in data infrastructure and data integration across multiple administrative data systems. So I think it's been really helpful in sort of setting up the internal seeds that sort of power a lot of these external partnerships. And also we have staff who have evaluation and research background in-house who can kind of embed closely with programs, make relationships with them, make these connections between external parties and programs, and kind of speak both languages and sort of help to kind of alleviate the fears you might, that program staff might have about, you know, some of the things David talked about of like, you know, adding data collection or tweaking program design to sort of facilitate evaluation. And I think that was very much the case with yipe, their, um, our prevention program for the acronym, um, and our ev uh, eviction prevental prevention and, um, rental assistance, program. And it, it also means that we are able to sort of see the opportunities. So, um, several times throughout the pandemic my team have, were sort of been, you know, the, the forest gumps watching history gone around. Us watching the program staff just move mountains to just make amazing things happen. And we're able to sort of see there's this natural experiment. We need to find someone, you know, David, come here, David, help us with this thing. And so being able to sort of like have the, the seat to see those opportunities and then also to say, Hey, at this data element, because if we flag this thing, then we can now study it. and so I think kind of building that internal data capacity is, is kind of, foundational to being able to do some of these bigger projects and invite in, our external partners. And I know, um, we've had great relationship with David and, and Leo, and I'm also trying to cultivate more relationships and it's sort of been inspiring. We, we participated in a, MIT collaborative that sort of brought us together in the first place several years ago. But now we're sort of reaching out to University of Washington partners. They have a homelessness institute, you know, trying to kind of have, um, multiple folks that, that we know of already so that when we see these opportunities, we can jump right on them.
Speaker 2I just really wanna underline a couple of things that Christina said from the researcher side. Just I, I, king County generally, and specifically King County Metro and, and DC Hs, have spent a lot of time building capacity in terms of the people and also the data systems and such, but especially the people who are involved in evaluation, that this stuff doesn't happen whenever, right. The, the world that we live in tends to say, uh, you know, a Notre Dame study says X and it puts my name and two other people's name on it, and reality of a project that like hundred different people at least have contributed to, right? And it's like whenever I present these things, the first in the presentation is always just like a thank you, just a gigantic list of names because of these. People for it to actually function. That I think it speaks a capacity, uh, both people on this call and just like dozens of people who are not on this, this Zoom thing right now, who without who like this, these things just absolutely don't happen. And, and so I think, yeah, I love your question, Tina, because that that internal capacity, both for actually producing the evidence and then receiving it and doing something useful with it is just so incredibly important to, to make sure that we're not just creating evidence for evidence, but actually creating it in a way that's gonna be useful.
Survey Timing & Response Rates
Speaker 3Thank you. So I'll, I'll just everybody to a comment that Carrie made in the chat, which was that Christina was, was referring to J Pal at mit, which, um, matched King County and Leo. Um, and there there's a link in case anybody wants to learn anything about that. So, maybe you'll let know if I mispronounced your name. Uh, but I saw that you had a, a good question that, um, I'm happy to share with the audience if you wanna go ahead and, and share and, and actually ask, Maria, Krista and David. so it's up to you. You want me to ask or
Speaker 10sure. I can jump in and give, I know it's a very specific question. Um, but, we have, uh, two experience with two different projects. One in which the evaluation was embedded at the very beginning. So when people were applying for, in this case was fair fares in New York City, we embedded a link there to a survey, asking about how they use transit, et cetera. And that's, they could answer that survey within 30 days of their application. And Maria just mentioned like. When someone application that requires so much, like thought and revising documents, et cetera, is that the best time to ask them even more questions? I guess our thinking was like, okay, they, they may not come back. And the, the other question is like, okay, I'm working on another project not related to transit, but the program already passed and we're gonna be contacting people, who participated in the program, six to a year, to six months to 12 months after they participated. And we're con concerned of whether they're gonna remember or not. So just wanted to hear your thoughts on how you may have, deal with these two different things of when you're asking the questions.
Speaker 7Yeah. And I can start answering this and then again, my colleagues can jump in. But one thing that, for the first question of, you know, I guess for the subsidized pass, we're actually trying to survey people before they enroll. So anyone who is already enrolled, we're not following up with them for that reason, that then we're not able to actually capture a good baseline survey. But we're trying to capture people who, would end up enrolling and that through the process of doing the base, doing the baseline survey, then enroll. it separates out that process a little bit. Um, the other question of how to increase engagement, one of the things that our partners here are, have been really great at is, making, yeah, so there's been a couple things. One is making sure that's available in more than just English. English that is translated in multiple languages. And they're also being very, very, very intentional about, pretesting the, the survey materials with a couple of people that would be filling up the survey to get a sense of. Hey, how does this sound? Does this make sense? Does, is this any of this scary or like, you know, turn turns you off? And this actually brings me to something else that I wanted to point out. As Sarah was talking about, privacy earlier, that was something that came up during the pretesting, which was originally we were like, oh, we're gonna connect your information to administrative data. And then the question was like, well, kind of an administrative data. And it did force us to pause and, reflect. And then we end up being more transparent about the data that we were collecting. as well as then also really examining which data are we collecting and which is actually necessary in order for us to be able to, make that impact evaluation that we're trying to learn. And which was more of good to know. But really if, if, participants didn't feel comfortable in getting that information, then we really shouldn't be asking if it's not actually moving forward our learning agenda. But yeah, so ultimately I think being able to have that kind of community testing, having that community review and weighing in on what that looks like, I think is gonna help our response rates. At least that's the hope we're about to start. So it's hard to say for sure,
Speaker 2maybe two little things to add in addition to those, which I very much agree with. One is, just on the practical side survey again highlighted our partner, good work on the survey side. and right, and one thing they nudge in the direction of is a good idea for this one is we're trying to follow up people a year later and one thing they nudge us, okay, let's have some contact between not just, you know, when you start off a year later, but also have lots of context in between to try some connection. That's one increase response. Another is recognizing before. Dozens of people in terms of the staff that are involved in these project. It's also the case that every respondent to a survey, right, is a fundamental contributor to this project and their time is valuable. And so to the extent that it's possible with budgets and getting support from external and stuff them for their time. And so like the people who are gonna taking survey, you know, they're, they're providing us and, and their community with lots of useful information and they're gonna get compensated for that. So I think that's, whenever that's possible is both from a respect point of view and from getting high survey response rate point of view is, is a great thing to do. the last thing I would say, and, and Maria also just underlining one thing that Maria said and maybe expanding a little bit is, sometimes with, use of administrative records can help a lot in a situation, particularly like the example we did with the, emergency rental assistance. We have to move really, really fast and you may not be able to put in all of the foundations that you would like to for a really high response rate survey. That being able to pair that also with some data that's already being collected. On folks and using that in a careful and responsible way and de-identify and so on, might allow you to learn something more about the whole group of people. Even if you can't learn everything, you can learn some things about the whole group and then learn, some more detail about a smaller group of people. those are some of the things that, that we've done to try to deal with the difficulty of just people are busy and they don't,
Closing Thanks & What's Next
Speaker 3I appreciate comments, so thank you all.
Speaker 10Thank you. So Tina, a big part of building great researcher, pro practitioner partnerships is being grateful and thanking folks and congratulating everybody on work. Super well done. So thank you. Thank you, thank you to all of our panelists. I also wanna call out Matt Friedman, who's in the audience, and, Chong, who is from one of our partners from the state, um, who have been really instrumental in this work. So thank you all. One of the things that's the things that's been so great about, our work together across researchers and practitioners is that each one of us is really putting the community first. And, you know, it's really recognizing that the service comes first and then let's organize our service in such a way that we can get the best information about the impact that's coming out of it. And so that's part of what has led to such, um, great, great partnerships. we'll be in touch with all of you soon about, our next event, part two of great researcher provider partnerships, coming up in January date, TBD. You'll hear, um, about another three or four projects, at King County where, um, we've got other research partners we're gonna bring in. David will certainly be part of that conversation. Matt will be part of that conversation. So, really looking forward to having you all there. And, um, in the meantime, enjoy the winter holidays and thanks everybody.