Evidence Matters, Part 8: The Power of Learning Agendas

Episode Topic: The Power of Learning Agendas

Unlock the blueprint for modern governance in this collaborative breakthrough featuring King County and Notre Dame’s Lab for Economic Opportunities (LEO). Learn how Learning Agendas transform evidence into equity-driven results. This recording is the essential toolkit for leaders seeking to accelerate organizational impact through community-centered policy evaluation. 

Featured Speakers:

• Diana Epstein, Evidence Team Lead, Office of Management and Budget (OMB)
• Keith Fudge, Director of Policy Management, Research to Action Lab, Urban Institute
• Dr. Calvin Johnson, Deputy Assistant Secretary for Research, Evaluation, and Monitoring, United States Department of Housing and Urban Development (HUD)
• Vincent Quan, Director of Strategic Partnerships and Innovation, J-PAL North America
• Dr. Eva Wong, Senior Maternal and Child Health Epidemiologist, Public Health—Seattle & King County

Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/15f08d.

This podcast is a part of the ThinkND Series titled Evidence Matters. 

Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.

• Learn more about ThinkND and register for upcoming live events at think.nd.edu.
• Join our LinkedIn community for updates, episode clips, and more.

Transcript

Welcome and Introduction

1 0:06

All right, well, why don't we get started? on behalf of King County and the Wilson Sheehan Lab for economic opportunities at Notre Dame, I wanna give a big welcome to everyone. Um, this is our fourth event in our series called Evidence Matters, and it's so great to see so many of you friends and, and colleagues here. We're recording the event and we'll make the recording and the materials available, sometime early next week. I'm Carrie Hawk and I'm, uh, evidence and Impact Officer here at King County. Uh, I'm passionate and have dedicated my career to advancing racial equity through bringing together the expertise of our fantastic staff here at King County, our research partners and our community partners. And we started this Evidence Matters, series with three purposes in mind. So first, to build skills and knowledge that focus on equitable impact and, community outcomes. Second, to celebrate our successes, in our evidence-based practice as well as share our challenges. And third, to build a community of practice and learn from one one another. It's been really great to connect with people through this event, and so I really appreciate you all hopping on and participating. Today we're gonna learn and we're gonna learn about learning agendas. Um, simply put, learning agendas are like a strategic plan for building knowledge that's needed to inform really key decisions. And as you'll hear today, learning agendas are increasingly being used as a tool that help agencies to accelerate their impact and advance equity. I'm so pleased we have such an incredible stellar, group of partners with us today. these are leaders who are working across research, policy and practice, and across all levels of government. And, um, yeah, sorry if there's a little feedback. I don't know what that's coming from, but hopefully it's not too distracting and we'll go away when I, when I stop talking. Um, so on our agenda today, um, we're gonna start with a brief overview of the what, why, and how of learning agendas from Keith Fudge. And, he's written one of the, the most cited articles on this topic. And then we're gonna move to a moderated panel discussion, that highlights how learning agendas are being developed and implemented at the federal, state, and local government levels. Each of our, each of our panelists, are going to relate their practical experience on building learning agendas and using them to advance equity in their organizational mission. I am sure that everyone here is gonna walk away inspired by the work that our panelists are doing, and that you'll also come away with some practical insights and resources that can get you started on learning agendas too, and your participation in the event is important to us as well. So, we have time at the end of, the panel for your questions and observations as well. Um, so as you listen to the panel, please drop your questions or observations for discussion into the chat. And Patrick, who, Patrick from Leo, who's partnering with me on this event, will collect those and attempt to answer them in the panel itself or, in the q and a session. So with that, let's, uh, get on with our agenda and lemme introduce you to our moderator today who is also, uh, my partner in this evidence matter series. Rachel Cher Dawson and Rachel's director of research operations at Leo King County and Leo have a very deep, partnership and relationship on many projects. And so I've worked with Rachel for, a few years now and have learned so much from her. She brings to her role at Leo a lot of, um, experience from having worked in the federal government at the Department of Education as well as on Capitol Hill. And she's also done a lot of consulting with nonprofit schools and state governments on education, policy, research and practice. So, Rachel, with that, I'm gonna turn it over to you.

Speaker 2 4:32

Thanks so much, Carrie. And. Welcome everyone. I would just echo what Carrie said in terms of being delighted that you're joining us and excited for what we have to offer you. So I was gonna start by just, Carrie, so nicely framed kind of the goals of our series in general, and, gave us a nice glimpse into the idea of a learning agenda. But we, we have kind of three specific goals for everyone for this session today. first we'd love for you to come away with an understanding of what learning agendas are and also, how they can support you and your organization and advancing equity and other goals. We'd also love, you know, we're happy for you to hear from people who have led learning agendas, the development, to learn about the practical benefits and the challenges that they faced. And then finally, we'd love for you to leave the SES session with some resources and advice on how to get started, if that's where you're at. So, without further ado, I'm pleased to introduce our, speaker. Our first speaker, Keith Fudge. And Keith is currently the Senior Policy Program manager in the Research to Action Lab at the Urban Institute. Prior to that, he managed technical assistance projects for Urban's Pay for Success Initiative and also managed the Evidence-based Policymaking collaborative. He holds a degree in English from Michigan and also Masters in Public Policy from Michigan's Sports School. So thanks so much for joining us, Keith.

Steps and Engagement in Learning Agendas

Speaker 3 6:02

Hi. Thank you, and thanks for the opportunity to talk to you today and just give a quick introduction on learning agendas. if you could advance the slide. one more. thanks. so what is a learning agenda? Uh, at a high level, a learning agenda, as Carrie said, is a set of prioritized research questions and activities that guides an organization's evidence building process and informs its decision making. A way to identify what is already known, what evidence gaps exist, and what research is needed to fill those gaps to improve an organization's work. And critically, it's a process, not just a product. Um, this is something that pretty much every person who's worked on a learning agenda, every piece of research about them will emphasize, but the steps required to develop a learning agenda can themselves make an organization better engaged with partners and more learning centered. the federal government has been key to the development and evolution of learning agendas as a model, as you'll hear more later in the panel. for instance, HUD's research roadmap was an early example of a learning agenda that really prioritized engagement with external parties. I had the opportunity to work a little on that first roadmap when I was at the agency, and you'll hear a lot more about it from Dr. Johnson. Um, and the Department of Labor was another early leader, um, with each of its 15 operating agencies, creating five year learning agendas because of this and all the advancements in recent years that you'll also hear more about much re much research discusses learning agendas in the federal context. But they've also been developed for other levels of government and in other sectors as you'll hear more today. And in terms of scope, mini learning agendas that focus on capturing research questions and activities across an organization, but others have focused a little more narrowly on a department or sub-agency, and still others have more broadly centered on cross cutting themes to promote learning across one or multiple policy areas or field. On next slide, please, please. So when done well, learning agendas have many benefits. They can help organizations to set priorities by taking census of the state of evidence on key topics, and learning from internal and external partners about the most important questions that need to be answered. Promote equity both in terms of process and what research is conducted. As I'll talk more about later, equitable engagement processes can ensure that people in communities who typically may not influence the type of research and organization conduct can bring their perspectives to what questions are asked and how they're prioritized. By reviewing existing research and developing learning questions with an explicit equity lens, an organization can more clearly see gaps in existing knowledge and conduct new research to better understand the root causes underpinning disparities so that its work can better support bipoc communities and others being harmed by structural racism and structural barriers. learning agendas can also garner buy-in from organizational leaders and partners by demonstrating how evidence will support them in their work, can align evidence. Building activities with organizational strategic plans, budgets, and program timelines can improve collaboration and coordination within and beyond the organization, and of course, generate new knowledge to help programs and processes operate more effectively and feed into that broader performance management goals. And they, of course, promote, uh, continuous learning by creating an environment that encourages individuals and offices to learn from new evidence and from others which can lead to adaptation and innovation. For these benefits to be realized. It's critical that a learning agenda not be seen by people within the organization as just a mandate or a box checking exercise. So it's essential to involve people through throughout an organization in the process. It can't just be limited to the core team leading the development of a learning agenda who often may sit in a research office or a similar function, and that means engaging staff throughout the process and demonstrating the value of the agenda to the organization's mission. Next slide please. Next, I'll just, um, quickly touch on components of a learning agenda. Document. These elements can and should vary to meet the needs of the organization developing it. I really encourage you to look at some of the different models and examples Our panel will point to. common EL elements though often include a background and methodology section explaining the process and organization use to develop their agenda. A description of the agenda's, goals and objectives, a set of learning questions, which are prioritized and often categorized within thematic areas. Associated learning activities often including both research-based activities and ones that draw on expertise from practitioners. And finally, learning products that support dissemination to a range of audiences. Next slide, please. The learning agenda process has several key steps, and different organizations might approach them somewhat differently, and it might be better to think of these steps as part of a cycle, since learning agendas are meant to evolve, to reflect new knowledge and priorities. But generally, a learning agenda process usually involves some of these following steps. First, conducting internal and external engagement. Though this actually occurs throughout the process and in a minute I'm gonna talk more about that in greater detail. Next, reviewing existing knowledge and evidence, building activities that are underway and creating a systematic approach to identify gaps in current knowledge, and then with partners developing and prioritizing learning questions and the research activities best suited to address them. Then of course, writing the learning agenda, getting it reviewed and approved by necessary leaders and disseminating it to key audience. To complement the full agenda document, organizations that develop payload products for different audiences, including web pages, infographics, and two pagers, briefings and events and other approaches to share the agenda beyond research, constituencies. But that's just the steps to release the learning agenda. After that, of course, comes the work of conducting the research activities that features, and then using new learning from that research to inform and improve the organization. Finally, the cycle completes with a plan for updating the learning agenda on a regular basis that might be annually every few years or longer. And with that said, many organizations see it as a continually evolving living document, not something static that's only updated every few years. Next slide, please. Now that I've, uh, described the steps of the process, I'd like to dive a bit more deeply into one really important piece, which is the role of engagement. Uh, when done effectively, both internal and external engagement can create a more equitable process and increase buy-in for the activities of learning agenda proposes. As USAID's landscape analysis captures, there are broadly three ways of engaging with interested parties. These methods can be used both within with internal audiences, such as various offices within an organization and with external ones, such as academic researchers, interest groups, practitioners, and the people and community served and impacted by the work of an organization. So the team leading an effort can inform by giving information about their learning agenda, consult by seeking feedback from these parties at a certain time to inform decision making or involve or collaborate with by actively working together with them to identify ideas for the learning agenda and the broader the set of parties that can be involved meaningfully in the process of building a learning agenda, the more effectively and equitably it can answer important questions. So let's talk first about internal engagement. By working with staff throughout an organization such as people managing programs or projects, you can get perspectives about the type of information that can best help them improve decision making in the future. Engagement can also generate staff buy-in toward using research and evidence. So it's important to recognize that staff may have different comfort levels and familiarity with the language of research and evaluation. Methods of engagement used with organizational staff often include individual and group meetings, surveys and internal advisory groups. And for large and distributed organizations, it's important to consider how best to get the input of staff beyond the the main office. Those who often have on the ground experience working closely with the implementation of programs and services. If they aren't engaged well, it can contribute to a feeling of a learning agenda, being a headquarter centric activity that doesn't value their perspective. And finally, involving leadership throughout an organization at each stage of the process is essential. If brought in early leaders can be champions for a learning agenda, creating resources to make it happen, and applying leverage where necessary to overcome roadblocks. And of course, leaders will have important perspectives on priority research topics and input on dedicated, dedicated funding to evidence building. When to external interested parties, it's especially important to be thoughtful about how you approach engagement. As a recent, OMB circular on Learning agenda says, robust engagement should advance equity and meet the needs of underserved communities and cannot be accomplished without intentional interactions with diverse stakeholders. And engagement should be conducted in a manner and using methods that are transparent, generate trust, and advance equity. I know that Dr. Epstein is gonna talk about that further. when developing a plan for external engagement, some key questions to ask are, who are you engaging? It can be easy to start outreach with the organizations and experts you know best, especially if you're short on time or staff resources, but that can further compound inequities of access. So consider how to reach beyond those audiences, especially the people in communities facing structural barriers, whose perspectives have likely been underrepresented, and when engaging, when engaging, be thoughtful about how to make sure the process is collaborative and not extractive. One important way to do this is to compensate people for their time and experience, if that's possible. two other related questions are, when in the process are you engaging and what type of engagement are you seeking? As I mentioned previously, engagement can and often should happen at multiple stages of developing a learning agenda. But the questions that you ask in the process you use when generating research questions will be different from the ones you use when getting input on prioritization. And too often external parties are only given the opportunity to react to priorities once they've been set or nearly so not to engage earlier in developing them. So the learning agenda process can be a great opportunity to really foreground the needs and perspectives of diverse audiences from the outset. And the last question is, how are you engaging? Organizations have used a wide range of methods to seek external input, including email, outreach websites, individual meetings, presentations, requests for information, round tables, convenings and conferences and advisory bodies of communities of practice. Again, many of these approaches, privilege, well connected people and organizations, the ones who are already on the email list, are invited to the conference. So drawing on research about participatory and community engaged methods can help you consider approaches that will garner wider input and, and will value lived experience alongside other forms of expertise. Again, engagement is such a rich aspect of learning agenda development. I'm excited to hear from the panel about the different forms it can take. and so with that quick overview, I'll pass it back to Rachel.

Panel Discussion: Federal Focus on Evidence and Equity

Speaker 2 16:14

Great. Thanks so much, Keith. You've, uh, set the stage very nicely and for all of us, and given I think all of us different things to think about. I in particular, I kind of jutted down notes about this idea of engaging, internal and external people as well as making it a collaborative process. And, so I'm gonna keep thinking about those things as we continue to hear from everyone. So, uh, I'm pleased to introduce now. Has built is our Allstar panel, rightly so. and I'm introduce them and then I will start with some questions, but as mentioned, please do put, um, your questions as they come up in the chat and we'll facilitate that through the q towards the end. So first we're joined by Diana Epstein. Um, Diana is the ev the evidence team lead for the Office of Management and Budget. And, as such, she's in charge of facilitating the learning agendas across multiple different, uh, federal agencies and will serve, kind of to inform all of us about the different, the importance of learning agendas in federal government. We also have, uh, joining us, Calvin Johnson. Dr. Johnson is the Deputy Assistant Secretary for research, evaluation and monitoring at the Department of Housing and Urban Development at hu. He is located within Office of Policy Development and Research, and so he serves as one of the principal advisors to the department department's leadership on research, demonstration and evaluation activities. We also have, um, my colleague Vincent Kwan, who is another, um, research works at a, peer research lab from Leo. He's, Vincent is the co-executive director of J North America. Together with Laura Feeney. His co-executive director Vincent Leads, leads the office's efforts to reduce poverty by ensuring policy is informed by rigorous evidence in the North America region, region. And then we also are joined by Dr. Evo Wong. Dr. Wong is an applied public health practitioner and researcher with her primary appointment as the senior maternal and child health epidemiologist, at Public Health Seattle and in and in King County. So that is our amazing panel for you. I'm gonna turn first to, um, to Diana. So Diana, my question for you, you know, the passage of the Evidence Act in late 2018 and then, the Biden Harris administration's executive orders and memoranda from their first week in office have set the stage for a strong focus across the federal government on both evidence and on equity. Can you give us a brief overview of that and of the role of OMB in coordinating across agencies, federal agencies?

Speaker 4 19:06

Sure. I'm happy to start with that and thanks so much for having me. It's really great to be here with you all today. so yes, as you said, I mean, since the first week in office, the Biden Harris administration has prioritized evidence-based decisions, um, rooted in the best available science and data. So that's starting with that presidential memorandum. On restoring trust in government through scientific integrity and evidence-based policymaking. and that called for expanding evidence, building and evidence use across the federal government so that policy and program decisions would be informed by the best available facts, data and research facts information. that executive order on advancing racial equity was also issued right away, and, and stated that it's the policy of the administration that the federal government should pursue a comprehensive approach to advancing equity for all. so yes, very, early statements on both. the commitment to evidence-based government is also apparent in the, the fiscal year 23 president's budget, which were, which was released, recently proposed a number of new investments in agency evidence, capacity proposals to fund programs that are evidence-based. Um, and you see a real through line, on equity in the budget as well. the commitment to building and using evidence and to equity are also apparent in the way that large new investments are being implemented, such as the American Rescue Plan and the bipartisan infrastructure law. So, as one example, um, you know, we're gonna be undertaking a large national evaluation of a RP programs. and that's, you know, a really unprecedented opportunity for the government to learn if and how these a RP investments are achieving their goals. you know, of which equity was primary. There's also really strong evidence language, in o b's guidance for the infrastructure law that was issued a couple weeks ago. Um, and I also wanna mention that the president's management agenda has a focus on equity and also has evidence-based policymaking. Um, as a through line. And, um, we've put out a draft president's management agenda, learning agenda to tackle some of these key unanswered management questions. So you see both of those, um, in many of the things I just mentioned, evidence and equity. Um, I'll spend a minute just to talk about the Evidence Act and since you asked about that too. So, as many of you may know, the Foundations for Evidence-Based Policymaking Act or Evidence Act has been a really important focus of our, of our efforts for the past few years. Um, it didn't start from scratch, right? So it builds on longstanding principles for federal data policy, analytics, infrastructure, um, but it really did create a new paradigm for agencies to more strategically think about how they use data, how they build evidence. it emphasizes collaboration and coordination across functions. And it's really designed to break down some of the longstanding silos that have really stymied this work. It puts in place a more strategic approach to building evidence as opposed to how it's traditionally happened in most federal agencies, which is really ad hoc, so often in response to a particular mandate or driven by a specific group of motivated, of motivated staff. I'll say two, the Evidence Act, also elevated program evaluation as a key agency function to compliment the systems we had already in place for statistics and performance. and this is really important because there's some kinds of questions, for example, as a programmer, policy working as intended, and for whom and, and why and under what circumstances. only evaluation can really answer those questions. I know we'll talk about learning agendas, I think more in a, in a minute, so maybe I'll pause on that for now. and just keep it at the higher level of coordination. So at OMB, we've been coordinating. Internally, you know, quite a bit across offices that we, that, you know, we haven't always worked together very well. Um, but we've been coordinating to issue guidance for agencies, in a few big chunks so that they can implement this law, in a meaningful way. So we've issued foundational guidance on learning agendas on the senior officials they need to designate. and then on documents like the annual evaluation plans, capacity assessments, we took a really flexible approach for these new requirements whenever possible. It's our hope that, being flexible will give agencies, you know, the space to drive the work themselves and to do it in a meaningful way that really reflects their priorities and makes sense for them. We, we do not want this to be a compliance or a reporting exercise just to check the box. Um, that has never been the goal from the start, and we've been doing whatever we can to avoid that. I'll say too, so last year we also issued some guidance in response to that presidential memorandum that I mentioned at the outset. and that guidance really makes it clear that evidence building is a mission critical function and it really emphasizes the need for agencies to build a culture of evidence. Um, I'd also, you know, be remiss if I, if I didn't mention that we have some great peer communities that are being built around these efforts. Um, we have new councils, we have peer groups of agency staff that are meeting, and I think that's also been a really effective strategy for folks to be working together. and then I also, I just wanna mention one of our latest exciting developments, which is an OMB partnership with the White House Office of Science and Technology Policy. Um, we held a, a White House Summit on evidence for action about a month ago, which was a great virtual event. Um, and served as a launch for our year of evidence for action. Um, and the goal of this year is really to share leading practices from federal agencies for generating and using evidence to advance, um, more equitable and better outcomes for all Americans, to, you know, promote consistent evidence-based decision making and to increase the connection among researchers, knowledge, producers, decision makers, community members inside and outside the federal government. We really, the hope here, equity is a real through line initiative in this initiative. we, we wanna create strategies to mobilize evidence that can make life, you know, healthier, safer, more equitable, um, better for everyone. So hope that gives you all kind of an overview of the ways that, evidence and equity are very much, um, working together for us.

Speaker 2 25:26

Yeah. That's so interesting. I'm glad you brought up the year of evidence in action too, because I think that compliments a lot of what we're talking about here with learning agendas. So I'd love to hear a little bit more, if you can dig in a little more on specifically how learning agendas, play a role in this work. So kind of how you're coordinating them across agencies. I'm just thinking of if each agency's operating in a silo, then those learning agendas mean one thing, but if there's more of this peer groups and other things you've been describing happening, then it would look very different. So I'd love to hear more about that.

Speaker 4 25:59

Yeah, so, you know, I'm really excited about learning agendas. Um, we were promoting this as a great framework even prior to the agency, the Evidence Act, but the Evidence Act really codified it, and gave us a way to have every agency working on a learning agenda. So, yeah, so we have been providing guidance and support and, peer groups and trainings and learnings, um, for, you know, our, for our evaluation officers, their staffs and others, um, for the past three years. So, you know, we've really been helping folks to build these documents from the ground up and to leverage, agencies that were already doing learning agendas really well. So, you'll hear from Calvin in a minute, but, you know, there, there of course, were agencies that were, were already doing this, and had a lot of lessons that could be shared with their peers. you know, I think one of the most exciting things about a learning agenda is, you know, it, it is a systematic plan that really encourages this deliberate strategic planning of evidence building activities. Um, and it elevates important mission, mission questions, and operational questions. I think both are really important. I, I should mention, I mean, it's getting a little bit into the bureaucratic weeds, but. Agendas by law, they're part of agency strategic plans. And so we said from the start, they should be aligned with an agency strategic goals and objectives. And this alignment of evidence with strategic goals hasn't really typically been done much before. But the Evidence Act offers us this framework in which, you know, evidence, building priorities are aligned with strategy and they're envisioned together from the start. And since you know, we're focusing here on equity, this is really important for equity because we've made it clear that both strategic plans and learning agendas need to reflect questions related to administration priorities, of which equity is clearly one. So this means that agencies integrated an equity lens into their learning agendas, as they should. And so we've actually been doing a cross-agency analysis of learning agenda questions, and in fact, um, we do see that agencies have many, many questions aimed at addressing equity issues. You know, and as Keith said, the evidence Act requires consultation with a range of interested parties as part of this learning agenda process. And we've elaborated this on our, in, in our implementing guidance that agencies need to meaningfully consult groups, internal to their agency and external, so the communities that their programs serve, from the outset to identify the right questions and throughout the learning agenda cycle. So it's really important that agencies are working with those that are affected by a policy or served by a program to understand how the resulting information would be most helpful for them. you know, I don't know that we've, we've figured this out the best way to do it. Um, but fortunately we have created some communities, of peers that we're helping each other learn along the way. What's working well in terms of engagement and what. Is an effective strategy. and then, you know, I, I wanna to mention too, that these are public documents. So learning agendas are public documents. They're posted on agency websites. They're linked on our new site evaluation gov, which promotes transparency, and accountability. So those are, you know, transparency is a good thing in and of itself. Um, but we hope this also provides new opportunities for partners, partnerships through more equitable and inclusive sharing of priorities. So this is really the first time that all federal agencies have communicated their evidence building priorities to anyone who's interested. So it offers a chance for academics, practitioners, community members, philanthropic foundations, other researchers, to, you know, to get involved and to align their own portfolios with these priorities. you know, we, we appreciate that agencies have asked tough questions. Um, they're not taking the easy path in most cases, but we, we also understand capacity is limited. Resources are constrained. agencies need help answering these questions, and we often do, we hear from researchers, they like to do policy relevant work, um, but they don't know where to start or what the agencies need. So I, I'm, I'm hopeful that this is a new, sort of transparent way for agencies to signal their evidence, building priorities and maybe break down some of these barriers, to getting people more engaged with government priorities. and I, I really also hope that this will draw in more practitioners, more researchers who haven't always had access to government data or access to even knowing who to talk to in government. we really want to expand the evidence tent, so to speak. Uh, and I'm hopeful that this is, this can be a first step to doing that.

Speaker 2 30:43

Great. Well thanks for sharing all of that. It's wonderful, wonderful to hear. So I'm gonna turn now to Calvin. Calvin, my question for you to start a, a soft, digging into your wealth of knowledge, um, you know, the department of, housing and Urban Development, um, plays a critical role in equitable access to affordable housing and building inclusive communities in the United States. And through the development of, of the its research roadmap in 2017, HUD was really an early adopter of using a learning agenda to help advance the agency's mission. Can you give us of an example of how the creation of this research roadmap and learning agendas has helped advance HUD's goals?

Speaker 5 31:26

Absolutely. thanks for having me here. And, um, it's nice to see many of the folk who I work with on, who I work with on a regular basis. Um. And really like the framing of this panel so far. I'm sitting here listening. I forgot that I haven't, you know, that I had to speak. So first, lemme just share with you, because I think it's important to think about mission. So HUD's mission, I'm quite certain you all know this, but I was just, I'm just gonna restate it. HUD's mission is to create strong, sustainable, inclusive communities and quality, affordable housing for all right? So we start off with that mission and we think about what is it that we need to know so that we can actually perform that mission. And so when we think about the roadmap, it is really, or the research roadmap, it's a strategic research plan. And so we started off with the strategic research plan. We've then transitioned it over to a learning agenda, which is really intended to help HUD to identify the department's highest priority research and the data needs and actual performance of our mission. So this is really about performance of mission, the alignment of the strategic plan, the goals. How is it that we can identify the highest priority research and data needs? We use the learning agenda extensively when it comes to requesting research funding from Congress. It is rare that we will take on research projects without funding from Congress. and for many of y'all who actually read the appropriations, you can see that in our appropriation, it spells out very specifically the research projects that we take on. And so the learning agenda is critically important because it helps us in justify our appropriations for research, research to actually support our mission, the actual performance of our mission. And when the high priority research projects are funded and actually complete it, we can then fill the research gaps in our knowledge. And that translates to actual program improvement and policy changes. And so there's this connection between, HU'S mission. the research roadmap slash learning agenda, or the learning agenda slash research roadmap, which we're transitioning to, and how we basically, improve our programs and not to mention how we get funded. So now, some of HUD's most, most, most impactful research from a policy perspective actually originated from the roadmap. For example, in the, um, 2014 2018 roadmap, it included this, um, you know, this idea to further study rapid, re rapid rehousing. Specifically, we wanted to look at the program model and how well it was doing to provide short-term rental assistance to people experiencing homelessness. That idea was picked up and actually Congress liked it. They funded us, and so we launched the study. That study gave us a lot of information, a lot of products about better understanding the program models, the outcomes that we should expect from rapid rehousing. And it provided us with a sense of what we still don't know, which then provides us a feedback, a natural feedback directly back into the learning agenda. And so, that process is a pretty standard process, for how we basically, uh, use the roadmap exercise or use the learning agenda exercise to actually advance our goals here at Hut.

Speaker 2 34:48

Thanks. Yeah, and I, I, um, that's super helpful. Uh, we actually are doing a, an additional study on rapid rehousing at LEO now, so, um, I'd like to, I loved hearing that context too for, knowing, you know, laying out what we know, but also mapping out what we still need to know for different kinds of programs and, um, programs that serve different populations. So I'd love to hear also about how equity influenced the development of the learning agenda, and then kind of vice versa, how can the learning agenda help support the agency's equity goals?

Focus on Equity and Evidence Building

Speaker 5 35:20

Yeah, so, the way that we developed the learning agenda, we developed it concurrently with the department's first equity assessment, which is what had to be developed under executive order 13 9 85. It was through this parallel process that HUD engaged stakeholders in issue specific action teams or issue specific action teams. Yes. And then they applied an equity lens to the learning agenda. Specifically we wanted to look at the learning agenda, the program evaluation policy, and our current research and evaluation portfolio. So we looked across all of those things through these, through the equity lens. Now, um, if, if, well, but yet I'm just gonna assume that folk haven't read our equity, plan yet. and so what I will do is I will say. there are two broad areas for research purposes that I wanna pay attention to. One of them is equitable access to home ownership and other wealth building activities. And the second one is equitable provision of homelessness services. So clearly those are areas where, you know, it's in our warehouse to basically not only focus on equity, but also build evidence. And so those are two areas that we spend a lot of time talking about and the research that we do here within PDN r. Now the learning agenda includes research questions that address those two focus areas and it also provide project description not only of those two areas, but there are, I believe there are like 10 domains where you'll find research questions and a description of the project so that you almost have a catalog of, alright, these are the questions that are important and here's a description of the project that we think would work. As well as, as, it provides, uh, data, the data enhancement priorities that we think are necessary to make certain that we're able to measure, and actually build the evidence that we need to. now the other thing about that is that it incorporates equitable outcomes. Um, and we look at the implementation and the impact of HU programs and policies as part of that process. Now, this focus is actually observable for, for example, in the learning agenda. Foundational learning questions are laid out, and the language for most of those questions is what we use to frame various aspects of equity, equal opportunity or, uh, serving underserved populations. And finally, the Learning Agenda team collaborated with the HUD equity Leadership Committee and the equity working group. Now, this is a whole lot of groups working together with any agency, right? But it's important. you cannot have silos. You cannot develop a learning agenda and have a learning agenda team that is talking about high priority research questions for the organization, not only with stakeholder engagement, internal stakeholder, holder engagement, but with other leadership committees and other working groups that are basically rolling in the same direction. So we had to bring those teams together and so they came together and they came together to ensure that the learning agenda fully supported the department's learning needs with respect to reversing inequities and federal housing policies and practice and prioritizing equity and, and all HUD's programs.

Speaker 2 38:37

Great. And speaking of, um, you mentioned stakeholders in that process, I'm interested, you know, obviously HUD has this massive national scope, um, in its work and I'm interested to hear kind of how you engage with. Across the United States as you develop the learning agenda and continue to inter iterate on it, you know, tapping into that local expertise to inform all of this.

Challenges and Opportunities in Stakeholder Engagement

Speaker 5 39:03

Yeah. So challenge here, but the opportunity is that HUD has regional offices and offices across the country. And so the importance is this making certain that we're leveraging those resources and our access to communities and stakeholders in a way that makes sense. And so the most direct way that, we tap the expertise of communities is through our stakeholder engagement process. We have, you know, going back to the days of the research roadmap, we have developed a process whereby we wanna ensure that we're tapping our stakeholder groups. And as Keith mentioned, making certain that we are mindful of people who might not be your traditional folk at the table. Uh, we wanna make certain that we're inclusive. We wanna make certain that we're reaching groups who we don't, you know, who we haven't and we don't traditionally reach. And so we try to make the learning agenda and the roadmap development process visible and transparent, right? We've held kickoff meetings, we sent out email blasts, and there's some limits that have already been mentioned about that. But we also set up, um, email, boxes and kind of, posting areas where people can actually provide feedback. And again, there are some limitations to that as well, because the assumption there is that everyone has access to email, right? But we also created online forms and we allow for folk to provide, feedback. we engaged in listening sessions as well. And so, you know, despite these best efforts, we know that we're still missing people, we're still missing groups of stakeholders. But we are cat, uh, we're, we're, cataloging, and documenting those areas where we perhaps miss stake. Uh, we miss stakeholder groups. and that's not only external groups, but that's also internal groups. I mean, you would think that an agency would do a really great job at making certain that we reach all of the internal stakeholders, but hu these are big organizations with organizations spread throughout the country, and the territories, uh, you know, the territories as well. And so we gotta make certain that we identify those areas or those groups that we might be missing. And so, despite us being actually intentional about the way that we target our stakeholders, we need to figure out how to, ensure that, we're being more inclusive. I mean, we already are inclusive, but being more inclusive. And so we are thinking about listening feedback through more listening sessions and trying to figure out ways of using technology and other forms of engagement that we have in traditional youth. Now we're gonna be, partnering with policymakers and researchers and practitioners to kind of determine from them or listen to them about, Hey, look, what are ways that. You've encountered folk actually expanding the reach of these, these like engagement activities. And we're hoping to learn from folk. Now, additionally, I can tell you about some things that we've done that we plan on doing more of in terms of reaching folk who aren't typically at the table. So, for example, now we had staff who were leading the learning agenda implementation efforts. Um, they gave a presentation to advocates from a range of lgbtq plus advocacy organizations doing the collaborative discussion sessions with other HUD offices. We need to do more of that. I mean, we need to find out more of the stakeholder groups that we haven't traditionally reached and figure out ways of reaching them. So we need to do more of that. And in new efforts, we've tied, we've tied our outreach and engagement activities, uh, specifically to the learning agenda, by seeking out and engaging groups with lived experience and HUD programs. Right? So we want to know, about the experience of folk who experience and encounter on a daily basis are programs. And we need to figure out strategies for doing more of that. And so I think that as we move forward, we are going to really develop, I think, like this, like in this, this evolving strategy for making certain that we identify groups who we don't traditionally, reach in terms of engagement that's internal and external and come up with, you know, creative strategies for actually engaging them.

Speaker 2 43:01

Thanks so much, Calvin. I'm gonna turn now to Vincent. Vincent, maybe you could start by just telling everyone a little bit more about JAL North America.

Speaker 6 43:12

Sounds great. Thanks very much Rachel. Very excited to be here to share more about J PAL North America and the work that we've done on a mobility from poverty learning agenda. For those of you who are not familiar, J PAL North America stands for the Abdul Latif Jamil Poverty Action Lab. We are a research and public policy center that's based within MIT's Department of Economics and our entire mission is focused on helping to improve the lives of people experiencing poverty through rigorous scientific research. So we as an organization specialize in the use of randomized control trials as a form of rigorous research to identify which programs and policies work and which programs and policies ultimately should be scaled up to benefit many more people, uh, around the country. Jal's a bit of a unique organization in that we have a global focus. I lead JAL North America, which is the regional office of JAL that focuses on. Everything that happens in the United States. And when we think about poverty, we think about it across a range of major policy areas that have a connection to poverty, including education, healthcare, criminal justice reform, economic and workforce development, and homelessness and housing stability as a, a few, couple of areas.

Speaker 2 44:31

Great. Thanks. And, you know, as, as JAL has taken on the development of a learning agenda for mobility from poverty across several state and local jurisdictions, I'd love to hear kind of the genesis of that project. And then also, you know, what benefit did you envision coming out of this multi jurisdiction learning agenda?

Speaker 6 44:53

So the idea behind doing this mobility from Poverty learning agenda really emerged a couple of years ago via what's called our state and local innovation initiative. So Chap Powell, state and Local Innovation Initiative, which is actually partnered with a number of folks I think on this call, including UH, care at King County, is really focused on helping state and local governments generate rigorous evidence on the impact of their programs. And over a couple of years we started to notice patterns emerging, both in terms of the specific research questions, uh, that state and local governments were asking, and also in terms of the specific challenges that they were hoping to address and recognizing this pattern. We thought that it would be really helpful to put together a learning agenda that could help. Create connections between state and local governments that are facing similar challenges and are thinking through similar innovations and interventions that if tested, could really help build a roadmap for an effective way to, uh, address poverty. So a couple of years ago, we decided to create this mobility from poverty learning agenda to help identify commonalities across, state governments, city governments, and county governments. And from there we were able to really identify a couple of key areas that many governments, regardless of if you're a government that was on the East coast or the West coast, a government in the Midwest versus a government, in in the South, a lot of commonalities and key themes in terms of what they thought would be really important to research. And from there we were interested in building out a blueprint and really a public good that if you were a member of a state government or a local government. You would be able to look at this blueprint and then ask yourself, what are some innovations that are actually being tried elsewhere? What are things that we already have a promising body of evidence and what are things that we don't really know that much about? But if we evaluated, could significantly advance their understanding for how to address poverty in a, in a specific way. And then from there we were also really interested in incorporating the research perspective. So J is a bit of a unique organization in that we also are a network of academic researchers that sit across many different universities all around the world. And we were really interested in getting the researcher perspective, uh, to identify areas where. If we could build out more evidence, we could really advance our understanding for how to address poverty and help people who are, uh, experiencing poverty. And I think one, one benefit of doing something that cuts across many different policy areas and covers many different geographic areas is that you are able to identify, uh, that a lot of the challenges are the same. And that collectively, if we run a lot of different evaluations at the same time and we try to advance evidence building efforts at the same time, focused on concentrated areas where we know that there is not enough evidence or we know that there could be more evidence generated, uh, we can really move the field forward, uh, at a much faster pace all at all at once.

Complementing Federal Learning Agendas

Speaker 2 48:06

Yeah. That's so great. And then, you know, because that, that's, that work is all with partners at the state and local level. Uh, how do you see that work complementing. The learning agenda work happening at the federal level that Diana and and Calvin have talked about.

Speaker 6 48:22

So it's been really exciting to see this big push around federal learning agendas. And it really does feel like it is the year for evidence in action. And I think the way that we think about it is that a lot of the federal priorities intersect a lot with state and local government priorities. And I think especially in the past two years with major pieces of legislation being passed, like the American Rescue Plan that is allocating a lot of money at the federal level down to the state and local level, there's a lot of opportunity for state and local governments to be testing new innovations that can help to answer some questions that are being laid out in federal, uh, learning agendas. So for example, federal learning agendas that are trying to understand the impact of the American rescue plan and whether or not programs that are being funded with these dollars are making a difference. At the end of the day, it's the state and local governments that are implementing a lot of these programs and by utilizing, uh, a state and local mobility from poverty learning agenda, we can also help state and local governments figure out what are the biggest priority questions that should be answered. If you are utilizing these federal dollars to test new programs, what are the programs that you should really pair up with rigorous evaluation that can help you understand the impact of your specific program, and can also create really generalizable knowledge that can be used elsewhere too. So other state and local governments can. look at a program that was tested, let's say in California, and then apply it to their specific context. And I think these findings from evaluations that are happening at the state and local level can feed back up to the federal level and help answer a lot of major questions, uh, that are being posed in, in, uh, federal learning agendas. And it's really a continuous cycle where the questions that are then being pros by federal learning agendas can inform the priorities of what state and local governments and also the, what the research community is really interested in, uh, in, in evaluating. I think the other thing that I'll mention is that a lot of, data also exists at the federal level. So with these evaluations that are being implemented at the state and local level, we actually are able to tap into federal administrative data to really understand the impact of these types of programs. So I do think there's also another connection and intersectionality there.

Speaker 2 50:39

Yeah, that's great. And, if you'd be willing to, I'm interested if there was any one or more kind of big challenges that you faced as you guys went to develop this kind of learning agenda.

Speaker 6 50:50

Definitely a lot of challenges. Mm-hmm. I think one big challenge is who you are able to get in touch with. So these learning agendas in some ways are not exhaustive in that they only represent who you're ultimately able to get information from. So you also have to ask yourself the question, who were you not able to engage as part of this process of developing this learning agenda? For this learning agenda. We mainly had worked with, city, state, and county government officials and also researchers. But we also recognized that there were quite a few limitations in our approach. We weren't able to engage with the community as extensively as we would've liked. We also weren't able to engage with tribal governments as part of this process, um, as extensively as we would have liked. So in some ways, while this mobility from poverty learning agenda is incredibly helpful in shining a spotlight on where we need more evidence, it only represents the views of the people that we ultimately were able to talk to. But at the same time, I think one side benefit is that we were able to also recognize where there are major shortcomings in terms of the specific research questions that are also being asked. Because we know that we had, we have some blind spots unfortunately, in terms of where we were gathering this information.

Speaker 2 52:06

Sure. Well, thanks so much. Uh, Vincent. I'm gonna turn it to Eva. Eva, I'd love to hear, especially, um, your experience from the more local perspective for local government. So, the Best Starts for Kids is one of the most comprehensive local initiatives, investing in early childhood and youth development in the United States. It's invested over$800 million. it's investing that,$800 million, uh, in the next six years. And it seems like some of the strong voter support for the initiative rests on its attention to both equity and evidence-based approaches. my understanding is that the Best Starts for Kids Initiative has a dedicated set aside for data and evaluation.

Speaker 7 52:52

Yeah, that's right. About 5%.

Speaker 2 52:55

Wow, that's great. And, and from its inception in 2015, you've led the development of the Best Start for kids. The bsk, I guess, is the acronym. Um, performance and evaluation plan, how is that different or similar to the learning agendas that we've been talking about today?

Speaker 7 53:11

Yeah, thanks so much for having me and for having me be a part of the panel. And I'm not quite sure how I agreed to be less because the, the other panelists have been a really, spectacular act to follow. So, thanks everybody. Yeah, there are definitely a lot of commonalities with our evaluation and performance measurement plan. We have a, a plan, it's a multi hundred page document that I think nobody here actually wants to read, but it really laid out for us what we're going to be doing, how we're going to be doing it, how we're going to deprioritizing, even with our 5% are still limited resources. Best Starts for Kids is doing, a range of programs that focus on direct service to families and individuals, up to, workplace focused initiatives. Acknowledging that children, particularly young children don't. In their own little bubble. They, they really need that community around them as well as their workforce. And then even up through policy systems and environment changes. So running across that span and then the, the many programs that we're doing, we set out a prioritization process in that plan where we knew we would do performance monitoring for everybody that would go and be funded. And I should clarify that while this is an initiative through King County government, we partner with hundreds of community-based organizations, that are, are deeply embedded in these communities to conduct the work. So there's performance measurement for everybody, and that gets reported to on an annual basis and we share out how we're doing. and then equity is something that's been discussed throughout this panel, and that is a huge part of what we're doing and making sure that what we're doing is actually in service and equity, and that what we're doing is working for everybody. And then finally we have some projects that we have prioritized for evaluation. I'll give an example from the home visiting portfolio specifically. Um, and that's just one tiny sliver of what we're doing. But in home visiting, we are, that starts for Kids is funding evidence-based programs, evidence, evidence-informed programs, and then community design programs. For those of you who are familiar with home visiting to be an evidence-based program, there's a really rigorous, evidence bar for that. We didn't need to spend our Limited King County resources to prove, again that something like Nurse Family Partnership is working. However, there are a number of community led programs, so, programs that the community based organizations have, have come up with or have a kernel of an idea for or have been doing for some time, but just simply haven't met that evidence based. And so we're actually focusing our resources on things like that. And, It's really around lifting up these programs that simply don't have enough evidence base, helping them to develop that evidence base and really focusing on, on the programs that work.

Community Involvement in Learning Agendas

Speaker 2 56:09

Thanks for sharing that. That's so interesting. So, you know, I know Best Starts for Kids Approach is also deeply embedded, uh, in working with communities where the needs are the greatest. How have you involved communities in the learning agenda and its activities?

Speaker 7 56:24

Yeah, that's a great question and I, I loved hearing Calvin talk about how you engage with communities from the very beginning. So, I would say this evaluation plan that I talked about, we got community engagement throughout the development of that process. And then in thinking about how we actually operationalize that, in working with community-based organizations that do the work, I would say we don't just start with them. Data and evaluation and, and really data from day one, we start day minus one. So we have a lot of information that's available on our website. We go out actively to share information, to build capacity for groups before they're even partners. we build their capacity to even use some of this data. And then once they are actually funded, we provide completely free to them trainings to build their internal capacity to do this kind of work. And then, also with partners, we'll have things like learning collaboratives or quarterly meetings, quarter, really around that peer learning, shared learning and continuous quality improvement. There's so much knowledge that are in the room when we have our, partners pull together and they can share among themselves what's working for them, new ideas, and really lead that innovation within, within their peer groups. And then thinking about, advisory groups. We have advisory groups for our evaluation activities and, whether that's, that's usually folks with lived experience as well as our community based partners there, representatives. And then they sit with us and we work side by side from the very beginning, designing the evaluation questions, designing our methodologies, reviewing the data that's collected, trying to review like, well, that doesn't really match what I thought would've thought, what else needs to be contextualized with that? and then really thinking about how to disseminate that information. there's a slightly different project that I've recently been working on where the feedback we got was, we want one page infographics and dissemination materials. And they helped us pick out the pictures. They helped us on every step of the way, and it was very much community led. And having folks with lived experience is key. There's an advisory group that I help lead and we actually have 50, 50% representation from 50% people with lived experience, and then 25% frontline staff and 25%, some, some external partners who are experts. So really intentional balancing of the lived experience in the room. the last point I wanna make, which folks haven't brought up, is the importance of qualitative data. a lot of evaluation and performance measurement is very traditionally quantitatively focused. And what we've really heard from our community partners is the importance of qualitative information and sharing their story, telling their story, making sure it's truly represented, and, making sure the parts that we're missing are, are included in the full story. there's a project right now where we're. We've used a lot of federally available administrative data and some of the local advocates, for people living with disabilities have said, Hey, we're completely not represented in what you're sharing. So what we've actually done is work closely with them and funded them. And we've done a series of listening sessions and surveys with them and, and we're currently in the middle of doing analysis. So those surveys were completely focused on people living with disabilities.

Speaker 2 1:00:09

So, great. That kind of connects back to what Keith was talking about at the beginning with the really learning agendas being engaged work and what you just described sounds like truly engaged work. so thanks for sharing all that. So, uh, we're now at the time where we're gonna turn questions and answers, you know, questions. Um, and we'll have our panelists, the answers. But I do wanna thank, Eva and. Keith and Calvin and Vincent and Diana for sharing everything so far. I'm happy to turn to some questions that people have, some people have put in the chat or sent us. So, the first question, uh, is for Diana. the question is, Diana, if you can share more about the new Analytics for Equity program that was announced at the White House Summit on evidence?

Speaker 4 1:00:58

Uh, yeah, so there's actually limited additional information I can share. Um, but this is sort of a first of its kind initiative, led by NSF, the Office of Science and Technology Policy and OMB, which is gonna, distribute targeted funding to external research teams that submit rigorous proposals, to directly answer federal agencies learning agenda questions that focused on equity. We're really aiming to, include participation from. Researchers with a real broad range of institutional affiliations and areas of expertise. So again, trying to really broaden the evidence tent here. beyond that, I mean, there's not a, a whole lot I can say right now. The call for proposals has not actually been posted yet, but we hope very soon. so I do hope very soon I, I can say some of our partners include, um, HHS, office of the Assistant Secretary for Planning and the CDC, uh, department of Labor's Chief Evaluation Office, and the EPA. So hope that like piques people's interest. Um, we're really excited about this and hope I can share more, very soon.

Speaker 2 1:02:05

Great. Thanks so much. So the next question, um, is to really, Diana and or Calvin, can you share any of the challenges that are emerging with agencies incorporating equity in their learning agendas that, that you've seen so far?

Speaker 4 1:02:22

Yeah. Calvin, do you wanna tackle that one first since you started to get at this? I think,

Speaker 5 1:02:26

yeah. So I think part, part of the issue that we find is that, we're, we're having to first of all acknowledge that there has been, a role that many of our policy, our past policies and practices have played in building up inequities. And so we have to engage people around the importance of research questions, policies and programs to kind of reverse those issues. And so that is a work, right? That is a piece of work, and we're having to not only engage external, parties, actually around that issues, uh, stakeholders, but we also have to engage internal stakeholders and bring everyone, um, to the table and have honest discussions about things that we can do and how we go about doing it. I think that if you look at our, um, our equity. Plan. And if you look at our learning agenda, you look at our strategic goal, you look at the alignment across those, I think that we're all rowing in the same direction. I think the challenge is making certain that we are being deliberate and that we're coming together on a regular basis to ensure that, that roadmap, that plan that we have in place across those, processes that we're being true to them and that we're working collaboratively, um, to get those done. And we need to ensure that we provide feedback to the stakeholders who are actually affected by that. So folk with lived experiences to include people who we haven't traditionally had at, who we haven't traditionally had at the table. And so as I was listening to Eva talk, I was kind of thinking about, you know, what else do we need to do? Like how do we, as we learn, As we learn, we bring those folks who have lived experience to the table so that they can benefit from what it is we're learning and can help contextualize some of those things that we're learning because we might not necessarily pick up on everything.

Advice for Developing Learning Agendas

Speaker 4 1:04:27

I'll just add too, I mean, something we're also thinking about is on the, for the backend of the learning agenda cycle. So once we've done some studies and, and generated some evidence, you know, I think there's a lot of, there's been a lot of barriers to how the information is shared and how it can be shared most effectively, effectively with the communities who are most affected, who need the information. It's, it's, you know, their information, it's their evidence. but I think folks, and we have not really, really figured out how to best share that back in a really, you know, authentic and meaningful way and on a regular basis. So. All these plans get implemented, you know, what we do with the information I think is also gonna be really important.

Speaker 2 1:05:15

Yeah, that makes sense. so here's a question kind of to to everyone, um, the panel and Keith, uh, what's one piece of advice that you would have advice for someone who wants to start the development of a learning agenda at their agency or organization?

Speaker 7 1:05:33

I'm gonna be bold and, and go first and really pick up a thing that Diana was talking about, which is building the culture of evidence and building the culture of data and making sure that's valued from, from day one. I'm so fortunate to work with people who come from a variety of backgrounds and experiences. but may not have been trained the same way I have been. And so how do we build that value for everybody and help them see that value? And that is not an overnight thing. That is not something that can be mandated. but it really takes relationship and partnership to help everyone see, the whys and how it, of how, the, the learning agenda can help.

Speaker 6 1:06:13

So I think building off of Ava's point, I think being able to connect the development of learning agendas to your overarching goals, to really be able to sell the why behind how evidence can actually get you closer to achieving whatever goal you have at the end of the day, whether or not it's trying to reduce homelessness by a certain rate, or improve learning outcomes by a certain rate. And just being able to tell that overall narrative of how evidence at the end of the day will give you much more clarity. As to how you can most effectively and efficiently achieve your goals. Um, it is probably where I would start, and I think in general, whenever we partner with state and local governments on this evaluation work, and we have to convince people why you should evaluate, why you should collect data and some instances why you want to conduct a randomized evaluation, starting with that big picture overall message of how this evidence is going to help you achieve your goals is, is typically a good starting point.

Speaker 4 1:07:19

I was gonna say actually something really similar, which is, you know, we, we often found it's helpful to frame it as what is it that you wanna know? what is one thing that if you knew that it would help you do your job better, right. I think that's like, that is something that we all do like in our daily lives. And so I think like using that as a, an opening as a starting frame can be really helpful.

Speaker 5 1:07:43

I was gonna pick up on that same thing. I mean, I think, um, you know, if you start off by saying what are the things that keep, you know, that keep you up at night? What are the things? I mean, I mean, that's the way that I was told, Hey look, you don't go in, uh, telling folk that you're the government, you wanna help them. Um, that that doesn't go far. But people have issues that they're dealing with, they have challenges that they're struggling with. And if you can just go in and say, look, what are the challenges that you're facing? And just collect those things and then gather your thoughts and figure out how you might be able to actually promote a strategy for helping them, I think is, is perhaps the way that I would start.

Q&A and Closing Remarks

Speaker 3 1:08:26

Yeah. I would echo what everyone has said and, and maybe just say, especially to a new organization that's considering this, you know, the example today have been. Big and broad federal agencies with huge mandates or, you know, upward mo, you know, mobility from poverty broadly. But, it can be daunting to think of collecting and assessing evidence and, and key questions across so many different, you know, huge, important topics. But it's okay. I think for a, an organization do the process to start small. Start with like a subset of questions and many of these same steps work for a narrower learning agenda. And ultimately it might be that by doing it in one office or one, you know, area that you're focused on, you can start to build some of those constituencies that will allow you to, you know, in the future do it organization wise.

Speaker 2 1:09:15

tips for everybody. This is wonderful. So I think we're just at time. I, I did wanna just thank again our speaker, Keith and our panelists and, um, everybody who showed up today. I'm gonna turn it back to Carrie. It's just about dinner time if you're on New York time, and I don't like to stand in the way of anybody in their dinner. So Carrie's gonna wrap it up. And, uh, I, I thank you again for joining us.

1 1:09:40

Yeah, thank you Rachel. And thank you to all of our panelists. Big round of applause to all of you. I learned so much in this session, session, um, session. One of the things that I definitely heard come through in everyone's, in everyone's, points is this common theme of using learning agendas as a way to put practitioners at the head on the engine of the evidence train to borrow, an analogy from our friends at Project Evidence Board here as well. I think that's, that's one of the great, the great benefits of investing in these learning agendas is that we can really set a research agenda for the research community that, The questions that are most important for policy decisions, in in the forefront. Keith, your work continues to inspire and be a great resource for all of us on this topic. I love your emphasis on this as not a check the box exercise, but a cycle of learning and your advice at the end about starting, you know, kind of small is I think, I think really important. Diana, like the, the work that you're doing across the federal government is just, it's astounding and, uh, very exciting. And, I loved how you really emphasized the, the breaking down of silos that can often stymie or evidence work and how you're really building really thoughtfully the strategy and the systems to, to align things and how much you know, this work and evidence and learning agendas. All of that is really critical for us in advancing equity. our communities who we have most marginalized, deserve our attention around evidence as much as every other community does. Um, Calvin, I'm, just, you know, in odd by, you know, the, the challenges that you all face at HUD and trying to reach all 330 million of us across, the United States. And, thank you for being so transparent about the fact that we're still missing people and we'll still miss people, and, you know, continuing to work every day, to, to address that. And I loved your advice about just, you know, what's keeping you up at night. That's what, you know, we want at the topical learning agenda. Vincent, you really, you know, inspired me to think more about how we can spread our learnings, not just within our own organizations, but across governments and, you know, across. Across so many places in the United States and, again, your transparency about who we're missing and tribal governments and you know, how what, how what we build just reflects who we can be in touch with, is a really important, piece for us to keep in mind. And Ava, I think your day minus one, tagline is like, for me, like the thing that will really stick with me, you know, as a few words coming out of this event, we really have tried, through best starts for kids, as Ava said, to like not just, you know, to start as early as possible that day minus one approach to using data and evidence and engaging and building the capacity of our organizations, um, to use that data and evidence, has been a huge focus and peer learning as a key component of our learning agenda. Data stories, is super important. So thank you all so much. This was, a stellar panel and it exceeded my expectations, for what we were able to, discuss and learn. So really my heartfelt thanks to all of you who, who, spoke with us and for all of you who came and joined us as well. I really appreciate your helping us to build this community that values evidence and equity together. And yes, there will be a copy of the recording that we will send out, probably early next week as well as the slides that you saw that have a lot of the links and resources to things that, our panelists talked about. So thank you so much everybody.