Category Archives: Evaluation

3 Tips to Use Your Data to Drive Program Improvement

Nationally, only about half of students who start college actually complete and earn college degrees, according to the National Student Clearinghouse. It is much worse for students who start at a community college or 2-year degree program. This means many students are increasing their debt (from student loans) and not reaping the financial benefits of a college degree and higher earnings. In addition, colleges are losing money when students dropout. It’s a lose-lose game.3 Tips to Use Your Data to Drive Program Improvement

Colleges have been getting increasing pressure from the federal government and others to improve their college completion rates. They have turned to data analytics to better understand how they could intervene earlier with students who might be at risk of not completing their degree and dropping out.

When I first heard this podcast from The Hechinger Report on how “Colleges are using big data to track students in an effort to boost graduation rates, but it comes at a cost” it made me want to listen closer.  As a #datalover and #datanerd, this topic certainly peaked my interest. I am also a first generation college graduate and can personally relate to this topic of college dropouts since many (over half) of my high school peers didn’t finish their college degree.

This is not a new strategy (“predictive analytics”) as companies like Amazon and Google are doing this all of the time with our digital footprint. However, it’s new for colleges and non-profit organizations.  

Georgia State is a case study example of how their university has embraced data analytics to improve college completion rates. Their university now has one of the highest rates of college graduation for public universities in the country, and they have closed the racial equity gap. Students of color are graduating at the same rate as white students at Georgia State. 

So how did Georgia State get there? They used their data to drive and inform program improvement. We’ve talked about this here and here.

The podcast also shares the struggles that other universities face in implementing these changes. It’s not enough to purchase the data analysis software, but you also need trained staff who are able to analyze and interpret the data to take action on it. 

We’ve put together 3 tips to get started with using your data to drive program improvement that’s not only based off of the success of Georgia State but our work with other clients who want to improve their impact.

  1. Have a system in place to collect and track meaningful data. Georgia purchased a data system to help them bring all of their data together and identify patterns. At Transform Consulting Group (TCG) we are big fans of using Tableau Software (see more here and here). We love Tableau, because it can make your data easier to review and understand. We are also adept at using whatever data systems our client has to pull out the information that we need to inform decision-making.
  2. Have trained staff who can analyze and interpret the data. Getting the data from your system is the first step. Then you need to have individuals who know how to identify patterns, ask inquisitive questions and develop recommendations. We are big fans of forming an “Impact Team” at an organization who is trained on analyzing and reviewing your data and can help drive action based on the results. Learn more here.
  3. Have a process to determine program improvement changes. Once you have your data and results, you are now at the fun part – you get to take action on changes to make to improve your results! We find that sometimes organizations don’t have a clear process in place to determine what changes – such as a new curriculum, staff training, client outreach – they will make based on the results. We follow a Continuous Quality Improvement Process using the “Plan – Do – Study -Act or (PDSA)” framework to determine what action steps we will take. Other similar processes might be “Lean/Six Sigma”. At Georgia State they decided to hire additional advisors to use the data to reach out to students at risk. This is one of the many changes they have made, based on their data results, to help students stay on track with college graduation. The goal here is to be intentional on how you will use your data to make programmatic changes and then study the result of those changes to know if it’s working or not.

What are some goals that you are not currently satisfied within your organization? Have you considered how data analytics could help you focus in on improving your results?  Bill Gates said, “Without measurement there is no shared accountability.“ We would love to help you improve your impact through data analytics. Contact us to learn more.

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Why Break Down Data?

When you’re using data to make decisions, are you also taking time to break down data to learn more? Perhaps you are struggling to understand the needs of different parts of the population you serve. Maybe you’re noticing different outcomes in different groups but don’t know why. When you break down data, you can see what’s hidden within your overall results. 

One important reason to break down data is to help your clients who are experiencing multiple adverse events. Our team at Transform Consulting Group worked with Coalition for Homelessness Intervention and Prevention (CHIP) to understand the challenges of people experiencing both homelessness and domestic violence. We took a close look at their data. Then we came up with recommendations on how to best meet the needs of this population.

A significant portion of individuals who are homeless have also experienced domestic violence. In Marion County, 21% of individuals in the Homelessness Management Information System (HMIS) had lived in households with reported domestic violence (DV). DV-homelessness-data

Based on national best practices and local data, CHIP and its partners identified potential system and policy improvements. They gathered feedback from domestic violence survivors experiencing homelessness via surveys and focus groups. 

The data and research revealed the need for targeted public policy and legislative protections for this population. When domestic violence survivors leave their relationships, they face economic hardships that put them at risk for homelessness. One policy solution allows survivors to remain in their rental home after the perpetrator is removed from the lease.

Breaking down the data led to this policy recommendation that is specific to domestic violence survivors. This policy change goes beyond what’s relevant for all individuals experiencing homelessness.

In other instances, breaking down data can be particularly helpful to ensure you meet the needs of the most marginalized people. Are children from high-income families able to access your programming more easily than other children? Are your participants of color seeing the same gains as your white participants? Do those in rural areas achieve the same positive results as urban communities? Looking at data in this way can help you focus on equity for your vulnerable populations. 

Researching national best practices revealed that domestic violence survivors, in particular, benefit from meeting with advocates in locations other than their office. Survivors face transportation and other logistical barriers. This can mean it’s much easier if an advocate comes to their home or neighborhood.

There are more details on all the data and findings in the Report on Domestic Violence Survivors Experiencing Homelessness in Marion County that Transform Consulting Group prepared for CHIP. CHIP-DV-report-cover

No matter what your organization’s mission is, breaking down data can help you learn more about different segments of the population you’re serving. Do you see better outcomes when participants have been in your program for more than six months? Is your curriculum more effective for younger children?

In addition to breaking down your data, check out our other blogs on making sure you’re data literate and putting data into context

If you aren’t using data to look at segments of the population you serve, then you might be missing what is (or isn’t) working well in your program. Let us know if you need help with data analysis or program evaluation!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

How A Needs Assessment Can Support Your Head Start Program

Most organizations that receive federal or state funding and even private funding are required to complete some type of needs assessment. This might be part of their grant application or annual program update. The purpose of the needs assessment (which we talk more about here and here) is to help organizations align their services to meet the needs of their targeted population and geographic service area.

Head Start and Early Head Start grantees are one type of grant program that must complete a comprehensive needs assessment every five years as part of their grant application. They are also required to complete an annual needs assessment update. In addition to the local grantees, each state has a Head Start Collaboration Office. They too are required to complete an annual needs assessment based on federal priorities to inform their annual plan and funding priorities.Blog Image

Transform Consulting Group has worked with Head Start and Early Head Start programs at every level from the local grantee level to the state collaboration office and even the federal Office of Head Start. With these partners, we have helped with writing grant applications, managing data systems, completing strategic plans, supporting implementation of new grants, and of course completing needs assessments. Based on our breadth of experience with Head Start, we have some tips to share in how to best complete and leverage your Needs Assessment:

  1. Gather Quantitative Data

The 5-Year Community Assessment must include a variety of data points such as community demographics, data about Head Start eligible children and families, education, health, social services, nutrition, housing, child care, transportation, community resources, and the list goes on. During the other 4 years of the grant period, local grantees must do a Community Assessment Annual Update. This update includes any significant changes in data around key areas such as the availability of prekindergarten, child and family homelessness, and other shifts in demographics and resources.

  1. Gather Stakeholder Feedback

We’ve talked a lot about stakeholder engagement in past blogs (here and here). The 5-Year Community Assessment includes gathering input from community partners, parents, and staff. We do this through the use of surveys (electronic or paper), focus groups, and interviews. This is a great opportunity to hear from your key stakeholders, build buy-in and engagement, and strengthen existing relationships.

  1. Create Visually Appealing Needs Assessment Reports

We pride ourselves on creating visually appealing reports that are user-friendly for all audiences and talk about it in this blog. You can see examples of our Head Start needs assessment reports here and here. We have also taken these reports to create fact sheets about the need for services across different service areas or to summarize the impact / footprint of the Head Start and Early Head Start program.

In more recent years, we have started developing data dashboards that summarize the community needs assessment. Organizations are putting these dashboards on their website like this example here.  By doing this, Head Start programs can be a great resource in the community of comprehensive data about young children and families that other partners can use for planning purposes.Screen Shot 2019-05-22 at 12.04.29 PM

  1. Share and Use Your Data

After your organization has invested all of this time and effort in completing your needs assessment you want to make sure you use it to drive programming and services. This is where having a visually appealing report, some infographic facts and / or a data dashboard are so important. It makes sharing them internally with your staff and parents, as well as externally with partners, that much easier! We love to share this information at policy council meetings, family events, and community partner meetings.  

Does this process sound overwhelming to you? Do you feel like you are in data overload? We can help! You don’t have to do this alone.

Head Start programs, like many federally funded programs, are tasked to track and monitor a lot of data and information especially for compliance purposes. Evidence can be seen of that in the reporting requirements of the needs assessments, along with other state and federal regulations. Most Head Start programs do not have one primary database, so data is often stored in many ways across several systems and staff members. TCG can help review these systems, provide recommendations, assist in analyzing data, and offer training to staff about data systems and best practices around data collection and analysis.

We have the Head Start knowledge and the data expertise to support your needs assessment and data management needs. Consider how TCG can help your Head Start program today. Contact us to learn more!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

4 Steps to Complete a Program Evaluation

Program evaluation helps organizations determine if the change that they set to accomplish actually occurred. Change can be knowledge gained, attitude change, or behavior change. For example, did a literacy tutoring program help the students who were not reading on grade level actually catch up to reading on grade level by the end of the program period? Or, are the low-income children who participated in a high-quality pre-K program ready for kindergarten?  

At Transform Consulting Group, we are data nerds and love helping organizations develop and implement evaluation plans to assess their change by using a four-step process.
4 eval steps

Program Evaluation in Four Steps:

  1. Establish clear outcomes
  2. Create or modify data tools and systems
  3. Analyze the data
  4. Use data to make informed decisions

The first step is to ensure there are clear outcomes in place that support an organization or program’s goals. We work to develop SMART outcomes: specific, measurable, achievable, relevant, and timely. In this step, we typically review or develop a logic model, which is just a technical description of aligning your programs to the change (or outcomes) that you want to accomplish. For example, a college-readiness program may want to increase the number of students who can (1) identify a major or career they are interested in pursuing after high school; or (2) understand how to apply for college financial aid.  

The second step focuses on having the right data tools and systems in place to measure and report back on the designated outcomes. Transform Consulting Group helps organizations determine the most appropriate tool(s) to collect, track, and monitor accomplishment of the identified outcomes. We consider organizational capacity (staff time, knowledge, and budget) when suggesting data tools. Some examples could include participant surveys, assessments, and student academic records. We work to have data tools that are valid and reliable and will provide the data necessary to monitor progress.

The third step is to analyze the data once it has been collected, and present the results in an easily-understood format. Data is measured to determine whether program outcomes were met and the change, if any, that occurred. This is often the step where organizations get stuck, because they don’t have the staff time or knowledge to complete the analysis. [Stay tuned for a separate post on simple ways to analyze your data!] We tell our clients that this is the fun part because we can see if what they set out to accomplish actually occurred!  

Some common research questions that drive many organizations to conduct program evaluations to get answers include:

  • Are program participants being reached as intended? If yes, why? If not, we also want to know why.
  • To what extent are desired program changes occurring? Was there a significant difference or just a small difference? Is there a specific group that is not being impacted?
  • Is the program worth the resources it costs? What is the “return on investment” for this program or service?

The fourth step is to discuss the results of the program evaluation and make informed decisions based on what the data tells us. We will compile a summary report and/or slide deck presentation of the evaluation data for stakeholders internally and externally to review the results and discuss the implications. Good evaluations often lead to recommendations for improvement such as enhanced professional development, diversified participant recruitment strategies, and/or program model changes. This is an opportunity to discuss data collected and implications for future programming, including ongoing program evaluation practices within the organization.

In today’s era of accountability, what gets measured gets done. If you don’t measure results, you can’t tell success from failure[1]. Transform Consulting Group equips organizations to celebrate their successes and inform opportunities for growth. Contact us today for more information on how Transform Consulting Group can help assess the impact of your organization.  
[1] Reinventing Government, Osborne and Gaebler, 1992.

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

4 Steps to Complete a Feasibility Study

Too often non-profits and government agencies immediately begin implementing a new program or service area. They see a need with their clients or a gap in the existing services, so they elect to help meet that need. This all sounds good, right? The challenge is that there has not been enough time to complete a comprehensive planning and assessment process to develop the program or service. One service we offer our clients to meet this need is completing a feasibility study.plan-act-do-study-cycle4

We follow the Plan-Do-Study-Act or “PDSA” continuous quality improvement cycle (learn more in this blog).  We help clients assess, design, launch and evaluate programs and services in order to meet community needs and apply the latest research. When following this approach, we most often find that clients tend to skip the first step “Plan” and jump straight to “Do” as mentioned above. We work to help our clients thoughtfully plan out their services, programs, and interventions before they implement them to get the impact and desired change they are working towards.

Implementing a feasibility study is a great tool to complete a thoughtful planning process. A well designed feasibility study will help an organization assess 1) if what they are thinking of implementing is possible and 2) how to consider implementing it.

Shoes at ArrowsWe worked with a group of community leaders in Jay County to complete the feasibility of converting an old elementary school building into an early childhood center. Like many rural communities, Jay County has a declining population that has impacted their local schools in continuing to operate multiple school buildings, which has resulted in school consolidations and closures. At the same time, their rural community also struggles with attracting new employers due to a lack of child care for a growing workforce. Their community leaders had the idea of converting a closed elementary school into an early childhood center but wanted assistance in completing a feasibility study first.

4 Steps to Complete a Feasibility Study

 

1. Market Analysis

During this step you want to gather key information about your targeted population. This includes collecting demographic information from online public sources. This helps create a composite of your targeted community and population. We also suggest completing a landscape assessment to identify any other organization providing similar services or working with the target population. Lastly, it’s important to gather some qualitative feedback from various key stakeholders in the community to determine what they think the needs and gaps are as well as build community will for possibly launching a new service. This can be done through focus groups, surveys, and key informant interviews.

The purpose of this step is to ensure that there is in fact a need for your proposed program/ service. Check out this blog for more insight on completing a community needs assessment!

2. Program Design

During this step you will want to complete some research on your targeted service area. For Jay County, we are gathering the latest research on early childhood program models and services that lead to the desired outcomes they are seeking. Our landscape scan is also looking at existing program models in the community so as to not duplicate existing options but to consider complementary program models that will meet the needs of communities. If you are seeking external funding, you may want to adopt or align your program around research-based models that have demonstrated outcomes. This will provide confidence to potential funders in implementing a new program.

The purpose of this step is to determine the best model and design for implementing your program. Check out this blog for more tips on finding evidence-based programs.

3. Business Model

The next step is to develop the business model for operating the program. During this phase of the feasibility study you will gather important financial information that will help you understand what it will cost to implement the program and potential sources of funding. You should create a budget and possibly complete some financial forecasting to show start-up costs and when the program would “break even” or be self-sustaining. This step should also assess the operations behind implementing the program, which includes the staffing model, materials and services, training, facility, technology, equipment and other program needs.

With Jay County, we completed walk-throughs of three possible locations with an architect and construction group to inform the best location to operate an early childhood center. This informed the potential capacity to serve children, the staffing needs and ultimately budget the break down for start-up costs versus ongoing maintenance costs. The purpose of this step is to think through all of the components needed to successfully implement the program.

Check out this blog for some tips to establish financial goals.

4. Communications Plan

The last (and sometimes forgotten) step is to develop a communications strategy if you decide to launch the new program. After spending all of this time assessing and planning the design of the program, you want to ensure that the targeted audience knows about the program and enrolls/ participates. The communications plan would include determining the current knowledge base in the community, so there might need to be some education and awareness about why you are providing this service especially if it is new and different.

In Jay County, we are created a PR Campaign through a series of op-eds penned by different key stakeholders (employers, teachers, judge, doctor, etc.) in the community all talking about why expanding early childhood is critical to meet the community’s needs. Your communications plan should include the different channels (social media, newspaper, radio, text, mailings, etc.) that residents use to gather information. In a parent survey (our potential client for early childhood services), we asked them where they get their information and their preferred method of communication. Based on this assessment, develop a start-up marketing plan and community education plan for the proposed new program that will meet participation goals and engage the key stakeholders and partners in the community.

Check out this blog for tips on creating an op-ed campaign and this blog for getting media attention.

Completing a feasibility study may seem unnecessary or slow down your timeline, but the time you invest up front will see a return in a well thought out model that will be set up for success and to accomplish your goals. Completing intentional design through the PDSA model is a critical differentiator for Transform Consulting Group and many clients point specifically to this process improving their own internal operations which accelerates impact. Contact us if we can help you complete a feasibility study!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Does Your Organization Have a Continuous Quality Improvement Process?

During a time where communities and policies are changing, it is important to ensure the programs and services within those communities are constantly evolving to meet the needs of families. The Continuous Quality Improvement (CQI) model is an ongoing process for organizations to be able to determine whether or not a change made led to an improvement in quality. In order to move toward making the necessary improvements, a review of what occurred is conducted through a CQI process like the Plan-Do-Study-Act (PDSA) cycle.

Steps to Complete a PDSA Cycleplan-act-do-study-cycle4

At Transform Consulting Group, we utilize this consistent approach when working with organizations to help them find solutions.

Plan:

Before beginning your PDSA cycle, identify the problem or issue you would like to address for quality improvement. The problem identified will guide your purpose for the review. Once you have decided what to focus on, consider the the following steps to plan to test your idea for change:

  • Define the goals
  • Define your research question(s)
  • Make predictions
  • Determine details for implementation of change or intervention
  • Plan of action for data collection

During this phase, we work with the client to thoughtfully plan to implement their new or current program/ service. So often, organizations jump immediately to step 2 – “Do” – without completing this critical first step. During the planning phase, we define what we hope to accomplish especially if we are proposing a change. Then we determine how the proposed change/ intervention will be implemented and work through all of the details. Lastly, we finalize how data will be collected.

Do:

This phase of the PDSA cycle requires you to conduct the test for the change or intervention. It is during this phase that you will complete the following tasks:

  • Carry out the intervention
  • Collect data
  • Begin data analysis

This step in the process is what most organizations know and are doing. Organizations are delivering interventions every day with their services. They might be intentionally or unintentionally modifying their intervention. The “Do” step in this process is not new to organizations. It is wrapping it around the other three steps that makes this work transformational!

Study:

The study phase of the cycle occurs after you have completed your intervention. You then analyze the data to study what did or did not occur. Organizations will want to review their predictions and assumptions before conducting the test. You will want to take the following steps during this phase of the cycle:

  • Complete data analysis
  • Compare data to predictions
  • Summarize the information

Organizations often skip over this step in the process or do not spend enough time thoughtfully reviewing the data. For some organizations, their data can be considered “high stakes” and there is a tendency to want to focus on the positive changes/ results that occurred and glance over the changes that did not occur or the benchmarks that were not met. During this phase, it is so important for an organization to be transparent and honest with themselves when reviewing the data.

Act:

Based on the summarized information, this last phase of the cycle allows you to determine what modifications may be needed to ensure that the goals you set will be met. Your organization may decide to modify a program element or change how a service is delivered; you may decide to target a different population or use a new curriculum. Once you have determined whether or not to adapt, adopt, or abandon your intervention, you will be prepared to do the following:

  • Plan next cycle
  • Decide whether the change can be implemented

During this last step, your organization takes all of the information gathered to make data-informed decisions that will ultimately improve your results. This is the exciting part of the process and one that you don’t want to skip. This step and the overall PDSA process will help your organization continue to improve the quality of services provided and impact in the community.

In this blog, “Is it time to redesign your program?”, we shared several examples of clients we helped use the PDSA process to test and implement new interventions/ modifications to improve their outcomes. The CQI process allows organizations to have a plan of action once a problem or service gap has been identified.

At Transform Consulting Group, we follow this consistent approach when helping you find solutions to accelerate your impact. If you are looking to improve the quality of a service or program to facilitate positive change, contact us today!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Putting Equitable Evaluation into Practice

Evaluation ProcessDoes your evaluation process accurately represent the population you serve? The marginalized and less privileged? We shared in this blog that evaluation is a process that helps organizations determine if the change that they set to accomplish actually occurred. We follow a 4-step evaluation process: (1) establish clear outcomes, (2) create or modify data tools and systems, (3) analyze the data, and (4) use data to make informed decisions. 

In all our work with clients, we want to ensure we’re best representing all populations when doing evaluation. That’s why we’re learning more about an emerging initiative that is transforming the way evaluators think about their work – equitable evaluation. 

At TCG, we’re always learning! We recently attended a lunch and learn titled “Equitable Evaluation: Whose Voice Counts?!” hosted by the Dorothy A. Johnson Center for Philanthropy at Grand Valley State University. The presenter Equitable evaluation presentationshared examples about how equity is undermined based on traditional evaluation practices. She then shared an overview of the core equitable evaluation principles. The work of evaluation is for and contributes to equity. Evaluations should answer critical questions about the effect of a program on populations, underlying drivers of inequity, and why history and culture might be affecting the work and potential changes. In addition, evaluation design and follow-through should be based around the values of equity, meaning multi-culturally valid and set toward participant ownership.

We [the audience] were challenged to explore the opportunities to implement equitable practices into our own evaluation work, so that’s what we [TCG] did!

This presentation got us intentionally thinking about our four-step process, how equity is included and opportunities for improvements.  

Putting Equitable Evaluation into Practice

Equity can be applied throughout our  four steps, creating a fair and impartial process from beginning to end. Some examples for before, during, and after an evaluation that help lead to an equitable evaluation include: equitable evaluation handout

  • Before the evaluation, ensure the program evaluation questions and design are equitable. This means that the evaluation is multi-culturally valid and geared towards participant ownership. 
  • This thought process can be carried on throughout your evaluation. During the evaluation process, consider equity when determining appropriate stakeholders and the method of collecting feedback from those stakeholders. What factors should be considered for different stakeholders (environment, demographics, politics, influences, etc.)? What methods will be most effective to gather feedback? What influencers may be impacting stakeholder responses, or lack thereof? 
  • After, review the feedback from the evaluation through a lens of equity. What and/ or who was missing from the process? Determine the next steps to follow through based on the results of the evaluation. 

How are you ensuring equitable evaluation in your practice? You can learn more about the Equitable Evaluation Initiative on the EEI website. If you’re stuck in your evaluation process or don’t feel like you have a system in place to effectively collect equitable data – let’s chat!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Learn About Indiana’s Youngest Children with the 2019 ELAC Annual Report!

2019-elac-annual-reportIndiana’s Early Learning Advisory Committee (ELAC) released its 2019 Annual Report. Each year, ELAC completes a needs assessment on the state’s early childhood education system and then recommends solutions.

We want to share some quick highlights and key takeaways from this year’s needs assessment.  ELAC focuses on ensuring early childhood education is accessible, high-quality, and affordable to all families. 

Are Children Ages 0-5 Receiving High-Quality Care?

  • Of the 506,257 children in Indiana ages 0-5, 64% need care because all parents are working. This includes both working parents who are single and households where both parents work outside the home. Figure 9
  • Of those children who need care, only 40% are enrolled in known programs. The other three fifths of children receive informal care—from a relative, friend, or neighbor.
  • Of the young children who need care, only 16% are enrolled in high-quality programs. A high-quality early childhood education program not only ensures that children are safe, but also supports their cognitive, physical, and social-emotional development. 

Are Children in Vulnerable Populations Receiving High-Quality Care?

  • Indiana makes funding assistance available for early childhood education for children from low-income families.
  • Indiana does not collect data on children in other vulnerable populations, such as children in foster care and children affected by the opioid epidemic.
  • Overall, due to lack of data, Indiana does not know the kind of care received by children in vulnerable populations.

What Trends Are There in Early Childhood Education?

  • Since 2014, Indiana has made progress by enrolling more of the children who need care in known early childhood education programs. 
  • Over the past 5 years, Indiana has consistently enrolled fewer infants and toddlers than preschoolers in known and high-quality programs. Figure 31
  • Compared to 2012, more early childhood education programs are participating in Paths to QUALITYTM, Indiana’s quality rating and improvement system.
  • In addition, significantly more programs have earned high-quality designations of either Level 3 or Level 4 since 2012.

What Trends Are There in the Early Childhood Education Workforce?

  • Indiana’s early childhood education workforce is more diverse than the K-12 workforce but not as experienced.
  • Nationally, the early childhood education workforce earns $4-$7 less per hour than the average hourly wage of all occupations.

What is the Unmet Need in the Early Childhood Education System?

  • There has been a persistent need in early childhood education programs for more available spots for infants and toddlers.
  • Despite overall improvements, there are still some communities in Indiana with no high-quality early childhood education programs.
  • The tuition cost of high-quality early childhood education programs remains unaffordable, and the available financial assistance for low-income families is insufficient.

How Can I Find Out More?

  • Read the 2019 ELAC Annual Report, which includes statewide data on Indiana.
  • ELAC also publishes an interactive dashboard that allows you to learn more about specific data points. You can also easily present data to stakeholders.
  • The interactive dashboard contains both state- and county-level data. Use the map to select your county, and hover over the data to learn more!

2019-elac-interactive-dashboard

Transform Consulting Group is proud to support ELAC’s work by pulling this needs assessment and interactive report together!

Does your organization, agency, or coalition need to better understand your community or a key issue, but you don’t know how to get started? We are skilled in collecting quantitative data from multiple data sources and pulling it together in a visually-appealing, user-friendly report. Contact us to learn how we can help you complete your next needs assessment!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

5 W’s of a Process Evaluation: Part 2

In a recent blog post, we introduced the first two W’s of a process evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation

This blog post will cover the remaining three W’s:

  1. What methods to use to conduct a process evaluation
  2. Where to conduct a process evaluation
  3. When to conduct a process evaluation
WHAT METHODS TO USE WHEN CONDUCTING A PROCESS EVALUATION

There are several different data tools and methods you can use during a process evaluation. It may be helpful to use a combination of these methods!

  • Review documentation: It can be helpful to review staff logs, notes, attendance data and other program documents during a process evaluation. This method will help you to assess if all staff are following program procedures and documentation requirements.
  • Complete fidelity checks: Many programs/curriculums come with fidelity checklists for assessing program implementation. This is especially important if you are implementing an evidence-based program or model. Programs may have a set number of required sessions and guidelines for how frequently they should occur. You can use fidelity checklists to assess if the program’s implementation is consistent with the original program model.
  • Observe: Observations can be especially helpful when you Y Observationshave multiple sites and/or facilitators. During observations, it’s crucial to have a specific rating sheet or checklist of what you should expect to see. If a program has a fidelity checklist, you can use it during observations! If not, you should create your own rubric.
  • Collect stakeholder feedback: Stakeholder feedback gives you an idea of how each stakeholder group is experiencing your program. Groups to engage include program staff, clients, families of clients and staff from partner programs/organizations. You can use interviews, surveys, and focus groups to collect their feedback. These methods should not focus on your clients’ outcomes, but on their experience in the program. This will include their understanding of the program goals, structure, implementation, operating procedures and other program implementation components.

In our evaluation project with the Wabash YMCA’s 21 Century Community Learning Center, we used a combination of the methods described above. Our staff observed each program site using a guiding rubric. Our team collaborated beforehand to make sure they had a consistent understanding of what components to look for during observations. We also collected stakeholder feedback by conducting surveys with students, parents and teachers. The content of these surveys focused on their experiences and knowledge of the program. After the program was complete, we reviewed documentation, including attendance records and program demographic information.

WHERE TO CONDUCT A PROCESS EVALUATION

You should conduct a process evaluation wherever the program takes place. To capture an accurate picture of implementation, an evaluator needs to see how the program operates in the usual program environment. It is important to assess the implementation in all program environments. For example, if a program is being implemented at four different sites, you should assess the implementation at each site.

In our evaluation project with the Wabash YMCA, we assessed the program implementation at three different school sites. This involved physically observing the program at each site as well as reviewing records and documentation from each site. Being in the physical environment allowed us to assess which procedures were used consistently among sites. It also helped us identify program components that needed improvement.

WHEN TO CONDUCT A PROCESS EVALUATION

An organization can conduct a process evaluation at any time, but here are a few examples of times when its use would be most beneficial:

  • A few months to a year after starting a new program, you can conduct a process evaluation to assess how well your staff followed the implementation plan.
  • When you’re thinking about making a change to a program, a process evaluation will help you determine in what program areas you need to make changes.
  • If your program is not doing well, conduct a process evaluation to see if something in your process is interfering with program success.
  • When your program is doing well, conduct a process evaluation to see what in your process is making it successful.
  • If you’ve had issues with staff turnover, conducting a process evaluation can help identify gaps in staff training, professional development and ongoing support that may be contributing to the turnover rate.

To determine when to conduct a process evaluation, it is also important to consider the capacity of your organization. Make sure that your staff will have enough time to devote to the evaluation. Even when using an external evaluator, staff may need to spend extra time meeting with evaluators or participating in focus groups/interviews.

We conducted our evaluation with the Wabash YMCA at the end of their first year of program implementation. Evaluating their first year of implementation allows us to provide them with recommendations on how to improve the program’s implementation in future years. We will conduct a similar evaluation during the next three subsequent years to track their operations and processes over time.

If your organization needs support in conducting a process evaluation, contact us today to learn more about our evaluation services!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

5 W’s of a Process Evaluation: Part 1

When it comes to program evaluation, people often think of evaluating the effectiveness and outcomes of their program. They may not think about evaluating how the program was administered or delivered, which may affect the program outcomes. There are several types of valuable evaluations that do not focus on outcomes. One type of evaluation, called “process or formative evaluation”, assesses how a program is being implemented.

In this two part blog series, we are going to cover the 5 W’s of a Process Evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation
  3. What methods to use to conduct a process evaluation
  4. Where to conduct a process evaluation
  5. When to conduct a process evaluation

In this first blog in the series we will cover the first two W’s. The next blog will discuss the other three.

WHY CONDUCT A PROCESS EVALUATION

Let’s start with the “why”. A process evaluation helps an organization better understand how their program is functioning and operating. Process evaluations also serve as an accountability measure and can answer key questions, such as:Screen Shot 2018-07-27 at 4.38.23 PM

  • Is the program operating as it was designed and intended?
  • Is the current implementation adhering to program fidelity?
  • Is the program being implemented consistently across multiple sites and staff, if applicable?
  • What type and frequency of services are provided?
  • What program procedures are followed?
  • Is the program serving its targeted population?

It is important to determine what you want to learn from your process evaluation. Maybe you want to assess if the program is being implemented as it was intended or you want to know if the program model is being followed. Whatever the reason, you want to be clear about why you are completing the process evaluation and what you hope to learn.

We are currently working with the Wabash YMCA’s 21st Century Community Learning Center to evaluate their program implementation. Each center is required to work with an external evaluator to conduct a process evaluation. Here is what we hope to learn and the why of this evaluation:

  1. The evaluation will assess if the program has been implemented as it was intended and if it is adhering to state standards;
  2. This evaluation will capture the population served through the assessment of attendance trends;
  3. The findings from the process evaluation will be used for program improvement in subsequent years.

WHO SHOULD CONDUCT YOUR PROCESS EVALUATION

When determining who will conduct your process evaluation, you have the option of either identifying an internal staff member (e.g., program manager or quality assurance) from your organization or hiring an external evaluator. Many organizations find that there are challenges with an internal team member: they may not be objective, they don’t have a fresh perspective, and they often have other job responsibilities beyond the evaluation.

For the reasons mentioned above, it is beneficial to have an external evaluator (like TCG!). An external evaluator will be able to assess the operations of your program from an unbiased lens. This is especially helpful if a program has multiple sites. An external evaluator can assess all sites/facilitators for consistency more objectively than a program staff member. (If you’re interested in learning more about how to evaluate multi-site programs, view our blog post here!).

In our evaluation project with the Wabash YMCA, the decision to conduct an evaluation with an external group was made by their funders. This decision ensures that the evaluation is high quality and objective.

The other three W’s will be discussed in a later blog post, so stay tuned! In the meantime, contact us today to learn more about our evaluation services!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin