Category Archives: Evaluation

Does Your Organization Have a Continuous Quality Improvement Process?

During a time where communities and policies are changing, it is important to ensure the programs and services within those communities are constantly evolving to meet the needs of families. The Continuous Quality Improvement (CQI) model is an ongoing process for organizations to be able to determine whether or not a change made led to an improvement in quality. In order to move toward making the necessary improvements, a review of what occurred is conducted through a CQI process like the Plan-Do-Study-Act (PDSA) cycle.

Steps to Complete a PDSA Cycleplan-act-do-study-cycle4

At Transform Consulting Group, we utilize this consistent approach when working with organizations to help them find solutions.

Plan:

Before beginning your PDSA cycle, identify the problem or issue you would like to address for quality improvement. The problem identified will guide your purpose for the review. Once you have decided what to focus on, consider the the following steps to plan to test your idea for change:

  • Define the goals
  • Define your research question(s)
  • Make predictions
  • Determine details for implementation of change or intervention
  • Plan of action for data collection

During this phase, we work with the client to thoughtfully plan to implement their new or current program/ service. So often, organizations jump immediately to step 2 – “Do” – without completing this critical first step. During the planning phase, we define what we hope to accomplish especially if we are proposing a change. Then we determine how the proposed change/ intervention will be implemented and work through all of the details. Lastly, we finalize how data will be collected.

Do:

This phase of the PDSA cycle requires you to conduct the test for the change or intervention. It is during this phase that you will complete the following tasks:

  • Carry out the intervention
  • Collect data
  • Begin data analysis

This step in the process is what most organizations know and are doing. Organizations are delivering interventions every day with their services. They might be intentionally or unintentionally modifying their intervention. The “Do” step in this process is not new to organizations. It is wrapping it around the other three steps that makes this work transformational!

Study:

The study phase of the cycle occurs after you have completed your intervention. You then analyze the data to study what did or did not occur. Organizations will want to review their predictions and assumptions before conducting the test. You will want to take the following steps during this phase of the cycle:

  • Complete data analysis
  • Compare data to predictions
  • Summarize the information

Organizations often skip over this step in the process or do not spend enough time thoughtfully reviewing the data. For some organizations, their data can be considered “high stakes” and there is a tendency to want to focus on the positive changes/ results that occurred and glance over the changes that did not occur or the benchmarks that were not met. During this phase, it is so important for an organization to be transparent and honest with themselves when reviewing the data.

Act:

Based on the summarized information, this last phase of the cycle allows you to determine what modifications may be needed to ensure that the goals you set will be met. Your organization may decide to modify a program element or change how a service is delivered; you may decide to target a different population or use a new curriculum. Once you have determined whether or not to adapt, adopt, or abandon your intervention, you will be prepared to do the following:

  • Plan next cycle
  • Decide whether the change can be implemented

During this last step, your organization takes all of the information gathered to make data-informed decisions that will ultimately improve your results. This is the exciting part of the process and one that you don’t want to skip. This step and the overall PDSA process will help your organization continue to improve the quality of services provided and impact in the community.

In this blog, “Is it time to redesign your program?”, we shared several examples of clients we helped use the PDSA process to test and implement new interventions/ modifications to improve their outcomes. The CQI process allows organizations to have a plan of action once a problem or service gap has been identified.

At Transform Consulting Group, we follow this consistent approach when helping you find solutions to accelerate your impact. If you are looking to improve the quality of a service or program to facilitate positive change, contact us today!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

3 Strategic Plan Tools to Create

Congratulations! You have journeyed through the 4 steps of our Strategic Planning Process and you’re ready for the final step: Create. (We covered step 1, step 2, and step 3 in previous blogs).

Screen Shot 2018-07-27 at 2.03.19 PM

The goal of a strategic plan is to develop timely, relevant and action-oriented plans for the future of your organization. Once you have a clear direction, it is time to make sense of the information and package it in a way that is meaningful and possible to implement.  

At TCG, we believe a strategic plan has little value if it is a report that sits on your shelf, never to be seen again. We don’t create long strategic plan reports that you can’t use. We want you to use it, share it and review it on a regular basis.

When working with clients, we recommend and create 3 different strategic plan tools:

1. One-page strategic plan – This is a one-page summary of your goals and top strategies. This tool can be shared externally with partners, funders, and other key stakeholders as well as internally with staff.

When creating a strategic plan for the Wabash County Early Childhood Education Committee, we wanted a one-page overview that highlighted the following key elements:Screen Shot 2018-07-27 at 2.03.33 PM

a. Stakeholders involved (especially since this is a collective impact, multi-sector plan)

b. Goals

c. Strategies

d. Outcomes

Each one-pager for the strategic plan that we create is unique to the client but essentially covers their top goals and strategies.

2. Strategic plan report – This report explains the process of how the strategic plan was completed, the information that was collected, and more details about the goals and strategies. This is typically an internal document that is shared with staff and the board to use when reflecting on the process. It’s especially helpful to document this information for when there are leadership transitions with the staff and board.

Screen Shot 2018-07-27 at 2.03.53 PM

3. Implementation plan – Too often we find that organizations get stuck with figuring out how to take the big picture elements in the strategic plan and make them operational. We create an “implementation plan” to unpack the strategic plan into actionable steps for staff, committees and the board. The main audience for the implementation plan is staff, board and committee members who are most likely responsible for implementation.  Screen Shot 2018-07-27 at 2.04.09 PM


This could be set up like a calendar or a chart that describes who is responsible for each step. We also love using Tableau to create a strategic plan dashboard to track and monitor action items and milestones. The point is that we want all parties involved to have a clear understanding of the timeline, so that they can put the plan in motion.

Is your organization ready to jump into a strategic planning process? Learn more about our strategic planning services here. Contact us today, and we’d love to chat about how our team can meet your needs.

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Putting Data into Context

At Transform Consulting Group, we are proud data nerds. Through our evaluation services, we help clients collect, analyze, and share meaningful data. In this blog post, we explained who to share your data with and why. In today’s post, we will go one step further by providing tips on how to present your data in a meaningful way. More specifically, we’ll discuss how to put your data in context and why it is important to do so.

Impact Image- blogWhen presenting your data, you shouldn’t share it in isolation. For example, an after school tutoring program might find that 75% of their students pass their required standardized tests. If the program shared this data point by itself, their audience might have a lot of unanswered questions, like:

  • How does this pass rate compare to other students who don’t receive tutoring services?
  • How does this rate compare to local and national data?
  • What standardized tests is the statistic referring to?

 

To avoid this problem and present their data it a meaningful way, it would be best for the tutoring program to cite outside data sources to provide comparison, credibility, and context. By including this additional information, the program could more fully illustrate their impact and outcomes.

We are currently working with the Center for Leadership Development to develop an evaluation plan. Through this process, we have helped them demonstrate their impact by presenting their data within context. Here are three tips we shared with them that can also help you use outside data sources to put your data into context.

1. Find credible data sources that add meaning to your data.

When citing outside data, it’s important to make sure the data is credible, accurate, and relevant to your organization’s work. When working with clients like CLD, we often provide a resource sheet listing different data sources they can cite for comparison and context. An example of a data source we shared with CLD is the Indiana Commission for Higher Education’s College Readiness Dashboard. This was an appropriate choice because it is a reliable interactive data set that can be used to compare the outcomes CLD students experience to other students in their state and county in similar demographic groups. Check out this blog post for a list of our go-to data sources. This list may help you identify which data sources you can cite to move your organization forward.

2. Benchmark similar programs.

In a previous blog post, we explained that you may want to benchmark the practices of organizations similar to yours when making a programmatic change or looking to diversify your funding. Benchmarking can also be helpful when creating an evaluation plan and reporting your data. Looking at the outcomes of similar programs gives you comparable data to assess your program’s efficacy.

When working with CLD, we benchmarked similar programs such as College Bound in St. Louis. Their programming aims to help low-income students get into and graduate from college. Not only were they a similar program for CLD to compare their outcomes to, but they are also a great example of an organization who puts their data into context to make it more meaningful. For example, they compare their data to St. Louis Public School data and low-income students across the nation:

94% of College Bound students have matriculated to college immediately after high school, compared to 66% of St. Louis Public School graduates and only 51% of low-income graduates nationwide.

By presenting this statistic in the context of the students’ school system and other low-income students, College Bound is displaying the impact they are having and the success of their students relative to their peers.

3. Make sure you’re comparing apples to apples.

We always tell clients to make sure they’re not trying to compare apples to oranges. This phrase refers to the comparison of items that aren’t really comparable. An example of this came up in our work with CLD when reporting their alumni’s postsecondary persistence rates. When comparing their persistence data to local and national data, we needed to make sure the outside data set was defining persistence in the same way they were. They define it as persisting from Freshman to Sophomore year of college. Other sources defined persistent students as those who were enrolled at any institution or had attained a degree 3 years after first enrolling. Therefore, these two data points aren’t really talking about the same thing and aren’t comparable. By finding the right data sources to compare your data to, you ensure that the data and context is meaningful.

If you need help presenting your data in a meaningful way and using it to make data-informed decisions, give us a call to see how we can help through this process!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

2018 Indy Big Data Conference Visualization Challenge Summary

Our TCG team recently presented at the Fifth Annual Indy Big Data Conference for the Visualization Challenge (read about some of our lessons learned here). This year’s challenge focused on improving Indiana’s talent pipeline and workforce. fullsizeoutput_3b6

Participating teams were provided several datasets to analyze and interpret, and then present to a panel of judges. Following the first round of presentations, the judges then awarded five teams the opportunity to present to 500+ conference attendees. Our TCG was one of those top five teams selected! Read more about the Visualization Challenge and data sets here: https://www.indybigdata.com/visualization-challenge/.

Our team used our Tableau expertise to create this interactive dashboard, and then we presented on our findings. The interactive dashboard addresses several key questions about the future of Indiana’s workforce and talent pipeline, including:

  1. What will be the top jobs in demand in 10 years?
  2. What is the education requirement for those top in demand jobs?
  3. What is the earning potential of those top in demand jobs?
  4. What is the return on investment on education for those top in demand jobs?

The following is a summary of TCG’s findings to these questions based on the data provided.

1. What will be the top jobs in demand in 10 years?

Screen Shot 2018-11-13 at 10.03.45 AM

Indiana’s top projected jobs are found primarily in the healthcare, food service and business industry. TCG found that the top projected jobs in 10 years varies depending on how you look at the data. For example, Wind Turbine Service Technicians looks like the top in demand job, because of a 71% expected increase in positions in 10 years. However, when you look at the actual number of new jobs for Wind Turbine Service Technicians,  there are only 66 total positions expected to be added over 10 years. This is not actually enough individual positions to create new higher education programs to encourage individuals to pursue this career. In this visual chart, you can scroll down to see the top projected jobs and how they compare. Hover over the number for more information about that occupation.

2. What is the education requirement for those top in demand jobs?

Screen Shot 2018-11-13 at 10.06.28 AM

The U.S. Census Bureau reports one-third of working adults ages 18-64 have an associate degree or higher. An additional 25% of this population have some education beyond high school. Interestingly, we found that six out of ten of the top 25 projected jobs only require a high school diploma. This doesn’t mean that we should stop encouraging young people to pursue higher education. Instead, the trend looks like the largest amount of new positions will be entry level or positions that do not immediately require an education beyond high school.

3. What is the earning potential of those top in demand jobs?

TCG created a box and whisker plot chart to show the earning potential by occupation. For all occupations the median annual salary is about $40,000. The top 25 projected jobs based on the number of new positions shows a median annual salary at $31,000 which is below the current median annual salary. Again, this somewhat relates to the education level requirements in the previous section. The largest increase in projected new jobs are not high earning jobs. Visit the interactive dashboard to explore the different wage levels, experience levels, and hourly rates of the projected top jobs.

Screen Shot 2018-11-13 at 10.07.08 AM

Screen Shot 2018-11-13 at 10.07.23 AM
4. What is the return on investment on education for those top in demand jobs?

The costs of education can feel like a burden to individuals, especially when the costs drag on through student loans for several years. The average timeframe expected to pay off the cost of postsecondary education is over two years. Our TCG team was interested in calculating the return on investment (ROI) of pursuing higher education for the top 25 occupations. You can rank the list by the jobs with the highest and lowest ROI. The average ROI for all jobs is $1.4 million.

Recommendations

TCG made several recommendations from the data analyzed. (This list can also be viewed on the second tab in the interactive dashboard).

  1. Review employment projections in multiple ways, like reviewing the expected growth based on the number of new jobs and the percentage growth of new jobs (e.g., Wind Turbine Service Technicians example) to ensure informed decision making. This can assist with decision making when developing or removing educational degree or certification programs and when encouraging individuals to enter a certain field.
  2. Offer more certificates and associate degrees both independently and as a part of bachelor’s degree curriculum. It would align with required educational needs of the workforce, contribute to less debt for students, and help improve higher education completion rate.
  3. Increase partnerships with employers (like IU Health) to do more direct career pipelines and apprenticeship programs. This could also potentially reduce the student cost. For example, Graduate Assistant positions are partially funded by the organization that receives a graduate student.
  4. Include more experiential learning in degree programs. It would help the student decide if they are in the right major or explore a secondary interest. It could also get industries to partner with other majors/ disciplines. For example, get the “creatives” into STEM activities – increasing the workforce pipeline.
  5. Consider offering incentives to attract individuals in the top projected jobs, especially those jobs with a low ROI. Look at ways to make these jobs more attractive to meet the future workforce needs.

Learn more about the Indy Big Data Challenge in this article by the Indiana Management Performance Hub. At TCG, we are data nerds. Check out our research and analysis services, and learn how we can develop visual reports and dashboards to inform your decision-making and help you tell your story of impact.

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

5 Tips to Implement an Evidence-based Program

When awarding funding, philanthropic funders want to invest in “what works” and is proven effective. Many funders show preference for programs and practices that are evidence-based. Implementing an evidence-based program is a great way for grant seekers to demonstrate that they are also committed to “what works”.

For example, the Richard M. Fairbanks Foundation recently awarded funding to over 20 schools and school districts as part of their Prevention Matters initiative.  Prevention Matters is a three-year grant initiative aiming to help Marion County schools identify, implement and sustain proven substance use prevention programs.

To apply for this funding, schools selected an evidence-based substance use prevention program that aligned with their needs. In their proposal, schools had to demonstrate that they had a strong plan for implementation and sustainability. Developing such a plan can be a daunting task, but is crucial for successful implementation. We worked with Bishop Chatard and the North Deanery Schools of the Archdiocese of Indianapolis to help them develop their implementation plan and proposal (Which was fully funded by the Fairbanks Foundation! Learn more about our fundraising services here.). Here are 5 tips we used to help them prepare to successfully implement their evidence-based program!

1. Select an Evidence-based ProgramWhat Works Image (1)

First, you need to find a program that aligns with the needs you are trying to address. For example, if you are a school looking to prevent substance use and violence, while also promoting positive youth development, you may choose to implement the Botvin LifeSkills Training curriculum.

Taking the time to research available programs is crucial to ensuring successful implementation and maximum impact. To learn more about how to find an evidence based program, check out this blog!

2. Assess your Organization’s Capacity

Once you have selected an appropriate evidence-based program, it is important to assess your current funding and staffing capacity. You want to assess if your current organizational capacity will allow you to implement the program with fidelity. Fidelity refers to the extent to which you deliver your program as the original program model intended. Evidence-based programs are  proven effective and that effectiveness relates to how the program is implemented. Therefore, fidelity to the model is crucial to successful implementation.

Completing a feasibility study is a great way to assess your capacity and readiness. A well designed feasibility study will help an organization assess 1) if what they are thinking of implementing is possible and 2) how to consider implementing it. Check out this blog to learn more about completing a feasibility study.

The assessment of your capacity may indicate that you need to make some organizational changes. For example, you might need to tweak your program budget to purchase necessary materials and/or hire additional staff. Making these operational and workforce investments will lead to more successful implementation and program outcomes.

3. Create an Implementation Plan

Next, it’s time to flesh out your implementation plan. This plan should include a timeline and should specify staff members’ responsibilities for program related tasks. Many evidence-based programs have a set number of required sessions and guidelines for how frequently they should occur. Make sure that your implementation plan aligns with program requirements.

4. Train and Prepare Staff

Once you create your implementation plan, provide training for staff involved in the implementation. Involved staff should have a clear understanding of the program goals, activities, and their responsibilities throughout implementation. Your implementation plan should also include continued professional development opportunities and training for staff, to ensure continued high quality implementation.

5. Establish Continuous Monitoring Procedures

Once you begin implementing the program, you want to continuously monitor your fidelity to the program model. Many evidence-based programs come with accompanying fidelity checklists. It is important to identify a staff member, or an outside evaluator, who will conduct observations of the program to evaluate the implementation. You can use observations and fidelity checklists to assess if the program’s implementation is consistent with the original program model.

If your organization is looking for support in choosing, implementing or evaluating an evidence-based program, contact us today to learn more about our program development and evaluation services!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

5 W’s of a Process Evaluation: Part 2

In a recent blog post, we introduced the first two W’s of a process evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation

This blog post will cover the remaining three W’s:

  1. What methods to use to conduct a process evaluation
  2. Where to conduct a process evaluation
  3. When to conduct a process evaluation
WHAT METHODS TO USE WHEN CONDUCTING A PROCESS EVALUATION

There are several different data tools and methods you can use during a process evaluation. It may be helpful to use a combination of these methods!

  • Review documentation: It can be helpful to review staff logs, notes, attendance data and other program documents during a process evaluation. This method will help you to assess if all staff are following program procedures and documentation requirements.
  • Complete fidelity checks: Many programs/curriculums come with fidelity checklists for assessing program implementation. This is especially important if you are implementing an evidence-based program or model. Programs may have a set number of required sessions and guidelines for how frequently they should occur. You can use fidelity checklists to assess if the program’s implementation is consistent with the original program model.
  • Observe: Observations can be especially helpful when you Y Observationshave multiple sites and/or facilitators. During observations, it’s crucial to have a specific rating sheet or checklist of what you should expect to see. If a program has a fidelity checklist, you can use it during observations! If not, you should create your own rubric.
  • Collect stakeholder feedback: Stakeholder feedback gives you an idea of how each stakeholder group is experiencing your program. Groups to engage include program staff, clients, families of clients and staff from partner programs/organizations. You can use interviews, surveys, and focus groups to collect their feedback. These methods should not focus on your clients’ outcomes, but on their experience in the program. This will include their understanding of the program goals, structure, implementation, operating procedures and other program implementation components.

In our evaluation project with the Wabash YMCA’s 21 Century Community Learning Center, we used a combination of the methods described above. Our staff observed each program site using a guiding rubric. Our team collaborated beforehand to make sure they had a consistent understanding of what components to look for during observations. We also collected stakeholder feedback by conducting surveys with students, parents and teachers. The content of these surveys focused on their experiences and knowledge of the program. After the program was complete, we reviewed documentation, including attendance records and program demographic information.

WHERE TO CONDUCT A PROCESS EVALUATION

You should conduct a process evaluation wherever the program takes place. To capture an accurate picture of implementation, an evaluator needs to see how the program operates in the usual program environment. It is important to assess the implementation in all program environments. For example, if a program is being implemented at four different sites, you should assess the implementation at each site.

In our evaluation project with the Wabash YMCA, we assessed the program implementation at three different school sites. This involved physically observing the program at each site as well as reviewing records and documentation from each site. Being in the physical environment allowed us to assess which procedures were used consistently among sites. It also helped us identify program components that needed improvement.

WHEN TO CONDUCT A PROCESS EVALUATION

An organization can conduct a process evaluation at any time, but here are a few examples of times when its use would be most beneficial:

  • A few months to a year after starting a new program, you can conduct a process evaluation to assess how well your staff followed the implementation plan.
  • When you’re thinking about making a change to a program, a process evaluation will help you determine in what program areas you need to make changes.
  • If your program is not doing well, conduct a process evaluation to see if something in your process is interfering with program success.
  • When your program is doing well, conduct a process evaluation to see what in your process is making it successful.
  • If you’ve had issues with staff turnover, conducting a process evaluation can help identify gaps in staff training, professional development and ongoing support that may be contributing to the turnover rate.

To determine when to conduct a process evaluation, it is also important to consider the capacity of your organization. Make sure that your staff will have enough time to devote to the evaluation. Even when using an external evaluator, staff may need to spend extra time meeting with evaluators or participating in focus groups/interviews.

We conducted our evaluation with the Wabash YMCA at the end of their first year of program implementation. Evaluating their first year of implementation allows us to provide them with recommendations on how to improve the program’s implementation in future years. We will conduct a similar evaluation during the next three subsequent years to track their operations and processes over time.

If your organization needs support in conducting a process evaluation, contact us today to learn more about our evaluation services!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

Getting the Most Out of Your Needs Assessment

Recently Transform Consulting Group finished the annual needs assessment for the Indiana Head Start State Collaboration Office (IHSSCO).   Each Head Start State Collaboration Office is required to annually submit a needs assessment, which informs their strategic plan goals and objectives.  

IHSSCO uses their needs assessment to inform their annual work plan, and all organizations should make the connection between a needs assessment and the organization’s strategic goals!  Whether you want to conduct a needs assessment, program or organization evaluation, or annual report, don’t miss the chance to do one of the following:  

  1. Use your assessment to solicit new feedback or data.

The IHSSCO needs assessment solicited new feedback this year.  We interviewed and surveyed external stakeholders and Head Start partners.  If you’re going to request feedback, make sure you show you’re doing something with it.  No one likes to provide feedback, and then see that nothing changes. For Head Start partners and stakeholders, they will soon be able to read the needs assessment report and see the recommendations for solutions that address some of their feedback.

  1. Take the time to learn from your data.

Data is collected and reported on, but beyond totals and percentages, what does your data say?  What questions does it raise to inform your assessment and planning efforts? For example, we wanted to know:

  • How do Indiana Early Head Start and Head Start programs compare to national statistics?
  • Where are Early Head Start and Head Start centers located across the state, and is it proportionate to the population and need?  Image
  • What percentage of children are being served?  
  • Is there more of a demand for Early Head Start and Head Start in rural or urban areas?

Besides the demographics of your program participants and the outputs of a program, look for issues and barriers, gaps or overlap in services or clients, layer the data with other relevant indicators, and don’t forget to look at outcomes as well!

  1. Make sure you share the report – internally and externally.  

A needs assessment can take a great deal of time, effort, and resources from multiple individuals.  Once the process is completed, it is easy to do a quick review of the findings with program staff and then put it on the shelf.  The needs assessment report and its findings are not only important to program staff; it can also provide insight to all staff, program participants, funders, and external stakeholders/partners.  Check out this past blog for more ideas!

Make the report accessible and relevant.  Many people may only be interested in reading an executive summary of the report or skipping straight to the recommendations.  Others may be more attracted to infographics or dashboards. Decide how to best present your data for your audience, and then post these materials on your website, link to them in a newsletter, or mention them on social media.
If you’re ready to do things differently with your needs assessment but are not sure where to start, contact us today to discuss ways Transform Consulting Group can help!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

3 Financial Health Indicators to Track for Your Organization

Is your organization financially healthy? How do you measure the financial health of your organization? Nonprofit, education and other community organizations can look at the financial health of their organization and its relationship to your impact. Financial health also helps make sure programs are being run efficiently and if any program changes need to be made.

For many organizations, financial information is used with staff and administrators and shared with boards, potential funders and clients. It is not always necessary to give each audience the detailed level view of an organization’s financial information. However, it is critical that the leadership of the organization is aware of and tracking some critical indicators that determine your financial health.

While working with ten early childhood education programs on reaching their financial goals, we first had to look at the overall financial health of the organization. We asked to see income statements, balance sheets, client/ service information and any other key financial indicators. It can be hard to make sense of all of this information, especially for staff and board members who are not accountants or financial advisors.

We saw a gap and need to summarize the most important financial indicators in a one-page overview that can be regularly (and automatically) updated for board meetings, funder reports, and other key reports. We were able to identify three key areas of information to highlight for different. While every organization is different, we believe these three financial areas are critical for every organization to track and monitor.

Three Financial Health Areas to Track for Your Organization

1. Participation

If you are an organization that receives funding based on participation or enrollment numbers, then this is a critical indicator for you to track. Just like a hotel tracks their vacancy rate, your organization should have set goals and benchmarks for participation or enrollment rates. These should be based on your budget and what is needed to “break even”. Then you can look more closely at your participation by different population groups and a calendar of enrollment.

The chart example below shows the trend in children served by month. The trend shows how the capacity fluctuated. It may be useful to take closer look at multiple years to see if the monthly trends are consistent by year. This will help organizations plan for the future.

OrgDash1

2. Revenue & Expenses

Looking at the overall revenue and expenses helps organizations see when more money is coming in compared to what is being spent. Comparing these funding sources helps make informed decisions on program changes when necessary.

For example, when we looked at the revenue sources for an early childhood education program, we saw a majority of funds were coming from government subsidized programs. This might be a risky revenue source to rely on for a program. We helped the leaders discuss what might be a healthier percentage to set as a goal for government funding and other funding sources. We encouraged the program to increase regular parent-pay enrollment and fundraising initiatives to create more balanced revenue streams.

The sample dashboard section below takes a deeper dive into the revenue and expenses of an organization. This includes:

  • Budget versus actual revenue and expenses,
  • Net income (revenue minus expenses) trend, and
  • Sources of revenue and expenses.

As seen in the donut chart, the revenue and expense sources are each categorized into four main buckets. This makes the categories more understandable for a variety of audiences.

OrgDash23. Account standings

Looking at the organization’s account standings is a simple way to monitor any large fluctuations in the account. It also helps decision makers see how much money the organization has to work with at any given time. For example, a fundraising event that will have to be funded from the account can be approved if the board knows the appropriate funding is available in the accounts.

The dashboard below shows fairly consistent account amounts. The dip in checking in September could raise some questions. This is an opportunity for the organization to go back to the original sources to understand why.

OrgDash3

We encourage organizations to collect information beyond these three areas. These areas are meant to give a snapshot and raise questions within the organization when necessary. As seen in the examples, we like to display these areas in a visually-appealing dashboard using Tableau Software. Interactive dashboards allow users to apply filters and easily update the dashboard every month or quarter. Organizations can use other methods to display information depending on the intended audience and the amount of information being shared. View our Dashboard Services page to find out how we can help you with your dashboard needs.

If your organization does not know where to start, this blog post may be helpful for understanding why financial goals are important. At TCG, we can help your organization identify what financial information to collect and recognize key areas to share with the intended stakeholders. If your program needs support evaluating your financial stability or creating a visually appealing financial dashboard, contact us today!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

5 W’s of a Process Evaluation: Part 1

When it comes to program evaluation, people often think of evaluating the effectiveness and outcomes of their program. They may not think about evaluating how the program was administered or delivered, which may affect the program outcomes. There are several types of valuable evaluations that do not focus on outcomes. One type of evaluation, called “process or formative evaluation”, assesses how a program is being implemented.

In this two part blog series, we are going to cover the 5 W’s of a Process Evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation
  3. What methods to use to conduct a process evaluation
  4. Where to conduct a process evaluation
  5. When to conduct a process evaluation

In this first blog in the series we will cover the first two W’s. The next blog will discuss the other three.

WHY CONDUCT A PROCESS EVALUATION

Let’s start with the “why”. A process evaluation helps an organization better understand how their program is functioning and operating. Process evaluations also serve as an accountability measure and can answer key questions, such as:Screen Shot 2018-07-27 at 4.38.23 PM

  • Is the program operating as it was designed and intended?
  • Is the current implementation adhering to program fidelity?
  • Is the program being implemented consistently across multiple sites and staff, if applicable?
  • What type and frequency of services are provided?
  • What program procedures are followed?
  • Is the program serving its targeted population?

 

It is important to determine what you want to learn from your process evaluation. Maybe you want to assess if the program is being implemented as it was intended or you want to know if the program model is being followed. Whatever the reason, you want to be clear about why you are completing the process evaluation and what you hope to learn.

We are currently working with the Wabash YMCA’s 21st Century Community Learning Center to evaluate their program implementation. Each center is required to work with an external evaluator to conduct a process evaluation. Here is what we hope to learn and the why of this evaluation:

  1. The evaluation will assess if the program has been implemented as it was intended and if it is adhering to state standards;
  2. This evaluation will capture the population served through the assessment of attendance trends;
  3. The findings from the process evaluation will be used for program improvement in subsequent years.

WHO SHOULD CONDUCT YOUR PROCESS EVALUATION

When determining who will conduct your process evaluation, you have the option of either identifying an internal staff member (e.g., program manager or quality assurance) from your organization or hiring an external evaluator. Many organizations find that there are challenges with an internal team member: they may not be objective, they don’t have a fresh perspective, and they often have other job responsibilities beyond the evaluation.

For the reasons mentioned above, it is beneficial to have an external evaluator (like TCG!). An external evaluator will be able to assess the operations of your program from an unbiased lens. This is especially helpful if a program has multiple sites. An external evaluator can assess all sites/facilitators for consistency more objectively than a program staff member. (If you’re interested in learning more about how to evaluate multi-site programs, view our blog post here!).

In our evaluation project with the Wabash YMCA, the decision to conduct an evaluation with an external group was made by their funders. This decision ensures that the evaluation is high quality and objective.

The other three W’s will be discussed in a later blog post, so stay tuned! In the meantime, contact us today to learn more about our evaluation services!

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin

3 Tips to Evaluate Multi-Site Programs

Evaluation is one of many services we provide for clients at Transform Consulting Group. Evaluation is important for any program because it helps organizations determine if a program is effective or if the change they set to accomplish actually occurred. In a previous blog we shared the 4 key steps to evaluation that we follow. These steps are (1) establish clear outcomes, (2) create or modify data tools and systems, (3) analyze the data, and (4) use data to make informed decisions.

Some of our clients offer their program Multisite Programservices across multiple sites. This program model can be useful because it can save time, money and resources (i.e. hiring one director to manage multiple sites). As evaluators, we appreciate multi-site programs as well. Here are some reasons why multi-site evaluations are important and useful for evaluators:

  • Increases knowledge about what works in different settings
  • Allows for larger sample size when gathering data
  • Creates opportunities for sharing experience and knowledge about what works
  • Helps evaluators see sustainability on a wider scale

At TCG, we are currently evaluating a coupleY Observations 21st Century Community Learning Centers (CCLC) Program grantees. The clients implement the program across multiple sites, usually different schools within a school district. We use the following tips to help us evaluate multiple sites for one program.

3 Tips to Evaluate Multi-Site Programs

  1. Observe each program site: Site observations help evaluators see how a program is implemented across each site. Is the approach and curriculum consistently followed? What is each site doing well compared to other sites? Where do improvements need to be made? Observations give evaluators important context and understanding when it is time to analyze the data. For the 21st CCLC program, the clients operate both summer, before and after school programs. We have built into our evaluation plan time to observe each site during the summer, at the beginning of the school year and the end of the school year.
  2. Collect data by site: Each site will have unique factors that may influence the program and overall data. When we are able to look at  data by individual site, it will help inform recommendations for improvement and possible best practices. For example, we recently worked with a multi-site program to assess their workforce retention data. They identified several locations that had really high turnover and a few sites with lower turnover rates. By disaggregating the data by each site, we were equipped to inform some strategic decision-making.  
  3. Solicit stakeholder feedback: Gathering input from key stakeholders is a critical component of our evaluation process (see these past blog posts: How to Engage Diverse Stakeholders in your Planning and 4 Tools to Collaborate with Key Stakeholders). For the 21st CCLC program, we sought feedback from program staff. We wanted to know if all staff understand the goals and outcomes of the program since it’s a new program for the organization and they have hired several new staff. If the responses differed, we can report back to the administrators that better messaging may be needed to ensure the program is being implemented effectively and meeting the state requirements.

Once data and information has been collected by site, the evaluator determines the best way to report the information. Depending on the audience for the evaluation report, the data may be presented differently.  For an external audience (public, the funder, etc.), program summary results may be reported. For internal audiences (staff, board of directors, volunteers), it may be helpful to present the data at a disaggregated level by each site or location. There may be steps to take to better align sites to meet program goals and outcomes.
4 step evalWhile there may be some additional steps needed to evaluate multi-site programs, our overall four-step approach to evaluation stays the same. Good evaluations often lead to recommendations for improvement. This is an opportunity to discuss data collected and implications for future programming, including ongoing program evaluation practices within the organization. If you like our approach, contact us today to learn about how we can help you with your next evaluation.

Share this article:Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Email this to someone
email
Share on LinkedIn
Linkedin