Category Archives: Evaluation

2018 Indy Big Data Conference Visualization Challenge Summary

Our TCG team recently presented at the Fifth Annual Indy Big Data Conference for the Visualization Challenge (read about some of our lessons learned here). This year’s challenge focused on improving Indiana’s talent pipeline and workforce. fullsizeoutput_3b6

Participating teams were provided several datasets to analyze and interpret, and then present to a panel of judges. Following the first round of presentations, the judges then awarded five teams the opportunity to present to 500+ conference attendees. Our TCG was one of those top five teams selected! Read more about the Visualization Challenge and data sets here: https://www.indybigdata.com/visualization-challenge/.

Our team used our Tableau expertise to create this interactive dashboard, and then we presented on our findings. The interactive dashboard addresses several key questions about the future of Indiana’s workforce and talent pipeline, including:

  1. What will be the top jobs in demand in 10 years?
  2. What is the education requirement for those top in demand jobs?
  3. What is the earning potential of those top in demand jobs?
  4. What is the return on investment on education for those top in demand jobs?

The following is a summary of TCG’s findings to these questions based on the data provided.

1. What will be the top jobs in demand in 10 years?

Screen Shot 2018-11-13 at 10.03.45 AM

Indiana’s top projected jobs are found primarily in the healthcare, food service and business industry. TCG found that the top projected jobs in 10 years varies depending on how you look at the data. For example, Wind Turbine Service Technicians looks like the top in demand job, because of a 71% expected increase in positions in 10 years. However, when you look at the actual number of new jobs for Wind Turbine Service Technicians,  there are only 66 total positions expected to be added over 10 years. This is not actually enough individual positions to create new higher education programs to encourage individuals to pursue this career. In this visual chart, you can scroll down to see the top projected jobs and how they compare. Hover over the number for more information about that occupation.

2. What is the education requirement for those top in demand jobs?

Screen Shot 2018-11-13 at 10.06.28 AM

The U.S. Census Bureau reports one-third of working adults ages 18-64 have an associate degree or higher. An additional 25% of this population have some education beyond high school. Interestingly, we found that six out of ten of the top 25 projected jobs only require a high school diploma. This doesn’t mean that we should stop encouraging young people to pursue higher education. Instead, the trend looks like the largest amount of new positions will be entry level or positions that do not immediately require an education beyond high school.

3. What is the earning potential of those top in demand jobs?

TCG created a box and whisker plot chart to show the earning potential by occupation. For all occupations the median annual salary is about $40,000. The top 25 projected jobs based on the number of new positions shows a median annual salary at $31,000 which is below the current median annual salary. Again, this somewhat relates to the education level requirements in the previous section. The largest increase in projected new jobs are not high earning jobs. Visit the interactive dashboard to explore the different wage levels, experience levels, and hourly rates of the projected top jobs.

Screen Shot 2018-11-13 at 10.07.08 AM

Screen Shot 2018-11-13 at 10.07.23 AM
4. What is the return on investment on education for those top in demand jobs?

The costs of education can feel like a burden to individuals, especially when the costs drag on through student loans for several years. The average timeframe expected to pay off the cost of postsecondary education is over two years. Our TCG team was interested in calculating the return on investment (ROI) of pursuing higher education for the top 25 occupations. You can rank the list by the jobs with the highest and lowest ROI. The average ROI for all jobs is $1.4 million.

Recommendations

TCG made several recommendations from the data analyzed. (This list can also be viewed on the second tab in the interactive dashboard).

  1. Review employment projections in multiple ways, like reviewing the expected growth based on the number of new jobs and the percentage growth of new jobs (e.g., Wind Turbine Service Technicians example) to ensure informed decision making. This can assist with decision making when developing or removing educational degree or certification programs and when encouraging individuals to enter a certain field.
  2. Offer more certificates and associate degrees both independently and as a part of bachelor’s degree curriculum. It would align with required educational needs of the workforce, contribute to less debt for students, and help improve higher education completion rate.
  3. Increase partnerships with employers (like IU Health) to do more direct career pipelines and apprenticeship programs. This could also potentially reduce the student cost. For example, Graduate Assistant positions are partially funded by the organization that receives a graduate student.
  4. Include more experiential learning in degree programs. It would help the student decide if they are in the right major or explore a secondary interest. It could also get industries to partner with other majors/ disciplines. For example, get the “creatives” into STEM activities – increasing the workforce pipeline.
  5. Consider offering incentives to attract individuals in the top projected jobs, especially those jobs with a low ROI. Look at ways to make these jobs more attractive to meet the future workforce needs.

Learn more about the Indy Big Data Challenge in this article by the Indiana Management Performance Hub. At TCG, we are data nerds. Check out our research and analysis services, and learn how we can develop visual reports and dashboards to inform your decision-making and help you tell your story of impact.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

5 Tips to Implement an Evidence-based Program

When awarding funding, philanthropic funders want to invest in “what works” and is proven effective. Many funders show preference for programs and practices that are evidence-based. Implementing an evidence-based program is a great way for grant seekers to demonstrate that they are also committed to “what works”.

For example, the Richard M. Fairbanks Foundation recently awarded funding to over 20 schools and school districts as part of their Prevention Matters initiative.  Prevention Matters is a three-year grant initiative aiming to help Marion County schools identify, implement and sustain proven substance use prevention programs.

To apply for this funding, schools selected an evidence-based substance use prevention program that aligned with their needs. In their proposal, schools had to demonstrate that they had a strong plan for implementation and sustainability. Developing such a plan can be a daunting task, but is crucial for successful implementation. We worked with Bishop Chatard and the North Deanery Schools of the Archdiocese of Indianapolis to help them develop their implementation plan and proposal (Which was fully funded by the Fairbanks Foundation! Learn more about our fundraising services here.). Here are 5 tips we used to help them prepare to successfully implement their evidence-based program!

1. Select an Evidence-based ProgramWhat Works Image (1)

First, you need to find a program that aligns with the needs you are trying to address. For example, if you are a school looking to prevent substance use and violence, while also promoting positive youth development, you may choose to implement the Botvin LifeSkills Training curriculum.

Taking the time to research available programs is crucial to ensuring successful implementation and maximum impact. To learn more about how to find an evidence based program, check out this blog!

2. Assess your Organization’s Capacity

Once you have selected an appropriate evidence-based program, it is important to assess your current funding and staffing capacity. You want to assess if your current organizational capacity will allow you to implement the program with fidelity. Fidelity refers to the extent to which you deliver your program as the original program model intended. Evidence-based programs are  proven effective and that effectiveness relates to how the program is implemented. Therefore, fidelity to the model is crucial to successful implementation.

Completing a feasibility study is a great way to assess your capacity and readiness. A well designed feasibility study will help an organization assess 1) if what they are thinking of implementing is possible and 2) how to consider implementing it. Check out this blog to learn more about completing a feasibility study.

The assessment of your capacity may indicate that you need to make some organizational changes. For example, you might need to tweak your program budget to purchase necessary materials and/or hire additional staff. Making these operational and workforce investments will lead to more successful implementation and program outcomes.

3. Create an Implementation Plan

Next, it’s time to flesh out your implementation plan. This plan should include a timeline and should specify staff members’ responsibilities for program related tasks. Many evidence-based programs have a set number of required sessions and guidelines for how frequently they should occur. Make sure that your implementation plan aligns with program requirements.

4. Train and Prepare Staff

Once you create your implementation plan, provide training for staff involved in the implementation. Involved staff should have a clear understanding of the program goals, activities, and their responsibilities throughout implementation. Your implementation plan should also include continued professional development opportunities and training for staff, to ensure continued high quality implementation.

5. Establish Continuous Monitoring Procedures

Once you begin implementing the program, you want to continuously monitor your fidelity to the program model. Many evidence-based programs come with accompanying fidelity checklists. It is important to identify a staff member, or an outside evaluator, who will conduct observations of the program to evaluate the implementation. You can use observations and fidelity checklists to assess if the program’s implementation is consistent with the original program model.

If your organization is looking for support in choosing, implementing or evaluating an evidence-based program, contact us today to learn more about our program development and evaluation services!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

5 W’s of a Process Evaluation: Part 2

In a recent blog post, we introduced the first two W’s of a process evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation

This blog post will cover the remaining three W’s:

  1. What methods to use to conduct a process evaluation
  2. Where to conduct a process evaluation
  3. When to conduct a process evaluation
WHAT METHODS TO USE WHEN CONDUCTING A PROCESS EVALUATION

There are several different data tools and methods you can use during a process evaluation. It may be helpful to use a combination of these methods!

  • Review documentation: It can be helpful to review staff logs, notes, attendance data and other program documents during a process evaluation. This method will help you to assess if all staff are following program procedures and documentation requirements.
  • Complete fidelity checks: Many programs/curriculums come with fidelity checklists for assessing program implementation. This is especially important if you are implementing an evidence-based program or model. Programs may have a set number of required sessions and guidelines for how frequently they should occur. You can use fidelity checklists to assess if the program’s implementation is consistent with the original program model.
  • Observe: Observations can be especially helpful when you Y Observationshave multiple sites and/or facilitators. During observations, it’s crucial to have a specific rating sheet or checklist of what you should expect to see. If a program has a fidelity checklist, you can use it during observations! If not, you should create your own rubric.
  • Collect stakeholder feedback: Stakeholder feedback gives you an idea of how each stakeholder group is experiencing your program. Groups to engage include program staff, clients, families of clients and staff from partner programs/organizations. You can use interviews, surveys, and focus groups to collect their feedback. These methods should not focus on your clients’ outcomes, but on their experience in the program. This will include their understanding of the program goals, structure, implementation, operating procedures and other program implementation components.

In our evaluation project with the Wabash YMCA’s 21 Century Community Learning Center, we used a combination of the methods described above. Our staff observed each program site using a guiding rubric. Our team collaborated beforehand to make sure they had a consistent understanding of what components to look for during observations. We also collected stakeholder feedback by conducting surveys with students, parents and teachers. The content of these surveys focused on their experiences and knowledge of the program. After the program was complete, we reviewed documentation, including attendance records and program demographic information.

WHERE TO CONDUCT A PROCESS EVALUATION

You should conduct a process evaluation wherever the program takes place. To capture an accurate picture of implementation, an evaluator needs to see how the program operates in the usual program environment. It is important to assess the implementation in all program environments. For example, if a program is being implemented at four different sites, you should assess the implementation at each site.

In our evaluation project with the Wabash YMCA, we assessed the program implementation at three different school sites. This involved physically observing the program at each site as well as reviewing records and documentation from each site. Being in the physical environment allowed us to assess which procedures were used consistently among sites. It also helped us identify program components that needed improvement.

WHEN TO CONDUCT A PROCESS EVALUATION

An organization can conduct a process evaluation at any time, but here are a few examples of times when its use would be most beneficial:

  • A few months to a year after starting a new program, you can conduct a process evaluation to assess how well your staff followed the implementation plan.
  • When you’re thinking about making a change to a program, a process evaluation will help you determine in what program areas you need to make changes.
  • If your program is not doing well, conduct a process evaluation to see if something in your process is interfering with program success.
  • When your program is doing well, conduct a process evaluation to see what in your process is making it successful.
  • If you’ve had issues with staff turnover, conducting a process evaluation can help identify gaps in staff training, professional development and ongoing support that may be contributing to the turnover rate.

To determine when to conduct a process evaluation, it is also important to consider the capacity of your organization. Make sure that your staff will have enough time to devote to the evaluation. Even when using an external evaluator, staff may need to spend extra time meeting with evaluators or participating in focus groups/interviews.

We conducted our evaluation with the Wabash YMCA at the end of their first year of program implementation. Evaluating their first year of implementation allows us to provide them with recommendations on how to improve the program’s implementation in future years. We will conduct a similar evaluation during the next three subsequent years to track their operations and processes over time.

If your organization needs support in conducting a process evaluation, contact us today to learn more about our evaluation services!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

Getting the Most Out of Your Needs Assessment

Recently Transform Consulting Group finished the annual needs assessment for the Indiana Head Start State Collaboration Office (IHSSCO).   Each Head Start State Collaboration Office is required to annually submit a needs assessment, which informs their strategic plan goals and objectives.  

IHSSCO uses their needs assessment to inform their annual work plan, and all organizations should make the connection between a needs assessment and the organization’s strategic goals!  Whether you want to conduct a needs assessment, program or organization evaluation, or annual report, don’t miss the chance to do one of the following:  

  1. Use your assessment to solicit new feedback or data.

The IHSSCO needs assessment solicited new feedback this year.  We interviewed and surveyed external stakeholders and Head Start partners.  If you’re going to request feedback, make sure you show you’re doing something with it.  No one likes to provide feedback, and then see that nothing changes. For Head Start partners and stakeholders, they will soon be able to read the needs assessment report and see the recommendations for solutions that address some of their feedback.

  1. Take the time to learn from your data.

Data is collected and reported on, but beyond totals and percentages, what does your data say?  What questions does it raise to inform your assessment and planning efforts? For example, we wanted to know:

  • How do Indiana Early Head Start and Head Start programs compare to national statistics?
  • Where are Early Head Start and Head Start centers located across the state, and is it proportionate to the population and need?  Image
  • What percentage of children are being served?  
  • Is there more of a demand for Early Head Start and Head Start in rural or urban areas?

Besides the demographics of your program participants and the outputs of a program, look for issues and barriers, gaps or overlap in services or clients, layer the data with other relevant indicators, and don’t forget to look at outcomes as well!

  1. Make sure you share the report – internally and externally.  

A needs assessment can take a great deal of time, effort, and resources from multiple individuals.  Once the process is completed, it is easy to do a quick review of the findings with program staff and then put it on the shelf.  The needs assessment report and its findings are not only important to program staff; it can also provide insight to all staff, program participants, funders, and external stakeholders/partners.  Check out this past blog for more ideas!

Make the report accessible and relevant.  Many people may only be interested in reading an executive summary of the report or skipping straight to the recommendations.  Others may be more attracted to infographics or dashboards. Decide how to best present your data for your audience, and then post these materials on your website, link to them in a newsletter, or mention them on social media.
If you’re ready to do things differently with your needs assessment but are not sure where to start, contact us today to discuss ways Transform Consulting Group can help!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

3 Financial Health Indicators to Track for Your Organization

Is your organization financially healthy? How do you measure the financial health of your organization? Nonprofit, education and other community organizations can look at the financial health of their organization and its relationship to your impact. Financial health also helps make sure programs are being run efficiently and if any program changes need to be made.

For many organizations, financial information is used with staff and administrators and shared with boards, potential funders and clients. It is not always necessary to give each audience the detailed level view of an organization’s financial information. However, it is critical that the leadership of the organization is aware of and tracking some critical indicators that determine your financial health.

While working with ten early childhood education programs on reaching their financial goals, we first had to look at the overall financial health of the organization. We asked to see income statements, balance sheets, client/ service information and any other key financial indicators. It can be hard to make sense of all of this information, especially for staff and board members who are not accountants or financial advisors.

We saw a gap and need to summarize the most important financial indicators in a one-page overview that can be regularly (and automatically) updated for board meetings, funder reports, and other key reports. We were able to identify three key areas of information to highlight for different. While every organization is different, we believe these three financial areas are critical for every organization to track and monitor.

Three Financial Health Areas to Track for Your Organization

1. Participation

If you are an organization that receives funding based on participation or enrollment numbers, then this is a critical indicator for you to track. Just like a hotel tracks their vacancy rate, your organization should have set goals and benchmarks for participation or enrollment rates. These should be based on your budget and what is needed to “break even”. Then you can look more closely at your participation by different population groups and a calendar of enrollment.

The chart example below shows the trend in children served by month. The trend shows how the capacity fluctuated. It may be useful to take closer look at multiple years to see if the monthly trends are consistent by year. This will help organizations plan for the future.

OrgDash1

2. Revenue & Expenses

Looking at the overall revenue and expenses helps organizations see when more money is coming in compared to what is being spent. Comparing these funding sources helps make informed decisions on program changes when necessary.

For example, when we looked at the revenue sources for an early childhood education program, we saw a majority of funds were coming from government subsidized programs. This might be a risky revenue source to rely on for a program. We helped the leaders discuss what might be a healthier percentage to set as a goal for government funding and other funding sources. We encouraged the program to increase regular parent-pay enrollment and fundraising initiatives to create more balanced revenue streams.

The sample dashboard section below takes a deeper dive into the revenue and expenses of an organization. This includes:

  • Budget versus actual revenue and expenses,
  • Net income (revenue minus expenses) trend, and
  • Sources of revenue and expenses.

As seen in the donut chart, the revenue and expense sources are each categorized into four main buckets. This makes the categories more understandable for a variety of audiences.

OrgDash23. Account standings

Looking at the organization’s account standings is a simple way to monitor any large fluctuations in the account. It also helps decision makers see how much money the organization has to work with at any given time. For example, a fundraising event that will have to be funded from the account can be approved if the board knows the appropriate funding is available in the accounts.

The dashboard below shows fairly consistent account amounts. The dip in checking in September could raise some questions. This is an opportunity for the organization to go back to the original sources to understand why.

OrgDash3

We encourage organizations to collect information beyond these three areas. These areas are meant to give a snapshot and raise questions within the organization when necessary. As seen in the examples, we like to display these areas in a visually-appealing dashboard using Tableau Software. Interactive dashboards allow users to apply filters and easily update the dashboard every month or quarter. Organizations can use other methods to display information depending on the intended audience and the amount of information being shared. View our Dashboard Services page to find out how we can help you with your dashboard needs.

If your organization does not know where to start, this blog post may be helpful for understanding why financial goals are important. At TCG, we can help your organization identify what financial information to collect and recognize key areas to share with the intended stakeholders. If your program needs support evaluating your financial stability or creating a visually appealing financial dashboard, contact us today!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

5 W’s of a Process Evaluation: Part 1

When it comes to program evaluation, people often think of evaluating the effectiveness and outcomes of their program. They may not think about evaluating how the program was administered or delivered, which may affect the program outcomes. There are several types of valuable evaluations that do not focus on outcomes. One type of evaluation, called “process or formative evaluation”, assesses how a program is being implemented.

In this two part blog series, we are going to cover the 5 W’s of a Process Evaluation:

  1. Why conduct a process evaluation
  2. Who should conduct a process evaluation
  3. What methods to use to conduct a process evaluation
  4. Where to conduct a process evaluation
  5. When to conduct a process evaluation

In this first blog in the series we will cover the first two W’s. The next blog will discuss the other three.

WHY CONDUCT A PROCESS EVALUATION

Let’s start with the “why”. A process evaluation helps an organization better understand how their program is functioning and operating. Process evaluations also serve as an accountability measure and can answer key questions, such as:Screen Shot 2018-07-27 at 4.38.23 PM

  • Is the program operating as it was designed and intended?
  • Is the current implementation adhering to program fidelity?
  • Is the program being implemented consistently across multiple sites and staff, if applicable?
  • What type and frequency of services are provided?
  • What program procedures are followed?
  • Is the program serving its targeted population?

 

It is important to determine what you want to learn from your process evaluation. Maybe you want to assess if the program is being implemented as it was intended or you want to know if the program model is being followed. Whatever the reason, you want to be clear about why you are completing the process evaluation and what you hope to learn.

We are currently working with the Wabash YMCA’s 21st Century Community Learning Center to evaluate their program implementation. Each center is required to work with an external evaluator to conduct a process evaluation. Here is what we hope to learn and the why of this evaluation:

  1. The evaluation will assess if the program has been implemented as it was intended and if it is adhering to state standards;
  2. This evaluation will capture the population served through the assessment of attendance trends;
  3. The findings from the process evaluation will be used for program improvement in subsequent years.

WHO SHOULD CONDUCT YOUR PROCESS EVALUATION

When determining who will conduct your process evaluation, you have the option of either identifying an internal staff member (e.g., program manager or quality assurance) from your organization or hiring an external evaluator. Many organizations find that there are challenges with an internal team member: they may not be objective, they don’t have a fresh perspective, and they often have other job responsibilities beyond the evaluation.

For the reasons mentioned above, it is beneficial to have an external evaluator (like TCG!). An external evaluator will be able to assess the operations of your program from an unbiased lens. This is especially helpful if a program has multiple sites. An external evaluator can assess all sites/facilitators for consistency more objectively than a program staff member. (If you’re interested in learning more about how to evaluate multi-site programs, view our blog post here!).

In our evaluation project with the Wabash YMCA, the decision to conduct an evaluation with an external group was made by their funders. This decision ensures that the evaluation is high quality and objective.

The other three W’s will be discussed in a later blog post, so stay tuned! In the meantime, contact us today to learn more about our evaluation services!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

3 Strategic Plan Tools to Create

Congratulations! You have journeyed through the 4 steps of our Strategic Planning Process and you’re ready for the final step: Create. (We covered step 1, step 2, and step 3 in previous blogs).

Screen Shot 2018-07-27 at 2.03.19 PM

The goal of a strategic plan is to develop timely, relevant and action-oriented plans for the future of your organization. Once you have a clear direction, it is time to make sense of the information and package it in a way that is meaningful and possible to implement.  

At TCG, we believe a strategic plan has little value if it is a report that sits on your shelf, never to be seen again. We don’t create long strategic plan reports that you can’t use. We want you to use it, share it and review it on a regular basis.

When working with clients, we recommend and create 3 different strategic plan tools:

1. One-page strategic plan – This is a one-page summary of your goals and top strategies. This tool can be shared externally with partners, funders, and other key stakeholders as well as internally with staff.

When creating a strategic plan for the Wabash County Early Childhood Education Committee, we wanted a one-page overview that highlighted the following key elements:Screen Shot 2018-07-27 at 2.03.33 PM

a. Stakeholders involved (especially since this is a collective impact, multi-sector plan)

b. Goals

c. Strategies

d. Outcomes

Each one-pager for the strategic plan that we create is unique to the client but essentially covers their top goals and strategies.

2. Strategic plan report – This report explains the process of how the strategic plan was completed, the information that was collected, and more details about the goals and strategies. This is typically an internal document that is shared with staff and the board to use when reflecting on the process. It’s especially helpful to document this information for when there are leadership transitions with the staff and board.

Screen Shot 2018-07-27 at 2.03.53 PM

3. Implementation plan – Too often we find that organizations get stuck with figuring out how to take the big picture elements in the strategic plan and make them operational. We create an “implementation plan” to unpack the strategic plan into actionable steps for staff, committees and the board. The main audience for the implementation plan is staff, board and committee members who are most likely responsible for implementation.  Screen Shot 2018-07-27 at 2.04.09 PM


This could be set up like a calendar or a chart that describes who is responsible for each step. We also love using Tableau to create a strategic plan dashboard to track and monitor action items and milestones. The point is that we want all parties involved to have a clear understanding of the timeline, so that they can put the plan in motion.

Is your organization ready to jump into a strategic planning process? Learn more about our strategic planning services here. Contact us today, and we’d love to chat about how our team can meet your needs.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

3 Tips to Evaluate Multi-Site Programs

Evaluation is one of many services we provide for clients at Transform Consulting Group. Evaluation is important for any program because it helps organizations determine if a program is effective or if the change they set to accomplish actually occurred. In a previous blog we shared the 4 key steps to evaluation that we follow. These steps are (1) establish clear outcomes, (2) create or modify data tools and systems, (3) analyze the data, and (4) use data to make informed decisions.

Some of our clients offer their program Multisite Programservices across multiple sites. This program model can be useful because it can save time, money and resources (i.e. hiring one director to manage multiple sites). As evaluators, we appreciate multi-site programs as well. Here are some reasons why multi-site evaluations are important and useful for evaluators:

  • Increases knowledge about what works in different settings
  • Allows for larger sample size when gathering data
  • Creates opportunities for sharing experience and knowledge about what works
  • Helps evaluators see sustainability on a wider scale

At TCG, we are currently evaluating a coupleY Observations 21st Century Community Learning Centers (CCLC) Program grantees. The clients implement the program across multiple sites, usually different schools within a school district. We use the following tips to help us evaluate multiple sites for one program.

3 Tips to Evaluate Multi-Site Programs

  1. Observe each program site: Site observations help evaluators see how a program is implemented across each site. Is the approach and curriculum consistently followed? What is each site doing well compared to other sites? Where do improvements need to be made? Observations give evaluators important context and understanding when it is time to analyze the data. For the 21st CCLC program, the clients operate both summer, before and after school programs. We have built into our evaluation plan time to observe each site during the summer, at the beginning of the school year and the end of the school year.
  2. Collect data by site: Each site will have unique factors that may influence the program and overall data. When we are able to look at  data by individual site, it will help inform recommendations for improvement and possible best practices. For example, we recently worked with a multi-site program to assess their workforce retention data. They identified several locations that had really high turnover and a few sites with lower turnover rates. By disaggregating the data by each site, we were equipped to inform some strategic decision-making.  
  3. Solicit stakeholder feedback: Gathering input from key stakeholders is a critical component of our evaluation process (see these past blog posts: How to Engage Diverse Stakeholders in your Planning and 4 Tools to Collaborate with Key Stakeholders). For the 21st CCLC program, we sought feedback from program staff. We wanted to know if all staff understand the goals and outcomes of the program since it’s a new program for the organization and they have hired several new staff. If the responses differed, we can report back to the administrators that better messaging may be needed to ensure the program is being implemented effectively and meeting the state requirements.

Once data and information has been collected by site, the evaluator determines the best way to report the information. Depending on the audience for the evaluation report, the data may be presented differently.  For an external audience (public, the funder, etc.), program summary results may be reported. For internal audiences (staff, board of directors, volunteers), it may be helpful to present the data at a disaggregated level by each site or location. There may be steps to take to better align sites to meet program goals and outcomes.
4 step evalWhile there may be some additional steps needed to evaluate multi-site programs, our overall four-step approach to evaluation stays the same. Good evaluations often lead to recommendations for improvement. This is an opportunity to discuss data collected and implications for future programming, including ongoing program evaluation practices within the organization. If you like our approach, contact us today to learn about how we can help you with your next evaluation.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

3 Steps to Establish Clear Outcomes

Evaluation is key in determining if your program is making the desired impact. While critical, evaluation can be an overwhelming and intimidating process for organizations. We have worked with several clients to help them embark on the journey of evaluating their program(s). At Transform Consulting Group, we follow a four-step evaluation process. The first step of establishing clear outcomes can be one of the most difficult. You know what your mission is and you know your vision for a better community, but how do these translate into measurable outcomes?

4 eval steps

 

1. Establish clear outcomes

2.  Create or modify data tools and system

3. Analyze the data

4. Use data to make informed decisions

 

Outputs vs. Outcomes

When determining outcomes, the conversation usually starts with program outputs. Outputs are what your program produces: activities, services and participants. Tracking, analyzing and reporting your program outputs is a valuable way of displaying an organization’s work! For example, let’s say an after-school tutoring program served 650 students during the 2017-2018 school year. You could further break that number down by age and frequency of services:

Age group Session Frequency Number of participants Total number of sessions provided
3rd-5th grades Weekly for 10 weeks 320 320×10=3,200
6th-8th grades Weekly for 15 weeks 330 330×15=4,950
Total tutoring sessions provided= 8,150

With a few simple calculations, we have a powerful representation of the work this tutoring team has accomplished! However, outputs alone don’t display programmatic impact.

Outcomes go one more step in showing impact. Outcomes are the changes in knowledge or behavior that you want your clients to experience as a result of your program. They are the “so what” of your services and activities. There are three levels of outcomes that you want to set and measure:

  1. Short-term: What changes in knowledge, attitude or behavior do you want to see in your clients by the time they complete your program or service?
  2. Intermediate: What changes do you want to see in client knowledge, attitude or behavior 6 months-12 months following program completion?
  3. Long-term: What changes do you want to see in client knowledge, attitude or behavior 1+ years after program completion?

IMG_5774

We recently worked with the Center for Leadership Development (CLD) to develop short-term, intermediate and long-term outcomes. They are focused on helping get more students of color to and through postsecondary education. Here are three steps that we used to help them establish clear outcomes that assess the impact of their organization.

1. Align to Organizational Mission and Purpose

When you set outcomes, you want to make sure that they align with your organizational mission and benchmarks. CLD’s programming and organizational benchmarks are centered around five principles for success: character development, educational excellence, leadership effectiveness, community service, and career achievement. We helped them establish several outcomes that aligned with their programs, missions, and key principles. 

2. Review Funder’s Priorities 

When receiving grant funding or large donations, organizations often make commitments about what they will accomplish with those funds. Therefore, you want to make sure that future outcomes still align with your current funding priorities and commitments. We worked with CLD to make sure that their many outcomes aligned with the commitments they had made with their current funders.

3. Develop SMART Outcomes

When working with clients to develop outcomes, we follow the “SMART” rubric. We plan to write a full blog to go more in-depth about the SMART rubric, but for now the main takeaway is that they are specific, measurable, achievable, relevant and timely.

One of CLD’s long-term desired outcomes is for 75% of their participants to earn a bachelor’s degree or credential within six years of high school graduation. This outcome aligns perfectly with their mission and funding commitments, but is it SMART? Let’s check!

Copy of Establishing Clear Outcomes draft (2)With their clear outcomes established, CLD now has a road map of where they want their participants to go. This road map not only helps CLD stay on course, but it also helps to paint a picture of their desired impact for their funders and supporters. Now they are ready to move on to the next step of their evaluation: Creating or modifying data tools and systems!

If you’re ready to evaluate your program, but are hesitant to take the first step, contact us today!

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

Training Your Team

As a new employee (week 8 on the job), I’m learning more and more about Transform Consulting Group. I have found that we don’t stop working where other consultants might.

At other consulting groups, the typical client engagement cycle is this: Research is finished. Meeting is over. Report is delivered. On to the next client.

Client:  Now what?

Consultant:  I wonder what they did with our recommendations.

TCG goes further – we help you implement change.

At Transform Consulting Group we go further in helping our clients implement change. One way we do this is through training our client’s staff.

CLD TrainingJust like no funder wants to adopt a grantee forever, no client wants to hire a consultant forever. We want to make sure that the work we have completed for the client will be sustained beyond our engagement with the client.  

We work with several clients to support their evaluation needs. This might include evaluating a program for a grant, developing an evaluation plan and process, or developing a data dashboard. Evaluation can be intimidating for many of our client’s staff, so we make sure we develop data tools and plans that they can manage and implement beyond our engagement. In order to accomplish this, we might embed a series of trainings for staff throughout our evaluation project.

We are currently doing this right now for the Center for Leadership Development. We have provided the following training topics mainly around analyzing data:

Our trainings include detailed slides, a recording of our training, and handouts. These are all useful tools that can be shared with new staff at an organization.

We also offer trainings on other topics for which we provide services. Some of our popular trainings include:CQI Training UW Central IN

Not only does the organization benefit, but according to research, employees want professional development!  It makes them feel that the company is investing in them and their development, which can increase employee retention.  A win-win!

At Transform Consulting Group, we understand the importance of evaluation and the critical step of implementation.  If your organization needs help developing its staff or would like to engage with a consultant that can help you through the entire evaluation process, contact us today to learn more!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn