Category Archives: Evaluation

4 Steps to Create a Dashboard

In today’s information age, organizations are overwhelmed with the amount of information that they collect, track and monitor.  Non-profit leaders must decipher through all the data to determine what is meaningful and relevant to share with staff, funders, Board of Directors and other community partners.  A dashboard is a great tool to bring all the critical elements together in a user-friendly report.

Through our program evaluation and research and analysis services, we help organizations create dashboards.  Here are a four steps to create an effective dashboard:

  1. Determine the audience for the dashboard. A dashboard is customized for the audience meant to view and use the dashboard, so first an organization needs to determine the intended audience. Then an organization needs to determine the key takeaways that you want the targeted audience to get from this dashboard.  Lastly, the organization should really focus on the information that is most important and relevant for this audience.
  1. Decide on what the dashboard is tracking now that the audience is determined. A dashboard is meant to communicate progress over time, such as monthly, quarterly or annually. In addition, data in the dashboard can be used to compare different data sets, such as geographic locations, sites or populations. These factors need to be determined to provide the appropriate context for decision makers.
  1. Determine the visuals that will be most effective in communicating the message. In most cases, we work to fit a dashboard on one-page. This does not provide much “real estate”, so you must be intentional about the visuals used to grab the audience’s attention and display the key messages [Sidebar: this is why we use Tableau!]. A dashboard does not have much room for wording and explanation.
  1. Determine the delivery of the dashboard. In most cases, dashboards are “static” or print reports that are shared via handouts or electronically. However, with the growing development of software programs more interactive dashboards are being created.  In some cases, there may be value in creating both a static dashboard that is completed annually and an interactive dashboard that is updated real-time.

Dashboard Blog imageAs we shared in this blog post about creating a needs assessment and annual report, we mentioned the state dashboard and county profile that we created for ELAC.  After conducting the first state needs assessment on young
children, ELAC realized that the amount of data and information was overwhelming. ELAC was inspired by the Annie E. Casey Foundation’s 16 indicator dashboard in their annual data book and used this to create ELAC’s dashboard.

Following the four steps above and inspired by the Annie E. Casey Foundation’s dashboard, we worked with ELAC to create a state dashboard that:

  • Identifies four focus areas related to children and families, high-quality care programs, education workforce, and kindergarten readiness with 16 key indicators.
  • Compares progress over time (dependent upon the release of new/ old data)
  • Uses arrows (a visual tool) to depict if the numbers represent improvement (arrow goes up) or worsening (arrow goes down).

After a few years of creating the ELAC state dashboard, communities across the state of Indiana were asking for this same data at the local level.  We worked with ELAC and our state data partners to gather the county level information to create a two-page county dashboard and profile.  The first page of the county profile mirrors the state dashboard with a few exceptions.

Instead of comparing progress over time, the county dashboards compare the county data to the state data.  Following steps #1 and #2 above, we focused on the audience for the county dashboards who said that having their data in the context of the state data would be helpful to know if they are doing better or worse.  Therefore, that ranked as a higher priority than comparing their data over time.

The second page of the ELAC county dashboard was new and provided the opportunity to add visuals (charts and graphs) to depPicture1ict the key findings in the full narrative report.  The visuals help to communicate complex information in simple charts.

Using a data visualization software program like Tableau is critical to not only help make the dashboards visually appealing but also to automate the process.  In this case, we created 93 unique dashboards for the state and all 92 counties.  While the ELAC dashboards are currently only static reports, there is the option and feature (with Tableau) of making them interactive like the Indiana Commission of Higher Education’s College Readiness Dashboard. One of our good friends, the M.A. Rooney Foundation, has also been working to transpose K-12 data for schools and community partners into meaningful dashboards.
Are you ready to get started in creating a dashboard for your organization? We would love to work with you to help you focus in on the key indicators important for your organization and create a dashboard that informs decision-making! Contact us today for help.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

4 Steps to Complete a Program Evaluation

Program evaluation helps organizations determine if the change that they set to accomplish actually occurred. Change can be knowledge gained, attitude change, or behavior change.  For example, did a literacy tutoring program help the students who were not reading on grade level actually catch up to reading on grade level by the end of the program period?  Or, are the low-income children who participated in a high-quality pre-k program ready for kindergarten?  

At Transform Consulting Group, we are data nerds and love helping organizations develop and implement evaluation plans to assess their change by using a four-step process.
4 eval steps

Program Evaluation in Four Steps:

  1. Establish clear outcomes
  2. Create or modify data tools and systems
  3. Analyze the data
  4. Use data to make informed decisions

The first step is to ensure there are clear outcomes in place that support an organization or program’s goals.  We work to develop SMART outcomes: specific, measurable, achievable, relevant, and timely.  In this step, we typically review or develop a logic model, which is just a technical description of aligning your programs to the change (or outcomes) that you want to accomplish.  For example, a college-readiness program may want to increase the number of students who can (1) identify a major or career they are interested in pursuing after high school; or (2) understand how to apply for college financial aid.  

The second step focuses on having the right data tools and systems in place to measure and report back on the designated outcomes.  Transform Consulting Group helps organizations determine the most appropriate tool(s) to collect, track and monitor accomplishment of the identified outcomes.  We consider organizational capacity (staff time, knowledge and budget) when suggesting data tools. Some examples could include participant surveys, assessments, and student academic records.  We work to have data tools that are valid and reliable and will provide the data necessary to monitor progress.

The third step is to analyze the data once it has been collected, and present the results in an easily-understood format.  Data is measured to determine whether program outcomes were met and the change, if any, that occurred.  This is often the step where organizations get stuck, because they don’t have the staff time or knowledge to complete the analysis. [Stay tuned for a separate post on simple ways to analyze your data!]  We tell our clients that this is the fun part, because we can see if what they set out to accomplish actually occurred!  

Some common research questions that drive many organizations to conduct program evaluations to get answers include:

  • Are program participants being reached as intended?  If yes, why?  If not, we also want to know why.
  • To what extent are desired program changes occurring?  Was there a significant difference or just a small difference?  Is there a specific group that is not being impacted?
  • Is the program worth the resources it costs?  What is the “return on investment” for this program or service?

The fourth step is to discuss the results of the program evaluation and make informed decisions based on what the data tells us.  We will compile a summary report and/or slide deck presentation of the evaluation data for stakeholders internally and externally to review the results and discuss the implications.  Good evaluations often lead to recommendations for improvement such as enhanced professional development, diversified participant recruitment strategies, and/or program model changes.  This is an opportunity to discuss data collected and implications for future programming, including ongoing program evaluation practices within the organization.

In today’s era of accountability, what gets measured gets done. If you don’t measure results, you can’t tell success from failure[1].  Transform Consulting Group equips organizations to celebrate their successes and inform opportunities for growth.  Contact us today for more information on how Transform Consulting Group can help assess the impact of your organization.  
[1] Reinventing Government, Osborne and Gaebler, 1992.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

Using A Needs Assessment Process to Plan a New Program

When a business plans to open a new store or restaurant, the business first conducts market research to ensure a good fit and a successful venture. In the public sector, we call this a “needs assessment”.

Whether your organization is a non-profit, hospital, school or government agency, more than likely you have been asked or initiated a needs assessment.  Why?  The needs assessment is a great opportunity to step back and build understanding about the target population (aka, intended client), their strengths, needs, concerns and goals, as well as taking a look internally at your own organization to ensure a good fit.

In Indianapolis, we have a Center for Working Families (CWF) program model that was developed by the Annie E. Casey Foundation to help low-income families achieve long-term financial stability. A group of local funders and partners help support and coordinate the CWF model to ensure a good fit with communities and strong fidelity at implementation.

Transform Consulting Group has helped a few organizations complete a needs assessment and project implementation plan to implement CWF in Indianapolis.  Most recently, we worked with Shepherd Community Center, a multi-service center on the Near Eastside of Indianapolis. Achieving CWF status would designate Shepherd Community Center as a location where low-income families could access a coordinated or bundled set of three key services to help lift them out of poverty and achieve long-term financial stability. In order to demonstrate the need for CWF in the neighborhood that Shepherd Community Center supports, Transform Consulting Group conducted a community needs assessment within the Near Eastside neighborhood of Indianapolis and completed an internal assessment of the organization.

Completing the needs assessment allowed Transform Consulting Group staff an opportunity to connect with the target audience through in-person focus groups and surveys at key community gatherings as well as meetings with key community partners.  We also gathered key indicators from the Census and other external data sources. The information collected included demographics, income levels, education levels and goals, goals, and perceived gaps in community services. Combined with the stakeholder feedback, we developed a comprehensive profile of the targeted community.   Data was analyzed and key findings shared with Shepherd Community Center leadership.  Results from the needs assessment not only demonstrated a strong need in the community, but will also serve as a baseline of information for future CWF planning and programming within this community.

Understanding the needs of your community or “market” is the foundation for successful organizational programming that will have the greatest, long-lasting impact. Transform Consulting Group looks forward to new opportunities helping organizations like Shepherd Community Center get a pulse on their community needs and effectively provide support. Contact Transform Consulting Group today to learn more about how we can assist your organization.

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

4 Tips for Getting Started with Tableau

Have you ever seen beautiful charts or dashboards that make the data “pop” in the report or presentation and wondered how could you do that?  At Transform Consulting Group, we have made a lot of charts and graphs to help our clients evaluate their programs and understand important information in a way that is easy to digest. We work to find the most efficient ways to assist our clients with the data that they need to make informed, timely decisions. 2016 Percent of Annual Income a family pays for high-quality careOne way to do this is staying current with data analysis and visualization software.

The data visualization software that we are crushing on these days is Tableau. It is essentially an accelerated version of “pivot tables”.  If any of you are familiar with Excel, then you know pivot tables.  A pivot table is a tool that we use to determine the relationship between two or more data points. For example, when we were working with TeenWorks, a college and career readiness program, to see if their students are enrolling and persisting in college, then we might want to dig deeper in understanding who the students are that are not persisting, what schools are they enrolled in, what type of school is it (public or private, 2-year or 4-year), what is their major, and what is their gender and other socioeconomic statistics.

These additional data points help tell the story of what change is occurring and how that could impact the program model, partnership development, target clients, professional development and so many other factors.  Tableau helped our team answer these questions and more to better understand the relationship of our client’s program to its intended outcomes.pg 19


Recently, Transform Consulting Group used Tableau to complete a statewide needs assessment on Indiana’s youngest children ages 0-5 by pulling together data from multiple agencies and partners. The analysis resulted in the Indiana’s Early Learning Advisory Committee’s (ELAC) help
2017 Annual Report. The intended audience for the report are policy makers who do not have a lot of time to read technical reports, but Tableau equipped out team with creating a visually-appealing report that draws attention to the key findings.

These are our top four tips of getting started with Tableau:

  1. Use Tableau support. There are many support options through Tableau. One option is the Tableau Community, which allows users to connect and ask or answer questions for each other. This can be a quick way to find answers to a common problem or question that users have.  For example, we were havProjected Employment Needsing difficulty with one of our state maps and Tableau Community had a solution that we were able to implement.
  2. Another option is to contact a Tableau consultant through Tableau. A consultant can provide customized personal training and guidance, which might be especially helpful for a new staff person using Tableau and/or a special project (like a dashboard). The consultant won’t do the work for you but is available along the way for further questions and guidance as you complete your project.  
  3. Organize your data. Tableau can be picky about how the original data is organized and certain charts require different data formatting. Before getting started, it is helpful to organize your data into one spreadsheet. Transform Consulting Group prefers to use Google Sheets because it allows multiple people to work in a document and view changes real time, but Excel or Numbers could also work.
  4. Work with a Tableau expert.  Your project might be beyond the capacity (time and knowledge) of your current team, so partnering with a group or individual who has used Tableau might be a more efficient and effective solution.

If your organization needs help with analyzing and visualizing your data, contact Transform Consulting Group for a free consultation!

Disclaimer: This is not a sponsored post.  We were not asked by Tableau to write this post.  This is our own opinion.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

5 Ways to Use Your Program Evaluation Data

Your organization or school has spent a significant amount of time, money and resources on collecting, tracking and analyzing important data about your programs and services. In working with our clients, we often find that many are not doing a good job of sharing this information beyond the traditional annual report. Here are some simple, yet effective ways to share the impact and value of your organization with key stakeholders:

  1. Annual Report – At a minimum, you should annually produce a report that summarizes your organization’s impact in the community. However, we strongly encourage you to rethink your traditional annual report. Check out these olds posts (here and here) for some inspiration!
  2. Email – The signature line in your staff’s email is a great communication tool. Think of all of the collective emails that your staff send out daily, weekly and yearly and the potential reach of those emails. Use the signature line to highlight key successes, which can be updated monthly. For example, if you operate a tutoring program this quick line could be added to all staff signature: “85% of participating students in ABC program increased their reading level by 3 months during our five week summer program.”
  3. Social Media – Similar to creating “data sound bites” in your email signature, similar data posts can be created for your organization’s social media pages on Facebook and Twitter. Just make sure to limit your jargon and make the post user-friendly.
  4. Collateral Materials – Too often, organization’s marketing materials focus on the services and programs (what you do) and not the result of those services and programs (aka, your outcomes!). Refresh your marketing materials to include both of these critical items – the programs offered and the impact that these programs have in the community!
  5. Grants and Fundraising – One of the best ways to increase an organization’s revenue and funding is to share the results with your funders via grant proposals, grant reports and fundraising events. The evaluation data can be useful to both highlight the great work you are doing (aka – give us more money to expand our impact) as well as justify the need for more money (aka – we need better staffing, curriculum, etc. to accomplish our goals).

Transform Consulting Group is passionate about helping organizations get clear on their mission and goals as well as have the right tools and systems in place to monitor the accomplishment of your goals. Do you need help evaluating your programs or communicating your impact? Contact us today for a free consultation!

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

New Report on the Status of Young Children in Indiana

Indiana’s Early Learning Advisory Committee (ELAC) has submitted its 2015 report of findings to Governor Mike Pence and the state’s Legislative Council. As previously discussed (in this blog post), ELAC was created in the 2013 legislative session with a mandate to provide a comprehensive progress report by June 30th of each year.

ELAC’s vision focuses on ensuring that children ages birth to 8 years and their families have access to affordable, high-quality early childhood education programs that keep children healthy, safe and learning.

The report includes a Needs Assessment section where the following key items are noted:

  • Indiana has approximately 500,000 children five years old and younger;
  • Nearly half (47%) of Hoosier families with children ages 0-5 are in poverty;
  • The majority (66%) of Indiana’s families are working and require child care;
  • Only 11% of Hoosier children are enrolled in high-quality, program-based care;
  • For a family of three living in poverty, having one child in quality care would cost more than a third of their annual income; and
  • The 2020 projected demand for early childhood education teachers outpaces the current supply.

The report also includes the following important information: 2014 Accomplishments, Recommendations, and an Appendix. In the Appendix, there are several tables with detailed demographic information at the county level.

Transform Consulting Group was honored to partner with ELAC in helping them gather the information and write the annual report.

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

USDA Farm to School Program: New Resources and January Webinar Kick-Off

 

This post is part of Transform Consulting Group’s blog series highlighting federal programs that provide education opportunities and/or youth development services in communities.

Farm to School

In an effort to bring more local and regional foods into school cafeterias, the U.S. Department of Agriculture (USDA) division of Food and Nutrition Services has developed a Farm to School Program. The program incorporates nutrition-based curriculum and provides hands-on learning in activities such as gardening, visiting farms, participating in culinary classes, and working with food-related education in the classroom.

Students have access to more locally grown foods while farmers have the opportunity to educate kids about local foods and agriculture. Since the Farm to School Program encourages schools to buy local, communities benefit from the sales, and schools are provided with fresh food grown nearby.

January Kicks off the USDA Farm to School Webinar Series. This 11-part webinar series will help build farm to school skills and develop new strategies for bringing local foods into the lunchroom. Featured topics will include team building, menu planning, program sustainability, and more! For the full listing of webinars, please visit the website: http://www.fns.usda.gov/farmtoschool/videos-and-webinars.

To find out more about farm to school activities, the USDA distributed a Farm to School Census to 13,133 public schools across the nation. In Indiana, approximately 655 schools with an estimated 402,732 children are participating in farm to school activities. During the 2011-2012 school year, Indiana schools spent an estimated $52,444,060 on school food with $7,224,159 directed locally, meaning $7,224,159 stayed in the local economy!

School districts across the nation spent an estimated $3,006,167,550 on school food in the same year. Of that, $385,771,134 went back to local communities. Click here for more info about the Farm to School Census data. For updates, follow Farm to School on Twitter at #USDAF2S or on Facebook at #farmtoschool.

  • To dive deeper into the data provided by the Farm to School Census, the Farm to School Census Explorer Tool can be used to sort data by school, product, sources for local food, grade level engaged, and more, at both the school district and state level.
  • The Farm to School Planning Toolkit is another resource to help answer questions and provide guidance for schools interested in starting or enhancing a Farm to School Program.

Transform Consulting Group has expertise collecting community data and analyzing it to identify trends, strengthen organizational outcomes, and suggest program improvements. Contact us today for a free consultation!

 

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

Building the Pipeline of STEM Workers

 

STEM-logoAccording to a new Brookings report Still Searching: Job vacancies and STEM Skills by Jonathon Rothwell, the need for STEM (Science, Technology, Engineering, Mathematics) workers exceeds the supply. Utilizing workplace skill and educational requirement data as well as advertising duration from millions of job openings, the report highlights the shortage of U.S. workers with STEM skills and its detrimental effects.

Rothwell explains STEM skills are in relatively high demand, however, in certain metropolitan areas the demand is even greater due to shortage. The report emphasizes the “consequences” such as income inequality, long-term gap in lifetime earnings, and unemployment rates among STEM workers and non-STEM workers. Other consequences include inequality among racial and gender groups.

The report found job openings for STEM positions take longer to fill compared to non-STEM positions due to the lack of qualified candidates. STEM job openings requiring a Ph.D. or other professional degree lasted an average of 50 days, compared to 33 days for non-STEM vacancies. The extended duration of advertising these openings creates high costs for companies, including overworked staff, lower output, and product delays.

As the economy continues to recover and job opportunities increase, skills common to STEM occupations will be even more highly sought after by employers. Because there is a wide variety of STEM jobs—accountants, actuaries, civil engineers, nurses, school psychologists, computer programmers—the skills required for computer, health care, and management jobs will have some variance. The report indicates computer skills are associated with the highest salaries.

Rothwell hopes continued research and reports such as this will help create awareness and eventually lead to the improvement of education and training practices. Labor market data can be very important when evaluating policy. Rothwell suggests without “major changes” in the training of the U.S. workforce, the shortage will continue to increase.

Transform Consulting Group can help you collect and analyze data to evaluate the impact of your organization and develop achievable goals. Contact us today for a free consultation!

 

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

Early Learning Advisory Council (ELAC) 2014 Report to the Governor and Legislative Council

 

On June 30, 2014, Indiana’s Early Learning Advisory Committee (ELAC) released its initial report to the Governor and Legislative Council.  ELAC was created as part of legislation passed during the 2013 session.  ELAC’s vision is to ensure that children ages birth to 8 years and their families have access to high quality early childhood education programs that keep children healthy, safe and learning.

The eight members of ELAC are accompanied by over 120 volunteers serving in 7 ELAC workgroups, addressing specific aspects of this work.  The volunteers represent higher education, the business community, legislative and executive branch participants, not for profit organizations, and state employees – all of whom see enormous potential from wise investments in accessible, affordable, high quality early childhood education.

The report includes the following:

  1. Profiles Indiana’s current situation in early childhood education;
  2. Outlines active steps underway to build on strengths and address gaps; and
  3. Provides recommendations informed by the work to date.

The report highlights the need for child care by two-thirds of Indiana parents, and that 4 out of 10 parents still use “informal care” by neighbors or friends instead of “formal” providers.  The report includes Next Steps and Recommendations, including: addressing additional data needs for ages 0-8; linking that data with longitudinal tracking of academic performance and career readiness; and taking full advantage of appropriate federal grant opportunities to accelerate Indiana’s efforts.

Transform Consulting Group is proud to support the work of ELAC to optimize the prospects of our youngest learners!

Similar to the support work that we have done for ELAC, Transform Consulting Group can help organizations support coalitions, analyze data and develop meaningful reports.  Contact Transform Consulting Group today and we’ll help you “move the needle forward.”

 

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn

It’s Raining Data in April

 

Over the last month, a deluge of data has poured out of the Indiana Department of Education concerning Indiana’s public schools, student performance, and teacher effectiveness. Parents, school administrators, researchers, non-profits, and other interested parties will find this data a great resource for various decision-making circumstances. Below are summaries of some of the new data that has been released this month:

Educator Effectiveness

Since the implementation of Indiana RISE, school corporations have been required to observe, rate, and publicize the effectiveness ratings of its teachers.  All teachers must be scored on scale of “highly effective,” “effective,” “needs improvement,” or “ineffective”; however, it should be noted that not all schools use the same evaluation criteria.  The educator effectiveness data from the 2012-13 school year reports that 87% of Indiana’s teachers scored highly effective or effective.

Four Star Schools

For the 2012-13 Four Star Schools, 311 public schools in the state of Indiana received a 4-star status. To earn a 4-star-status, a school must 1) score within the 25th percentile of ISTEP+ English and ISTEP+ Math of state scores for each grade level tested, 2) score within the 25th percentile of percentage of students passing both ISTEP+ English and ISTEP+ Math, 3) test at least 95% of their students with ISTEP+, 4) score within the 25th percentile of ECA English 10 and ECA Algebra 1 of state scores, 5) score within the 25th percentile of percentage of students passing both ECA English 10 and ECA Algebra 1, and 6) test at least 95% of their students with both ECA exams.

Graduation Rates

The state’s graduation rate for the 2012-13 school year was also released in April.  The Indiana graduation rate in 2012 was 88.38%, up from 86.61% in 2011. Of those graduates, 81.72% of students passed their ECA exam and 6.88% graduated with a waiver.  Little has changed from the previous year, though Superintendent Glenda Ritz claims that the raw data shows an increase in the number of students graduating without a waiver.

IREAD 3

The IREAD 3 scores continue the trend seen in previous years of growing literacy in Indiana. The IREAD 3 is administered in the spring to every third grade student in order to benchmark literacy levels before students enter the fourth grade.  In 2012, 85.58% of third grade public school students passed the IREAD 3. In 2011, the passing rate was 80.07%.

School A-F Accountability

Indiana public schools also received their A-F scores from the State Board of Education for the 2012-2013 school. The rating incorporates student academic growth, graduation rate, and college and career readiness.  Beginning with the 2010-11 academic year, the State Board of Education changed the labels for school categories based on student performance from the terms Exemplary, Commendable, Academic Progress, Academic Watch and Academic Probation to the letter grades (A, B, C, D and F).

Transform Consulting Group works with organizations to use data to improve their programming and services, align to best practices and research and increase funding.  To receive assistance in how your organization incorporates data into decision-making, contact us for a free consultation!

 

 

Share this article:Share on FacebookShare on Google+Tweet about this on TwitterEmail this to someoneShare on LinkedIn