Teach

In this lesson, you will learn about the importance of program evaluation as a source for determining what services to children and families are working well and what aspects of the program are in need of improvement. All staff and families may participate in the program evaluation. Program evaluation is typically completed on an annual basis in order to determine whether the program is effectively meeting its goals. The findings of the program evaluation are typically shared with all of the program’s stakeholders: families, staff, and in some cases the community. In military child and youth programs these reports are also shared with the military leadership or higher headquarters within your Service.

The management team works with staff and families to collect relevant data, analyze it, and use the findings to make changes that will improve overall program quality. They also use this information to create a formal report or description of the program for those outside the program who are interested partners (families, advisory board members, funders, inspection teams, accrediting bodies, etc.). Program leaders will want to use both formative and summative evaluation methods when organizing the program evaluation process.

Formative Evaluation

Formative evaluation is used during the daily operation of the program to examine ongoing processes and to help improve the program. A formative evaluation examines day-to-day successes and deficits. Formative evaluation provides a fast feedback loop to influence program decisions and make necessary changes.

Formative Example: Tori, the T&CS, decides to create a new schedule for herself where she observes two of the toddler classrooms each morning during their snack routine and leaves a brief note summarizing her observation in each teacher’s staff mailbox. Both of the toddler rooms were starting a new routine at snack time and the teachers wanted Tori to observe snack several times to provide feedback about how it was working. At the end of two weeks Tori met with the toddler teachers to discuss her observations and to help them decide if the new routine was working well for the children and staff.

Summative Evaluation

A summative evaluation is typically conducted at the end of a program or after a program has been in existence for some time. The summative evaluation is often shared with those outside the program (advisory board, interagency council, funders) to provide data about the effectiveness of the program.)

Summative Example: Each July, Maria, the Program Manager writes an evaluation report. In it, she summarizes child and youth data, parent event data, child and parent satisfaction data, and the overall program budget report. This report also highlights progress on the program’s goals (e.g., increase the number of volunteers in the after-school program). In many ways, a summative evaluation report may be of interest to anyone affiliated with the program, but is also of importance to individuals outside the program.

Although a summative evaluation report may only be written and shared once per year, the information that is used to create the report is collected across the year at many different points and includes the input of many different stakeholders (e.g., staff, parents, coaches, advisory board members).

Evaluations may be conducted by a staff member who is internal to the program (e.g., T&CS or Program Manager) or the evaluation may be conducted by someone who is external to the program (e.g., paid consultant, university professor with expertise in child care program management, assessor for accreditation, or state licensing representative with expertise in conducting program evaluations). The Program Manager is typically responsible for conducting summative program evaluations, or coordinating and working in collaboration with external program evaluators; however T&CSs may assist at the request of the manager (e.g., to help gather some relevant data).

Planning the Program Evaluation

When planning a program evaluation the main question is "What is the purpose of the evaluation?"

• Do you want to demonstrate why the program needs more child development or school-age program staff? 
• Is the purpose to show children's growth and development?
• Is the evaluation being done to gather feedback from others about the program?

The purpose of the evaluation can be different for different stakeholders and will drive the types of questions the evaluation will answer:

• Are we (the leadership team and the staff) doing what we are saying we're doing?
• What is the program's impact on children and their families?
• Do all families feel welcome and part of the program community?
• Does the program comply with DoD certification requirements?

There are typically three major areas of consideration in Child Care and Youth Program Evaluations:

1. Program quality
2. Program outcomes (for children and youth)
3. Consumer and stakeholder satisfaction with the program

It is important that those participating in an evaluation understand the purpose of the evaluation and how their responses will be reported. Planning and designing an evaluation includes creating a matrix in which the major aspects of the evaluation are organized. If the evaluation is not carefully designed then the results may not be accurate or useful to report. The resources in the Apply section will help you more deeply understand the program evaluation process, including what documentation or data may be most appropriate based on the questions you seek to answer, as well as different classroom- or program-quality assessment tools (e.g., environment rating scales) that can offer additional data about the program.

Data used to answer the evaluation questions may be collected through online surveys, focus groups, one-on-one interviews, use of classroom- or program-quality assessment tools, and document reviews (e.g., teacher's written lesson plans, children's progress reports or assessment data, budget summaries, program supply orders).

Model

Sharing and Using Program Evaluation Data

When doing a formative or summative evaluation it is important to plan how the results will be reported and used. Parents will want to know if their time and attention to responding to survey questions results in higher quality child development and school-age programming for their children. Program leaders must decide the best way to share the evaluation results with those who participated in the data collection. Results may be posted on the program's website, in the newsletter, and in other places where families may access it. Hard copies may be made available to staff and advisory board members. It is important for the program manager to also discuss the program evaluation findings with stakeholders. Discussions may take place during a regularly scheduled staff meeting or advisory board meeting. The use of the evaluation results to improve the program should be clear to the stakeholders. Staff and other relevant stakeholders should be made aware about how progress will be monitored.

Some programs are asked to share program evaluation data during interagency council meetings or with other groups. For example, if the school-age program staff and the Program Manager apply for and receive a special grant from an organization (e.g., the Rotary Club) for a substantial amount of money to enhance the science activities offered during the school-age program, then the staff and Program Manager have a responsibility to evaluate the activities and youth learning and report how the money was spent. Again, choosing the most relevant evaluation questions is critical to planning the evaluation, data collection methods, and timeline. The evaluation report should be written, shared and discussed with the funders.

Observe

Watch the video below to hear how T&CSs and Program Managers use program evaluations to document success and guide changes for future improvement.

Model an Approach to Continuous Quality Improvement

The staff looks to the Program Manager and T&CS for leadership. As the leadership team, they focus on a continuous quality improvement outlook. Programs that are committed to consciously improving their services to children, youth, and families demonstrate an openness to learning new knowledge and skills. They have a clear understanding and commitment to quality. They know what a high-quality program should provide for children, youth, and families. They are aware of what the program is doing well and what could be done better. This awareness is based on observation, data collection, and use of valid and reliable program evaluation tools.

There are many different approaches to continuous quality improvement. The below is an example of a process for childcare programs in the state of Rhode Island. (Source: Exceed Rhode Island’s Early Childhood Commitment. (2015). Exceed tip sheet: Continuous Quality Improvement and The Quality Improvement Plan.)

1. Assess your Program: What are your program’s strengths and weaknesses? How do you know? In other words, what data do you have about your program, child outcomes, family satisfaction, staff qualifications, etc.?
2. Develop and Prioritize Goals: What are your goals for the program?
   1. How did you arrive at these goals? In other words, what is the current state of the program and how do you know this?
   2. What tools, documentation or data did you use to understand how your program currently functions?
3. Determine Action Steps:
   1. What are the specific action items to help address these goals?
   2. What activities or professional development training will help you meet these goals?
4. Access Resources: What resource do you need to achieve your goals?
5. Complete the Continuous Improvement Plan (CIP):
   1. Your Service, licensing agency, or accrediting body, may require a specific format for your continuous improvement plan. The Apply section of this lesson provides a sample of the Continuous Quality Improvement Plan (CQIP) for Early Childhood Environment Rating Scale-3 (ECERS-3) from the Illinois State Board of Education and a sample of the CIP used in Ohio. Although each agency, state or Service may be slightly different in how they want the information formatted or delivered, they are typically seeking the same kinds of information:
   2. Who is responsible for the particular action items listed? For many goals, there may be multiple people responsible, as direct care staff members may be responsible for enacting new practices in their classrooms, while T&CSs are responsible for providing training relative to this new practice and observation and coaching to ensure it is happening appropriately.
   3. What is the timeline for implementation of the specific action items?
6. Evaluate Progress and Revise the Plan as Needed: How will you know your goal was achieved?

You will need to gather information about your progress towards your goal. This may be similar to the information you gathered to identify goals in the first place. For example, if you used the ECERS-3 to set a goal around family engagement, you would also use the ECERS-3 to determine whether you met your goal. Continuous quality improvement entails:

• A commitment to lifelong learning
• Program changes that result in better quality care-giving
• Self-reflection
• Shared leadership
• Embedded job supports

When T&CSs and Program Managers consistently articulate and demonstrate a commitment to continuous quality improvement, they provide a model for all staff to reach for excellence within themselves. The leadership team is always ready to celebrate the program’s successes, but also keeps striving to reach the next goal.