Program Evaluation Design

Photo by Pixabay on Pexels.com

Reading Recovery Program Evaluation

Step 1: Program Context

Describe the focus of the program, size of staff, goals, resources, community demographics, and other details important to conducting a program evaluation.

Reading Recovery is a short-term intervention of one-to-one tutoring designed to help the lowest achieving first graders become confident readers. Individual students receive a half-hour lesson each school day for 12 to 20 weeks with a specially trained Reading Recovery teacher. As soon as students can meet grade-level expectations and demonstrate that they can continue to work independently in the classroom, their lessons are discontinued. Reading recovery was developed in New Zealand and is used in several different countries but for this evaluation, we will focus on the York school board in Ontario, Canada.

Goals

The main goal of the program is for children who have not responded well to classroom instruction in Kindergarten and grade one to make accelerated progress and meet grade level expectations with individual instruction by trained teachers.

The program also aims to minimize costly, long-term remedial classes in upper grades.

Staff

Each school chooses the teachers who will participate in Reading Recovery training. The training is one school year long and includes classwork while also working with Reading Recovery students. Once the training is completed, teachers receive a certificate and enter their field year. During this year, teachers attend at least 6 professional learning sessions and participate in a group of continuing contact teachers for support and peer learning. This year will include teaching, observing and discussing live lessons that are taught behind a one-way mirror. The training is provided by a Teacher Leader who has been a trained Reading Recovery teacher and has had professional development to train teachers.

Reading Recovery teachers need to be able to teach 4 individual 30 minute lessons each day in addition to any other roles they might have. These teachers are often Kindergarten or Primary teachers, Special Education teachers, English Language Learner teachers, literacy coaches, or administrators.

Resources

Implementing Reading Recovery involves resources and costs which include an annual site fee (approx. $3500), developing a teacher training site, Reading Recovery approved books and materials, training teachers, one or more teacher leaders and ensuring teachers and teacher leaders are able to participate in professional learning.

A Principal’s Guide to Reading Recovery in Canada, published in 2018 by the Canadian Institute of Reading Recovery is a full colour, 80 page guide with details for every Principal involved in the implementation of Reading Recovery in their school.

Community Demographics

Teachers are trained to use the Observation Survey of Early Literacy Achievement to determine which children are suited for participation in Reading Recovery. This tool and Canadian norms are used to interpret scores and determine which children are at the most need for an early literacy intervention.

In Canada, Reading Recovery is offered mostly in English but is also offered in some French First Language and French Immersion Schools.

Step 2: Purpose and Evaluation Questions

Describe the purpose of the evaluation and identify specific evaluation questions to help guide the evaluation design.

The purpose of this program evaluation is to determine if the program is successful in improving student literacy proficiency and confidence. Because the school boards work on a strict budget, the program needs to show that results are worth the money and resources required by ensuring underachieving students meet or exceed grade level expectations.

Process Evaluation

The process approach is used to evaluate if the program is being implemented as planned. Was the testing done properly to determine who enters the program? Are the students getting the 30 minutes a day the program requires? Are the teachers properly trained? Is the teacher fulfilling the duties the program requires? Are approved resources being used? Are parents and admin supporting the program requirements?

Impact Evaluation

The impact approach is used to evaluate short term goals. At this point, we could look at things such as attitude, behavior, environmental factors and policy. Have the students’ attitudes changed? Are their reading skills improving? Has it impacted their behavior in the classroom? At this point, teachers would complete another benchmark assessment to determine if the child’s reading level has improved.

Outcome Evaluation

Outcome in this case may be more difficult to evaluate. This approach focuses on long term goals and is not feasible to collect for this program evaluation design.

Step # 3 Construct a program theory: Building on your learning in Module 2, construct a program theory: theory of action and theory of change. Select a format that best communicates the goals and orientation of the program being evaluated.

Program Theory

If the Reading Recovery program is implemented using proper resources and trained staff, then the struggling first grade students will become confident readers and writers in just weeks.

Descriptive Assumptions

Students who struggle to read and write in their early education will continue to have challenges within the education system and on into adulthood. 

Prescriptive Assumptions

Students learn better when given daily individualized instruction based on their needs.

Theory of Change

Goals and Outcomes (Logic Model below)

LOGIC MODEL

Reading Recovery Program – York School Board – Ontario, Canada

Program IntentionsProgram Activities and Resources (Inputs)Short-term OutcomesShort-term Indicators – OutputsLonger-term OutcomesLong Term Indicators-Outputs
Teachers are certified and work collaborativelySeveral days of intensive training on the administration and interpretation of the Observation Survey of Early Literacy Achievement followed by bi-weekly classes   Learn to observe and record student behaviours as they read and write   Learn to use a variety of teaching procedures   Learn to tailor teaching decisions to maximize student learning   Collaboration sessions   Shared information   Schedules organized   Live sessions observed with reflection session following   Teacher Leader visits each teacher-in-training 5 times during the training year  Teacher is certified   Teachers feel confident in implementing the program   Teachers have a clear plan   A schedule is created # of teachers who have completed training   # of program hours scheduled   Teachers completed a exit interview when program is completedTeachers get a deeper understanding and share best practices   Reflection will continue to better the program# of Teachers who become training instructors   The program improves based on feedback (changes are implemented)
Students make accelerator progress in readingOne on One instruction for 30 minutes daily with a trained instructor   Lessons designed to offer students opportunities for problem solving and practice   Lessons include reading familiar books, taking a running record on a book introduced in the previous lesson, working with letters and/or words using magnetic letters, writing a story, assembling a cut up story and reading a new book          Students learn to hear and record sounds and to work with spelling patterns   improvement in Comprehension, independent problem solving and reading fluency   Students use learned strategies    students are able to read increasingly more difficult texts   Reading levels improve based of assessments completed before and after the program   # of students at grade level        Students continue to read at grade level in later years# of students reading at grade level who participated in the program
Students can continue to learn with supportive classroom instruction.One on One instruction for 30 minutes daily with a trained instructor   Reading recovery teacher communicating and collaborating with classroom teacher   Creating an interactive, engaging and fun learning environment based on the student’s interestsStudents are more motivated to read   Reading skills used in class   Students show more confidence   Students require less support in class# of books checked out by students who participated   Curriculum expectations being met   Students participate more in class   Feedback from classroom teacher    Students become leaders   Students advocate for the program   Students teach others# of students in leadership roles   Survey   Students involved in reading clubs, programs, or other reading related groups

Determinants

Students who participate in this program have skill deficiencies in reading and were not able to grasps taught skills in group settings and/or were not ready. Students need individualized teaching daily to accelerate to grasp these skills.

Interventions or Treatments

Daily 30-minute Reading Recovery lessons that are tailored to meet the specific needs of the student and are delivered by specially trained teachers for several weeks.

Step #4: Identify, describe, and rationalize your evaluation approach: Based on your learning in Modules 1 and 2, review your program theory and evaluation questions to identify an evaluation approach. Provide a description of how that approach will operate in your specific program context. Also provide a rationale for why your approach is a good fit with your program context.

For this program evaluation design plan, the focus will be on the implementation and process stages. This program has been in place for many years around the world but data for long term outcomes is not often published in Canada. As suggested by Jennifer Green, this evaluation is an evaluation for learning and use because the purpose is to contribute to the quality of the program. The purpose of this program is to ensure the participants make accelerated progress is reading skills and the quality criteria of an evaluation for learning and use is that it is a meaningful and consequential experience for program participants. (Greene, 2017)

I will use an action Model – a systematic plan for arranging staff, resources, settings, and support organizations in order to reach a target population and deliver intervention services to plan for this evaluation. (Chen, H.-T. 2005) The Reading Recovery Program includes many resource pieces that need to fit together like a puzzle. Trained staff, schedules and resources need to be created and allocated in an organized and effective manner in order for the program to be properly implemented. The program is aimed at specific targets (early years struggling readers) and is meant to be an intervention program to help students be more successful later in life.

The evaluation approach being used is the action model approach because “In the action model are found the bases for answering questions such as the following: What are the crucial elements of the intervention? What kind of organization is needed to deliver the services? Who is best qualified to deliver them? How will implementers be trained? What is the target group? How will the target group be reached?” (Chen, H.-T. 2005) and I believe these questions are important when evaluating the implementation and process stages of a program.

This program evaluation is used to determine if this program is worth the resources that are being used. It’s important to understand the descriptive and prescriptive assumptions made to create and implement this program because “if the action model is based on invalid assumptions and is thus poorly constructed or unrealistic, the program is not likely to meet with success” (Chen, H.-T. 2005)

The action model consists of 6 main elements (Intervention and Service Delivery Protocols, Implementing Organizations, Program Implementers, Associate Organizations/Community Partners, Ecological Context, and Target Population) (Chen, H.-T. 2005) and these are the elements I will be focusing on when creating and evaluating through the action model approach.

5. Identify data collection methods and analysis strategies

Based on the first four components of your evaluation plan, identify specific methods to collect information that will respond to your evaluation questions. Also think about and articulate how you might analyze the data you collect.

The purpose of this program evaluation is to determine if the program is successful in improving student literacy proficiency and confidence. Because the school boards work on a strict budget, the program needs to show that results are worth the money and resources required by ensuring underachieving students meet or exceed grade level expectations.

Process Evaluation

The process approach is used to evaluate if the program is being implemented as planned. Was the testing done properly to determine who enters the program? Are the students getting the 30 minutes a day the program requires? Are the teachers properly trained? Is the teacher fulfilling the duties the program requires? Are approved resources being used? Are parents and admin supporting the program requirements?

Impact Evaluation

The impact approach is used to evaluate short term goals. At this point, we could look at things such as attitude, behavior, environmental factors and policy. Have the students’ attitudes changed? Are their reading skills improving? Has it impacted their behavior in the classroom? At this point, teachers would complete another benchmark assessment to determine if the child’s reading level has improved.

Process Evaluation Questions:

Question to AnswerData Collection MethodsAdditional Notes
Was the testing done properly to determine who enters the program?Reviewing tests completed and interviews with test administratorsSampling Size – One Random Teacher from each school in the board Review 5 Tests from each School
Are the students getting the 30 minutes a day the program requires?Reading recovery teachers record session times to determine # of minutes students receivingCreate a Table with all results from the reading recovery teachers in the board
Are Teachers receiving proper training?Survey, interview with trainer ObservationsObserve one session by each trainer Interview one teacher per school Survery all reading recovery teachers and combine the data to use for the PowerPoint presentation
Is the teacher fulfilling the duties the program requires?Interview with reading recovery teacherInterview one teacher per school  
Are approved resources being used?Checklist of resources, check off which are approvedChecklist from each school
Are parents and admin supporting the program requirements?Survey for parents and adminCombine results to Add to Powerpoint

Impact Evaluation Questions:

Question to AnswerData Collection MethodsAdditional Notes
Are their reading skills improving?Benchmark assessment testing before and after5 samples from each school, creating a growth chart to add to the PowerPoint
Have the students’ attitudes changed?Student and teacher interviews2 students and 1 reading recovery teacher from each school
Has it impacted their behavior in the classroom?Interview with classroom teachers1 classroom teacher per school

Data will be analyzed by starting with interpretation of findings. First, we will use any previous evaluations the board has completed as a starting point. When organizing the data, we will separate it into process and impact data sources as they are different stages of the program and it will be easy to see the changes that could implemented in each stage. The surveys, interviews and benchmark assessments will be presented to stakeholders and program providers. We will look at the data as a team to determine the answers to our questions. We can look at the growth of the students, the implementation of the program and create recommendations for program change, if needed. Charts, graphs and tables will be used to organize the information to ensure it is easy to understand by all involved. As collaboration is important to this evaluation, a PowerPoint Presentation will be created with all of the data collected and compared with a summary of findings at the end to determine areas of weakness and what changes need to be made.

6. Describe approach to enhance evaluation use

Write a statement on how you will ensure that your evaluation will have impact on the program. Stipulate reporting strategies and methods for enhancing evaluation use

Through researching different approaches and the reading recovery program, the best method to approach this evaluation is a collaborative approach with teachers and administrators who plan and implement this program. This program is used throughout the school board and many members collaborate to deliver this program. For this reason, the collaborative approach will ensure the data needed is collected and stakeholders believe in the evaluation enough to implement change where needed. This could include administration, teacher trainers, reading recovery teachers, classroom teachers, students and parents.

By using this approach, the team can also decide on reporting strategies. The evaluation should be usable and involving stakeholders in the process will cause them to be more invested in the information presented.

The evaluation design checklist mentioned that is it important to “make clear to the client and other stakeholders what realistically can be accomplished in the projected evaluation, given the context and relevant constraints, and agree on an appropriate scope for the study” (article link). This will be discussed upfront, so all involved have realistic expectations of the evaluation.

When presenting data, making it easy to read is important. According to Shulha and Cousins, “frame findings in ways that are inherently meaningful to these users” (Shulha and Cousins, 1997). When sharing the findings of the evaluation, using easy to read graphs and charts, celebrating goals achieved through the program and showcasing the areas that should be improved will make the information useful to those who can use the information to better the program.

7. Commitment to Standards of Practice

Write a statement that describes how your evaluation plan adheres to the Standards for Program Evaluation.

Standards for Program Evaluation

Utility Standards:

In this evaluation, stakeholders have been identified and are an important part of the process and their needs have been considered. The evaluation is transparent and clearly describe the program being evaluated. Considerations have been made to ensure data is presented in a way that is easily understood. Collaboration with the team will encourage follow-through by stakeholders as the will have more buy in by being part of the process.

U1 Stakeholder Identification Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed.

U5 Report Clarity Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood.

U7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased.”

Feasibility Standards:

The resources needed to complete the evaluation are minimal and can be obtained easily. As this program was run through a school board, the budget for the evaluation is small and therefore choices were made with costs considered.

F1 Practical Procedures The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained.

F3 Cost Effectiveness The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified.”

Propriety Standards:

In the beginning of this process, the team will meet to ensure that each member understands their obligations and meeting notes will be taken and agreed to by all parties. When communicating with anyone included in the process, they will be treated with dignity and respect at all times. All findings will be included and disclosed to the evaluation team while keeping the privacy of those involved. When sharing student data, names will not be used.

P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of the agreement or formally to renegotiate it.

P4 Human Interactions Evaluators should respect human dignity and worth in their interactions with other persons associated with an evaluation, so that participants are not threatened or harmed.

P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results.

Accuracy Standards:

On going documentation will be completed clearly and accurately throughout the process. Purposes and procedures will be described in detail and shared with those involved. Quantative and qualitative data will be used appropriately and analyzed to be presented to the team. Conclusions will be based on the data, objective and presented in a clear and concise manner.

A1 Program Documentation The program being evaluated should be described and documented clearly and accurately, so that the program is clearly identified.

A3 Described Purposes and Procedures The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

A8 Analysis of Quantitative Information Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A9 Analysis of Qualitative Information Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A10 Justified Conclusions The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can assess them.

Sources:

Chen, H.-T. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage Publications.

Greene, Jennifer.(2017) “Approaches to Evaluation.” YouTube uploaded by Education at Illinois, April 12,  https://www.youtube.com/watch?v=zpwB_nSv6HM

“Reading Recovery in Canada.” Canadian Institute of Reading Recovery, February 5, 2020. http://rrcanada.org/.

Saunders, M. (2012). The use and usability of evaluation outputs: A social practical approach. Evaluation, 18(4), 421-436.

Stufflebeam, D. L. (november 2004). EVALUATION DESIGN CHECKLIST. The Evaluation Center Western Michigan University. doi:www.wmich.edu/evalctr/checklists

The Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair (ed.): The Program evaluation Standards, 2nd edition. Sage Publication, Thousand Oaks, USA, p.23-126 (see http://www.wmich.edu/evalctr/jc/)

“What Is Reading Recovery®?” Pages – Reading Recovery, www.yrdsb.ca/Programs/ReadingRecovery/Pages/default.aspx.