HMH Blended Professional Learning: Research Results, 2018–2019

Working with schools and districts, the Houghton Mifflin Harcourt (HMH) Professional Services team offers tailored professional learning services including: courses, collaborative coaching, reporting and data services, and research-based approaches to instruction. A central aspect of HMH Professional Services is instructional coaching geared to both individual teachers and teams of teachers. HMH offers job-embedded coaching through a student-centered and data-driven coaching model that involves an iterative process of data analysis, goal-setting, planning, implementation, and reflection. HMH coaching is provided in-person, online (via live, synchronous sessions using video and screen sharing), and through a blended approach that combines the two. The HMH Coaching Studio platform was recently introduced to provide additional coaching assistance through asynchronous online interaction.

RMC Research conducted a study to document the implementation and effectiveness of HMH Professional Services coaching in four school districts during the 2018/19 academic year. Coaching focused on assistance to individual teachers implementing Read 180 Universal and System 44/System 44 Next Generation. The study was designed to provide evidence of impact on participating teachers’ knowledge, attitudes, engagement, mindset, instructional practice, and on outcomes for their students. The study also examined implementation factors associated with impact, including the nature and extent of teacher and student program participation and usage. This report presents selected findings from the study¹, focusing on outcomes for teachers in two districts who received two types of coaching: (1) blended coaching (including in-person coaching, synchronous online coaching, and access to the HMH Coaching Studio) and (2) in-person only coaching. These findings focus on the experiences of 17 blended coaching teachers in 16 schools and 30 in-person only coaching teachers in 12 schools. Findings are based on qualitative data collected via interviews, focus groups, and classroom observations and quantitative data including teacher surveys, student achievement and demographic data, and HMH program usage data.

This study focused on the extent to which coaching-enhanced professional learning offered by HMH affects teacher practice and student achievement and whether there are differences in impact based on the nature and extent of coaching.

Findings in this report focus on 17 blended coaching teachers and 30 in-person only coaching teachers and their students. Most teachers were female and White and teacher characteristics were similar among blended coaching and in-person only coaching teachers. Teachers who participated in blended coaching had less teaching experience and less experience with HMH programs, included a slightly higher proportion of African American teachers, and included a slightly lower proportion of teachers who taught special education students. The percentage of students belonging to each race/ethnicity group was similar among students of blended coaching and in-person only coaching teachers with the majority of students identified as Hispanic. A slightly lower proportion of students of blended coaching teachers were identified as English learners (49%) and as eligible for special education services (40%) compared to the in-person only coaching group (59% and 45% respectively).

All blended coaching teachers and 91% of in-person coaching teachers reported using Read 180 Universal. System 44 usage was lower among both conditions with 64% of blended coaching teachers and 62% of in-person coaching teachers reporting using the program. Data for 1,761 students were matched to study teachers; approximately 31% of these students (n = 706) were excluded due to lack of HMH program usage data for 2018/19². A small number of students (7%) who were assigned to teachers who participated in both blended and in-person only coaching were excluded from the sample. The resulting sample included 981 students. Student data were available for 43 of the 47 teachers (93%).

Qualitative data sources included teacher focus groups and interviews, HMH staff interviews, school administrator focus groups and interviews, district staff interviews, classroom observations, and online coaching observations. Teacher engagement data included coaching records, HMH Teacher Central usage, Coaching Studio usage, and teacher survey data. Student data sources include student achievement data, program assessment data³, student demographic data, and student program usage data. Teacher Central data were available for 88% of all teachers and HMH Coaching Studio data were available for 41% of blended coaching teachers. HMH student program data were available for 88% of blended coaching teachers and 97% of in-person only coaching teachers. Presurvey data were available for 59% of blended coaching teachers and 37% of in-person only coaching teachers. Postsurvey participation was low with only six surveys completed by blended coaching teachers (35%) and two by in-person only coaching teachers (7%).

Qualitative data were analyzed using an approach that closely follows methods described by Miles, Huberman, and Salda (2019). Descriptive statistics were used to summarize the nature, extent, and frequency of program use at the teacher and student levels. Descriptive statistics are reported for the study sample and disaggregated by district, grade, and student subgroup for cells with sufficient sample size to maintain respondent confidentiality. Growth was assessed using paired-samples t-tests⁴. When sample sizes permitted, differences between blended coaching and in-person only coaching teachers were assessed using independent-samples t-tests.

Online Coaching Participation

Blended coaching teachers were scheduled to complete six synchronous online coaching sessions in 2018/19. While all teachers participated in at least one session, no teachers completed six sessions based on analysis of coaching records. Teachers completed between one and four coaching sessions with an average of 2.5. Postsurvey data about coaching experiences were available for six blended coaching teachers. These teachers reported having attended between two and four synchronous online coaching sessions and between three and eight in-person coaching sessions. Almost all teachers indicated that their in-person coaching experiences were one-to-one coaching sessions.

Respondents expressed differing perceptions about the extent of district and school support for online coaching. In one district respondents indicated that the support was evident based on the budget given for HMH coaching and additional support provided by the district in the form of instructional leadership specialists who served as onsite coaches. In the other district, district/school staff believed that reading teachers received plentiful instructional support for online coaching and the Read 180 program in general from the district through monthly professional learning activities as well as weekly Professional Learning Communities (PLCs) in which Read 180 teachers met together for planning. However, one member of the district/school staff characterized administrative support as “hit or miss,” saying the support varied greatly from school to school and administrators were not familiar with the content of online coaching unless teachers told them about it. HMH staff agreed, saying some administrators displayed a keen interest in coaching activities while others appeared to have little interest in the coaching work.

Topics Covered in In-Person and Online Coaching

Teachers were asked on the postsurvey to indicate the level of emphasis that 2018/19 in-person and online synchronous coaching sessions and non-HMH professional development placed on 15 topics covering instruction, data, and school culture. Most blended coaching teachers indicated that moderate or great emphasis was placed on the majority of topics. Relative to in-person coaching, online coaching placed slightly greater emphasis on applying learning in the classroom, planning for how to help students meet specified learning targets, strengthening content knowledge, developing content skills, using additional print or digital resources, and planning instruction in response to assessments. In-person coaching placed slightly greater emphasis on lesson modeling, designing or refining assessments, and reviewing progress with individual students. Non-HMH professional development focused less on lesson planning and modeling and more on leadership and school culture topics. An HMH coach indicated that teachers selected goals for the online sessions from a list of options and used student data as a basis for each 30-minute discussion. Topics covered in the online coaching sessions included finding evidence in the text to support ideas; teaching daily writing texts; using organizing questions; utilizing the student app for the appropriate length of time; helping students to demonstrate comprehension by scaffolding instruction; preparing students for assessments; classroom management; data analysis; and systems for having students rotate between sessions. Teacher comments about the content of coaching appear below.

It was an extension of what was happening with the in-person coaching. With the online, it seemed as though the data had already been pulled and analyzed by the person I was working with and she was able to give me some insight as to what the numbers meant and what I should be focusing on . . . she was able to look at each individual student and give me some ideas about what I should be working on with them. —Teacher

We talked about one of my groups where I had to do Read 180 within a 2-hour block instead of a 40-minute period. We talked about guided reading groups and how to make that go best with the students. Our school is really big on reading [and] the students did really well. —Teacher

It was really just about the running of the program. [The online coach] helped me assign things to my kids. I feel like I got more strategies to use. —Teacher

HMH Coaching Studio Usage

Among blended coaching teachers, 41% accessed the HMH Coaching Studio and 57% accessed the platform more than once. On average, teachers logged into the HMH Coaching Studio almost six times during the year. The most frequently used feature of the HMH Coaching Studio was the resource library.

Coaching Quality

Teachers were asked on the postsurvey to rate the quality of coaching sessions. Ratings were positive with all responding blended coaching teachers rating all online sessions and all but one of the in-person sessions positively (good or excellent). Ratings of in-person coaching sessions were similar with all sessions receiving average ratings of good or above.

Extent to Which Coaching Objectives Were Met

Blended coaching teachers were also asked to rate the extent to which objectives were met during each in-person and online coaching session. For four of the six in-person coaching sessions and all of the online coaching sessions, all teachers indicated that objectives were mostly or completely met.

Quality and Utility of Coaching Topics

Respondents were also asked to rate the quality and utility of each of four topics covered in coaching (instruction, instructional content and resources, data topics, and leadership and school culture). Blended coaching teachers rated all topics as being of between good and excellent quality, with the highest ratings given for instructional content and resources and data topics. These teachers also identified all the topics as moderately or very useful with the exception of leadership and school culture (survey ratings also suggest that few teachers addressed this topic in online coaching). Highest utility ratings were provided for instructional content and resources. Teacher comments about the quality and utility of coaching appear below.

The fact that it’s specific to Read 180 is helpful. Most coaching is too general; this is specific to the teachers’ needs. – District or school staff member

I think the coaching has helped [teachers] know what to look for [and] has been a big help to them. They talked about how their individual meetings with students have helped those students do better. – District or school staff member

In terms of knowing the Read 180 program, it has helped tremendously. It has helped teachers to use the data more effectively. – District or school staff member

Relative Utility of Online and In-Person Coaching

Blended coaching teachers were asked to indicate which coaching format they found most useful when targeting the following topics: instruction, instructional content and resources, data topics, and leadership and school culture. A majority of respondents reported that online coaching was similarly or more useful when focused on instructional content and resources, and data topics. Half of respondents reported that coaching focused on instruction and on leadership and culture was more useful when provided in person; the other half indicated that online coaching was similarly or more useful.

Teachers were asked to rate their confidence in their content knowledge, pedagogical content knowledge, teaching beyond grade level, and in responding to difficult questions from students. On average, blended coaching teachers reported being between moderately and very confident in all areas on the presurvey with the highest ratings provided for content knowledge. Ratings of confidence were similar on the postsurvey, with a slight decrease in ratings for teacher ability to teach beyond their grade level. Teachers reported that the blended coaching developed their ability to look at and understand Lexile scores and they use this knowledge to provide more effective individualized instruction to students. They also indicated that the coaching helped them to locate and use the wealth of online Read 180 resources. Teacher quotes about the impact on their content knowledge and attitudes appear below.

Without the coaching, there are things I would not have implemented that . . . allowed students to become successful. I would not have done the data walls without the coaching. [I gained] the ability to look at their Lexile tests and find out what those questions looked like so I could create my own ones that would help out. That definitely helped with their success.—Teacher

We were able to dig in, especially looking into System 44, into phonemes, phonemic awareness, and vocabulary in a different way. I learned how to reach more students with the same lesson but just [am approaching] it in different ways so that more students get more out of it. I’ve learned so much more about just the language and how to help students access it, especially my English learners, and how to make sure they are included right from the start.—Teacher

Teachers were asked to rate their ability to gauge student comprehension of a lesson, craft good questions, adjust lessons to the proper level for individual students, use different assessment strategies, and provide alternative explanations or examples when students are confused. Blended coaching teachers, on average, rated their ability between fair and good for four of the five items and their ability to provide an alternative explanation as good on the presurvey. These teachers rated their ability lowest for using a variety of assessment strategies. Ratings were similar on the pre and postsurvey. Teachers reported implementing much of what was shared through the online coaching. They repeatedly discussed having a much better understanding of the data and using it to change small groupings or conduct conferences with students. Teachers also described possessing greater confidence in using data to guide instruction as well as speak to students about their progress in reading. Teachers also said their expanded grasp of data usage allowed them to differentiate instruction and work in groups more effectively because of the variety of resources introduced to them during online coaching.

Respondents consistently reported that the degree of engagement in online coaching depended upon the commitment of individual teachers. Several school and program staff in one district noted that it was easier for teachers to cancel online sessions than in-person ones since teachers felt less responsibility toward someone who was never in their classrooms and with whom they never interacted in person. Both teachers and HMH staff felt that the online coaching, while generally perceived as helpful, was not as effective as it might have been because of the late start in the academic year and the lack of time teachers experienced due to their many other instructional commitments. In the other district, there was consistent agreement among respondents that teachers who participated in online coaching were highly engaged in the sessions and benefited from repeated contact with a coach, who helped walk them through data and brainstorm approaches to use with students. Teachers said they found the information shared in the coaching sessions to be applicable to their classrooms and felt supported by the coach.

HMH program data were used to examine the number of days that teachers used Teacher Central and the total number of Teacher Central logins during 2018/19. All blended coaching teachers and 83% of in-person coaching teachers accessed HMH’s Teacher Central Platform. On average, blended coaching teachers accessed Teacher Central on 10 more days (M = 57.9) and 19 more times (M = 110.2) than in-person only coaching teachers (M = 47.6 and M = 91.7 respectively). Differences in blended coaching teacher and in-person only coaching teacher usage were not statistically significant (n.s.), t_Days (40) = 0.85 n.s.; t_Logins (40) = 0.45 n.s.

Students of study teachers participated in System 44/System 44 Next Generation, Read 180 Universal or both during 2018/19. A higher percentage of students of blended coaching teachers (10%) than students of in-person only coaching teachers (< 1%) participated in System 44 while a higher percentage of students of blended coaching teachers (40%) than students of in-person only coaching teachers (17%) participated in System 44 Next Generation. In addition, a higher percentage of students of blended coaching teachers (81%) participated in Read 180 Universal than students of in-person only coaching teachers (71%). A small percentage of students in each condition (< 3%) participated in both System 44 and Read 180 Universal, and between 8% and 11% of students participated in both System 44 Next Generation and Read 180 Universal.

System 44 and System 44 Next Generation Usage

HMH program data were used to summarize student engagement with System 44 and System 44 Next Generation during 2018/19, including information about the total number of sessions, total session time, median session time, average sessions per week⁵, total number of topics completed⁶, the total number of topics actually completed by the student, the number of topics fast tracked⁷, and average minutes per topic during the 2018/19 school year. Students of blended coaching teachers engaged with System 44 for an average of 60 sessions and 1,052 minutes with an average session length of 15 minutes. On average, students completed 28 topics during the year and spent 42 minutes completing each topic.

Students of blended coaching teachers who engaged with System 44 Next Generation did so for an average of 35 sessions and 627 minutes during the year with an average session length of 17 minutes. Students of in-person only coaching teachers engaged with System 44 Next Generation for an average of 32 sessions and 469 minutes during the year with an average session length of 15 minutes. Significant differences were observed in the number of minutes t (106) = 2.2, p = .033, and the average session length, t (115) = 2.5, p = .013.

Students of blended coaching teachers completed an average of 17 total topics while students of in-person only coaching teachers completed 14 total topics. Students of blended coaching teachers completed significantly more topics than those of in-person only coaching teachers, t (104) =2.5, p = .015. The average time per topic for students of blended coaching teachers was 45 minutes and the average time per topic for students of in-person only coaching teachers was 44 minutes.

Read 180 Universal Usage

HMH program data were also used to summarize information about the total number of sessions, total session time, average session time, average sessions per week, total segments completed, and the average number of sessions per segment in Read 180 Universal in 2018/19. Students of blended coaching teachers engaged with Read 180 Universal for an average of 50 sessions and students of in-person only coaching teachers engaged with the system for an average of 56 sessions during the year. On average, students of blended coaching teachers engaged with Read 180 Universal for 732 minutes and students of in-person only coaching teachers spent 911 minutes engaging with the system. Students of blended coaching teachers engaged with Read 180 Universal for an average of 38 minutes per session and almost three sessions per week while those of in-person only coaching teachers had an average session length of almost 46 minutes and averaged just under three sessions per week.

Students of blended coaching teachers completed an average of just over three segments while students of in-person only coaching teachers completed an average of just under five segments. On average, students of blended coaching teachers took almost 22 sessions to complete a segment while those of in-person only coaching teachers took just over 17 sessions to complete a segment. Students of in-person only coaching teachers engaged with Read 180 Universal for significantly more sessions (t (737) = -2.6, p < .01), and time (t (702) = -4.6, p < .001), had significantly higher average session time (t (722) = -3.2, p <.01), and number of sessions per week (t (739) = -3.3, p < . 001), and completed more segments (t (625) = -4.8, p < .001) in Read 180 Universal. Students in the blended coaching condition took significantly more sessions to complete segments than students in the in-person only coaching condition (t (683) = -2.9, p < .001). Differences in the time students of blended coaching and in-person only coaching teachers spent in each zone in Read 180 Universal was also examined. In all cases, students of in-person only coaching teachers spent significantly more time in each zone than students of blended coaching teachers.

Embedded System 44 and System 44 Next Generation Assessments

HMH tracks performance of students who participate in System 44 and System 44 Next Generation including Word Score, Spelling Score, Comprehension Score, Decoding Accuracy, Decoding Fluency, and Number of Words Read. Embedded assessment scores were available for at least 77 students of teachers who interacted with System 44 and for at least 70% of students who interacted with System 44 Next Generation. No significant differences were observed for any embedded program assessment scores among students of blended coaching and in-person only coaching teachers.

Embedded Read 180 Assessments

Read 180 Universal tracks student performance within the system through end-of-workshop and interim assessment scores. In addition, the HMH system tracks the number of independent reading assessments students take and pass. The average end-of-workshop and interim assessment scores include all assessments completed during the year. Statistically significant differences in performance were observed for the average end-of-workshop score: Level B, interim assessment scores (Level A and Level B), and the number of independent reading tests passed. Students of in-person only coaching teachers scored an average of 8 points higher on Level B end-of-workshop assessments (M_blended = 59, M_in-person = 67) and 12 points higher on Level B Interim assessments (M_blended = 45, M_in-person = 58), while students of blended coaching teachers scored an average of 12 points higher on the Level A interim assessments (M_blended = 65, M_in-person = 54) and passed one more independent reading test (M_blended = 6, M_in-person = 5), on average.

Phonics Inventory

Students who engaged with System 44 or System 44 Next Generation completed the Phonics Inventory as part of their participation in the program. Table 1 presents the average scores on the Phonics Inventory and each section of the assessment for students of blended coaching teachers and in-person only coaching teachers who had at least two scores that occurred at least 8 weeks apart. These students completed between two and six Phonics Inventory assessments, averaging between two and three assessments per student (M_blended = 2.4; M_in-person = 2.5). Students of blended coaching teachers had an average of 141 days elapse between their first and final Phonics Inventory assessments while those of in-person only coaching teachers had an average of 129 days elapse between their first and final Phonics Inventory assessment. On average, at the first assessment students in both groups scored as Beginning Decoders. At the final assessment, average decoding status increased slightly to between Beginning and Developing Decoders. Almost 75% of students of blended coaching teachers and 72% of students in-person only coaching teachers were classified as pre-decoders or beginning decoders at the first assessment. At the final assessment, just over 66% of students of blended coaching teachers and 57% of students of in-person only coaching teachers were classified as Pre-Decoders or Beginning Decoders.

Linear regression analyses were conducted to assess the impact of being taught by blended coaching teachers on student outcomes and program assessments. All analyses included an indicator for the blended coaching participation, the initial score on the assessment and controlled for covariates including student grade level, gender, race/ethnicity, free or reduced priced meals (FRM)⁸ eligibility, special education status, and English language learner (ELL) status. Three statistically significant relationships were found. After accounting for pretest scores and student covariates, being taught by a blended coaching teacher was associated with an average:

• Increase in Decoding Status of 0.28 levels;
• Increase in Sight Word Fluency score of 1.90 points; and
• Increase in Total Fluency score of 2.76 points.

No effect of blended coaching participation was observed for Sight Word Accuracy, Nonsense Word Accuracy, Nonsense Word Fluency, or Total Accuracy scores (see Table 2).

Reading Inventory

Students who interacted with Read 180 Universal completed the Reading Inventory to assess their reading skills as part of the program. Table 3 presents the average scores on the Reading Inventory and reading level for the first and last completed assessments of students who had at least two valid scores that occurred at least 8 weeks apart. Students of blended coaching teachers completed an average of 2.9 tests and students of in-person only coaching teachers completed an average of 2.7 tests during the year. A significant difference between groups on the number of Reading Inventory assessments completed was observed, t (627) = 2.56, p = .01. Students of blended coaching teachers had an average of 160 days elapse between their first and final Reading Inventory assessments while those of in-person only coaching teachers had an average of 139 days elapse between their first and final Reading Inventory assessment. A significant difference in the time elapsed between first and final Reading Inventory assessments, t (627) = 6.41, p < .001, was observed between groups. On average, at the first assessment students scored as Below Basic in reading. At the final assessment reading level were generally stable for each group. Almost 77% of students of blended coaching teachers were classified as Below Basic in reading at the first assessment while just under 70% of students were classified in this way at the last assessment. Over 85% of students of in-person only coaching teachers were classified as Below Basic in reading at the first assessment while just over 78% were classified in this way at the final assessment.

Linear regression analyses were conducted to assess the impact of teacher participation in blended coaching on student Reading Inventory scores. All analyses included an indicator for blended coaching participation, the initial score/level on the assessment, time elapsed between first and last Reading Inventory assessments, and controlled for student level covariates including indicators for student grade level, gender, race/ethnicity, special education status, and ELL status. No effect of blended coaching participation was observed for student scores on the Reading Inventory after controlling for demographic covariates.

State assessment scores were provided for students in both participating districts. Because this sample included students from two different states, scale scores on the state assessments were standardized using z-scores for analyses. Both states included in the analyses classify their assessment scores into four performance levels with similar descriptions. Level 1 indicates that a student is performing below grade-level standards or not meeting grade-level expectations. Level 2 indicates that a student is performing on grade level but not on track for success or are partially meeting expectations. Level 3 indicates that a student is performing at grade level or proficient, and Level 4 indicates that a student exceeds grade-level expectations or is performing at an advanced level. On average, students in both conditions scored between Level 1 and Level 2 on both the spring 2018 and spring 2019 state ELA assessments. Between 60% and 70% of each group scored at Level 1 on the state ELA assessment each year (see Table 4).

Linear regression analyses were conducted to assess the impact of teacher participation in blended coaching on student state ELA assessment scores. All analyses included an indicator for the blended coaching participation and the initial score/level on the assessment and controlled for student level covariates including indicators for student grade level, gender, race/ethnicity, special education status, and ELL status. No significant impact of being taught by a blended coaching teacher was observed for ELA scores or performance level.

Respondents reported that it was difficult to isolate the impacts of blended versus in-person only coaching. However, district staff noted documented gains in mid-year and end-of-year assessments among students whose teachers had received coaching through HMH and reported that more students were moving toward proficiency. Several teachers also noted substantial gains in Lexile scores that they attributed to the help they received through both in-person and online coaching. In the other district, most respondents believed that the coaching efforts were making a positive change in students’ reading ability and school engagement, and that classes with fewer positive outcomes resulted from teachers not implementing the program with fidelity. Quotes about the impact of coaching on students appear below.

I definitely believe there is a lot of improvement with achievement, and I think that is because I can use the data more effectively. – Teacher

Some of the strategies they taught me helped with student engagement. The videos have been very helpful. My classrooms are very diverse and there are lots of levels in my room. – Teacher

I have gone to some meetings where I have seen significant changes in the reading levels of the students. – District or school staff member

A series of additional regression models was used to examine whether student characteristics influenced the relationship between teacher participation in blended coaching and student outcomes on the Phonics Inventory, Reading Inventory, and state assessments. Specifically, these models included an indicator for blended coaching participation, initial assessment score/level, elapsed time (when significantly different between groups), student demographic indicators and interaction effects for any demographic covariate that was significant in an earlier model. Only one statistically significant and substantively meaningful interaction effect was found suggesting that there was a stronger effect of blended coaching on Phonics Inventory Sight Word Fluency scores for male students.

Student Achievement

A series of regression models was used to assess if level of blended coaching teacher participation in online coaching sessions was related to student outcomes on the Phonics Inventory, Reading Inventory, and state assessments. These models included the pretest assessment and an indicator of the number of online coaching sessions the students’ teacher had completed. These models were run for students of blended coaching teachers (see Table 5).

Extent of teacher participation in online coaching was not significantly related to scores on the Phonics Inventory or to Lexile scores on the Reading Inventory. Teacher participation did, however, have a small positive effect on student reading level scores. For each coaching session a teacher participated in, student reading level increased by an average of .06 points, suggesting that if a teacher engaged with all six planned coaching sessions student reading level scores would increase by over a third of a point, on average.

Extent of teacher participation in online coaching sessions was also significantly related to performance on state ELA assessment scores. For each coaching session a teacher participated in, scale scores on the state ELA assessment increased by .25 standard deviations, suggesting that with only four online coaching sessions student scores can, on average, be increased by a standard deviation. In one district this represents an average change of approximately 17 points for every four online coaching sessions, and in the second district this represents an average change of 15 points for every four online coaching sessions. In the first district, achievement levels span from 22 to 56 scale points suggesting that a change of 17 points would represent anywhere from a change of almost a third of an achievement level to over three-fourths of an achievement level. In the second district, achievement levels span from 30 to 31 scale points suggesting that for every four online coaching sessions a teacher participates in students would, on average, see an increase that is reflective of one half of an achievement level.

A significant positive impact was also observed on ELA performance level with, on average, a .07 increase in performance level for each online coaching session completed.

Teacher Survey Responses

Pearson correlations were used to assess the relationship between the extent of participation in online coaching and teacher ratings of confidence and ability on the postsurvey. No significant relationships between extent of coaching participation and survey responses were observed.

Teacher Central Usage

Pearson correlations were used to assess the relationship between online coaching participation and Teacher Central usage (number of logins). No relationship was observed between the extent of online coaching participation and Teacher Central usage for teachers.

HMH Coaching Studio Usage

Pearson correlations were used to assess the relationship online coaching participation and use of the HMH Coaching Studio. No relationships were observed between online coaching participation and HMH Coaching Studio usage for teachers.

While these findings offer rich information about HMH coaching and program usage and encouraging findings related to teacher and student outcomes, several limitations should be noted. The ability to draw strong conclusions about differences in coaching modalities is limited by relatively small sample sizes and differences in implementation approaches across districts. These findings also reveal substantial variation in teacher engagement in coaching activities and student engagement in HMH programs during 2018/19. For example, no blended coaching teachers participated in six online coaching sessions as anticipated. If participation were more consistent, stronger and more consistent outcomes may have been observed. Further, detecting effects of teacher professional learning participation on student achievement is difficult and additional time (beyond one year) is often needed to see consistent effects.

1. Encourage teachers to take advantage of available online coaching sessions. Most teachers did not participate in their full allotment of online coaching sessions. Given the observed relationships between the extent of participation in online coaching and student outcomes it may be beneficial for teachers to complete all allotted sessions.

2. Have coaches further support teachers in their use of the HMH Coaching Studio. Overall teacher use of the HMH Coaching Studio was low, and teachers did not use many of the available HMH Coaching Studio functions. Having coaches support HMH Coaching Studio usage may allow teachers to more fully use this resource.

3. Encourage teachers to login to Teacher Central regularly to monitor student progress. The rate at which teachers logged into Teacher Central varied widely and some teachers never logged into the system. Additional use of Teacher Central could support teachers to better monitor student progress and reinforce coaching activities.

4. Encourage teachers to have students interact with HMH programming regularly. The online coaching provided by HMH was intended to support classroom usage of HMH programming, however, student program usage varied widely. It may be beneficial for coaches to more closely monitor program usage and encourage teachers to use HMH programming during coaching sessions.

5. Continue to examine online coaching implementation and its utility. These findings suggest that online coaching was considered to be aligned with teacher objectives, and to have high quality and utility. Teacher ratings indicated that it was generally as effective as in-person coaching, suggesting great promise for this delivery approach. Further monitoring of teacher coaching experiences will ensure that needs are met and help to highlight any areas of needed improvement.

Meyer, S., Fredericks, L., & Brown, S. (2019). Study of HMH professional services online coaching interim report of findings from mid-year case study data collection: 2018/19. Denver, CO: RMC Research Corporation.

Meyer, S., Fredericks, L., & Brown, S. (2019). Study of HMH professional services online coaching final report of findings from case study data collection: 2018/19. Denver, CO: RMC Research Corporation.

Miles, M. B., Huberman, A. M., & Salda a, J. (2019). Qualitative data analysis: A methods sourcebook (4th ed.). Thousand Oaks, CA: Sage.

Weston-Sementelli, J. L., & Meyer, S. (2020). Study of HMH professional services online coaching report of findings from quantitative data analysis: 2018/19. Denver, CO: RMC Research Corporation.

1. This report summarizes findings from a study examining HMH Professional Services Coaching in four districts during 2018/19. For complete findings see Meyer, Fredericks, and Brown, 2019; Meyer, Fredericks, and Brown, 2020; and Weston-Sementelli and Meyer, 2020.
2. These students were either users of other HMH programming or had used System 44, System 44 Next Generation, or Read 180 Universal during prior school years.
3. This category includes the Reading Inventory, Phonics Inventory, and embedded program assessments.
4. A paired samples t-test is a statistical measurement used to determine differences between matched responses for a group. It computes the difference between the two ratings or scores for each case, and tests to see if the average difference is significantly different from zero.
5. Average sessions per week was not available for System 44 users.
6. This is the total number of topics a student is credited with completing, it is a sum of topics students actually completed and Fast Tracked topics.
7. Topics can be either fully or partially fast-tracked (skipped) for students based on their performance on the Phonics Inventory or on Fast Track assessments. Performance on these assessments is used to determine which skill students have already mastered. A students’ teacher can also choose to fast track a student in a given topic.
8. FRM is only included in analyses for the Phonics Inventory because only one district provided this information, and this was the only district to have students of both blended coaching and in-person only coaching teachers who had Phonics Inventory results.