HMH Into Math Full-Year Impact Study 2021SY

This study included teachers from three public schools from one school district in an urban Southwest state that implemented HMH Into Math^® during the 2020–2021SY. This urban school district consists of three schools, serving over 1,200 students in grades K–8. Teachers volunteered to participate in the study in exchange for access to additional professional learning sessions. Only teachers that signed an informed consent form and agreed to participate were included in the study.

During the 2020–2021 school year all teachers implemented HMH Into Math as their core mathematics program at grades 1-8. The school district first implemented the program during the 2019–2020 school year.

All participating teachers were provided with implementation guidelines that outlined the essential program components and logic model of the program, and program specific training to ensure that all teachers had sufficient knowledge and skills to successfully implement the HMH Into Math program

HMH Into Math incorporates whole-class, small-group, and computer-adaptive instruction into clear, purposefully structured lessons, empowering teachers to create a learning community where students can experience success and persevere through mistakes. The HMH Ed platform provides access tools and resources that support whole group instruction, small group learning, and independent practice – with the flexibility to be used in a classroom setting as well as for remote learning.

Across all three schools, a total of 14 teachers agreed to participate in the study during the 2020-2021SY. Across these classrooms, there were a total of 509 students.

This sample consisted of slightly more male students (n = 268, 53%) than female students (n = 241, 47%). A majority of students (92%) were identified as Hispanic, with 6.0% being African American. Sixty-eight percent of students were eligible for free or reduced lunch. Thirty-four percent of the sample were identified as English Language learners and 8% of students were labeled as Special Education.

HMH MATH GROWTH MEASURE

The HMH Growth Measure is a computer-adaptive test (CAT). The test adjusts item difficulty based upon students' responses. The assessment is designed to provide timely insights to support student grade-level placement decisions in fewer than 45 minutes per student, with the assessment consisting of approx. 35 items.

Student performance can be measured in terms of a scaled score. Scaled scores range from G1 to G99 where G represents the Grade Level of the test administered. The scaled score makes it possible to compare individual student performance and to aggregate student performance within grade level classrooms.

The HMH Growth Measure provides benchmark assessments up to three times a year, and it is recommended that districts use the HMH Growth Measure at the beginning of the year (Fall), in the middle of the year (Winter), and at the end of the year (Spring).

NWEA MAP GROWTH

This adaptive assessment creates a personalized assessment experience that accurately measures performance. The level of difficulty of each question is based on the student’s answers on previous questions. The test is untimed, but is typically completed within 45-55 minutes and uses approximately 50 questions to determine a student’s score.

Every question on the assessment is calibrated to the proprietary Rasch Interval Unit (RIT) scale. The scale measures in equal intervals, regardless of a student’s grade or performance—and it remains stable over time. Because the equal-interval scale is continuous across grades, it can be used to track longitudinal growth over a student’s entire career. The RIT scale ranges from 100–350.

This assessment was administered to all students in the district at the beginning and end of the school year.

The teachers completed an online teacher survey at the beginning and end of the study period to collect information on typical math practices, and teacher background. The end of year survey also collected information on typical math practices with Into Math and feedback on the Into Math components and features.

As well, the survey gathered information about teachers’ distance learning practices and the programs usability during distance learning. Teacher surveys were completed by all but one teacher from the participating district (93%).

Study teachers completed a bi-monthly implementation log to report on lessons completed as well as the components and resources utilized. The Implementation Teacher Logs allowed the researchers to monitor fidelity of Into Math usage during the study period. These logs were developed as a Google^® worksheet to provide teachers with easy “anytime” access in a user-friendly format. Online implementation logs were completed by all but one teacher from the participating district (93%).

Teachers were asked to rate the HMH Into Math program’s perceived impact on student learning throughout the school year. Nearly 70% of participating teachers felt that HMH Into Math was a significant contributor of their student’s academic improvement.

Teachers were surveyed on the relative influence HMH Into Math had on student learning exhibited by 21st-century skills and life skills, with a majority of teachers indicating the program had a positive effect on students’ critical thinking/problem solving (84%) and engagement in independent learning (75%).

Teachers also reported that usage of HMH Into Math was associated with a number of effective instructional strategies. All teachers reported that using the program lead to “moderate or considerable” usage of evidence-based practices, such as promoting and supporting procedural fluency and the use formative assessment to inform instruction.

Further, teachers indicated that the use of HMH Into Math was associated with additional uses of several effective classroom practices (see Figure 1).

All teachers began the 2020-2021 school year via remote or distance learning with a couple of exceptions: 2 teachers indicated using a hybrid teaching method at the beginning of the year. By March of 2021, all teachers reported offering in-person learning.

During remote/distance learning, teachers provided an average of 212 minutes per week of total math instruction. During in-person learning, an average of 194 minutes a week was devoted to math instruction.

With regards to module and lessons covered throughout the school year, teachers completed an average of 70% or more of the program during the school year.

Teachers that indicated they used essential program components for most or all lessons on average, were classified as having met minimum implementation expectations for program component usage. All other teachers were classified as not meeting implementation criteria.

Of the fourteen participating teachers, 7 were classified as having met minimum implementation expectations and 7 were classified as having not met minimum expectations

With regards to implementation of the key program components (see Table 2), Build Your Understanding, Step it Out, Check Your Understanding, Module Review and Module Test were used by the most teachers with high fidelity (used for all or most lessons).

Data was available for a total of 114 students who completed both the beginning of year (BOY) and end of year (EOY) HMH Math Growth Measure. Among these students, there were significant gains from the beginning of year (M = 38.17 SD = 13.29) to the end of year (M = 45.12 SD = 13.37),t (113) = --5.847, p < .001).

Figure 2 displays the gains made on HMH Math Growth Measure Scaled Scores from the beginning of the year to the end of the year for each grade level span. Follow-up analyses indicated that there was statistically significant growth (p< .05) at all grade level bands.

Teachers participating in the study were classified as having either met or not met implementation expectations for the study. Student BOY to EOY Math Growth Measure and MAP scores from classes that met implementation criteria were compared to those that did not meet the criteria.

As shown in Figure 3, there is a significant relationship between growth and implementation level on the Math Growth Measure, F(1, 112) = 7.56, p<.05. Students in classes that met the minimum implementation levels grew an average of 11 points over the course of the school year while students in classes that did not meet the implementation criteria only grew an average of 4 points.

Similar results were demonstrated when researchers examined the NWEA MAP Rasch Interval Unit (RIT) scores.

Statistical analyses indicated students made significant gains from the beginning of the year administration (M = 194.39, SD = 23.922) to the end of year administration (M = 197.37, SD = 22.80), t(393) = -5.94, p < .001.

Further analyses of the MAP RIT scores found that students with teachers that met implementation levels witnessed significantly greater gains (4 points) as compared to student gains among those with teachers who did not meet minimum implementation levels (2 points), F (1, 392)= 5.55, p<.05.

When examining the impact of HMH Into Math on students from different subgroups, both assessments yielded positive results. As measured by both the HMH Math Growth Measure and the NWEA MAP scores, students labeled as eligible for Free or Reduced Lunch, English Language Learners, and Special Education²made significant gains over the course of the school year (all ps < .05).

This exploratory impact study of HMH Into Math found that most teachers covered a majority of the program lessons and typically spent around 40 minutes a day engaged in math instruction. Teachers did report spending more time covering math when using distance/remote instructional models when compared to in-person instruction.

Almost all teachers were teaching in distance/remote learning models at the beginning of the year. By the Spring, all teachers were back to teaching full-time in the classroom.

Teachers reported that the usage of the program was associated with the implementation of several effective instructional practices and strategies. Teachers indicated that the utilization of these strategies and technology increased over the course of the school year.

While teachers reported covering a large portion of the program, implementation of the essential core components for most or all of the lessons did vary. Half the teachers implemented the program with minimum fidelity; the other half of the teachers did not meet minimum implementation levels.

Educators should strive to implement the program with greater adherence to the Learning Models and HMH guidelines, as such HMH Into Math instructional practices were associated with greater student growth.

This finding was true for two independent, reliable, and valid standardized mathematics assessments.

This initial research provides encouraging results suggesting that using HMH Into Math is associated with improved student math achievement scores and improved teaching outcomes. More rigorous research is needed to determine the context and conditions in which the program's usage leads to greater increases in student achievement.