System 44: Charles Armstrong School

The Charles Armstrong School in Belmont, California, is a private school designed to educate students with language-based learning differences such as dyslexia. The Charles Armstrong School serves approximately 240 second- through eighth-grade students with a range of ethnic backgrounds[1]: African American (2%), Asian (2%), Caucasian(78%), Hispanic (4%), Native American (2%), and students with multiple ethnic backgrounds (12%). Approximately 25% of students receive financial aid.

^{[1] Charles Armstrong School Quick Facts accessed June 2019 www.CharlesArmstrong.org.}

Charles Armstrong School students in Grades 2 through 8 who were struggling with foundational reading skills as indicated by the HMH Phonics Inventory and other measures worked in the software portion of the System 44 intervention. System 44 students completed approximately 20 minutes in the personalized online software between two and five days a week. Students continued to receive standard core reading instruction in the classroom (utilizing the Wilson curriculum and program materials) in addition to System 44 instruction. Teachers met with HMH consultants for an overview of the System 44 program and ongoing instructional coaching sessions while implementing System 44.

All students who completed at least 125 minutes of the System44 software who did not also complete Read 180^®were included in the analysis each year (N = 207).

The demographics of System 44 students varied slightly by year (see Table 1); ethnic backgrounds included African American (3%–7%), Asian (4%–5%), Caucasian (48%–63%), Hispanic (2%–13%), Native American (0%–1%) and students with multiple ethnic backgrounds (9%–15%). Of these students, 46%–53% were male, 47%–53% were female, and 0%–1% were non-binary. All students were classified as students with disabilities (SWD) each year. Specific disability diagnoses included dyslexia, specific learning disorder with impairment in reading, convergence insufficiency, attention deficit hyperactivity disorder (ADHD), unspecified neurodevelopmental disorder, other health impairment, and speech or language impairment.

Software Usage Data

Student software usage data was collected as students used the online System 44 student application. Software usage data included the number of completed topics, the number of completed sessions, the average time spent in each session, and the number of sessions averaged per week.

Phonics Inventory

The HMH Phonics Inventory measures proficiency in the foundational reading skills of phonological decoding and sight-word reading for students in Grades 3 through 12. The Phonics Inventory is used to identify whether students with low reading comprehension achievement also lack the skills needed to decode new words (leading to placement in System 44) or are best served by an intervention to develop reading comprehension strategies, text-analysis skills, and background knowledge (leading to placement in other interventions). Assessment results include an accuracy score (range of 0–60) based on accurately reading sight words and nonsense words, a fluency score (range of 0–60) based on reading accurately as well as quickly, and a decoder status (pre-decoder, beginning, developing, or advancing). Charles Armstrong students completed the Phonics Inventory before beginning instruction in System 44; they completed it again at least twice each school year (fall and winter, fall and spring, or winter and spring).

Reading Inventory

The HMH Reading Inventory measures reading comprehension proficiency for students in Grades K–12. The Reading Inventory
uses adaptive technology to determine a student’s reading comprehension level on the Lexile^® Framework^® for Reading; the higher the Lexile score, the more challenging reading material the student can comprehend. Test item difficulty ranges from items appropriate for developing readers to items requiring a reading proficiency indicating preparedness for college-level texts. This difficulty range allows for measurement of skill growth regardless of the students’ initial ability. Assessment results include a Lexile scale score that indicates reading ability at a level of text complexity and a performance level of below basic, basic, proficient, or advanced, indicating achieved reading comprehension compared to grade-level expectations. System 44 students completed the Reading Inventory at least twice each year (fall and winter, fall and spring, or winter and spring).

Read Naturally Reading Fluency Progress Monitoring

Track My Progress is an online, computer-adaptive test designed to assess math and reading skills aligned to the Common Core State Standards in kindergarten through eighth grades. Track My Progress tests have been shown to be both reliable and valid measures of Common Core State Standard proficiency, normed to a nationally representative U.S. sample. Student progress in meeting skills is tracked as students take four 20-minute tests each year, and teachers have access to reports on specific subject and domain proficiency. Charles Armstrong School student data included a percentile score (range 0–99) for reading comprehension. A percentile score above 41 is considered to reflect grade-level ability.

An independent evaluator from Forge Research Group analyzed student academic achievement using data provided by the Charles Armstrong School and Houghton Mifflin Harcourt. System 44 students’ ELA performance was examined before and after System 44 implementation using multiple independent outcome measures.

Charles Armstrong School teachers began implementation of System 44 at different time points throughout the 2018–2019 school year, from November 2018 to May 2019. The 2019–2020 school year data collection efforts ended after the winter quarter, by January 2020, due to school closures during the coronavirus pandemic. Charles Armstrong School students resumed use of System 44 for a portion of the 2020–2021 school year and then continued use for the entire 2021–2022 school year. As such, the length of student participation in the program varied.

Students who participated in System 44 during the 2018–2019 school year (see Table 2) completed an overall average of 33.0 topics over an average of 33.6 total sessions, averaging 2.2 sessions per week and totaling 607.2 minutes in the System 44 online student application. In the 2019–2020 school year, students completed an overall average of 26.3 topics over an average of 45.0 total sessions, averaging 2.2 sessions per week and totaling 619.7 minutes in the System 44 online student application. In the 2020–2021 school year, students completed an overall average of 21.2 topics over an average of 44.3 total sessions, averaging 2.0 sessions per week and totaling 490.3 minutes in the System 44 online student application. During the 2021–2022 school year, students completed an overall average of 27.8 topics over an average of 54.8 total sessions, averaging 2.1 sessions per week and totaling 606.5 minutes in the System 44 online student application.

Phonics Inventory

Charles Armstrong School System 44 students completed the Phonics Inventory in the fall, winter, and spring of the 2018–2019 school year, the fall and winter of the 2019–2020 school year, and the fall, winter, and spring of the 2020–2021 and 2021–2022 school years. Overall, students who completed the Phonics Inventory at two timeframes within the school year (fall and winter, fall and spring, or winter and spring) demonstrated a statistically significant overall gain in Fluency Scores, averaging increases from 7.6 to 10.9 (2018–2019), from 7.8 to 8.9 (one semester of 2019–2020), from 7.7 to 9.5 (2020–2021), and from 9.9 to 12.5 (2021–2022) from first to last testing in the school year (see Graph 1). Disaggregation of the data indicated that students in both primary and middle school grades, both males and females, achieved statistically significant Phonics Inventory Fluency Score gains (see Appendix Table 1) from pre- to post-System 44 instruction. When results were disaggregated by ethnicity, Caucasian and multiracial students achieved statistically significant Phonics Inventory gains; the samples were too small to capture the true statistical significance of gains of students in other ethnic groups.

When students were disaggregated by specific disability diagnosis (available during the 2021–2022 school year) students with dyslexia and specific learning disorder with impairment in reading demonstrated a statistically significant overall gain in fluency scores, averaging increases from 11.2 to 13.8 and 7.4 to 10.0, respectively (see Graph 2). Although the sample size is too small to capture the true statistical significance of gains, students with ADHD also demonstrated notable fluency score growth (from 6.8 to 8.6).

System 44 students’ Phonics Inventory performance levels also reflected these increases in foundational reading skills (see Graph 3). The percent of students achieving an advancing performance level (indicating readiness to progress to a more complex reading intervention) increased from first to last testing each year: from 0% to 11% in the 2018–2019 school year, from 4% to 6% from fall to winter of the 2019–2020 school year, from 0% to 9% in the 2020–2021 school year, and from 6% to 13% in the 2021–2022 school year. Further, the percent of students scoring at a pre-decoder performance level (indicating little or no foundational reading skills), decreased from first to last testing each year: from 3% to 0% in the 2018–2019 school year, from 4% to 1% from fall to winter of the 2019–2020 school year, from 8% to 5% in the 2020–2021 school year, and from 2% to 1% in the 2021–2022 school year. The increase in students’ overall performance levels on the Phonics Inventory was statistically significant during each full year of System 44 implementation (the 2018–2019, 2020–2021, and 2021–2022 school years).

Notably, increased use of the System 44 online software was a statistically significant predictor of Phonics Inventory Fluency score growth in the 2019–2020 through 2021–2022 school years (when implementation group sizes were sufficient to test the statistical significance of differences in gains between groups). This correlation was evident both before and after correcting for selection bias (see Appendix Table 3 for details). On average, System 44 students who completed more topics in the System 44 online software also achieved higher Phonics Inventory Fluency score gains (see Graph 4).

Reading Inventory

System 44 students completed the Reading Inventory in the fall and winter of the 2019–2020 school year and in the fall, winter, and spring of the 2020–2021 and 2021–2022 school years. Overall, students[1] who completed the Reading Inventory at two timeframes within the school year (fall and winter, fall and spring, or winter and spring) demonstrated a statistically significant overall gain in Lexile (L) scores, averaging a 134L (2019–2020), 114L (2020–2021), and 177L (2021–2022) increase from first to last testing in the school year (see Graph 5). Disaggregation of the data indicated that students in Grades 2–7, both males and females, achieved statistically significant Reading Inventory Lexile gains (see Appendix Table 2) from pre-to post-System 44 instruction. When results were disaggregated by ethnicity, Asian, Caucasian, Hispanic, and multiracial students achieved statistically significant Reading Inventory gains; the samples were too small to capture statistical significance of gains of students in other ethnic groups.

When students were disaggregated by specific disability diagnosis (available during the 2021–2022 school year), students diagnosed with dyslexia, specific learning disorder with impairment in reading, and ADHD all demonstrated a statistically significant overall gain in Lexile score, averaging increases from 507.6 to 680.4, 281.5 to 512.0, and 289.2 to 435.2, respectively (see Graph 6).

In addition to demonstrating statistically significant Lexile score gains, System 44 students demonstrated accelerated Lexile score gains compared to the average annual growth demonstrated by an initially same-scoring national sample[1] (see Graph 7). Notably, students made almost two times the expected gains (133.7L compared to 75.3L) from fall to winter of the 2019–2020 school year and significantly more gains than would be expected (176.7L compared to 153.2L) during the 2021–2022 school year, demonstrating accelerated growth toward grade-level performance. In the 2020–2021 school year, primary grade students made less than expected gains (101.8L compared to 200.0L) while middle school students made significantly more gains than would be expected (149.5 compared to 100.9L).

Reading Inventory Lexile score growth was generally dependent on the extent of System 44 online software use (see Appendix Table 4). On average, System 44 students who completed more topics in the online software also achieved higher Reading Inventory Lexile score gains; students who completed 16 or more topics achieved 1.3–2.1 times more Lexile score gains than students who completed 15 or fewer topics (see Graph 8).

System 44 student Reading Inventory performance levels also reflected increases in reading proficiency (see Graph 9). The percent of students achieving at least a proficient Reading Inventory performance level, indicating an ability to access grade-level curriculum, increased from first to last testing each year: from 0% to 13% from fall to winter of the 2019–2020 school year, from 11% to 30% in the 2020–2021 school year, and from 20% to 39% in the 2021–2022 school year. Further, the percent of students scoring at a below basic performance level decreased from first to last testing each year: from 85% to 72% from fall to winter of the 2019–2020 school year, from 66% to 55% in the 2020–2021 school year, and from 62% to 41% in the 2021–2022 school year. The increase in students’ overall performance levels on the Reading Inventory was statistically significant in each year of this analysis.

Of note, from fall to winter of the 2019–2020 school year, 67% of students met or exceeded average Reading Inventory growth, with 36% of students increasing one grade level, 13% of students increasing two grade levels, and 8% of students increasing three or more grade levels in reading proficiency (see Table 3). In the 2020–2021 school year, 40% of students met or exceeded average Reading Inventory growth, with 38% of students increasing one grade level, 10% of students increasing two grade levels, and 9% of students increasing three or more grade levels in reading proficiency. In the 2021–2022 school year, 62% of students met or exceeded average Reading Inventory growth, with 37% of students increasing one grade level, 19% of students increasing two grade levels, and 14% of students increasing three or more grade levels in reading proficiency.

Read Naturally Reading Fluency

System 44 students demonstrated statistically significant growth in oral reading fluency (ORF) on the Read Naturally Reading Fluency Progress Monitor (see Graph 10). System 44 students averaged a 37% increase (from 44.9 to 82.0) in wcpm from the fall to winter of the 2018–2019 school year, a 13% increase (from 52.9 to 66.1) in wcpm from first to last test of the 2019–2020 school year, and a 19% increase (from 54.5 to 73.2) in wcpm from the first to last test of the 2021–2022 school year.

Track My Progress

During a partial-year System 44 implementation, students scored better than 34% of their peers in quarter 1 and 35% of their peers in quarter 2 of the 2019–2020 school year (see Graph 11). After a full year of System 44 instruction, Charles Armstrong School students showed accelerated growth in Track My Progress reading comprehension percentile scores compared to their initially same-scoring peers. On average, students scored better than 31% of their peers during the first testing but better than 34% of their peers at the last testing of the 2021–2022 school year, indicating accelerated growth compared to a national sample. At the end of this full-year System 44 implementation, 30.7% (31/101) of students tested at the 41st percentile or greater, indicating grade-level reading comprehension proficiency.

The Charles Armstrong School assigned students who were struggling with foundational reading skills to participate in the System 44 reading intervention during the 2018–2019 through 2021–2022 school years, with full-year implementation beginning in the 2021–2022 school year. In a correlational analysis across a four-year time span, Charles Armstrong School students in Grades 2–8 who completed topics in the System 44 online software demonstrated statistically significant increases in multiple independent ELA measures.

System 44 students at the Charles Armstrong School demonstrated yearly statistically significant increases in Phonics Inventory fluency scores. Disaggregation of the data by student category indicated that the use of System 44 was associated with significant gains for both males and females, for students in both primary and middle school grades, and for students with a specific disability diagnosis of dyslexia or specific learning disorder with impairment in reading. Additionally, 6%–13% of System 44 students achieved an advancing performance level each year, indicating readiness to advance to more complex reading interventions.

System 44 students also demonstrated yearly statistically significant increases in Reading Inventory Lexile scores and met or exceeded expected Lexile score growth during the 2019–2020 and 2021–2022 school years. Disaggregation of the data indicated that the use of System 44 was associated with significant gains in Reading Inventory Lexile scores for each student category of gender, grade level, and disability diagnosis of dyslexia, specific learning disorder with impairment in reading, and ADHD. Students also increased in assessment performance status: 13%, 30%, and 39% of students achieved at least a grade-level proficient performance status on the Reading Inventory in years 2, 3, and 4, respectively, indicating an ability to access grade-level curriculum. Importantly, 34% of System 44 students increased at least two grade levels in reading comprehension after one full year of instruction in the 2021–2022 school year.

Students worked through varying amounts of the self-paced online software. A higher number of System 44 topics completed in the school year predicted greater Phonics Inventory Fluency score growth. Additionally, System 44 students who completed 16 or more topics achieved 1.3–2.1 times more Reading Inventory Lexile score growth than students who completed 15 or fewer topics.

In addition to gains measured by HMH metrics, System 44 students showed growth on other measures of reading fluency and comprehension. System 44 students showed statistically significant growth in oral reading fluency words correct per minute as measured by the Read Naturally Reading Fluency Progress Monitor. System 44 students also demonstrated accelerated growth in reading comprehension compared to a national sample as measured by the Track My Progress reading comprehension assessment. After one full year of instruction in the 2021–2022 school year, 31% of students achieved at least a 41st percentile, indicating grade-level reading comprehension proficiency.

This study provides evidence that System 44 is an effective program for accelerating ELA and literacy gains among students with dyslexia and language-based learning differences struggling to achieve grade-level ELA proficiency.