System 44: Lincoln Unified School District

Read 180 now incorporates the comprehensive foundational literacy skills scope and sequence from System 44.

Achieving literacy by Grade 3 is correlated with future success on state English Language Arts assessments (CCRSC, 2013). Students who do not meet or exceed benchmark scores on state and national tests are less likely to graduate from high school, are less likely to persist in or successfully complete future academic and workplace training endeavors, and are overall less likely to be on track for future academic and workplace success (CCRSC, 2013). The Lincoln Unified School District identified the need for an intervention to accelerate the district’s lowest performing Grade 3 readers to grade-level reading proficiency.

System 44 is an intervention program designed to build foundational reading and decoding skills for the most challenged readers. This blended learning solution includes daily reading, writing, talking, and critical-thinking exercises. Materials feature modeled and independent reading with age-appropriate books that target decoding skills and strategies to promote comprehension, build vocabulary, and increase content-area knowledge. Using adaptive technology, students complete a personalized learning progression through five strands that include The Code, Word Strategies, Sight Words, Success, and Writing. 

Students in the third grade experienced the Double Period Instructional Model for 90 minutes approximately four days a week in an afterschool setting.

Lincoln Unified School District (LUSD) in Stockton, a mid-sized city in California’s Central Valley, serves a diverse population of students. During the 2017–2018 school year, the majority of the district’s 9,420 K–12^th grade students were Hispanic (48%), 22% were White, 13% were Asian/Filipino, 12% were African American, 3% were multi-race, and 2% were Native American/Native Hawaiian/Pacific Islander. In addition, 15% of LUSD students were classified as English learners (EL), 11% were classified as students with disabilities (SWD) and 63% were eligible to receive free or reduced-price lunch (FRPL) through the National School Lunch Program.

During the 2017–2018 school year, all schools serving third-grade students in the LUSD utilized System 44 as a Tier 3 reading intervention during Reading Academy, an afterschool reading program. Because the class size was limited to 15 students per school, not all students who qualified were invited to participate. Those students who scored in the “Beginning” or “Developing” range on the HMH Phonics Inventory^® qualified to participate in System 44 instruction dependent on a teacher recommendation based on an analysis of factors, such as attendance and behavior issues. Of the 151 students who tested in the eligible range for the intervention, 36 students did not participate either because the teacher did not recommend inclusion or the parent did not decide to allow the student to participate. In an analysis of participants compared to non-participants, students did not differ based on initial HMH Reading Inventory^® Lexile^® score, race, EL status, FRPL eligibility, or disability classification. However, females were more likely to participate than males, with 81% of eligible females and 65% of eligible males participating in the intervention.

The analytic sample includes all students who received System 44 instruction and had a pre- and post-test (n = 108); 7 students without either a Reading Inventory or Phonics Inventory post-test (6%) were excluded from this analysis. Of the students included in this analysis, 39% were male and 61% were female, 82% were eligible to receive free or reduced-price lunch, 32% were classified as ELs, and 26% were classified as Students with Disabilities. Student disabilities included autism (4%), speech or language impairment (32%), specific learning disability (54%), and other health impairment (11%).

LUSD students in third grade were identified as struggling with foundational reading skills based on a pattern of below-grade-level DIBELS^® Next scores, CORE Phonics Survey scores, and Reading Inventory Lexile scores. Students who scored less than 400 Lexile measures (L) on the Reading Inventory completed the Phonics Inventory, which is used to determine eligibility and placement in System 44. Students were invited to participate in the afterschool intervention based on literacy assessment performance, teacher recommendation, and parental consent (detailed above).

Students continued to receive standard core reading instruction in the classroom in addition to System 44 instruction in the afterschool program. Students received 90 minutes of instruction in the afterschool program using the System 44 curriculum approximately four times a week.

LUSD teachers were hired specifically to run the afterschool reading intervention. Teachers met monthly with the System 44 coordinator and had regular follow-up with the curriculum coach as well as early site visits.

Student software usage data was collected as students used the online student application during System 44 instruction. Software usage data included number of completed topics, number of completed sessions, average time spent in each session, and number of sessions averaged per week. 

The HMH Reading Inventory measures reading comprehension proficiency for students in Grades K–12. The Reading Inventory uses adaptive technology to determine a student’s reading comprehension level on the Lexile Framework for Reading; the higher the Lexile score, the more challenging reading material the student can comprehend. Test item difficulty ranges from items appropriate for developing readers to items requiring a reading proficiency indicating preparedness for college level texts, allowing measurement of skill growth regardless of the students’ initial ability. Assessment results include a Lexile scale score that indicates reading ability at a level of text complexity and a performance level of below basic, basic, proficient, or advanced, indicating achieved reading comprehension compared to grade-level expectations. LUSD students receiving System 44 instruction completed the Reading Inventory in the fall of 2017 (August through October) before beginning instruction, again in the winter (November through January), and again in the spring of 2018 (April or May) following instruction.

The HMH Phonics Inventory measures proficiency in the foundational reading skills of phonological decoding and sight word reading for students in Grades 3–12. The Phonics Inventory is used to identify whether students with low reading comprehension achievement also lack the skills needed to decode new words (leading to placement in System 44) or are best served by an intervention to develop reading comprehension strategies, text analysis skills, and background knowledge (leading to placement in other interventions). Assessment results include a fluency score and decoder status of pre-decoder, beginning, developing, or advancing. LUSD students with Reading Inventory scores at 400L or below completed the Phonics Inventory before beginning instruction in System 44, and every couple of months following instruction.

The Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next are one-minute fluency measures for assessing the acquisition of early literacy skills. DIBELS Next scores are predictive of reading proficiency and are used to provide frequent feedback on student progress. Benchmark tests are used to determine which students are at risk for not meeting end-of-school-year reading expectations and can include subtests such as letter-naming fluency (LNF), oral reading fluency (ORF), nonsense word fluency (NWF), and word reading fluency (WRF). Scores indicate number or percentage correct and are compared to the benchmark to indicate if students are on track for meeting school year reading goals. LUSD students completed the DIBELS Next tests three times a year.

As the students participated in the four-day-a-week reading intervention voluntarily and with parental consent, some LUSD students participated in the System 44 intervention for less than a semester (n = 2), some participated for only one semester (n = 20), and other students participated for the entire school year (n = 86). Students who participated in System 44 during the 2017–2018 school year completed an overall average of 18.42 System 44 topics (SD = 12.30) over an average of 23.59 total sessions (SD = 23.98), averaging 14.90 minutes (SD = 4.57) per session for 2.44 (SD = 0.63) sessions per week in the System 44 online student application. 

An independent evaluator from Forge Research Group analyzed student academic achievement using test score data provided by the Lincoln Unified School District and program usage data provided by Houghton Mifflin Harcourt^®. System 44 students’ English Language Arts (ELA) performance was examined pre- and post-implementation using multiple independent measures of reading. This analysis included scores on the Reading Inventory, the Phonics Inventory, and DIBELS Next. System 44 students demonstrated significant gains in ELA proficiency during the 2017–2018 school year on each of these reading measures. Notably, because of the self-paced nature of the program and the variability in participation, individual student gains were inconsistent. As would be expected, student gains varied by level of implementation: students who completed at least 19 topics in the System 44 online student software (Moderate or High Implementation) consistently demonstrated greater gains than students who completed 18 or fewer topics in the System 44 online student software (Minimal Implementation).

System 44 students completed the Reading Inventory as a pre- and post-implementation test; the Reading Inventory is sensitive enough to capture below-grade-level ELA reading comprehension ability and growth not usually captured by other assessments (including negative Lexile scores). System 44 students who completed the Reading Inventory in both the fall of 2017 and the spring of 2018 demonstrated a statistically significant overall gain in Lexile (L) scores, averaging a 346L increase from 12L in fall to 371L in spring (see Figure 1). Further, student gains varied by Minimal, Moderate, and High Implementation, averaging a 330L increase, 322L increase, and 432L increase, respectively, from fall to spring. Of note, students with Minimal and Moderate Implementation averaged a lower, but still significant, increase.

Averaging over all participants, System 44 students demonstrated accelerated Lexile score gains (achieving 125% of the expected score gains) compared to the average annual growth demonstrated by an initially same-scoring national sample (see Figure 2). Student gains varied by Minimal, Moderate, and High Implementation, averaging a 114%, 126%, and 153% larger Lexile score gain, respectively, as would be expected from beginning to exiting instruction. Of note, all students received standard reading instruction during the school day, but those students with standard instruction combined with even minimal participation in the System 44 intervention demonstrated accelerated growth towards grade-level performance.

In addition to Reading Inventory Lexile gains, System 44 students who completed the Reading Inventory in both the fall and spring also achieved increased performance levels (see Figure 3). Overall, the percentage of students achieving a Proficient Reading Inventory performance level increased from 1% to 17% and the percentage of students scoring at a Below Basic performance level decreased from 89% to 29% from 2017 to 2018. The increase in students’ performance levels on the Reading Inventory was statistically significant. Further, the percentage of students with at least Moderate Implementation (Moderate or High) achieving at least a Proficient Reading Inventory performance level increased from 0% to 20%.

System 44 students who completed the Phonics Inventory in both the fall and spring also demonstrated statistically significant overall gains in Phonics Inventory fluency scores (see Figure 4). Students increased from an average of 6.5 on first fall assessment to 10.0 on the spring assessment. Student gains varied by Minimal, Moderate, and High Implementation, averaging a 2.2 score increase, 3.5 score increase, and 5.9 score increase, respectively, from fall to spring. 

In addition to Phonics Inventory fluency score gains, System 44 students also achieved increased fall to winter and fall to spring decoder status on the Phonics Inventory (see Figure 5). The percentage of students achieving a “Developing” decoder status on the Phonics Inventory increased overall from 20% to 47%, while the percentage of students scoring at a “Beginning” decoder status decreased from 77% to 51% from 2017 to 2018. For students with at least Moderate Implementation (Moderate or High), the percentage of students achieving a “Developing” decoder status increased from 16% to 56%.

Summary Growth Metrics (see Figure 6) show the significant extent of reading comprehension growth during the 2017–2018 school year. With just one year of use, the majority of System 44 students achieved a Lexile gain on the Reading Inventory (95%) and 53% met and an additional 14% of students nearly met (achieved 80–99% of) end-of-year growth expectations based on Estimated Average Annual Growth adjusted for the number of days in the program. Additionally, 19% of students gained two times the end-of-year growth goal, and 61% increased at least one level on the Reading Inventory College & Career Performance Levels. Of note, 100% of those students with a High Implementation (completed 31 or more System 44 topics) achieved a Lexile gain on the Reading Inventory, 65% met end-of-year growth expectations, 29% gained two times the end-of-year growth goal, and 71% of students increased at least one level on the Reading Inventory College & Career Performance Levels. 

Further, increased use of the System 44 online software was a statistically significant predictor of reading Lexile score growth and fluency score growth on their Phonics Inventory, both before and after correcting for selection bias (see Appendix Tables 2 and 3 for details). On average, System 44 students who completed more topics in the System 44 online software also achieved higher gains in the Reading Inventory Lexile score (see Figure 7) and the Phonics Inventory fluency score (see Figure 8). Students who completed 31 or more topics in the online software (High Implementation) achieved an average of 1.6 times more Lexile score gain (432L) compared to students who completed 15 or fewer topics (268L) in the online software (Minimal Implementation). Similarly, students who completed 31 or more topics in the online software achieved an average of 4.6 times more Phonics Inventory fluency score gain (5.5) compared to students who completed 15 or fewer topics (1.2). 

Notably, disaggregation of the data indicated that System 44 was associated with significant Reading Inventory Lexile gains for all categories of students during the 2017–2018 school year (see Figure 9). When results were disaggregated by gender, both males and females achieved statistically significant Reading Inventory gains from pre- to post-instruction. Likewise, when results were disaggregated, students with disabilities, students classified as English learners, and students eligible to receive free or reduced-price lunch achieved statistically significant Reading Inventory gains. When results were disaggregated by ethnicity, African American, White, and Hispanic students achieved statistically significant Reading Inventory gains. Although the sample was too small (n = 5) to capture the true significance of gains statistically, Asian students gained an average of 185L on the Reading Inventory, approaching a statistically significant gain.

System 44 students’ DIBELS Next scores showed accelerated growth in Oral Reading Fluency (ORF) compared to third grade LUSD non-System 44 students. From the beginning to end of the 2017­–2018 school year, System 44 students averaged a 70% increase in Words Correct on the ORF test compared to an average 47% increase in Words Correct achieved by non-System 44 students who were also initially below the ORF benchmark (see Figure 10). Non-System 44 students averaged an increase from 96% accuracy to 98% over the school year; while System 44 students began the school year with much lower accuracy (89%), the gap was closed (96% Accuracy) by the end of the school year (see Figure 11). On average, Non-System 44 students slightly exceeded the benchmark throughout the school year; System 44 students began the school year achieving a much lower percentage of the benchmark (51%) but achieved 69% of the benchmark by the end of the school year, demonstrating accelerated growth towards grade-level achievement standards (see Figure 12).

This study demonstrates that using System 44 to provide an afterschool reading intervention is an effective method of increasing literacy achievement for students struggling to attain grade-level ELA proficiency. Multiple independent measures demonstrate that students who received System 44 instruction in an afterschool program made statistically significant improvements in English Language Arts and Literacy achievement.

After one year of instruction, third grade students in the Lincoln Unified School District demonstrated statistically significant increases in Reading Inventory Lexile scores and HMH Phonics Inventory fluency scores. In addition to statistically significant year-to-year gains, students achieved accelerated Reading Inventory growth and improved Lexile scores by approximately 125%, more than would be expected based on the average yearly growth of an initially same-scoring national sample. Students also increased in assessment performance levels: the percentage of students achieving a Proficient Reading Inventory performance level increased from 1% to 17% and the percentage of students achieving a “Developing” decoder status on the Phonics Inventory increased overall from 20% to 47%. Disaggregation of the data by gender, SWD status, FRPL eligibility, ethnicity, and EL classification indicated that use of System 44 was associated with significant gains in Reading Inventory Lexile scores for all categories of students.

Students worked through varying amounts of the self-paced, personalized online instruction components – increased completion of System 44 topics was a significant predictor of reading Lexile gains and phonics fluency score gains during the 2017-2018 school year. Importantly, while students with even Minimal Implementation of System 44 instruction achieved significant gains in phonics fluency and reading comprehension, students with at least Moderate Implementation (completion of 16 or more System 44 topics) consistently achieved higher gains on ELA measures and demonstrated accelerated growth in phonics fluency and reading comprehension.

In a comparison between third-grade LUSD System 44 and non-System 44 students, System 44 students averaged a greater increase in Words Correct on the DIBELS ORF test from beginning to the end of the school year. While initially achieving a lower average percentage of accuracy on the DIBELS ORF test, System 44 students increased accuracy to nearly match end-of-year non-System 44 students’ scores and achieved accelerated growth in meeting the ORF benchmark. 

College and Career Readiness and Success Center (CCRSC). (2013). Predictors of Postsecondary Success. Washington, DC: American Institute for Research.