The Houston County School District in Houston County, Georgia, is a predominantly urban public school district that served 30,243 K–12 students in the 2021–2022 school year and 30,631 students in the 2022–2023 school year on 37 campuses. The district served students with a range of ethnic backgrounds in the 2021–2022 and 2022–2023 school years^[1]: African American (38%–41%), Asian (2%–3%), Caucasian (38%–42%), Hispanic (9%–11%), Native American (<1%), Pacific Islander (<1%) and students with multiple ethnic backgrounds (6%–7%). Approximately 55–59% of students’ families were economically disadvantaged, 3% were English learners, and 14% were classified as Students with Disabilities.

^{[1] Houston County Data Dashboard}

Houston County School District (HCSD) students in Grades 6–12 completed writing assignments in the Houghton Mifflin Harcourt^® (HMH^®) Writable^® software program throughout the 2021–2022 and 2022–2023 school years. Teachers reported varying amounts of writing instruction each week, from less than 30 minutes to more than 2 hours. Teacher use of Writable during that instruction time also varied greatly, with some teachers using Writable as the primary means of writing instruction and others using Writable for each end of unit assignment or for blocks of writing days. Students utilized the Writable program in conjunction with standard curriculum and teacher-created supplements. The majority of HCSD Grade 6–12 students used Writable in the 2021–2022 (13,668 students submitted 61,604 Writable assignments) and the 2022–2023 (14,078 students submitted 62,031 Writable assignments) school years (see Table 1). The majority of teachers participated in Writable professional development by attending face-to-face coaching, online coaching, follow-up training delivered by the HMH team, and/or using the embedded Writable Live Chat and direct screen shares with the Writable team functions.

All students who made at least one non-zero graded submission in the Writable software and had at least one type of school testing data (see Table 2) were included in the analysis. In the 2021–2022 school year, student ethnic backgrounds included Asian (2%), African American (35%), Caucasian (36%), Hispanic (9%), Native American (<1%), Pacific Islander (<1%), students with multiple ethnic backgrounds (6%), and undisclosed ethnicity (12%). Of these students, 44% were male and 45% were female, 2% were classified as English Learners (EL), 38% were from socioeconomically disadvantaged households (eligible for free or reduced-price lunch), and 12% were classified as Students with Disabilities (SWD). Similarly, in the 2022–2023 school year, student ethnic backgrounds included Asian (3%), African American (41%), Caucasian (40%), Hispanic (10%), Native American (<1%), Pacific Islander (<1%), students with multiple ethnic backgrounds (7%), and undisclosed ethnicity (<1%). Of these students, 49% were male and 51% were female, 2% were classified as English Learners (EL), 39% were from socio-economically disadvantaged households (eligible for free or reduced-price lunch), and 11% were classified as Students with Disabilities (SWD).

Student software usage data was collected as students used the online student application during Writable instruction. Software usage data included number of sessions (times the Write page was accessed), words written, assignments submitted, teacher reviews, peer reviews, self-reviews, and number of revisions. An overall Writable engagement score sums the total number of practice assignments completed, revisions, teacher reviews, peer reviews, and self-reviews.

Teachers utilized the grading rubrics in the Writable software to score student writing submissions on a scale of 0%–100%. Mastery levels corresponded to the following score ranges: below (0%–64%), approaching (65%–74%), meets (75%–89%), and exceeds (90%–100%).

ACCESS for ELLs, a part of the WIDA^® suite of English language development tests, was designed to annually assess the progress of EL students towards English language proficiency. ACCESS is aligned to WIDA English language development standards and assesses progress in the four domains of listening, speaking, reading, and writing English. Assessment results include an overall scale score comparable across grades (from 100-600) and a grade-specific performance level for each domain (Entering, Emerging, Developing, Expanding, Bridging, and Reaching) as well as an overall composite score (35% Reading, 35% Writing, 15% Listening, and 15% Speaking) and related overall proficiency level. Proficiency levels of Bridging and Reaching indicate grade-level proficiency. HCSD EL students complete the ACCESS assessment in the spring of each school year.

The Georgia Milestones Assessment System is an end-of-year summative assessment program designed to measure student progress in achieving state-adopted content standards. Students in Grades 3 through 8 complete a computer-based English Language Arts (ELA) assessment. High school students complete an End-of-Course (EOC) assessment upon completion of the American Literature and Composition course. Assessment results include a scale score that corresponds to estimated norm-referenced performance ranges based on grade-level cutoff scores (beginning learner, developing learner, proficient learner, and distinguished learner). HCSD students complete the Georgia Milestones ELA and EOC assessments in the spring of each school year.

Measures of Academic Progress (MAP^®) Reading is a computer-adaptive interim (fall, winter, and spring) assessment developed by the Northwest Evaluation Association that measures reading comprehension. The primary score produced is the Rasch Unit (RIT) scale, which is a stable equal-interval vertical scale. This grade-independent RIT score indicates the level of question difficulty a given student can answer correctly about 50% of the time. National percentile ranks for the MAP are also provided. HCSD students completed the MAP Reading assessment in fall and spring each year.

An independent evaluator from Forge Research Group analyzed student academic achievement using data provided by the Houston County School District and Houghton Mifflin Harcourt. Writable students’ ELA performance was examined pre- and post-Writable usage using multiple independent outcome measures.

In the 2021–2022 school year, students who submitted an assignment in Writable (see Table 3) accessed the Write page an average of 22 times (sessions). Writable students wrote an average of 642 words while submitting an average of 5 assignments (including re-submissions). Students received an average of 1 teacher review per assignment. Demonstrating the same pattern of use in the 2022–2023 school year, students who submitted an assignment in Writable accessed the Write page an average of 21 times (sessions). Writable students wrote an average of 668 words while submitting an average of 5 assignments (including re-submissions). Students received an average of 1 teacher review per assignment.

In the 2021–2022 school year, the majority of Writable students (64%) accessed the program to complete writing assessments, while 28% also completed one practice assignment and 9% completed two or more practice assignments (see Figure 1). Very few students received at least one peer review (4%), while more than half (58%) completed at least one self-review. Nearly all students (98%) received one teacher review per graded assignment. The majority of Writable students (59%) did not submit any assignment revisions, while 27% submitted one revision, and 14% completed two or more revisions. In the 2022–2023 school year, the majority of Writable students (67%) again accessed the program to complete writing assessments, while 19% also completed one practice assignment and 14% (an increase from 9% in the 2021–2022 school year) completed two or more practice assignments. Slightly more students received at least one peer review (8% compared to 5% in the 2021–2022 school year), and more than half (60%) again completed at least one self-review. Nearly all students (99%) received one teacher review per graded assignment. The majority of Writable students (60%) did not submit any assignment revisions, while 27% submitted one revision, and 13% completed two or more revisions. Overall, Writable students averaged similar levels of overall engagement with revision and practice components in both school years (M = 6.71 and 7.06, respectively).

ELA performance was analyzed utilizing two assessments that measure writing skills directly (Writable Proficiency and ACCESS), one assessment whose score reflects a combination of 78% reading skills and 22% writing skills (Georgia Milestones), and one assessment that measures reading comprehension skills (MAP Reading). Based on literature summarizing the results of effective writing interventions for secondary students that explicitly teach writing technique or integrate writing with reading, effect sizes ranging from .10 to 1.32 are expected following substantial writing progress^[2]. 
 

^{[2] Based on the average magnitude of change (mean effect size=.23 [weighted by study sample size]) across 10 writing interventions that meet What Works Clearinghouse rigorous group design standards (total N=12,860)}

HCSD Writable students completed a variety of writing assignments throughout the 2021–2023 school years, including argumentative essays, nonfiction narratives, informative or research essays, letters or emails, literary analyses, analytical essays, personal narratives, journal or quick write assignments, and unit tests. Overall, students who submitted a Writable assignment in both the fall and spring demonstrated a statistically significant overall teacher-assigned proficiency score gain, averaging an increase of 4 percentage points from first to last submission in the 2021–2022 school year, effect size of Cohen’s d = .21, and an increase of 2 percentage points from first to last submission in the 2022–2023 school year, effect size of Cohen’s d = .14 (see Figure 2). As would be expected with sustained writing skill improvement, Writable students demonstrated a larger fall to spring proficiency gain in the first year of program use and began the next school year with a higher average proficiency score. Disaggregation of the data indicated that students in each ethnic group, students with a disability designation, socioeconomically disadvantaged students, EL students, and both males and females achieved statistically significant overall proficiency score gains from pre- to post-Writable instruction (see Appendix Table 1 and 2).

Notably, Writable students’ mastery levels also significantly increased during the 2021–2023 school years (see Figure 3). Overall, students who submitted a Writable assignment in both the fall and spring demonstrated a statistically significant mastery level gain in both the 2021–2022 school year, effect size of Cohen’s d = .26, and the 2022–2023 school year, effect size Cohen’s d = .15. The overall percentage of students who met or exceeded grade-level standards increased from 70% to 82% in the 2021–2022 school year and increased from 78% to 81% in the 2022–2023 school year. The percentage of students who were below grade-level standards decreased from 14% to 8% in the 2021–2022 school year and from 10% to 9% in the 2022–2023 school year. Of students who completed at least two assignments both school years, 42.1% of students increased at least one proficiency level from fall 2021 to spring 2023.

Writable students’ fall to spring proficiency score gains were statistically significantly related to their use of Writable practice and revision components, with and without correction for selection bias (see Appendix Table 2) in the 2021–2022 school year. There was not a linear progression in total component usage, as the majority of students did not use each component, and equal groups of students at varying levels of overall engagement could not be examined. Examining the growth trends overall, in both years, students who engaged with Writable  demonstrated overall proficiency score and mastery level gains (see Figure 4). Of note, students with a very high overall Writable engagement score (a summation of the total number of practice assignments completed, revisions, teacher reviews, peer reviews, and self-reviews) demonstrated a very large fall to spring proficiency score growth (increase of 8.1%) in the 2021–2022 school year. Specifically, students who completed at least one practice assignment or peer review demonstrated statistically significantly higher proficiency score growth as compared to Writable students who did not, however, this pattern did not repeat in the 2022–2023 school year.

Writable EL students completed the ACCESS in spring of 2021, before Writable implementation, in spring 2022, and in spring 2023. On average, students who took the assessment in two years (see Figure 5) demonstrated statistically significant year-to-year increases in overall composite scale scores, from 347 to 363 in the 2021–2022 school year (effect size Cohen’s d = .72), and from 346 to 358 in the 2022–2023 school year (effect size Cohen’s d = .58). Likewise, Writable EL students demonstrated statistically significant year-to-year increases in writing domain scores, from 336 to 350 in the 2021–2022 school year (effect size Cohen’s d = .39), and from 331 to 344 in the 2022–2023 school year (effect size Cohen’s d = .41). Of note, EL students who used Writable both school years (N = 131) averaged a larger 21.1-point ACCESS overall composite score gain and an 18.5-point writing domain score gain from fall 2021 to spring 2023 (Cohen’s d = .84 and .46, respectively), demonstrating continued growth with continued Writable usage. Disaggregation of the data indicated that students in each Grade 6–11, students in each ethnic group, students with a disability designation, socioeconomically disadvantaged students, and both males and females achieved statistically significant ACCESS scale score gains from pre- to post-Writable instruction.

Writable EL students demonstrated a statistically significant increase in ACCESS overall proficiency levels (see Figure 6) from spring 2021 to spring 2022 (effect size Cohen’s d = .44) and from spring 2022 to spring 2023 (effect size Cohen’s d = .23). Likewise, Writable EL students demonstrated a statistically significant increase in ACCESS writing proficiency levels from spring 2021 to spring 2022 (effect size Cohen’s d = .69) and from spring 2022 to spring 2023 (effect size Cohen’s d = .19). Of students who took the assessment in two years, the percentage who achieved an overall weighted proficiency level of 4 (expanding) or higher increased from 20% in spring 2021 to 34% in spring 2022 and from 18% in spring 2022 to 24% in spring 2023. Notably, the percentage who achieved a writing proficiency level of 4 or higher increased from 16% in spring 2021 to 28% in spring 2022 and from 13% in spring 2022 to 18% in spring 2023. The percentage who demonstrated an entering (level 1) writing performance level decreased from 28% in spring 2021 to 3% in spring 2022 and from 14% in spring 2022 to 9% in spring 2023. Of note, EL students who used Writable both school years (N = 131) demonstrated gains of significant magnitude in both overall and writing domain proficiency levels from spring 2021 to spring 2023 (effect size Cohen’s d = .31 and .55, respectively). 

Writable EL students’ large year-to-year increase in writing proficiency level contrasts simultaneous decreases in statewide EL students’ writing proficiency levels (see Figure 7). The percentage of statewide EL students who demonstrated at least a 3 (developing) proficiency level decreased from 24% in spring 2021 to 14% in spring 2022 and decreased further to 13% in spring 2023. During this same time period, the percentage of Writable EL students who demonstrated at least a 3 (developing) proficiency level increased from 20% in spring 2021 to 34% in spring 2022 and from 13% in spring 2022 to 18% in spring 2023.

Writable EL students’ year-to-year growth in writing domain scale scores was statistically significantly related to their overall use of Writable practice and revision components (see Appendix Table 4), with and without correction for selection bias. Students with a high overall Writable engagement score demonstrated 4.8 times more year-to-year writing scale score growth (28 points, effect size Cohen’s d = .87) compared to students with low overall Writable engagement (6 points, effect size Cohen’s d = .15) from spring 2021 to spring 2022. Similarly, students with a high overall Writable engagement score demonstrated 1.2 times more year-to-year writing scale score growth (15 points, effect size Cohen’s d = .38) compared to students with low overall Writable engagement (12 points, effect size Cohen’s d = .37) from spring 2022 to spring 2023. Over the course of two years from spring 2021 to spring 2023, students with a high overall Writable engagement score demonstrated 3.4 times more writing scale score growth (26 points, effect size Cohen’s d = .62) compared to students with low overall Writable engagement (8 points, effect size Cohen’s d = .20).

Statistically significantly more ACCESS writing score growth was achieved by Writable EL students who engaged with the practice assignments, peer reviews, and self-reviews as compared to Writable EL students who did not (see Figure 9). Further, EL students demonstrated writing score gains with use of all of the revision and practice components. From spring 2021 to spring 2022, large writing scale score gains were achieved by students who completed practice assignments (25 points, Cohen’s d = .73), revisions (19 points, Cohen’s d = .45), peer reviews (33 points, Cohen’s d = 1.20), and self-reviews (19 points, Cohen’s d = .57). Likewise, from spring 2022 to spring 2023, large writing scale score gains were achieved by students who completed practice assignments (18 points, Cohen’s d = .54), revisions (14 points, Cohen’s d = .36), peer reviews (45 points, Cohen’s d = 2.13), and self-reviews (19 points, Cohen’s d = .53).

HCSD Writable students completed the Georgia Milestones ELA assessment in spring 2021, 2022, and 2023. HCSD students averaged a scale score markedly higher than the state average in each year. Overall, Writable students’ consecutive scale scores decreased from 522 in spring 2021 to 520 in spring 2022 and increased from 519 in spring 2022 to 522 in spring 2023 (see Figure 10). In both years, students’ scores decreased from Grade 6 to Grade 7 and increased from Grade 7 to Grade 8 testing.

Despite the year-to-year decrease in Grade 6 to Grade 7 ELA scale scores, which is common when initial scores are above average, Writable students achieved scores greater than the state average. In contrast, the year-to-year increase in Grade 7 to Grade 8 ELA scores (see Figure 11) from spring 2022 to spring 2023 is notable during a time period of statewide maintenance in ELA scale scores and for students whose initial scores were above average.

Overall, from spring 2021 to spring 2023, (see Figure 12), the percentage of students who attained a proficient or above performance level remained fairly stable (with a slight nonsignificant decrease from 50% to 48%). Mirroring statewide decreases, the percentage of HCSD Grade 7 Writable students that attained a proficient or above performance level decreased (from 51% to 46%). In contrast, there was a statistically significant decrease (from 20% to 14%) in the percentage of Grade 8 Writable students who attained a beginning performance level, with a higher percentage of students attaining a developing performance level (increase from 32% to 36%) or a proficient or above performance level (slight increase from 48% to 50%). 

The Georgia Milestones ELA scale score is calculated based on a combination of questions measuring reading skills (78%) and writing skills (22%) in the form of argumentative/informative extended response questions. Writable students’ year-to-year growth in Georgia Milestones ELA scale scores was somewhat related to their use of Writable practice and revision components (see Figure 13). Statistically significantly more GA Milestones ELA score growth was achieved by Writable EL students who completed peer reviews as compared to Writable EL students who did not (see Figure 13). Notable differences in Georgia Milestones ELA scores from spring 2021 to spring 2022 were demonstrated between students who completed revisions (decrease of  -.4 points) and those who did not (decrease of -2.3 points) and differences were seen again from spring 2022 to spring 2023 between students who completed revisions (increase of 3.0 points, Cohen’s d = .08) and those who did not (increase of 2.4 points, Cohen’s d = .09). 

Likewise, notable differences in Georgia Milestones ELA scores from spring 2021 to spring 2022 were demonstrated between students who completed peer reviews (decrease of -.5 points) and those who did not (decrease of -1.6 points) and differences were seen again from spring 2022 to spring 2023 between students who completed peer reviews (increase of 4.8 points, Cohen’s d = .15) and those who did not (increase of 2.4 points, Cohen’s d = .08). Notable differences (p < .05) in Georgia Milestones ELA scores from spring 2021 to spring 2022 were also demonstrated between students who received multiple teacher reviews per submitted assignment (increase of 8.1 points) and those who received only one teacher review per submitted assignment (decrease of 1.8 points) and differences (p = .09) were seen again from spring 2022 to spring 2023 between students who received multiple teacher reviews per submitted assignment (increase of 9.2 points, Cohen’s d = .33) and those who received only one teacher review per submitted assignment (increase of 2.6 points, Cohen’s d = .08).

HCSD high school students complete the American Literature and Composition End-of-Course (EOC) test as a final exam for the course. HCSD Writable students averaged a scale score of 509 in the spring of 2022 and 499 in the spring of 2023; statewide, high school students averaged similar scale scores of 511 in spring 2022 and 507 in spring 2023 (see Figure 14). For Writable students, overall Writable engagement score was positively correlated with the EOC scale scores in both the spring 2022 (r = 0.06, p = .03) and spring 2023 (r = 0.12, p < .001) testing. 

HCSD students in Grades 6 to 10 take the Measurement of Academic Progress (MAP) Reading assessment in the fall and spring. MAP assessments are aligned to the Common Core State Standards (CCSS) and measure student growth in reading comprehension skills. HCSD Writable students demonstrated a statistically significant overall increase in MAP Reading RIT Scale Scores, averaging a 4-point gain in the 2021–2022 school year with Cohen’s d = .56 and a 6-point gain in the 2022–2023 school year with Cohen’s d = .67. (see Figure 15). Disaggregation of the data indicated that students in Grades 6–10, students in each ethnic group, students with a disability designation, socioeconomically disadvantaged students, EL students, and both males and females achieved statistically significant MAP Reading RIT Scale Score gains from pre- to post-Writable instruction.

Notably, HCSD Writable students exceeded national MAP Reading fall to spring growth averages based on grade level (see Figure 16). Writable students achieved an overall average 4.4-point gain in the 2021–2022 school year and a 5.5-point gain in the 2022–2023 school year, demonstrating statistically significantly greater growth than would be expected (3.5 points) based on the average growth of a national sample. Though each grade level exceeded expectations, Grade 8 students demonstrated nearly 1.4 times the national average growth in the 2021–2022 school year. Further, all grades demonstrated notable differences in expected to actual growth in the 2022–2023 school year, demonstrating from 1.4 to 2.2 times the national average growth.

Though research supports the idea that writing instruction improves reading outcomes, the expected impact is small, and less impactful than writing instruction on writing outcomes (Graham & Hebert, 2011). Writable students at all levels of Writable engagement demonstrated greater than average growth in fall to spring MAP Reading RIT scale scores. This growth was statistically significantly related to use of specific Writable components in the 2022–2023 school year (see Figure 17). Notable differences in MAP Reading RIT scale scores from spring 2022 to spring 2023 were demonstrated between students who completed practice assignments (increase of 6.4 points, Cohen’s d = .74) and those who did not (increase of 5.1 points, Cohen’s d = .64) and between students who completed peer reviews (increase of 6.4 points, Cohen’s d = .89) and those who did not (increase of 5.4 points, Cohen’s d = .65).  

The Houston County School District (HCSD) adopted the HMH Writable software in the 2021–2022 and 2022–2023 school years to conduct district writing assessments and provide teachers with access to practice writing assignments. Multiple independent ELA outcome measures support the idea that HCSD Writable students in Grades 6–12 made significant improvements in English language arts achievement during the 2021–2022 and 2022–2023 school years. Of note, statistically significant gains were achieved on the measured writing outcomes by students in each demographic category of grade level, gender, Students with Disabilities status, EL status, and students classified as socioeconomically disadvantaged.

HCSD Writable students achieved significant gains in measured writing ability from pre- to post-Writable instruction. Students who submitted at least one writing assignment in fall and spring demonstrated statistically significant increases in overall Writable proficiency percentages each year. Further, the percentage of students achieving a meets or exceeds mastery level increased from 70% to 82% in the 2021–2022 school year and from 78% to 81% in the 2022–2023 school year. Greater proficiency score gains were achieved with greater usage of the Writable practice assignments and peer review revision component.

In both school years, Writable EL students demonstrated statistically significant increases in ACCESS overall composite scores (16-point and 12-point gains) and writing domain scale scores (15-point and 14-point gains). Writable EL students achieved continued growth in both overall composite scores (21-point gain) and writing domain scale scores (19-point gain) with two years of Writable usage. The percentage of Writable EL students who achieved a 4 or 5 writing domain proficiency level, indicating progress in meeting grade-level standards, increased from 16% in 2021, pre-Writable implementation, to 28% in 2022, post-Writable implementation, and from 13% in 2022 to 18% in 2023. In contrast, statewide ACCESS writing domain proficiency levels decreased or remained stable during this same period. Greater ACCESS writing score gains were achieved with greater usage of the Writable practice assignments and peer review and self-review revision components.

The Georgia Milestones ELA assessment score is partially (22%) determined by argumentative/informative extended writing response questions. Though HCSD Writable students demonstrated a small overall average decrease in ELA scale scores from spring 2021 to spring 2022, HCSD Writable students with a high overall Writable engagement score demonstrated an average 3.9-point scale score increase, and ELA scale score growth was correlated to growth on Writable argumentative/ informative assignments. Writable students demonstrated a small overall increase in Georgia Milestones ELA scale scores from spring 2022 to spring 2023, and greater score gains were achieved with greater usage of the Writable peer review and teacher review revision components. Likewise, Georgia Milestones Literature and Composition EOC scale scores were positively correlated with overall Writable engagement scores in both school years.

Though not directly measuring writing proficiency, reading comprehension assessment scores are expected to improve when writing skills improve (Graham & Hebert, 2011). HCSD Writable students at each grade level achieved MAP Reading RIT scale score gains greater than the national average in both the 2021–2022 and 2022–2023 school years. In the 2022–2023 school year, greater MAP Reading RIT scale score gains were achieved with greater usage of the Writable practice assignments and peer review revision component.

HCSD Writable students submitted a varying amount of writing assignments and utilized a varying amount of Writable revision and practice components in each school year. Overall, the student usage data shows that Writable students may benefit from all kinds of engagement with the Writable software, as two-year writing assessment score gains were predicted by overall Writable engagement scores, both with and without correction for selection bias (effect size Cohen’s d = .62–.87). Further, usage of Writable to complete practice assignments (effect size Cohen’s d = .54–.73) and peer reviews (effect size Cohen’s d = 1.20–2.13) was accompanied by writing assessment score gains of large magnitude. Taken together, this pattern of results supports the idea that use of Writable increases writing proficiency among secondary school students.

Graham, S., & Hebert, M. (2011). Writing to read: A meta-analysis of the impact of writing and writing instruction on reading. Harvard Educational Review, 81(4), 710–744. https://doi.org/10.17763/haer.81.4.t2k0m13756113566