Outcomes
Student Outcomes
Types of ISAT Studies
Cross Study Implications
What one finds in research has a lot to do with the questions asked and the outcome variables of interest. Across the different ISAT studies focused on the CMSI, researchers asked many questions and used a range of outcome and predictor variables. In other words, the body of ISAT studies is diverse. The body of research has been produced over time, which has had important implications for understanding the results. The studies did not cover common time periods. Some studies are cross-sectional and cover only a single year. Some are longitudinal covering several years. Some covered the entire six-year period of the reform, 2002-2008; others covered "implementation" periods. The latter, while longitudinal in nature, focused on shorter time frames often built around two-year implementations periods for each cohort.
The findings are further complicated by the different outcome measures examined, the two most common measures being the percentage of students meeting or exceeding state standards (%ME) and scale scores. In addition to these two outcome variables, CMSI research involved analyses of derivative of these measures. These included annual change (in scale score or %ME), "momentum," or "rate of change," and it involved independent variables associated with CMSI program implementation, e.g., registration for CMSI training sessions, attendance rates at CMSI sessions, and number of sessions attended.
Further, the research methods varied from study to study. Some studies used minimal controls but covered relatively long periods. Others used highly controlled multivariate models, but covered relatively short time periods. While each set of studies separately has limitations, taken together they provide a stronger, more diverse, research base. Taken together, these analyses lead to a number of conclusions, namely:
- Since 2000, the percentages of CPS students meeting or exceeding ISAT math standards in the district have risen steadily. Of the three grades examined (3rd, 5th, and 8th), the greatest increases in the %Meeting/Exceeding standards in mathematics have occurred in the 3rd and 5th grades.
- Implementation of CMSI mathematics instructional materials and supports has been effective in increasing both the percentage of students meeting or exceeding state standards and ISAT scale scores. The magnitude of the effectiveness depends on the outcome measure being used, the grade level, and other factors. However, the preponderance of evidence supports this finding.
- Most of the research looking at scale scores, annual meets or exceeds percentages (%ME), or annual changes in these measures suggest the CMSI had small positive effects on student outcomes.
- Longitudinal analyses and those in which the dependent variable of interest focused on "rates of change" or upward "momentum" over time suggest that the effect of CMSI was large. That is to say, all of the CMSI "treatment" groups were found to outpace their comparison group counterparts, and at times by sizable margins, especially in comparison to the size of gains reported in the research literature for other large scale education reform efforts.
- The findings do not indicate that any single CMSI cohort clearly outperformed the others. The performance of individual cohorts, which received different supports, was mixed and depended on the outcome measure used in the study and the level of control.
- All of the CMSI implementation groups either outperformed or performed as well as their respective comparison groups when the outcome measures of interest were: long-term change, proportionate growth (i.e. proportionate changes in intercepts), "momentum" and/or "rates of change" over time. This finding is not true when the outcome measures employed were based on annual scores and/or annual changes in scores.
