HONOR ROLL METHODOLOGY
California methodology
Texas methodology

California Honor Roll Methodology

Top

Educational Results Partnership uses a data-driven approach called multilevel latent class modeling to identify California Honor Roll schools. This statistical technique assumes that:

  • (a) Hidden groups (latent classes) of schools exist among a larger set of schools,
  • (b) A set of observable criteria that represents various dimensions of school performance can distinguish these hidden groups from one another,
  • (c) Schools within the same district may be more similar to one another than typically assumed because they are affected by district-wide socioeconomic factors, and
  • (d) No single criteria is presumably better than another in distinguishing high performing schools among a larger set of schools; the pattern of results will determine which are the more important distinguishing criteria.

To identify Honor Roll schools, the models first estimated the characteristics (i.e., the average and spread of scores on each criterion) for each hidden group. Then, the models estimated the degree to which each school was similar to the estimated characteristics of the respective groups. Finally, each school was classified into one of several groups according to their highest probability of group membership. We interpreted the group with the highest scores on the majority of the criteria in the respective models as the Honor Roll group. The schools that were classified as members of that group were designated Honor Roll schools.

Seven different models identified candidates among schools that served students at various grade segments and levels of poverty. Data from the National Center for Education Statistic’s Elementary / Secondary Information System was used to identify elementary, middle, high, and “other” schools. Elementary, middle, and high schools were further split into two categories:

  • STAR Schools: schools with greater than or equal to 33% of students designated as socioeconomically disadvantaged (high poverty)
  • Scholar Schools: schools with fewer than 33% of students designated as socioeconomically disadvantaged (low poverty)

The number of Honor Roll schools in each category is not predetermined; the results of the model determine the number.

Six major criteria for schools in all grade segments were derived from public data at the California Department of Education (CDE) website:

  • Achievement in English, Math, and Science
    • The percentage of tested students that met or exceeded standards for their respective grade levels in the 2014-2015 California Smarter Balanced Tests (English and Math) and California Standards Tests (Science).
  • Equity in English, Math, and Science
    • We defined equity as high achievement among students from six significant subgroups in comparison to their peers across all significant subgroups in the same county.
    • For each school, we examined the achievement rates from four ethnic minority subgroups (African Americans, American Indian/Alaskan Natives, Asians, and Hispanics) and two disadvantaged subgroups (socioeconomically disadvantaged and English Language Learners).
    • Each school’s equity score in the respective subjects was calculated in two steps:
      • Subtract the achievement score among all students in the same grade segment and county who were outside the selected school from the achievement score of each subgroup in the selected school to generate a set of difference scores for each school.
      • Divide the sum of each school’s difference scores by the number of subgroups with available data to produce an equity score.
    • The equity score provides a general indicator about the degree to which students in significant subgroups were outperforming (more positive score), underperforming (more negative score), or near (score closer to zero) their respective county levels.
    • Schools that did not have a achievement score for any of the significant ethnic minority and disadvantaged groups did not receive an equity score for the subject.

Data from the CDE provided three additional criteria for high schools:

  • Percentage of graduates who met the 2013-2014 UC/CSU course requirements
  • Cohort graduation rate in 2013-2014
  • Cohort dropout rate in 2013-2014

Models for elementary, middle, and high schools accounted for two shared factors across schools within the same district:

  • Proportion of district spending on instructional staff in 2013-2014
  • Proportion of district spending on pupil support services in 2013-2014

A single-level latent class model identified Honor Roll schools in the “other” category because it mostly contained charter and alternative schools. Charter schools typically operate as independent school districts. Financial data was not available for most schools in this category. There was insufficient data to analyze high and low poverty schools in this grade segment separately.

STEM Schools

  • The top 10 percent among high poverty schools in math and science achievement within each grade span that were also recognized as Honor Roll schools. Both subjects were equally weighted in the determination.

Notes

  • We were unable to assess subject improvement in English and Math this year due to the launch of California Assessment of Student Performance and Progress (CAASPP) system in 2014-2015. The CDE guidelines state that results from the new test system may not be compared to results under the old system. Next year, the Honor Roll will incorporate measures of improvement on each subject with respect to a school in general as well as their specific ethnic minority and disadvantaged groups.
  • Honor Roll candidates must perform at or above the average achievement level within their grade segment and poverty level to receive recognition. Schools with lower levels of achievement on any tested subject were disqualified after the end of the modeling process.
  • Missing data did not automatically disqualify a school from Honor Roll consideration. Based on the pattern of relationships among available criteria scores from all schools in the same grade segment and poverty level, the models estimated each school’s likely group membership using their available data.

The modeled data was current as of April 22, 2016.

  1. California Assessment of Student Performance and Progress. (2016). Research Files for Online Assessments [Data file]. Retrieved from http://caaspp.cde.ca.gov/sb2015/ResearchFileList
  2. California Department of Education. (2016). 2015 CAASPP Paper-based Test Results [Data file]. Retrieved from http://caaspp.cde.ca.gov/caaspp2015/ResearchFileList.aspx
  3. California Department of Education (2016). Student & School Data Files (Downloadable) [Data file]. Retrieved from http://www.cde.ca.gov/ds/sd/sd/

Texas Honor Roll Methodology

Top

Educational Results Partnership uses a data-driven approach called multilevel latent class modeling to identify Texas Honor Roll schools. This statistical technique assumes that:

  • (a) Hidden subgroups (latent classes) of schools exist among a larger set of schools,
  • (b) Schools within the same district are more similar to one another than typically assumed because they may be similarly affected by district-wide factors such as demographics and finance, and
  • (c) A set of observable criteria that represents various dimensions of school performance can distinguish subgroups from one another,
  • (d) No single criteria is presumably better than another in distinguishing high performing schools among a set of schools; the pattern of results will determine which are the more important distinguishing criteria.

To identify Honor Roll schools, these models first estimate the characteristics (i.e., the average and spread of scores on each criteria) for each hidden subgroup. Then, the models estimate how likely individual schools are members of each subgroup by comparing the school’s characteristics to each subgroup’s. The Honor Roll subgroup had the best average score on the majority of school performance criteria while accounting for how schools are organized into districts and district-level influences on school performance. The schools that were classified as more likely a member of the Honor Roll subgroup than any other subgroups were designated Honor Roll schools.

Separate models identified Honor Roll schools within seven different categories of schools. For elementary, middle, and high school, there were two categories of Honor Roll schools:

  • Scholar Schools: schools with fewer than 33% of students categorized as socioeconomically disadvantaged
  • Star Schools: schools with equal or more than 33% of students categorized as socioeconomically disadvantaged

Due to a small number of K-8 schools with a low proportion of socioeconomically disadvantaged students, K-8 Honor Roll schools were combined into a single category.

The number of Honor Roll schools in each category was not predetermined; the results of the model determined the number.

The five major criteria for school performance were calculated using data from the State of Texas Assessments of Academic Readiness (STAAR). These criteria were calculated for schools in all grade spans:

  • Weighted average of percent proficient for tested grades in English and Language Arts
  • Weighted average of percent proficient for tested grades in Math
  • Weighted average of percent proficient for tested grades in Science
  • Improvement in proficiency scores in the tested subjects relative to other schools in the same grade span across the state
    • Change in proficiency scores across subjects over three years was placed into percentile rankings for all schools in the state of similar grade spans (i.e., elementary schools were compared only to elementary schools).
  • Equity – for each school, their proportion of significant ethnic minority and disadvantaged student subgroups that outperformed the average among all students in the region.
    • For each significant ethnic minority and disadvantaged subgroup’s subject score that was available in the data, the subgroup score’s was compared against the regional average among all students on that subject. Equity was defined as the number of above-average subgroup scores divided by the number of available subgroup scores in each school. Schools that did not have a score in any subject for significant ethnic minority and disadvantaged groups did not receive an equity score.

Data from the Financial Allocation Study for Texas (FAST) provided nine additional criteria for high schools:

  • Seven indicators of college readiness measured by the proportion of students in each school who
    • Met the English and Language Arts standard for higher education readiness from the Texas Success Initiative
    • Met the Math standard for higher education readiness from the Texas Success Initiative
    • Took ACT/SAT college entrance exams
    • Met the criterion on ACT/SAT exams
    • Took Advanced Placement / International Baccalaureate (AP/IB) exams
    • Met the criterion on Advanced Placement / International Baccalaureate exams
    • Completed at least one academically advanced course (AP/IB)
  • Graduation rate
  • Dropout rate

All of the models accounted for the organization of schools in their respective districts and two district-level measures of spending from the FAST data:

  • Proportion of district spending on instructional staff
  • Proportion of district spending on student support services

Notes

  • Missing data on any criteria did not automatically disqualify a school from Honor Roll consideration. Based on the estimated average scores and pattern of relationships between criteria for all schools in the same category, the model estimated each school’s likely subgroup membership with their available data.
  • An expert panel validated the list of the Honor Roll candidates identified by the statistical models. The panel evaluated schools that had missing or low scores for equity and those among the lowest third in subject proficiency within each grade span on a case-by-case basis

STEM Schools

  • The top 10 percent among STAR schools in math and science proficiency within each grade span. Both subjects were equally weighted in the determination.


EDUCATIONAL RESULTS PARTNERSHIP
2300 N STREET, SUITE 3, SACRAMENTO, CA 95816
P: 916-498-8980