State Department of Education officials proudly proclaimed earlier this month that Idaho students were “outperforming their national peers in early literacy” — even though their proficiency levels on the Idaho Reading Indicator exam backslid by about three percentage points from the previous spring.
It was a head-scratching conclusion — one made without evidence to back it up — because the IRI is specific to Idaho, which would seemingly make results difficult to compare nationally. Officials said the assertion was based on a report from Istation, the state’s IRI vendor.
EdNews requested the report. After reviewing it, we found that the comparisons between Idaho students and their national peers were often apples-to-oranges — they compared results from different years, or using unlike metrics, or from students with different economic backgrounds.
Plus, an Istation document said that “nationwide trends are similar to what are seen in Idaho” — not that Idaho performed markedly better.
The evidence provided by the SDE also seemed to undermine its argument that a new norming system might have caused the decline in proficiency levels from spring 2022 to spring 2023 — students outside of Idaho were just as likely to perform the same as or better than they did with the old norms.
Here’s what we learned:
The IRI results were compared to results from a similar — but different — test
Istation administers tests called the Istation Reading Formative Assessment to 493,000 students across 42 states. Scott Graf, communications director for the SDE, said those tests are similar to the IRI in content, question type and length, but are not exactly the same.
Results were compared using different metrics
The chart below, provided to EdNews, helps show the different metrics. At first glance, it’s easy to see that 65% of Idaho kindergarteners were reading at grade level, while only 42% of students nationally reached “Tier 1”. However, Tier 1 means that students performed better than 40% or more of their peers — which is not the same as “at grade level.”
Idaho students’ 2022-23 results were compared to national students’ 2018-2019 results
The SDE based its claims on another dataset as well: that of average percentile rank.
In Idaho, students had “higher mean percentile ranks at the end of the year” than their peers, as the graph below shows.
However, Idaho students were not compared to peers’ performance this year — instead, Istation compared them to peers’ performance in 2018-2019. So what the graphs really show is that Idaho students are doing better now than their peers were four years ago.
Idaho students were compared to peers who are not “nationally representative”
Istation did compare Idaho students with their peers in 2022-2023 for percentile growth — and they outperformed them. However, Istation notes that the peers Idaho students are compared to are not “nationally representative” because “there are more students from Title 1 schools than are typical in a national sample.”
Title 1 schools have a large population of students from low-income families and receive federal funding.
Half the time, new norms didn’t result in declines outside of Idaho
The charts show something else interesting, too. SDE officials claimed that the lower exam results may have occurred because of new norming — something that takes place periodically “to address changes in the population.”
This time, Istation based the new norms on the 2018-2019 school year “due to the impact of the COVID-19 pandemic on the educational system and the learning lags evident in reading.”
“This was to avoid having norms that were too easy and thus would not serve the purpose of identifying students at risk of reading failure,” a document provided to EdNews read.
However, the charts provided by the SDE show that outside of Idaho, student scores were just as likely to increase or stay the same with the new norming system as they were to decrease.
Pay attention to the black and blue bars (representing old norms and new norms, respectively) in the slides below. You’ll see that on 12 of 24 of the bar graphs, the scores increased or stayed the same with the new norms.
Because the datasets between Idaho students and their peers in other states are so different, the claim that Idaho students outperformed their peers is questionable at best.
