Test Scores: The RBI of Student Data

#Data Read time:
In 2011, the movie Moneyball brought mainstream attention to the concept of sabermetrics, originally defined by Bill James as “the search for objective knowledge about baseball.”

Statistical analysis has long been a driving force behind decision making in the sport, with executives and writers deferring to a “numbers don’t lie” attitude when managing talent and comparing the accomplishments of the game’s brightest stars.

This kind of empirical analysis governs our lives in ways we don’t even consider. Big data is driving the business world and (as a result?) the education system is not far behind. Instead of pushing back against the data movement in schools, many educators are working overtime to figure out how they can use this data to improve the learning experience for their students.

The first step in the process is to acknowledge that educational data is years behind other industries in terms of practical application. Sure, we track test scores, discipline infractions, and attendance, but does that really tell the whole story about how effectively we are preparing students for college and a career? Recent trends, such as flexible learning environments, PBIS, and the de-emphasis on letter grades all show early promise, but we've only just begun to scratch the surface.

Any casual baseball fan can visit a glut of websites to easily find the splits, trends, and relative production of their favorite team’s backup shortstop, yet it can be hard for educators and administrators to identify even the basic information needed to take action and drive results in our schools. Priorities, anyone?

Educational data is used as a source for funding decisions, school report cards, and legislative action, but only the most analytical-minded leaders are finding ways to apply this data where it's needed most. Teaching and leading both require a combination of art and science. Striking the right balance can result in positive, lasting change.


 

Learning from the RBI

For the first 100+ years of baseball’s existence, the run-batted-in (RBI) was a sacred cow in the baseball world, right there next to batting average and home runs in the pantheon of baseball card statistics. Triple digit RBI numbers were a sign of stardom, and front office personnel everywhere shelled out big bucks to attract top “run producers” to hit in the middle of their lineups.
 
Then, everything changed. With the boom of sabermetrics in the late 20th century, baseball statheads began to realize that the RBI was really more a product of luck, environment, and opportunity than a defining attribute of a hitter. Dozens of new metrics emerged that painted a more holistic picture of a ballplayer’s impact on his team and rendered the RBI irrelevant, except, of course, among baseball purists who frowned on these new, seemingly complex analysis systems. Sound familiar?
 
Today’s educational data landscape closely mirrors that of pre-sabermetric Major League Baseball. Swap "assessment performance" for "RBI" and "student" for "hitter" in the above paragraphs and see what happens. We have more tools than ever before to measure our students’ progress and identify any number of variables that might be contributing to their success (or their struggles), yet we still lean heavily on traditional, stand-alone testing data that paints a very limited picture of student abilities.

Standardized test scores are the RBI of the K-12 system, ripe for exploitation and lacking in context. This single data point, when placed on an island, offers little value to the pursuit of improving outcomes.


 

Applying Sabermetric Concepts to K-12

There are still some who decry the use of educational data and question its efficacy, while a few data-minded administrators are quietly using the information at their disposal to make small changes with big impact. Knowledge of something as simple as a correlation between a certain student’s lunch period and performance can prove invaluable. The same holds true for class size, discipline, and teacher effectiveness.

Let’s look at some hypothetical examples of advanced educational metrics and how they might be used to identify trends and improve student outcomes:

PER (Parent Engagement Ratio)

If a superintendent wants to measure parent involvement efforts across campuses, there are more than a few metrics to looks at. Are teachers posting future assignments on the parent portal, or just using it to store grades that have already been issued? Do certain schools show a higher usage rate of digital communication tools? Are parent volunteers more abundant in certain population centers? 

The value in a metric like this is twofold - first, it can help district leaders make decisions about outreach and support mechanisms; second, it can serve as a valuable coaching and feedback tool in conversations with principals about priority initiatives. 

Ex: "PER is significantly lower at North High School than South. That tells me that we need to consider a nontraditional approach to parent outreach and raise awareness of the tools and resources available to these families. Maybe a Wi-Fi kiosk at a local community center or an assigned liaison for this particular neighborhood would be a step in the right direction." 
 

AIV (Assessment Input Variance)

Compare student test/assignment scores against input methods (electronic, paper, oral presentation, etc.). This will require some A/B testing in a low-stakes environment, but the results should help paint a picture of digital literacy levels in your district so you can determine whether it is the content of the assessment itself or the delivery method that is impacting your students' performance.

Ex: “Jefferson Elementary’s electronic AIV over the past two years is -.36. This number indicates that electronic test score results in this school are not indicative of conceptual knowledge, but are rather a result of relatively low digital literacy rates. We are planning a phased 1:1 rollout and technology coaching program to help close the gap.”
 

TLO (Teacher Leader Output)

Analyze teacher leader impact based on professional development and peer improvement measures. If you are an administrator, odds are you rely heavily on your strongest teachers to share their instructional tips with less experienced staff. Why not add objective measurements to aid in teacher evaluations and professional development?

Ex: “Jason’s 2.34 TLO is indicative of the impact he has had on the entire English department. We need to get him more involved in the professional development process district-wide. Let's see if he would be willing to host a workshop at a neighboring school next year.”
 

SVP (Student Voice Profile)

Analyze class participation, extracurricular involvement, and student portal activities in order to easily identify your least engaged and/or introverted students. Tailor your instructional approach to include additional one-on-one coaching and strategic lesson plans designed to help these students become more comfortable and raise their level of achievement.

This approach is already happening in classrooms today thanks to teacher awareness and intervention, but objective measurements can reduce the possibility of observational bias and ensure these methods are being applied universally.

Ex: “Dolores has a relatively low SVP and her grades appear to be suffering as a result. I will set aside class time to work with her individually and in small groups. I will also make a point to encourage her to participate in our discussions, both in class and online."


 

What's the Point of this Exercise?

Large volumes of data and complex analyses can be overwhelming, which is why it is important for individuals in every role and every unique situation to narrow their focus to the metrics that matter to them. This makes strategic data planning at all levels especially important.
 
Data collection for its own sake has little value. The three Ts of data use – technology, training and transparency – can make or break your educational data practices. The examples we used above were arbitrary and the only factor they had in common was the inclusion of more than one variable as opposed to test scores on an island.
 
The point is, teachers, administrators, and central office personnel will all find different metrics that help them make decisions to improve student outcomes. School districts are already collecting most or all of the data required for these exercises, but few district leaders have the training or technology needed to unlock its potential. The policies and practices for data use in schools are still in their infancy.
 
As our understanding of effective educational data practices grows, it seems ever more likely that we will need to take a lesson from Major League Baseball's RBI and phase out our emphasis on stand-alone assessment as the foundation of decision-making processes.
 
Schools, populations, and individual students are subject to an abundance of variables that have a measurable impact on “achievement” metrics above and beyond the standardized test. Isn't it time we started prioritizing a smarter approach to data?


For more on strategic data planning, contact us today to find out how Skyward can help you find the information you need to make better informed decisions. 


 
Share this story: