Communities across the country have serious questions about how their students were affected by the rapid transition to distance learning wrought by the COVID-19 pandemic. How many attended online classes? How much did they really learn through virtual instruction? How far behind might they be in September?

Thoughtful data collection and analysis will be needed to give comprehensive answers and help define the path forward. As school reopening plans are developed and refined, district leaders will rely on available data to inform decisions about instructional modifications, grade promotion, allocation of resources and more. Principals, teachers, families and students will also be trying to make sense of the available information to shape their decisions.

The instability caused by COVID-19 raises new questions about what is most important to measure, how to collect accurate information, and the best ways to support people to use that information. But many of the fundamental challenges of effectively using and clearly communicating data are unchanged.

Over the past several years, the School District of Philadelphia (SDP), with support from the William Penn Foundation and others, has been working with partners to more effectively share data with school staff, families, district and city leaders, philanthropic organizations and other community members. While efforts are still underway, three insights from this pre-COVID-19 effort have become clear.

These lessons will help shape SDP’s response in the coming months and may help other districts grappling with how to address the upheaval caused by the pandemic.

Define how a concept is measured

Many people assume terms like “percentage of students reading on grade level” or “graduation rate” have a single, stable definition. But districts often define terms differently and make different decisions about what data are “in” and “out” of analyses.

Let’s use the graduation rate as one example. Despite being widely shared as a success metric, there is no single definition of graduation rate across districts. To help stakeholders understand how SDP’s graduation rate compares to other large districts, the SDP Office of Evaluation, Research, and Accountability recently published an analysis illuminating the many decisions that go into calculating graduation rate. It shows significant variation among comparable school districts.

Even as they adhere to state and federal accountability rules, districts have flexibility in the way they calculate the graduation rate for their local communities. From including or excluding students in charter schools and alternative schools to focusing on the 4-, 5- or 6-year graduation rate, differing formulas for calculating graduation rate stymie comparisons across districts. In spite of this, policymakers, politicians, practitioners and parents are likely to make decisions based on indicators that seem like they should be comparable but are not.

In the weeks and months ahead, education stakeholders will look at the leaders and laggards across a host of outcomes and will try to plan their future efforts accordingly. For example, which districts seem to have the highest attendance rates during virtual instruction? Which saw continued growth in reading proficiency despite the quarantine?

While there will be many lessons to glean from the experiences of schools and districts across this country, those lessons will only be accurate if we are clear about the definitions to ensure “apples-to-apples” comparisons where possible.

Craft a research agenda for the problem 

Regardless of the decisions professionals and parents will have to make in a COVID-informed world, it is unlikely that a single study or data point will suffice. Together, we face adaptive and technical challenges and will need a comprehensive set of information on each of them.

Continuing with the high school graduation example, it is important to understand the many points at which a student may be on or off a path to graduation. Opportunities for intervention and support are numerous but cannot be strategically targeted unless we understand the full trajectory.

SDP worked with a local research consortium to conduct several studies designed to understand what keeps students on track to graduation. Each piece of research answered some questions and raised others. And, while SDP’s graduation rate has been on an upward trajectory, there was no single piece of research that made that possible.

As we seek to recover from the COVID crisis and build a system of supports for students that is even more robust and equitable than the one we had before, we need to be cognizant to build on the research and evidence we have and design a research agenda rather than a single study that will give us the information we need to improve.

Make research results clear and usable

Thoughtful decision-making about metrics and research is only useful if it is communicated in ways audiences understand. Over the past couple of years, SDP has worked to communicate school and district data in ways families and community leaders can use.

Using an iterative process that included input from family focus groups, SDP has continuously refined its school profile pages so they include displays and language that make sense to these audiences. New outreach and training materials will help community members access the information. As parents make decisions about where to send their children to school, and as school leaders make decisions about where to focus limited resources in a post-COVID world, this kind of easily consumed information is essential.

School districts and families face critical decisions about how to balance health and safety precautions with learning and economic imperatives in the fall. We want them to make those decisions with the best information possible. That will only be possible if we remember the three lessons above: define your terms, develop a research agenda, and share clearly. Any one of these steps without the others is unlikely to support a more effective and equitable education system.