How do we know if distance learning is working?
Could data be the answer?

School closures and distance learning are upon many of us.

There will likely be more in the foreseeable future.

We are all inundated with policy implementation, setup and delivery. But what about efficacy? We know data helps us make the leap from opinion to fact. This post presents an overview outlining how to use this as an opportunity for making data relevant to maximize distance learning results.

If your school is already using data then this is already part of your process, otherwise, it can push your organization forward by light years. If you’re already engaging with learning analytics and data visualization this challenging time can demonstrate even further how useful timely data examination is to improving outcomes across your school.

Gathering the right data and knowing how you will share it is the best way for ensuring meaningful learning continues during a school closure. Of course closures are an unfortunate interruption but let’s do what we so often talk about and leverage this disruptive opportunity for trying new things.

By enacting a data collection approach early on a community of learners will be more strongly equipped to deal with the pivots that occur throughout a distance learning experience.

Distance learning provides the opportunity to rethink the way students learn and demands we take risks, but how do we know what risks were worth taking?


  • How will we measure the progress towards this rethinking of learning for our students at this time?

  • How can we preemptively plan for success and measure our progress towards that success?


Identifying the Metrics to Measure that Lead to Successful Distance Learning Implementation


Identifying desired outcomes and imagining what success looks like is the fundamental first step. Once the desired outcomes of your distance learning experience are decided on, you can then use them to make meaningful choices on how you will measure progress towards success.


Research and identify what success looks like

  1. Engage in conversations with schools in your network who have already begun. You are not alone!

      1. Research and brainstorm what other schools are doing/planning/have done.

  2. How do your findings apply to your school's context?

  3. Create a distance learning plan based on your institution’s success parameters

      1. CONSIDER: Should there be specific versions for specific stakeholders and/or divisions?


Transparency around what success looks like and what you will be measuring

Are teachers clear on what successful distance learning (and planning) looks like?


  1. Share what you will be measuring with your teachers ahead of time, keep them in the loop, and ask for feedback on all questions. In the end you can’t have 100 questions so narrow them down to 10 or less for each stakeholder for triangulation.

  2. One underlying opportunity with data collection is that what you measure and what data you collect becomes important to your community. Your community will want to have successful results against the metrics you plan on measuring.


Are students and families clear on the success you are looking for?

Examples

      1. All families have a clear understanding of the procedures and expectations

      2. Students know where to find updated information about their learning experiences



This approach to identifying what works will provide insight as we rethink student learning. With the data gathered we can develop and demonstrate greater instructional agility while simultaneously delivering more timely support procedures for students, teachers, and parents.


What data? Creating Data Structures

Now that your desired outcomes are identified and transparent, you can now decide on what monitoring tools will be used to measure & assess progress.


3 key measurement tools that you may consider are:

  1. Surveys

      1. Wellness Surveys

      2. Academic Surveys

  2. Digital Classroom Walkthroughs

  3. Student Log-in and Engagement Data


Crafting Wellness Questions

A few wellness questions that you may consider are:


  • Rate your level of motivation to learn on a scale of 1-10.

  • Rate your level of stress on a scale of 1-10

  • How much sleep are you currently getting on average each night? (1-10 hours)

  • How happy are you with how your family is getting along? (0(low) - 10(high))

  • How happy are you with maintaining connections with your friends? (0(low) - 10 (high))

  • How happy are you with how well you are balancing your time? (0(low) - 10(high))


Counselors can use this data to guide conversations with students. For example, a student may report that on a scale of 1-10 they are not stressed about learning and motivation is high but they are not sleeping enough and they are not getting along with their family. This would lead a counselor to ask questions about sleep and family relationships and have a targeted conversation.


Student wellbeing data is not necessarily feedback on the learning experience, rather it is the students letting the school know how they are doing personally. By not making this anonymous it allows for questions that can be followed up with counselors and potentially teachers that want to know more about certain students that they are worried about. You may consider fielding these surveys weekly.

Academic Feedback

It is important to gather information from all your stakeholders around what is or isn’t working well during your distance learning experience. Consider fielding similar questions to parents, students, and teachers so that you may triangulate the data and know best what your community's perception is with distance learning.


You may have already identified what success looks like and this will drive the creation of your feedback questions.


  • Ask teachers, students, and parents.


Here are some options below that you may consider:

A few Key Performance Indicator (KPI) questions may be:

  • How positive has your experience been with distance learning this past week?

  • How satisfied are you with the distance learning experience being offered by <<insert school>>?


A few questions around the experience students are getting may be:

  • How many separate times have you had the opportunity to collaborate?

  • How many separate times have you received feedback from your teachers?


You may be planning on a certain amount of time that students are spending on distance learning:


  • How many hours are you spending on your distance learning work? 0,1-2,3-4,5-6,7+...


*One of the biggest challenges of distance learning is gauging how long tasks take. Some students take more or less time just as they do in the classroom so the results will always vary, however by collecting this data it shows the community that we care about this metric, are paying attention to it, and working towards doing the best we can to avoid overloading students.


Outlining strategies for identifying effective data sets for visualizing efficacy

What are we learning from the community's perception that will help us do even better?


Teachers are already very busy so we need only share with them the data that will be useful to them. When collecting data, providing demographic filters will allow you to visualize your data for the right stakeholders:

  • Example: What grade are you in? What building/division are you in?


Keeping feedback surveys about learning and experience anonymous while keeping student perception of well-being not anonymous is important.


The feedback data on the learning experience needs to be anonymous so that students, teachers, and parents are honest and provide feedback to the school so that they may improve.


Digital Classroom Walkthroughs: Looking for learning. (Find and Celebrate what’s working)

Teachers will be using different mediums to deliver content and create distance learning experiences. Find out what the learning actually looks like and look for celebrations around the metrics you are measuring. This helps you tell your story of the distance learning experience your students are having and empowers teachers to know what they can do to make it even better for their students.


Consider looking for these points in your survey questions to guide your improvement planning:


          1. Clarity of instructions and learning targets

          2. Opportunities for student collaboration

          3. Interactions with teacher

          4. Updated posts

          5. Students know where and when they can get support

          6. How students are going to receive feedback


Use the information you find to share back out to the community the different learning experiences that align to your desired outcomes. You can gather your own data by tallying up the look fors you find so that you can see what what "look fors" are most prominent in the community (Celebrate) and which ones least prominent in the community (Provide PD).


Student log in and engagement data

Your learning management system may have login information. Consider checking in on students that have not signed in for more than 2 days.

Some LMSs have more usage data than others. Have a look a what your LMS offers and see if it is a place where you can find and celebrate success.

Using Questions to Unpack it all for Shared Understandings: Using the data

When sharing your findings back with faculty and parents consider looking for celebrations and what is working. We are all learning through this experience and avoiding what is not working will allow teachers to continue to take risks and try new things.


Avoid “gotcha” situations.


We must avoid the perception of data as a way of catching or outing weakness, rather it should be seen as an effective means of learning about our community so that we can ask questions to improve our distance learning practice. (*You might consider offering PD on an area of weakness rather than pointing it out as a low metric.)


Stay away from comparison data as it is counterproductive to engendering an environment that encourages risk taking. As an administrator, you may want to know how grade levels compare, but a 5th grade teacher will only do better if they see only their 5th grade level data as there are too many factors that come up when you compare and even if there is something that you should be comparing it won't help during this distance learning experience because teachers are learning and making mistakes and need to stay empowered to continue to try new things and do what we keep on talking about: “Doing even better for your students.”


When looking at your survey data you may consider:


  • Sharing out to your faculty celebrations and what is working.

  • Providing PD around the metrics you hope to improve.

  • Sharing data you have learned about students with the right people to help them.

  • Sharing to counselors students with low wellness scores.

How do we Maximize This Opportunity for Growth Once we Get Back to Regular Life?


How does this make your school even better when students come back?

What did we learn from our experience as learners and as educators? Consider fielding more surveys and discussion around the following questions:


Learners: What surprised you about taking this new approach to learning?

Educators: How does this impact your teaching as you are now back?


Example of what we might learn:


  • While doing distance learning, teachers learn about a lot of tech tools that could be used in the classroom.

  • Teachers realize that there is a lot of what they do face to face that is unnecessary for their learning targets that they can cut.


Gathering even more data: How are you planning on finding out what tech tools are working during distance learning so you can find out what learning strategies improved student learning and keep it going when students are back?


Yes, this is the disruption we didn’t know we were waiting for.


Let’s dig in!


Feel free to contact us with any immediate questions to help through this challenging time.