Analyzing (Coding) Qualitative Data
Overview:
We need to get more insight from surveys than accessible through multiple choice questions
Open response questions are the best way to understand how our audience feels
Open response questions are difficult to gain insight from, however, by analyzing open response questions through coding, we can quantify our qualitative data in order tell its story (By coding in this sense we mean the process of labeling and organizing qualitative data to identify different themes)
There are two common ways to code the data
Have survey respondents "self code" the data by asking multiple choice questions that follow up with an open response explanation
Code the data with our SDFS spreadsheet coding tool, in order to dive deeper, identify themes, and quantify them in bar charts
Are multiple choice questions on surveys actionable?
Multiple choice questions on surveys are some of the easiest questions to answer and analyze, however they provide us with only very limited information. These results can help us know where we may choose to focus our energy, but don't provide us the how we may be looking for.
Why are multiple choice surveys not enough?
The results from the above question will let us know the general consensus of how well the class supported learning. It seems that most students feel as though their learning has gone quite well, but we don’t really know why. We can assume and draw some conclusions but we won’t really know for sure unless we follow up.
We don’t know why students felt as though the class has done extremely well.
We don’t know why there are some students that the class is not supporting with their learning well at all.
Open response questions are key
The best way to get quality information from your audience in surveys is to incorporate open response questions. The qualitative data in open response surveys can be very powerful. The best part of open response questions is there is no predetermined set of answers. This allows the respondents to tell us what is really going on.
Examples of open response questions:
What is your best idea for how groups of people at this school could get along with each other better?
What are two things that this school could do to improve? Please be as specific as possible.
What are two things that this school does well that it should continue to do? Please be as specific as possible.
Please explain your answers to the multiple choice questions.
Open responses are difficult to analyze & draw key insights
The problem with these open responses is that they are often difficult to analyze. We find when reading open responses, the information is so all over the place that it can be hard to decipher what is really going on.
In a class feedback survey, one student may say that they didn’t learn at all, while another student said that they had the best class ever.
If we are surveying more than 100 people, it can often a daunting task to read all open responses while at the same time having evidence to explain your insights and themes found.
How can we better analyze this qualitative open response survey data?
To better analyze open response data we need to code the data and identify themes.
By coding the data we are turning our seemingly overwhelming qualitative data into a quantifiable product from which to share and draw meaning. When we code open response data, what we are really doing is labeling each open response with a theme. As they are labelled with themes, we can then count how many times each theme was mentioned and then graph these metrics. By organizing the open responses into themes it is easier to gain insight and talk to the quantitative data that occurs the most.
Two ways to analyze (code) open response data
Respondents Auto Code Within Survey
Self Code Open Responses with SDFS Spreadsheet Tool
First way - Respondents Auto Code Within Survey (Easier)
Use a follow up question to your multiple choice question
The easiest way to code data is have respondents code it for you. You can do this by asking a multiple choice or qualitative question followed up by an open response question such as:
Please explain why you answered the way they did.
This way, when we look at the results we can filter our open response data by the way our respondents answered the multiple choice questions. This can save considerable time when you are looking for insights into positive and negative responders.
For example (Video overview)
The video to the right shows how this method can be used in combination with Google Data Studio.
If we use a Google form to collect survey data, we can connect the results to a google sheet.
Because the results are in a google sheet, we can connect those results to Google Data Studio.
We can then share Google Data Studio with others to gain insights from the open response data by clicking on the graph to filer the open responses.
We were able to find out that students where it did not go well at all needed more feedback and students that were happy with the class liked the groups and intro video.
Interact with the Live Dashboard Below
Click on a part of the bar graph to filter by the open responses by that metric.
With an interactive results page like this you can dig deeper into why people answered the survey the way they did.
What do you notice?
NOTE: It is not our intention to illustrate how these data visualization dashboards are made at this time. Contact us if you would like some help or sign up for our Google Data Studio course (coming soon).
Second Way - Self Code Open Responses with SDFS Spreadsheet Tool (Better)
Identifying themes and coding in SDFS coding tool
We may want to dig a little bit deeper and code the open response data ourselves or we may not have had respondents code it for us. The second way to code this data is to use a coding spreadsheet tool that we created at SDFS.
This tool helps you identify themes and count those themes which will allow you to graph the count of each theme.
Video: How to use the SDFS Coding Tool
Graphing our theme counts
We now have turned our qualitative data into a quantified graph that counts the number of times themes in the open responses were identified.
What we have discovered is that many students enjoyed the collaboration/grouping and warm up activity, but there were at least 5 students that felt that they did not get enough feedback from the teacher.
In our tool we can also organize each open response answer by theme if we need to reference this data in the future or to show/discuss others our findings.