One of Ascentum’s pubic involvement tools is the Choicebook – a deliberative experience where participants learn about issues, tough questions, and recommend options or choices. These are built into larger engagement processes that may include in-person events and other online tools, like crowdsourcing. Depending on the engagement objectives, participants can be asked a variety of open- and close-ended questions, in a Choicebook resulting in the collection of reams of quantitative and qualitative data for analysis.
While we use SPSS to analyze our quantitative results, the thousands of words of text that we collect through open-ended questions is analyzed using a specialized tool called NVivo. As an analyst, I use both tools to help dissect and understand the views of the publics we engage. During a recent project, I was responsible for reading through 85,335 words of comments (about the same length as the second Harry Potter book, “Chamber of Secrets”), contributed by over 850 participants. Deploying software like NVivo allows me to ensure that participant feedback can be analyzed and presented in a systematic way.
But, NVivo is just a tool. Getting true insights from qualitative data is as much about process and how the tool is used. Here is how I approach analysis:
- For each open text question (for example, “Share a positive experience you’ve had with a government centre”), I import a Word document containing all of the responses.
- After reading a response, I can highlight certain elements of a response (e.g. “agents are very knowledgeable” or “for me, it’s about quick and easy renewal of my permits”) and drop them into ‘buckets’ I’ve created, which are known in the program as ‘nodes’.
- After analyzing (or ‘coding’) about 20 responses, I can get a sense of the themes arising (“knowledgeable staff” or “quick service”), and can start creating sub-themes or sub-nodes (“quick permit renewals” and “quick processing of applications”).
It’s almost like using a handful of coloured highlighters to classify data. The program not only allows me to get a sense of recurring themes, it provides me with a way of quantifying qualitative data in real time (“the most recurring theme when participants spoke about their positive experiences in the government information centre was the breadth of knowledge of the staff, mentioned 83 times.”)
It gives me a true sense of what the majority is saying, without losing the views of the minority. Participant feedback can then be neatly presented, and enhanced through the use of charts, to get a sense of the relative popularity of themes and quotations, to illustrate these themes (and ensure that the voice of those engaged finds its way into our client’s reports).
- Stephen Telka -
Interesting to see that during two of the quite large online discussions on Change.gov (back in 2008/2009) the average comment size, according to my estimates, was 116 and 136: http://bit.ly/ieyeWu
Of course, the conveners never bothered to do any kind of data analysis, neither then nor for any of the other open government consultations that have followed since.
Thanks for your comment, Tim. This a problem we often hear about. That’s where a well-designed process comes into play – ensuring that all of the engagement tools (whether it’s a world café, crowdsourcing site or open space) have some sort of data collection and reporting work built into them (in terms of time, staff, and budget).