Interesting yet painful: recently the NOS news headlined, based on a survey, that there is more support for " farmer protests" than for "climate action". Is that really the case? Motivaction's senior researcher Cecilia Keuchenius questioned this. And decided to see if the survey result is partly based on the name you give to the people who participate in these protests in the survey question (framing).
How you ask a question in questionnaire research affects the answers you later consider to be objective results. What words do you use in your query text? What frame do you choose (unconsciously) to go with it? Important, because the "research result" based on it is adopted by the media and public opinion.
Framing: what is the feeling you get with the term 'activist' and what is the feeling with the term 'farmer'?
In a questionnaire, you always choose certain words, you have to. In this case: do you call people demonstrating on the highway against nitrogen policy "Dutch people exercising the right to demonstrate," "demonstrators," "activists" or "farmers"? These words evoke different associations and associated feelings. 'Farmers' may have more positive feelings than 'activists.'
And the people on the highway demonstrating against the lack of climate policy? Are they "Dutch people exercising the right to demonstrate," "demonstrators," "activists" or something else?
What do these frames do?
- The expectation is that choosing the term 'farmers' as an occupational group in a question text will attract more sympathy. By doing so, you frame these protesters as people who practice a profession that many in the Netherlands are proud of.
- By choosing the term 'activists' for the protesters at the climate demonstrations, the expectation is a more negative reaction. By doing so, you frame these demonstrators as people who go against public opinion and do something that is not "allowed," which can count on less sympathy.
To get a better handle on this, Motivaction put this to the test and presented two designations to the same target group, via a representative sample from this, to see if the frames produce different results. To clarify, the two frames:
The frame used for the NOS news headline, and expectedly, more controlling:
- 'farmers' versus 'climate activists'
The more neutral, evenly phrased, frame:
- Demonstrations 'against nitrogen policy / farmers' protests' versus demonstrations 'against lack of climate policy / climate protests'
Result
For the neutral frame, support for both is roughly equal (18% and 21% support them mostly or completely, respectively). But choosing 'farmers' versus 'climate activists' creates difference. Support for 'farmers' demonstrations suddenly rises to 26% (versus the 18% it was before) and support for climate demonstrations by the term 'activists' goes to 17% (versus the 21% it was before).
And with that, the news may be true that there is more support for farmers' protests than for climate protests: farmers versus activists. A difference created by framing in the questionnaire and not actual support.
Note: this is not meant to lash out at the party that conducted the earlier survey using the terms 'farmers' and 'climate activists'. There is always something to be said about questionnaire surveys, surely including the language chosen with the best of intentions in the more neutral frame. But what this does show is that you have to be extremely careful about the wording you choose. That your own worldview might play a role in it or the framing predominantly used in the media. That you should indeed put your words on a scale before you perhaps unknowingly choose research that leads to a news headline that has more to do with framing than reality.
Want to know more about doing good research, and how to frame a questionnaire in the least disruptive way possible? Then you can contact Cecilia Keuchenius at Motivaction.
