Lors de la Journée Nationale des études organisée le 20 janvier 2008 par l'Adetem et l'UDA, nous avons eu l'occasion de faire un exposé dont le titre était "Pour une approche plus ouverte des questionnaires online". Notre intervention a clôturé la matinée après notamment que Helen Zeitoun de GFK ait évoqué la nécessité de revoir la notion d'échantillon représentatif sur internet. Notre exposé était une bonne réponse au problème de motivation des interviewés soulevé par Michel Guidi de SSI. Michel Guidi a souligné le très fort taux d'attrition des panels (qui perdent 60% de leurs effectifs en 1 an) et la baisse des taux de retour. Son hypothèse était que les filtres démotivent les interviewés et que les questionnaires sont souvent initéressants voire repoussants pour les interviewés.
Dans ce contexte de remise en cause d'une démarche d'interrogation industrielle et machinale qui nie les répondants en tant qu'êtres humains sensibles, notre message d'ouverture arrivait au moment idéal. Après avoir fait le point sur le background culturel des études (hégémonie d'une culture scientifique, informatique dérivée du CATI), nous avons évoqué la motivation des interviewés. Nous avons montré que pour qu'une enquête online soit une expérience culturelle enrichissante il fallait passer par des questionnaires ouverts avec des stimuli. Nous avons illustré la présentation avec un questionnaire panoramique en rappelant que les origines du online étaient plus à chercher du côté de l'auto-administré papier que du téléphone.
Matthieu Guével a évoqué quelques livres de sociologie et de psychologie cognitive pour expliquer les conditions d'un développement durable des études.
Voici le document de présentation : Prez JNE QualiQuanti
Pour compléter cette lecture, je vous recommande l'article ci-dessous extrait de Quirk's magazine :
Should an online interview be fun?
Editor’s note: Stefan Althoff is team manager, market research at Lufthansa Technik, a Hamburg, Germany, subsidiary of Lufthansa German Airlines. Bill MacElroy is president of Socratic Technologies Inc., a San Francisco research firm.
Why do people surf the Internet? Many go to hunt for information, to shop or do online banking. In the end, the search for entertainment - in the widest sense - is often most important. Online games or sites like YouTube are popular. To meet other people, even in a cyber sense, is another motivation. The value of entertainment extends to the ad realm, where ads that are perceived as funny are often judged to be more effective than those that don’t elicit laughs.
What motivates people to participate in a research study? A kind-looking interviewer at the front door, a well-dressed and polite mall researcher or a telephone interviewer with a pleasant voice can convince almost anyone to complete a survey.
Of course, this is different in the online market research world because of the self-administration aspect: The e-mail invitation and the questionnaire must motivate the potential participant. This is the same with registered online panel members: The registration in the panel and the general interview willingness are not enough.
Online research is becoming more and more popular worldwide as a channel for gathering market data and customer feedback. More than 50 percent of all interview projects in the U.S. are conducted online. Online research has lost the aura of novelty; therefore it is no surprise that response rates have dropped since its introduction in the mid-1990s. In addition, the number of people who start Web surveys only to abandon them midway has also grown significantly.
As a consequence, the main target in the analysis of non-response has shifted. In the past, the focus was on the interviewers and what they could do to increase response rates. Now the interviewees themselves and their motivation for participation are the target of multiple analyses.
Many of the techniques to increase response rates for Web surveys are not rocket science. The so-called “Anita effect” (“Does the survey sender’s gender matter?” Quirk’s, February 2007) is an example for a simple method: The response rate - especially in a male-dominated environment - is increased by using an obviously female sender in the invitation e-mail. Furthermore, there are strategies to make the experience of interviewee as pleasant as possible, e.g., with the PRD technique (“Using the PRD technique online” Quirk’s, January 2008) by making the recall of the previous assessments easier (a finding of Lufthansa Technik Market Research, proven by a comparison study by Socratic Technologies).
In addition, there is much known about the most effective strategies for reaching the best potential participants in online research. The control of the e-mail timing alone can influence the success of a study. It can be of importance which day of the week the invitation e-mails are sent out, and that also makes a difference whether you are inviting respondents to a business-to-business or a business-to-consumer online survey.
The degree of personalization within the invitation mail and the survey itself can be of importance: For a B2B survey it is better to invite on behalf of a well-known counterpart instead of a CEO who people only know by name. This is similar to the problem with invitations automatically signed by the company president. Lufthansa Technik found that the response rates are significantly higher in more personalized survey designs: The average completion rate rose from around 30 percent to 40 percent.
Number of options
The online medium offers a number of design options. Since so many people use the Internet for entertainment, why not make survey participation fun? An interesting question was posted in the German XING.com expert forum on online research in the summer of 2007: Is the use of engaging technology enough to motivate participants or do these experience-enhancing extras only serve to frustrate survey participants?
Nowadays nearly all online surveys have a familiar layout. The times of directly adapting a paper questionnaire for use on the Web should be long over. An attractive design, matching the corporate identity, is now standard. Furthermore, multimedia elements are being increasingly integrated into interviews (e.g., entering single and multiple answers by clicking on pictures instead of check boxes). However, does this help with response rates?
Socratic Technologies analyzed which factors influence participation and dropout rate with the aim of solving the problems, if possible. One major finding was that the design of surveys must take into account the interaction between the variables of the burden and the intrinsic personal returns one experiences from participating. These tend to be related to three factors surrounding survey design (together with the extrinsic factors like interest in the topic and affinity with the sponsor):
1. Length of the questionnaire (both in terms of time to complete and number of questions).
2. Incentive (either total incentive offered as prize package or the approximate value of the incentive on an individual basis).
3. Engagement level (the degree to which the survey is perceived to be entertaining and/or intellectually stimulating).
A combination of these three factors influences the number and proportion of survey avoiders and mid-survey abandoners. While a great deal of attention has been paid to the length of the survey and the level of incentive, less study has been devoted to the in-survey experience as an intrinsic motivator.
Following are findings and recommendations based on Web-based studies from 2004 to 2006. All of the studies were with business and consumer technology-related decision makers and included American, European and Asian respondents. The total number of respondents included in these surveys was 28,437, with a median sample size of 422.
These studies examined the level of engaging activities, which ranged from low-engagement, indicative of a standard questionnaire-based survey with little interactivity beyond simple skip patterns and some calculated variables, to high-engagement, in which Flash-based animations and game-like activities were used as data collection vehicles.
The analysis focused on the level of mid-terminates - an indication of the point at which respondent fatigue, boredom or lack of perceived value becomes critical. Socratic set the critical threshold of fatigue at the point where surveys have a mid-terminate rate of more than 30 percent.
Findings from these studies indicated that no significant differences can be found across different geographies, meaning that engagement appears to be a global phenomenon that influences people of many cultures in a similar way.
Socratic found that while incentives can help get people into a survey, the length of the survey and the degree to which it is perceived to be boring causes participant attention to slip and leads to survey abandonment. As might be anticipated, the degree of overcoming mid-termination by successfully embedding engaging elements is related to how long the survey is to begin with: The more screens/questions, the greater the number of mid-terminates.
• Specifically, surveys that exceed 30 screens/questions are predicted to exceed the maximum acceptable level of dropouts.
• Other studies have shown that if the survey requires more than 55 clicks, the threshold level for abandonment is met.
• This phenomenon is also related to time. Surveys should ideally last no longer than 17 to 18 minutes to prevent more than 30 percent of people from mid-terminating.
When attempting to offset the length and duration of surveys, animation and interactivity are only effective within a certain range. For surveys less than 10 minutes in duration, animation produces no significant effects on dropout rates. This is probably due to the fact that most people will self-engage in a survey activity for the required length of time without additional stimulation.
The primary range of effect (Figure 1) appears to be between 17 to 42 minutes, where significant improvement in completion rates is seen with animation added, compared to surveys of that length without additional stimulation. For surveys beyond 45 minutes (which are not recommended), animations, interactive exercises, etc., have little effect on reducing dropout rates. (As a point of reference, none of the survey satisfaction factors - interest in the topic, level of incentive, sponsor affinity, etc. - are effective for retaining respondents in surveys lasting more than 45 minutes.)
This research indicates that the single greatest factors influencing survey avoidance and mid-termination are respondent fatigue and the perception that online surveys are boring. Socratic Technologies has experimented with game-like or engaging environments (where appropriate to the subject matter) and has found that certain interactive and animated elements lead to higher satisfaction with surveys and higher likelihood of future participation.
This is a starting point for advanced Web survey design. Examples of animated activities that increase satisfaction include sorting and arranging tasks, dragging and dropping elements, videos, demonstrations of product functions, and selection tools that have the look and feel of arcade-style games. Many companies now employ some form of animation within their survey environments.
But this does not necessarily mean complex Flash programming! Many surveys from Lufthansa Technik contain picture strips, and even in standardized questionnaires it is possible to use animated GIF files to replace the pictures or to randomize their display.
In addition to the entertainment value, advanced animations can also help track the experience and actions of the user. Ancillary variables that can be collected and analyzed include how much time someone spends examining an object, how many times they change their minds and how many items they explore.
One word of caution, however, is that using technology for the sake of technology can lead to poor results. Not everything that can be done from the technical point of view must be done. This is the techno trap, which occurs when technology is applied to a research problem when it is unnecessary. Elements such as extraneous animated characters, flying spaceships, blinking lights, annoying sounds, etc., tend to have the opposite of the desired effect: creating distraction and annoyance instead of focus and engagement. Even for simple questions the design can influence the answers. Using random animations and sounds also can increase survey costs, increase the chances of survey technical problems and tends to create overly complex or confusing situations for the respondent.
In the end, everything depends on the design and content of the survey. A good questionnaire is the foundation of good research. It should be the aim of all online researchers to make the job for the participants as easy and pleasant as possible.