Online cognitive interviewing. a contribution to data quality

Abstract

Survey data quality depends on the choices made during the questionnaire design process. Flaws in the questionnaire design can be often identified by conducting a pretest prior to the survey. Cognitive interviewing is a pretesting method useful for exploring hidden and overt problems respondents have in answering survey questions because of the sequence of questions, their wording, or the structure of responses. This interview is usually face-to-face, but the spread of the Covid-19 pandemic has imposed to find innovative solutions. In this paper we report on one of the first experiences of online cognitive interviewing run at the Italian National Institute of Statistics, which took place in 2020 in preparation for the Survey on discrimination at work against LGBT people. Subjects were recruited by the chain sampling method, starting from some acquaintances. They were asked to complete a screening questionnaire, sent via WhatsApp, and to invite other people to do the same. Eligible respondents were contacted by e-mail or by phone to arrange an appointment for the interview. The Whereby videoconferencing system was chosen, as it requires neither installation nor account registration, and therefore minimizes the response burden. Based on this experience, conclusions have been drawn about the benefits of conducting online cognitive interviews. Relying on videoconferencing system allows to reach people spread throughout the territory in a short time and at low cost. In addition, it encourages the participation of those who, due to lack of time, might otherwise refuse. The freedom in choosing the time and the place of the interview, along with the physical distance from the interviewer, also foster a perception of confidentiality and can reduce the interviewees’ reluctance to deal with sensitive topics. Finally, under certain setting conditions, the interaction dynamics do not seem to differ significantly from those in presence. Therefore, the spontaneity of the response process and the interviewee's ability to faithfully reconstruct the thought processes are not compromised.