Cognitive testing reports from our survey implementation teams show that overall, the venue operator and user survey instruments have no serious comprehension problems. The testing was designed to find out if survey respondents would have problems understanding the survey questions and whether our concepts (e.g. “impact,” “sharing”) translated accurately across national contexts.

 The teams completed cognitive testing at the end of May.  Their reports draw attention to a number of areas where the survey questions could be improved. We can attribute the comprehension issues identified in the testing to a variety of factors including:

• Inappropriate respondent to answer the question (e.g. lacks knowledge to answer questions about a venue’s computer equipment or balance sheet).
• Use of technical jargon (e.g. log server).
• Inaccurate characterization of situation (e.g. asking for a single fee amount for internet access whereas venue has a tiered fee system).
• Inapplicability of question to respondent (e.g. does not engage in the activity a question asks about).
• Too many answer categories, making it diffcult to remember.
• Confusing wording of question.
• Complicated nature of information requested (e.g. respondent needs to estimate budget allocation to different venue expenses).
• Complex question structure (e.g. requires answers in multiple layers).
• Inaccurate translation from English to local language.

Some of these issues require revising the questionnaire, some require revising the language translations, and others require developing strategies to make the survey easier to implement (e.g. using visual aids for answer categories). We paid particular attention to those issues that occurred in more than one country, while trying to find appropriate compromises for items that were specific to individual countries. The Survey Working Group has revised the instrument and survey design where needed, taking into account recommendations supplied by the  survey implementation teams. The instruments are now moving on to field testing.