Evaluating Pilot Data

Overview

In this lesson, students analyse responses from a pilot questionnaire to decide what to change before the final survey. They practice statistical thinking about variation in answers and use evidence from pilot data to justify improvements.

Learning intentions

• WALT analyse pilot responses to identify patterns and issues in questionnaire questions.
• WALT evaluate whether response options and wording collect the intended information.
• WALT decide refinements to improve clarity, bias, and usefulness of the questionnaire.

Success criteria

• I can summarise pilot data using suitable measures and displays (e.g., frequency tables, averages where appropriate).
• I can identify at least two question improvements with evidence from pilot responses.
• I can justify changes by linking them to the survey’s aim and the information needs.
• I can produce an “issue → evidence → revision” plan for the next questionnaire draft.

Curriculum links

• NZC Mathematics and Statistics: statistics investigations using clear questioning, planning, collecting data, and interpreting results to inform decisions.
• NCEA AS91263: use the questionnaire design process, including desk review, pilot(s), and refinement, with documentation of decisions.
• NCEA AS91266 (light-touch in class practice): evaluate a statistically based report by identifying sampling/survey method issues and non-sampling issues, and justifying conclusions using evidence.
• Statistical inquiry and data sense-making: interpreting data, recognising sources of variation, and communicating a conclusion.

Lesson structure (60 minutes)

1. 0–5 min · Launch & purpose

• Teacher shows the survey aim on the board and reminds students this is Lesson 8: pilot evaluation before finalising.
• Students reread the questionnaire’s intended information needs and underline what the pilot was meant to test.

2. 5–15 min · Quick recap: what to look for

• Teacher leads a short discussion: common pilot findings include unclear wording, poor response coverage, uneven distribution, missing answers, and misunderstanding.
• Students complete a “checklist brainstorm” on paper: for each question they’ll ask, “Is it easy to answer? Are options complete? Are answers usable?”

3. 15–25 min · Data familiarisation (mini-task)

• Teacher distributes a pilot data sheet (pre-prepared class resource) with responses for 6–8 key questions, including a few marked problematic responses.
• Students sort pilot issues into two categories: “response data problems” (e.g., missing/rare selections) and “question design problems” (e.g., wording/format).

4. 25–40 min · Analyse with evidence

• Teacher models one worked example: create a small frequency table, comment on distribution/blank responses, and propose a specific wording revision.
• Students work in pairs on two pilot questions: calculate frequency summaries, identify patterns, and record “issue → evidence → possible revision” statements.

5. 40–52 min · Whole class improvement decisions

• Teacher facilitates a gallery share: each pair posts their issue/evidence/revision and the class votes on which changes are most justified.
• Students contribute by asking “How does this link to the survey aim?” and “Would this reduce misunderstanding or bias?”

6. 52–60 min · Exit ticket: refine plan

• Teacher collects an exit ticket with three prompts and gives a reminder of what will be used next lesson (final questionnaire drafting).
• Students submit: (1) one improvement they recommend, (2) the evidence from pilot responses, (3) a brief justification linked to information needs.

Resources

• Printed pilot response sheet (class set) with selected questions and responses
• Student copy of the questionnaire aim and information needs statement
• Data analysis template: “issue → evidence → revision” table
• Coloured pens/highlighters (or digital annotation)
• Calculator or spreadsheet access (teacher decides based on available devices)
• Display board or projector showing one worked example frequency table
• Dyslexia-friendly text strips version of key instructions (large font, minimal clutter)

Assessment

• Formative check during pair work: teacher circulates, listens for correct use of evidence and clear links to the survey aim.
• Formative check during gallery share: students use respectful questioning and reference specific pilot evidence.
• Exit ticket (quick, individual): verify students can state an improvement, cite evidence, and justify it for the information needs.

Differentiation

• Guided questions for support: provide sentence starters such as “The evidence shows…”, “This suggests…”, “I recommend changing… because…”.
• Reduce cognitive load: assign students fewer questions to analyse (e.g., 1–2 instead of 3–4) while keeping the same quality of justification.
• Dyslexia-friendly reading options: offer large-font printed extracts, or allow audio read-through of the pilot data instructions and question wording.
• Extension for advanced learners (teacher selects): require a second layer—identify whether any issues likely create non-sampling error and propose an additional revision to address it (e.g., reorder options, add clarification, adjust measurement units).