Designing a Pilot Evaluation

#Define what the pilot must learn

A pilot is a limited evaluation used to learn whether and how a proposed activity can be carried out. It may examine recruitment, workflow fit, training or data collection. State its main questions before starting, rather than introducing a system and deciding afterwards which favourable observations count as success.

Distinguish feasibility questions from questions about benefit. Showing that staff can operate a tool does not show that it improves patient outcomes. A small pilot may provide little information about uncommon harms or effects in other settings. The scope of any conclusion should match the design and information collected.

#Choose measures and safeguards in advance

Define the setting, participants, inclusion criteria and activities involved. Select measures that directly address the questions, and explain how they will be collected and interpreted. Include workload, unintended consequences and differences in participation where relevant. Collect enough context to understand whether results depend on unusual support or local conditions.

Establish the appropriate governance route before activity begins. A pilot label does not determine whether work is research or remove requirements for review, permission or consent where applicable. Specify oversight, incident reporting, fallback arrangements and who can pause the evaluation. Participants should understand the activity and its relevant uncertainties.

#Set progression rules and report limitations

Agree what findings would support proceeding, require changes or justify stopping. These progression criteria should reflect safety and practical delivery, not only enthusiasm or usage. The evaluation period and sample need to suit the questions; there is no single size or duration that makes every pilot informative.

Report incomplete participation, missing data, problems and changes made during the pilot. Without a suitable comparison, observed improvements may reflect other changes or differences in participants. Even a successful feasibility evaluation may leave substantial questions about effectiveness, cost and sustainability. Wider deployment requires a separate decision informed by those remaining uncertainties.

#Common misunderstandings

A pilot is not simply a smaller effectiveness study. Its main purpose is to test whether an approach can be delivered and evaluated as intended. Positive feedback, high attendance or improved scores may be encouraging, but they do not, by themselves, show that the approach caused a benefit.

Likewise, finding no serious problems does not establish safety. A small, short pilot may miss uncommon harms, delayed effects or difficulties affecting people who were not included. Participants and staff may also receive extra support that would not be available in routine care.

Meeting recruitment or delivery targets does not automatically mean an approach is ready for wider use. It may instead justify a larger, more rigorous evaluation. Conversely, a missed target need not mean the approach has no value: it may reveal a practical problem that can be addressed. The distinction is between learning how to proceed and proving that something works.

#Questions worth asking a clinician

• What specific feasibility questions will this pilot answer, and how will you measure whether recruitment, delivery, and follow-up are practical?
• What criteria will you set before the pilot to decide whether to stop, adapt, or proceed to a larger evaluation?
• How will you assess participant burden, safety concerns, and whether the pilot includes people representative of those who would use the intervention?
• How will you explain the uncertainty in pilot outcomes so early findings are not mistaken for evidence that the intervention works?
• What additional evidence would you need after this pilot before recommending the intervention for routine care or wider use?