A SAP is usually structured to reflect the protocol but will provide more granularity regarding the statistical methodology. Ideally it includes and addresses the following elements in detail:
Defining primary and secondary objectives is important to avoid 'data dredging'. A hypothesis is the product of deductive reasoning, going from general premises to specific results one would expect if those general premises are indeed true. This usually involves a set of possible relationships between a set of variables. It should be clearly stated how each outcome will be measured. Negative findings may be equally important as positive findings.
Outcome variables based on historical data may involve complex transformations to approximate clinical variables not explicitly measured in the dataset used. These transformations should be discriminated from those made to improve the fit of a statistical model. In either case the rationale should be given. In the latter case this will include which tests of fit will be used and under what conditions a transformation will be used. Next to the outcomes, also the variables used in the study need to be further formalised; formatting (e.g. categorisation, dichotomisation), modifications or derivations with a special attention to time-dependent variables (e.g. age, BMI).
The SAP should make explicit the data source(s) from which the expected variation of relevant quantities and the clinically relevant differences are derived. It should be noted that in observational studies on data that already exist and where no additional data can be collected, sample size is not preclusive and the ethical injunction against 'underpowered' studies has no obvious force provided the results, in particular the 'absence of effect' and 'insufficient evidence', are properly presented and interpreted.
If considered, interim analyses can be beneficial. Criteria, circumstances and possible drawbacks for performing an interim analysis and possible actions (including stopping rules) that can be taken on the basis of such an analysis should be presented.
This section includes a description of the study data sources and linkage methods, inclusion and exclusion criteria, withdrawal/follow-up, baseline patient characteristics and potential confounding variables and analysis population.
This section should describe effect measures and statistical methods used to for each primary and secondary objective; how the achieved patient population will be characterised; handling of confounding and assessing bias; statistical methods to handle missing data; assessing goodness of fit; sensitivity analysis considered.
When false positives are a greater concern, a smaller confidence interval should be considered. Any planned adjustment of the significance level to control for type 1 error that can arise from comparisons across multiple subgroups or analysis of multiple predictors or outcomes (secondary analyses) should be presented.
If decisions are drawn from the study results, a section of the SAP should explain the different outcomes that might be selected for each decision, which statistics influence the decision making process and which values of the statistics will be considered to support each outcome.
Often statistical analyses will employ standard procedures incorporated in statistical packages that provide outputs seen as implicit decision criteria – for instance default p-values (i.e. 5%) or confidence intervals (i.e. 95%). However, different objectives of the study may require lower or higher strength of evidence – for instance, policy recommendations regarding drug licensing may require a lower chance of false positive decisions than the classical one when deciding whether further investigation is needed for a product safety issue. Hence, consideration of decision-making criteria with explicit reference to the type of decision to be made is beneficial.
Further reading on how to draft a SAP tailored to observational studies, see DEBATE-statistical analysis plans for observational studies (BMC Med Res Methodol. 2019;19(1):233) and The value of statistical analysis plans in observational research: defining high-quality research from the start (JAMA. 2012 Aug 22;308(8):773-4).