Use an existing design

It may be possible to save time and work by using a pre-existing set of audit questions and tools, modified if necessary. This also makes it easier to compare results between institutions. Even if an existing design is used, a small pilot study should still be performed.

Design a new audit

Build the frame of reference, which should be based on the most recent, available, relevant documentation, including:

• Regulations: European directives, national laws and decrees
• Professional documentation: Clinical guidelines, consensus conferences, scientific literature, expert opinion.
• International (ISO, EN), national or professional standards

Select criteria

A critical step is to reach agreement on the criteria against which practice is to be audited. Although guidelines for many aspects of transfusion process exist, they are frequently based on inadequate evidence. There may be local barriers – such as individual clinicians’ opinions – to general acceptance of a guideline. The process of negotiating an agreed and measurable objective standard of care that can be endorsed by all stakeholders and used in the audit process can be an extremely valuable means of encouraging clinicians to review their practice (R29 document attached below).

Prospective or retrospective?

Prospective audit is based on the collection of information about patients during their process of care. It permits more reliable and complete clinical data collection since the data required is pre-defined and can be validated and errors corrected while the data collection is in progress. A possible disadvantage is that practice could be altered if staff are aware that they are being observed and data is being collected.

Retrospective audit is generally based on review of records of discharged patients. This may provide information that is more representative of day-to-day practice, but it is more difficult to obtain complete data on every subject in the sample. Retrospective audit may make use of computer databases provide the data they contain is of adequate quality.

Develop the audit criteria

A criterion is a principle or standard by which something may be judged. Audit criteria should describe the aspect of care that is being measured. Explicit means SMART:

• Specific: Unambiguous. Relates to a specific area of care and states specific boundaries
• Measurable: There must be objectively measurable aspects to allow comparison
• Achievable: Must be achievable either within available resources, availability of cases, etc.
• Research based: Wherever possible, there should be sound research evidence that shows the best available treatment or method for the aspect of care being audited
• Timely: Criteria should reflect current practice

Each criterion included in the frame of reference should ideally be derived from the reference guidelines or regulations. It should be designed to identify clearly whether practice conforms or does not conform to the guideline or regulation.

Write the protocol

The protocol should include:

Objectives
• Sources of information used to build the frame of reference
• Definition of the audited hospitals, teams or individuals
• Criteria for inclusion and exclusion (when patients’ files are concerned)
• Type of study (prospective, retrospective)
• Type of data collection (observation by external auditor, self-evaluation, interviews, data collection from patients’ files, from hospital blood bank and from blood transfusion establishment.)
• Description of the role of each person involved.
• Any requirements related to consent, confidentiality or ethical issues

Develop or adapt tools for data collection

Data collection must be simple to perform, valid and reliable.
Data collection forms (often called Case Report Forms or CRF) must have a unique identifier, clearly identify the person who completes the document, and show the date of completion. Questions should require a simple unambiguous response (e.g. YES / NO/ Information not available). Responses such as ”not applicable” should not be permitted. The units in which quantitative results are recorded must be stated e.g. “Hb concentration to be recorded as g/l”. Clinical terms must be defined explicitly (e.g. “bleeding”, “cardiovascular disease”). Questions that require free text response should be avoided or strictly limited as they are difficult to analyse (R24 document attached below).

User manual

For a simple audit the user manual should be short and simple, explain exactly how data collectors should obtain the data items for the audit, how they should enter the information in the case report form or equivalent, and how completed CRFs should be submitted to the audit team.

Statistical support

It is strongly advised to engage the help of a statistician right from the planning stage and throughout the audit to provide expert advice on design of the study including aspects such as sample size, power, sampling methods, development of the analysis plan, conduct of the analysis and presentation of findings. If the results of the audit are to change practice, the results and analysis must be robust and also simple enough for everyone in the care process to understand.

The statistician should generally be a co-author in any report prepared for publication (R23 document attached below).

Sample size

Where a research study will need large numbers of subjects to show which intervention is best, clinical audit only needs to determine the extent to which practice complies with standards or criteria. Smaller sample sizes can often provide the information. The information collected from the audit sample should be representative, i.e. should allow 95% confidence that the results will be within 5% of the results that would be obtained from the relevant population. Sample size calculators are useful when determining an appropriate sample size (R31 document attached below).

Sampling

There are several methods of choosing which cases to include in an audit, including:

Random Sampling: Assumes your audit population will remain the same throughout the audit period and that each subject will have an equal chance of being chosen, either by drawing names out of a hat or choosing every nth subject from a list (e.g. choosing every 3rd or 5th patient).

Interval Sampling: Assumes your audit population will change over the period of the audit. In these circumstances, the audit sample is often determined by a period of time e.g. all patients transfused during May and June.

Stratified Sampling: Is a method used to ensure that the proportions of different groups in the population are reflected in the sample. For example, if investigating donor deferrals, and if male blood donors make up 40% of the donor population, you would ensure that 40% of your sample are male.

Rapid Cycle Sampling: Small data sets are audited to improve and monitor care. This approach can make the change cycle quicker and is useful if a problem is suspected and results are needed quickly. Auditing a small sample can show the nature of the problem. After implementing the action plan for improvement, a repeat the audit on another small sample can quickly show if improvement has been achieved.

Because a poorly chosen sample can skew results and give inaccurate information, the advice is repeated to seek advice from statistician or audit department (R31 document attached below).

Pilot testing

The data collection process should always be pilot tested before the full implementation of the audit. This will often lead to improvements in the data collection forms and improve the final result.