Table 2 Considerations for the “Process phase” of translational simulation. Examples of tools and techniques for data collection and analysis*
Direct observation | Observers selected for expertise; may need training in assessment tools Assessment tools Procedure-specific assessment tools, e.g. arterial blood sampling [37] Global rating scale for procedural skills [37] Teamwork, e.g. Team Emergency Assessment Measure (TEAM) [38] Time-to-event, e.g. time to CT scan for trauma patients [8] Safe design goals observer tool [39] Ethnographic observation [6] |
Monitoring | Video and/or audio recording and streaming Motion tracking [5] Eye movement tracking [40] Other ergonomic assessment tools (e.g. heart rate monitoring, strain measurements) [41] |
Learning conversations | Debrief approaches Rapid cycle deliberate practice [42] (can be modified to improve processes as well as individual performance) Systems-focused Promoting Excellence and Reflective Learning in Simulation (PEARLS) framework [43] SAFEE debriefing tool [44] (based on evidence-based design principles) Pluralist walkthrough [41] with iterative discussions ‘Brainstorm’ sessions [45, 46] (e.g. with participants having the opportunity for quiet reflection and labelling the environment and equipment with sticky notes as a starting point for discussion) Documentation Whiteboards, sticky notes, and photography Technology-enhanced (e.g. TrelloTM [47] as a virtual ‘sticky note board’) Video recording, audio recording and transcription |
Post-event data | Review of video recording, audio recording and transcripts Interviews and focus groups [45, 46] Surveys, e.g., Relational Coordination Survey [6] Artefact analysis (e.g. guidelines, cognitive aids, checklists, debrief reports) [41] |
Analysis | Qualitative analysis of interviews, focus groups, surveys and artefacts Statistical analysis of quantitative data (e.g. time to completion, survey data) Human factors/ergonomics Failure Modes Effect Analysis (FMEA) [15, 39] to risk stratify threats Hierarchical task analysis [41] to understand task steps Cognitive task analysis [41] to understand cognitive processes during tasks Charting techniques [42, 44, 45], e.g. process charts, decision action guidelines Mental workload assessment techniques [41], e.g. NASA Task Load Index Situation awareness measurement techniques [41] Team assessment methods (see also above) Interface analysis [41], e.g. walkthrough analysis Performance time assessment techniques [41], e.g. Critical Path Analysis Design techniques [41], e.g. rapid prototyping, think aloud protocols Quality improvement Gathering information [45, 46], e.g. stakeholder analysis, benchmarking Problem solving [45, 46], e.g. Five Whys Understanding variation [42, 44, 45], e.g. statistical process control Simulation-based Quality Improvement Tool (SQOIT) [15] Incident reporting and root cause analysis [45, 46] (e.g. latent threats identified by ISS) Cost-benefit analysis (60, 62) |