Trustworthiness of field notes in Family Medicine residency training: does field note content provide evidence to support the validity of our decisions about residents’ competence?

Introduction: The Calgary Family Medicine (FM) Residency Program implemented a competency-based curriculum in 2012 (Triple C). To meet the College of Family Physicians’ accreditation standards, the Program also implemented a new assessment program. Field notes (FNs) were introduced to record feedback and to provide data for decisions around Resident competence and progress. Validation of inferences from data collected in field notes is sparse, particularly in relation to how the data can be extrapolated to competence and professional practice. This study investigates the quality and trustworthiness of FNs when their content is used to make decisions about a Resident's competence.

Methods: Assessment data from over 3100 FNs, 99 in-program progress decisions, and scores on the the SOOs and SAMPs components of the College of CFPC Certification Examination in FM were analysed for 16 randomly selected Residents who had successfully completed the Urban FM Residency Program in Calgary under the Triple C Curriculum. Six independent raters (FM community preceptors) were recruited to review copies of the same sets of FNs that were originally used by in-program preceptors to assess Resident progress. 2 independent blinded raters were randomly assigned to each set of FNs. Raters were asked to use FN data to decide on Resident progress, and indicate their level of confidence in their decisions. This was compared with the actual in-program progress decision previously made for each Resident. Quality of FNs was assessed using a Formative Feedback Evaluation Tool (FFET), and were scored 1-5.

Results: The quality of the FN data was found to be sub-optimal (mean 2.27). The consistency of raters’ progress decisions was high (89%). Correlation analyses indicated a significant weak positive relationship between quality of FNs and raters’ confidence, r(196) = .201, p = .005; a significant moderate positive linear relationship between number of FNs and raters’ confidence, r(196) = .30, p < .001; and a significant moderate positive linear relationship between total number of FNs received by a resident and the residents z-scores in the SAMPS component of the CFPC exam, r(14) = .55, p = .026

Conclusion: The results provide evidence supporting the validity of assessment decisions based on Field Note data. The quality and number as well as the quality of the FNs appears important in supporting the trustworthiness of summative progress decisions.