NATIONAL EXAMINATION OF BRAZILIAN RESIDENTS AND SPECIALIZATION TRAINEES IN RADIOLOGY AND DIAGNOSTIC IMAGING: A TOOL FOR EVALUATING THE QUALIFICATIONS OF FUTURE RADIOLOGISTS

PURPOSE

This is a study of performance based on an In-training Examination for Radiology and Diagnostic Imaging targeting residents (R) and specialization trainees (ST) in Radiology. The radiological training may differ between R and ST in some centers. The authors present their experience and thoughts regarding the first three years of application of the In-training Examination administered by The Brazilian College of Radiology.

METHODS

Three hundred and eight-six tests were analyzed in 1999, 715 in 2000, and 731 in 2001. The yearly tests consisted of multiple-choice answers, some with interpretation of digital images, and were divided into 9 specialties: neurology, thorax, physics, pediatrics, digestive system, urinary system, musculoskeletal system, mammography, and gynecology-obstetrics. Each specialty was analyzed separately. The tests were given simultaneously in 12 Brazilian cities. The subspecialty scores of examinees at different stages of training were compared (1^st, 2^nd, and 3^rd year residents and specialization trainees), by the Kruskal-Wallis test (P<0.001).

RESULTS

The 1999 examination did not reveal any significant difference between the R and ST groups (P>0.05). Generally, in 2000 and 2001, R achieved higher scores than ST (P<0.001). The performance in physics was poor for both groups for the 3 years covered by the study.

CONCLUSION

The performance of residents was better than that of the specialization trainees in the majority of the subspecialties, mainly in the last two years. The In-training Examination provides a system for evaluating future specialists and identifying the centers that need to revise their teaching methods and the regional differences in radiological training.

RESUMO

OBJETIVO

Estudo comparativo entre o desempenho dos residentes e especializandos em radiologia por meio da Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem (PNRERADI), durante os três primeiros anos de sua aplicação. O ensino nos centros de formação em radiodiagnóstico pode diferir entre residentes e especializandos.

MÉTODOS

Foram analisadas 386 provas em 1999, 715 em 2000 e 731 em 2001. As provas foram divididas em nove subespecialidades: neurologia, tórax, digestivo, física, pediatria, urinário, músculo-esquelético, mamografia e ginecologia-obstetrícia, cada uma delas avaliada separadamente, constando de testes de múltipla escolha, algumas com interpretação de imagens digitalizadas. As provas foram aplicadas simultaneamente em 12 cidades distribuídas no território nacional. As subespecialidades foram comparadas nos diversos níveis (residentes e especializandos de 1°, 2° e 3° anos) através do teste não-paramétrico de Kruskal-Wallis (p<0,001).

RESULTADOS

Na prova de 1999 não foi observada diferença significativa entre residentes e especializandos na maioria das áreas. Em 2000 e 2001, no geral, os residentes apresentaram aproveitamento superior aos especializandos. O aproveitamento na área de física foi ruim para residentes e especializandos, durante esses três anos de aplicação da prova.

CONCLUSÕES

O desempenho dos residentes em relação aos especializandos foi melhor na maioria das subespecialidades, principalmente nos dois últimos anos avaliados. A PNRERADI representa um meio de avaliação dos futuros especialistas, identificando os centros que necessitem de revisão do modelo de ensino, além de prover informação da diferença regional no treinamento da radiologia.

INTRODUCTION

Until the early 1970s, radiology progressed relatively slowly compared to some other medical specialties. The advent of ultra-sonography (US), computerized tomography (CT), and magnetic resonance imaging (MRI) provided a marked stimulus for the specialty, broadening the diagnostic tools and the inclusion of radiological procedures in medicine. These new methods renewed, revolutionized, and modified the diagnostic protocols for a significant number of diseases.

Besides the technological progress, medical schools have made an effort to investment in training for these new tools. Otherwise, it would be impossible to provide the technical-scientific information required for the students who would choose radiology as a medical specialty.¹

Medical demand for radiology has been increasing, as well as for the subspecialty of diagnostic imaging. With the increasing expansion of medical knowledge, sub-specialization has become a natural course in medicine based on the available equipment or areas of the human body². The resulting growth in radiological technology led to the establishment of sub-specialties.³

However, the growth of sub-specialization has resulted in a complex structural organization of departments with frequent under-specialization or over-specialization in certain organs and systems or diagnostic techniques.⁴ The expansion of radiological knowledge has been followed by greater reliance on the newer diagnostic techniques and their clinical applications.

Residency training provides adequate assistance for radiology specialty; however, the assessment of this training is not always duly analyzed, either due to theoretical-practical or interpretative and personal parameters. Although the educational training for residents (R) and specialization trainees (ST) should be the same and based in a program accredit by the National Commission of Medical Residence (Portuguese, “Comissão Nacional da Residência Médica – CNRM”), in some centers there are no official teaching programs for ST group.

In 1999, the National Commission of Teaching and Improvement of the Medical Residency of the Brazilian College of Radiology (Colégio Brasileiro de Radiologia - CBR) proposed that the proper way to evaluate R, ST, and radiology centers would be by testing them. This is how the National in-Training Examination for Radiology and Diagnostic Imaging (Portuguese, PNRERADI: “Prova Nacional dos Residentes e Especializandos em Radiologia e Diagnóstico por Imagem”) came into being. The exam is administered yearly during the medical radiological residency, permitting follow-up of the learning progress, and it has since been applied to all Brazilian R and ST groups.

Purpose

A comparative study of the performance of the residents and specialization trainees in the various sub-specialties of radiology was undertaken by the PNRERADI during the first 3 years of its application.

METHODS

Brazilian R and ST were able to register for the annual tests administered in 1999, 2000, and 2001, at 12 national cities.

The multiple choices tests included 9 sub-specialties: neurology, thorax, physics, pediatrics, urinary system, musculoskeletal system, digestive system, mammography, and gynecology-obstetrics. Each exam had 180 questions per year, 20 per sub-specialty, with separate analysis of image interpretation questions. The image interpretation questions were digitized, with most of the questions on the second and third examinations.

Head-neck, US, CT and MRI sub-specialties were not included separately in the study, although questions referring to these techniques and area were included in all tests.

To ensure secrecy, the test seals were broken at the time and place of the examinations, in the presence of the R and ST, by the presidents of the regional CBR societies, who were responsible for execution and delivery of the written tests to the CBR.

All the tests were scored electronically. The 1999 and 2000 grades were computed via a table, in which each group's highest grade was considered equivalent to 100% and the remainders were proportional to the highest grade. In 2001, each test was analyzed based on absolute values and percentages relative to grades at each level.

Every R and ST received his absolute grade and was made aware of his level of training in each sub-specialty and his rank compared to the others at his level. The grades were mailed to each candidate to ensure confidentiality.

The various areas of the sub-specialties (n=9) were compared applying the Kruskal-Wallis non-parametric pos test. The level of rejection of the null hypothesis for all tests was set at 0.05 or 5% and the significant values were marked with an asterisk. The SPSS software (9.0 version) was used for application of the statistical tests.

RESULTS

The results displayed for 1999, 2000, and 2001 tests are in Tables 1 through 9, taking into account the comparative analysis of the examinations among the R and ST at each level and in each area according to the average absolute value of points (Portuguese, “VAMDP”-valor absoluto da média de pontos) and the minimal significant difference (Portuguese, “DMS”-diferença mínima significante).

Table 1.

A comparative analysis of the average correct answers in mammography imaging in 1999-2001 (*P< 0.001)

Area	Mammography 1999 VAMDP (DMS)	Mammography 2000 VAMDP (DMS)	Mammography 2001 VAMDP (DMS)
R1 X ST1	NS 18.79 (56.56)	SS 117.56 (67.07)*	NS 15.60 (73.72)
R2 X ST2	NS 16.74 (66.20)	SS 104.23 (80.69)*	SS 107.98 (76.68)*
R3 X ST3	NS 49.58 (85.12)	NS 93.08 (99.87)	SS 125.01 (95.05)*

SS = statistically significant; NS = not statistically significant

SS increase in the performance of R1 compared to T1 and of the R2 compared to T2 in 2000.

SS increase in performance of R2 compared to T2 and of the R3 compared to T3 in 2001.

Table 2.

A comparative analysis of the average correct answers in pediatric imaging in 1999-2001 (*P< 0.001)

Area	Pediatric 1999 VAMDP (DMS)	Pediatric 2000 VAMDP (DMS)	Pediatric 2001 VAMDP (DMS)
R1 X ST1	NS 26.73 (56.56)	SS 85.10 (67.10)*	SS 159.32 (73.72)*
R2 X ST2	NS 84.04 (66.20)	SS 104.98 (80.57)*	SS 159.38 (76.68)*
R3 X ST3	NS 60.58 (85.12)	SS 136.82 (99.73)*	SS 137.72 (95.05)*

SS = statistically significant; NS = not statistically significant

SS increase in performance by the residents compared to the trainees at the same level in the 2000 and 2001 tests.

Table 3.

Comparative analysis of average correct answers in thorax imaging in 1999-2001 (*P< 0.001)

Area	Thorax 1999 VAMDP (DMS)	Thorax 2000 VAMDP (DMS)	Thorax 2001 VAMDP (DMS)
R1 X ST1	NS 46.54 (56.56)	SS 143.82 (67.10)*	SS 110.63 (73.72)*
R2 X ST2	SS 98.75 (66.20)*	SS 158.06 (80.57)*	SS 148.59 (76.68)*
R3 X ST3	NS 82.50 (85.12)	NS 84.65 (99.73)	SS 133,84 (95.05)*

SS = statistically significant; NS = not statistically significant

SS increase in performance of R2 compared to T2 in 1999.

SS increase in performance of R1 compared to T1 and of the R2 compared to T2 in 2000.

SS increase in performance of residents compared to trainees at the same level in the 2001 tests.

Table 4.

Comparative analysis of average correct answers in musculoskeletal imaging in 1999-2001 (*P< 0.001)

Area	musculoskeletal 1999 VAMDP (DMS)	musculoskeletal 2000 VAMDP (DMS)	musculoskeletal 2001 VAMDP (DMS)
R1 X ST1	NS 29.65 (56.56)	SS 99.35 (67.23)*	SS 123.45 (73.72)*
R2 X ST2	SS 85.79 (66.20)*	SS 133.34 (80.31)*	SS 126.15 (76.68)*
R3 X ST3	NS 63.13 (85.12)	SS 104.44 (99.73)*	SS 148.38 (95.05)*

SS = significant; NS = not statistically significant

SS increase in performance of R2 compared to T2 in 1999.

SS increase in performance of residents compared to trainees at the same level, in 2000 and 2001tests.

Table 5.

Comparative analysis of average correct answers in digestive imaging in 1999-2001 (*P< 0.001)

Area	Digestive System 1999 VAMDP (DMS)	Digestive System 2000 VAMDP (DMS)	Digestive System 2001 VAMDP (DMS)
R1 X ST1	NS 41.53 (56.56)	SS 80.92 (67.27)*	SS 168.21 (73.72)*
R2 X ST2	NS 62.79 (66.20)	SS 125.87 (80.65)*	SS 85.75 (76.68)*
R3 X ST3	NS 46.39 (85.12)	SS 107.54 (100.29)*	SS 133.76 (95.05)*

SS = significant; NS = not statistically significant

A SS increase in performance by R compared to ST of the same level in 2000 and 2001 tests.

Table 6.

Comparative analysis of average correct answers in physics in 2000 (*P< 0.001)

Area	Physics 2000
R1 X ST1	SS 82.75 (65.25)*
R2 X ST2	NS 47.23 (76.00)
R3 X ST3	NS 48.57 (93.46)

SS = statistically significant; NS = not statistically significant

SS increase of performance of R1 compared to ST1 in 2000.

In 1999 (P=0.171) and 2001 (P=0.034), there was NS difference between the average of test grades given to R and ST.

Table 7.

Comparative analysis of average correct answers in neurological imaging in 1999-2001 (*P< 0.001)

Area	Neurological System 1999 VAMDP (DMS)	Neurological System 2000 VAMDP (DMS)	Neurological System 2001 VAMDP (DMS)
R1 X ST1	NS 47.73 (56.18)	SS 110.12 (67.10)*	SS 111.38 (73.79)*
R2 X ST2	SS 95.69 (65.35)*	SS 96.94 (80.57)*	SS 114.64 (76.47)*
R3 X ST3	NS 56.36 (84.02)	NS 58.70 (99.73)	SS 124.29 (94.79)*

SS = statistically significant; NS = not statistically significant

SS difference in performance of R2 compared to ST2 in 1999.

SS increase of performance by R1 compared to ST1 and of the R2 compared to ST2 in 2000.

SS increase in performance by residents compared to trainees of the same level in the 2001 test.

Table 8.

Comparative analysis of average correct answers in gynecology-obstetrics imaging in 1999-2001 (*P< 0.001)

Area	Gynecology-Obstetrics 1999 VAMDP (DMS)	Gynecology-Obstetrics 2000 VAMDP (DMS)	Gynecology-Obstetrics 2001 VAMDP (DMS)
R1 X ST1	NS 9.65 (56.40)	NS 42.91 (67.34)	SS 116.69 (73.72)*
R2 X ST2	SS 91.72 (65.86)*	NS 25.39 (80.08)	SS 106.69 (76.68)*
R3 X ST3	NS 44.78 (84.68)	NS 85.85 (99.45)	NS 93.34 (95.05)

SS = statistically significant; NS = not statistically significant

SS increase in performance of R2 compared to ST2 in 1999.

SS performance of R1 compared to ST1 and of R2 compared to ST2 in 2001.

Table 9.

Comparative analysis of average correct answers in urinary imaging in 1999-2001 (*P< 0.001)

Area	Urinary System 1999 VAMDP (DMS)	Urinary System 2000 VAMDP (DMS)	Urinary System 2001 VAMDP (DMS)
R1 X ST1	NS 35.58 (56.56)	SS 89.78 (67.27)*	SS 131.22 (73.62)*
R2 X ST2	SS 93.06 (66.20)*	SS 135.85 (80.65)*	SS 188.29 (76.75)*
R3 X ST3	NS 48.29 (85.12)	SS 100.74 (100.29)*	SS 185.42 (94.92)*

SS = statistically significant; NS = not statistically significant

SS increase in performance by R2 compared to ST2 in 1999.

SS increase in performance of R compared to ST at the same level in tests applied in 2000 and 2001.

No significant difference in the majority of areas was detected between R and ST in the 1999 PNRERADI (P>0.05). Nevertheless, it should be noted that the number of ST registered was one-third the number of residents. This was the first test applied and few trainees took it. It is not known whether these trainees came from programs that were accredited by the National Commission of Medical Residence (Portuguese, “Comissão Nacional da Residência Médica – CNRM”). This topic is not discussed in this report.

A significant increase in performance by the R was evident in 2000 (P<0.001), particularly in the areas of pediatrics, urology, musculoskeletal, and digestive systems compared to ST of the same level for all three years. In mammography and thorax, the R performed better than the 1^st and 2^nd year ST (P<0.001).

In 2001, a significant increase in performance by R compared to ST of the same level occurred over the 3 years in pediatrics, thorax, neurological, urological, musculoskeletal, and digestive systems (P<0.001). The 1^st and 2^nd years R performed better than ST in gynecology-obstetrics (P<0.001). In mammography, R exhibited a better performance than 2^nd and 3^rd year ST (P<0.001). The superior performance by R, especially in their 2^nd and 3^rd year, compared to ST was confirmed in almost all areas.

No significant difference was observed in physics on the tests grades between the three levels in 1999 (P=0.171) and 2001 (P=0.034). It is important to point out that these results were similar in these years, although the tests were equal for all levels with various degrees of difficulties. Only the ST for the 1^st year did a better performance then 1^st R in 2000 test. The ST group did not achieve a better performance in no other area during the 3 years of testing.

DISCUSSION

The education in some radiological centers could be influenced not just by the length of the medical residency for R and ST, but also by the quality of the teaching. For example, the performance of both R and ST in physics was poor, even though the candidates of the 3^rd year. The low scores in physics could be due to a lack of interest of the radiologists, inadequate preparation, or a combination of both, since the grades in this area remained consistently low at all levels over the 3 years. Lately, the CBR has been promoting physics courses and virtual physics classes for interested radiology centers in an attempt to improve radiological physics for future specialists.

The quality of radiology teaching should be considered. The intention to evaluate doctors trained in radiology residency centers began with the examination taken to obtain the Certification of Specialist in Radiology and Diagnostic Imaging (Portuguese, “Título de Especialista em Radiologia e Diagnóstico por Imagem”–TERDI). In briefing, this title could indicate the quality of the candidate's residency center. The skills which radiology students should acquire during their training were the subject of Koch's thesis in 1987.⁵

In Souza's thesis on the evaluation of radiology residency programs the following mandatory areas were suggested: mammography, conventional radiology, US, CT, MRI, and interventional radiology. Optional areas were also suggested including: densitometry, nuclear medicine, and radiotherapy. Other issues discussed by Souza were: examinations under supervision, manuals of routine protocols, exchanges with residents at other hospitals, priority for scientific events, medical ethics/professional responsibility, and participation in meetings, classes, and case discussions.⁶

Although there is a standard medical residency program in radiology, the experience gained by both R and ST varies among centers and often among students from the same center. The ST groups represent a heterogeneous team. Some radiology residencies accredited by the CNRM offer specialization in Radiology and Diagnostic Imaging (RDI). In most of these centers, the ST activities are similar to those of the R group. However, in some unaccredited centers, radiology training is offered with undetermined teaching methods, sometimes resulting in inadequate specialty training.

Even though radiology residencies were instituted at the end of the 1960's, structured information about medical residencies filed with the CNRM only became available in 1985. FUNDAP (Portuguese, “Fundação Nacional de Apoio e Pesquisa”) data show that from 1985 to 1998, the number of accredited institutions offering medical residency programs grew at a rate of about 51%, principally at the expense of private entities (90%). During this time, medical residency programs in RDI grew by 37%.⁷

Many teaching techniques are employed for radiological residency education. A commonly utilized system is review by assistants of radiological reports issued by residents. A study by Seltzer et al.⁸ found that residents under this study system exhibited improved performance as early as the first 6 months of practice. In addition to improving the residents’ education, it also ensures greater accuracy in the assistants’ final reports. The reports by the second and third year residents were more accurate than those issued by first year residents.

Night shifts are also necessary and are considered an essential educational experience in radiological residencies and are used in the majority of the 166 programs analyzed in the USA.⁹ In the majority of cases, 1^st year residents together with 2^nd year residents are on call during the first 6 months and only after this period are allowed to take over some responsibilities on their own.^10,11

A study correlating the level of discrepancies in interpretation of emergency body CT scans among residents and assistants reported that this rate ranged from 1.2% to 6.5%, for major and minor discrepancies, respectively. Major differences were considered as those likely to affect the patient's therapy and minor ones were those that did not meet this criterion. The residents were more inclined to overlook some details of the image such as incidental findings rather than interpreting normal images as pathological ones. This data suggests good interpretation of emergency images by residents.¹²

The PACS (Picture Assistance Computer System) is an opportunity for radiologists to increase their interaction with other specialists, in addition to providing better diagnostic service.¹³

Many studies evaluated the effect of the introduction of PACS on diagnostic productivity and its effect on teaching during medical residency. Studies revealed an improved in academic activities productivity by teaching/research. Besides a change in the profile of residents who modify the position from active medical participation in reports to observer assistance. In the study by Redfern et al.,¹⁴ the proportion of images interpreted by residents dropped from 38% with film images to 17% when using digital images. Additionally, 45% of the proportion of images interpreted by residents fell to 4% with image digitalization. The effects of this change on teaching are still unknown. Nevertheless, PACS has the advantage of storing a greater number of images for teaching purposes.¹⁴

A study of 92 radiology residency programs approved by the Accreditation Council for Graduate Medical Education has shown that development of research should be an important priority to meet the Council's requirements. About 53% of residency programs require research by residents; however, only 23% of them dedicate specific time for it.¹⁵ It is up to the resident to take advantage of the residency for doing research, giving lectures, honing skills, and finding additional educational opportunities.¹⁶

Interestingly, a study examining why junior academic radiologists leave academia for private practice showed that significant changes should be made to motivate junior radiologists to stay in academia. Academic radiologists should not be expected to read private practice volumes and still have time to produce quality research and teach. In this situation, at least 1 day a week should be protected academic time.¹⁷

The impact of the course by the Armed Forces Institute of Pathology (AFIP/Washington DC, USA) regarding residents’ scores on written tests of the American College of Radiology (ACR) has demonstrated better results by residents who took the ACR test during the AFIP course compared to those who had not yet taken the course. This may affect the residencies that supervise the accomplishment of residents for the American Board of Radiology examination.¹⁸

An educational activity based on case studies was well received by students and met its educational objectives. Activities of this type have potential for widespread application in radiology education. Application of this method in radiology education centers reminded the importance of providing clinical histories when ordering radiological examinations.^19,20

Radiology education is heavily dependent on visual images. Currently, hardcopy film is being replaced by digital radiological images in teaching hospitals and an electronic teaching file library would be desirable.^21,22 The ACR “Index for Radiological Diagnoses” is hierarchically organized and thus an ideal classification system for digital teaching files.²³

The digital revolution in radiology continues to advance rapidly. Whereas the implementation of an institutional radiological digital image database can require significant developmental effort and programming expertise, there are a number of web-based solutions which are freely available and can be relatively easily employed to establish a contemporary electronic image library.^24,25

Radiologists are responsible for providing patients with proper assistance in spite of the demands of their work schedule; for example, imaging specialists should be able to view general radiology studies. There are differences between radiologists working in urban and rural environments, between minor and major services, and between academic and private practice.²⁶

The PNRERADI is not only an annual method for evaluating future radiologist, but is included in a pedagogical training process of continuing education for R, ST, medical residencies, and teaching committees.

The proposal of this evaluation is to guide R and ST through their most difficulties in RDI. Additionally, it should stimulate interest and provide motivation for them to obtain the Certification of Specialist in Radiology and Diagnostic Imaging (TERDI) in Brazil.

This test has been applied since 1999 and the number of R and ST has been increasing over the years. The training of good professionals and their continuous evaluation are fundamental for the best practice of radiology.

CONCLUSION

The performance of the R compared to that of the ST was better for the majority of the radiology specialties, demonstrating that R may receive better overall training, enabling them to obtain better average scores on tests. The PNRERADI provides a system for evaluating future specialists and for identifying the centers that need to revise their teaching methods. It could be used to inform accreditation decisions and to identify regional differences in radiological training, in addition to providing support for improvement of the imaging specialist.