This study aims to describe our experience using ChatGPT to create exam multiple-choice questions (MCQs) and describe the students' opinions.
Material and methodsWe used ChatGPT to create 55 MCQ. Furthermore, we conducted a questionnaire to evaluate students' perceptions.
Results89% of the students considered the questions concise and comprehensible; 91% mentioned language was clear; 76% described it as “simple”.
ConclusionChatGPT has the potential to support teachers in generating MCQ and students´ perception of the test of syntactic comprehension of the exam was positive.
El objetivo de este estudio es describir nuestra experiencia usando ChatGPT para crear un exámen de opción múltiple y describir la opinión de los estudiantes.
Material y métodosUsamos ChatGPT para crear 55 preguntas. Posteriormente, realizamos un cuestionario para evaluar la percepción del estudiante.
Resultados89% de los estudiantes consideraron las preguntas concisas y comprensibles; 91% que el lenguaje era claro y 76% «sencillo».
ConclusiónChatGPT tiene el potencial de apoyar a docentes en la generación de exámenes de opción múltiple y la percepción de los estudiantes sobre la comprensión sintáctica del exámen fue positiva.
The demands of fulfilling teaching responsibilities, along with increasing workloads in healthcare, result in detrimental effects on the labor of healthcare personnel who attempt to carry out these multiple tasks simultaneously, including the augmented risk of burnout syndrome, stress, and reduced job performance.1,2 Therefore, with the increasing workload, it is crucial to acknowledge the clear need to implement or develop tools that can assist healthcare personnel in managing said responsibilities. This is where AI-powered tools in medical education, such as ChatGPT, come into play, as they can be a helpful resource to mitigate some of the negative effects of workload in healthcare academics.
Previous studies have shown promising outcomes of large language models (LLM) like ChatGPT in the creation of exercises and quizzes,3 as well as generating high-quality MCQs that are comparable to those created by expert teachers,4 which are activities that can demand a lot of time from professors. However, it is still important to explore the student's perception of the questions generated by these tools in real settings. The current study aims to describe our experience using ChatGPT-3.5 to create MCQ exams and the students' opinions at a university in Mexico.
MethodsWe used ChatGPT-3.5 to create an MCQ exam of 50 questions for the anatomy and kinesiology course in one group of 33 students of the physical culture and sports degree program. The exam topics were cellular biology and muscular and skeletal systems. Three different commands were used to request ChatGPT-3.5 to create the questions (Table 1). Since the prompts were originally written in Spanish, we relied on ChatGPT-3.5 to translate both the prompts and the MCQs into English. A total of 55 questions were created (Table 2). Furthermore, the professor designed the exam´s final version, and no changes were made to the questions created by ChatGPT-3.5. Two external professors reviewed the exam before its application. In accordance with the Delphi process, we developed a satisfaction questionnaire employing a Likert scale to evaluate the assessment’s redaction, clarity, coherence, and simplicity. Additionally, we inquired about the pretest preparation of the alumni (Table 3), subsequently we administered it using a Google Form. The results of both the exam and the perception survey were analyzed with Google Sheets and Google Forms. None of the students knew that ChatGPT-3.5 was used in exam creation.
Topics included in the exam and prompts given to ChatGPT-3.5 to generate MQCs.
| Topic name | Prompts used to MCQs generation |
|---|---|
| Cellular biology | Imagine you are a professor from the course of anatomy and kinesiology of the degree in physical culture and sports at a university in Mexico. Create 15 MCQ about cellular biology, each question should have 4 options and display me the correct answer. |
| Muscular system | Imagine you are a professor from the course of anatomy and kinesiology of the degree in physical culture and sports at a university in Mexico. Create 20 MCQ about the muscular system, each question should have 4 options and display me the correct answer. |
| Skeletal system | Imagine you are a professor from the course of anatomy and kinesiology of the degree in physical culture and sports at a university in Mexico. Create 20 MCQ about the skeletal system, each question should have 4 options and display me the correct answer. |
MCQ, multiple choice questions.
MCQs generated by ChatGPT-3.5.
| Topic | Questions |
|---|---|
| Celular biology | Which of the following organelles is known as the “control center” of the cell?(a) Ribosomes(b) Nucleus(c) Lysosomes(d) MitochondriaCorrect answer: (b) Nucleus |
| What organelle is responsible for protein synthesis in the cell?(a) Lysosomes(b) Rough endoplasmic reticulum(c) Golgi apparatus(d) VacuolesCorrect answer: (b) Rough endoplasmic reticulum | |
| Which of the following statements about eukaryotic cells is true?(a) They have no defined nucleus.(b) Their genetic material is found in the cytoplasm.(c) They have a nuclear membrane.(d) They lack mitochondria.Correct answer: (c) They have a nuclear membrane | |
| What is the main function of ribosomes in the cell?(a) Storing energy(b) Performing photosynthesis(c) Synthesis of lipids(d) Synthesis of proteinsCorrect answer: (d) Synthesis of proteins | |
| aWhich of the following structures is common in plant cells but not in animal cells?(a) Lysosomes(b) Cell wall(c) Centrioles(d) Smooth endoplasmic reticulumCorrect answer: (b) Cell wall | |
| What organelle is known as the “powerhouse” of the cell?(a) Nucleus(b) Lysosomes(c) Mitochondria(d) Golgi apparatusCorrect answer: (c) Mitochondria | |
| Which of the following organelles is involved in the degradation of cellular waste and recycling of materials?(a) Nucleus(b) Ribosomes(c) Lysosomes(d) MitochondriaCorrect answer: (c) Lysosomes | |
| What is the main function of the Golgi apparatus in the cell?(a) Protein synthesis(b) Protein transport(c) Water storage(d) Nutrient digestionCorrect answer: (b) Protein transport | |
| Which of the following cell structures is responsible for regulating the passage of substances into and out of the cell?(a) Nucleus(b) Cell membrane(c) Ribosomes(d) Rough endoplasmic reticulumCorrect answer: (b) Cell membrane | |
| What function does the cytoplasm serve in the cell?(a) Storing genetic material(b) Carrying out protein synthesis(c) Providing structural support(d) Controlling the passage of substances into the nucleusCorrect answer: (c) Providing structural support | |
| What type of cell lacks a nucleus and other membranous organelles?(a) Eukaryotic cell(b) Plant cell(c) Animal cell(d) Prokaryotic cellCorrect answer: (d) Prokaryotic cell | |
| aWhat organelle is responsible for photosynthesis in plant cells?(a) Mitochondria(b) Rough endoplasmic reticulum(c) Chloroplasts(d) LysosomesCorrect answer: (c) Chloroplasts | |
| What is the main function of peroxisomes in the cell?(a) Protein synthesis(b) Lipid metabolism(c) Substance transport(d) Energy storageCorrect answer: (b) Lipid metabolism | |
| aWhat organelle is responsible for intracellular digestion and degradation of macromolecules?(a) Mitochondria(b) Smooth endoplasmic reticulum(c) Lysosomes(d) RibosomesCorrect answer: (c) Lysosomes | |
| What cell structure regulates vesicle trafficking and protein transport in the cell?(a) Golgi apparatus(b) Nucleus(c) Ribosomes(d) LysosomesCorrect answer: (a) Golgi apparatus | |
| Muscular system | What is the largest muscle in the human body?(a) Biceps(b) Quadriceps(c) Deltoid(d) TrapeziusCorrect answer: (b) Quadriceps |
| Which of the following muscles is responsible for elbow flexion?(a) Brachialis(b) Triceps brachii(c) Deltoid(d) Pectoralis majorCorrect answer: (a) Brachialis | |
| What muscle is located in the back of the thigh and is important for hip extension?(a) Rectus abdominis(b) Gluteus maximus(c) Quadriceps(d) TrapeziusCorrect answer: (b) Gluteus maximus | |
| Which of the following muscles is located in the shoulder region and is responsible for arm abduction?(a) Deltoid(b) Pectoralis major(c) Trapezius(d) Rectus abdominisCorrect answer: (a) Deltoid | |
| aWhat muscle is located in the front of the thigh and is important for hip flexion?(a) Tibialis anterior(b) Biceps femoris(c) Quadriceps(d) GastrocnemiusCorrect answer: (c) Quadriceps | |
| Which of the following muscles is responsible for wrist extension?(a) Triceps brachii(b) Brachialis(c) Deltoid(d) Pectoralis majorCorrect answer: (a) Triceps brachii | |
| What muscle is located in the front of the neck and is responsible for head flexion?(a) Trapezius(b) Sternocleidomastoid(c) Deltoid(d) Pectoralis majorCorrect answer: (b) Sternocleidomastoid | |
| Which of the following muscles is a calf muscle and is used for plantar flexion of the foot?(a) Tibialis anterior(b) Biceps femoris(c) Quadriceps(d) GastrocnemiusCorrect answer: (d) Gastrocnemius | |
| What muscle is located in the back of the arm and is responsible for elbow extension?(a) Brachialis(b) Triceps brachii(c) Deltoid(d) Pectoralis majorCorrect answer: (b) Triceps brachii | |
| Which of the following muscles is located in the chest and is responsible for arm flexion?(a) Deltoid(b) Trapezius(c) Pectoralis major(d) Rectus abdominisCorrect answer: (c) Pectoralis major | |
| What muscle is located in the back of the leg and is responsible for knee flexion?(a) Rectus abdominis(b) Gluteus maximus(c) Quadriceps(d) HamstringsCorrect answer: (d) Hamstrings | |
| Which of the following muscles is located in the abdominal region and is important for trunk stabilization?(a) Brachialis(b) Triceps brachii(c) Deltoid(d) Rectus abdominisCorrect answer: (d) Rectus abdominis | |
| What muscle is located in the back of the shoulder and is responsible for external rotation of the arm?(a) Deltoid(b) Pectoralis major(c) Trapezius(d) InfraspinatusCorrect answer: (d) Infraspinatus | |
| Which of the following muscles is located in the upper arm and is responsible for forearm supination?(a) Pronator teres(b) Biceps brachii(c) Triceps brachii(d) DeltoidCorrect answer: (b) Biceps brachii | |
| What muscle is located in the groin region and is responsible for hip flexion?(a) Gluteus maximus(b) Rectus abdominis(c) Sartorius(d) TrapeziusCorrect answer: (c) Sartorius | |
| Skeletal system | Which of the following bones is part of the vertebral column?(a) Femur(b) Tibia(c) Vertebra(d) HumerusCorrect answer: (c) Vertebra |
| What is the longest bone in the human body?(a) Radius(b) Ulna(c) Femur(d) TibiaCorrect answer: (c) Femur | |
| Which of the following bones forms the back part of the skull?(a) Mandible(b) Scapula(c) Occipital(d) MaxillaCorrect answer: (c) Occipital | |
| Which of the following bones forms the front part of the rib cage?(a) Sternum(b) Humerus(c) Femur(d) RadiusCorrect answer: (a) Sternum | |
| What bone is known as the “arm bone” and is located between the shoulder and the elbow?(a) Humerus(b) Radius(c) Ulna(d) ClavicleCorrect answer: (a) Humerus | |
| aWhich of the following bones forms the shoulder joint?(a) Femur(b) Radius(c) Scapula(d) TibiaCorrect answer: (c) Scapula | |
| What bone forms the lower part of the vertebral column?(a) Cervical vertebra(b) Lumbar vertebra(c) Thoracic vertebra(d) SacrumCorrect answer: (d) Sacrum | |
| Which of the following bones is part of the hand and is located in the wrist?(a) Femur(b) Radius(c) Scapula(d) CarpalsCorrect answer: (d) Carpals | |
| What bone forms the top of the skull and is known as the “cranial vault”?(a) Temporal(b) Parietal(c) Occipital(d) MaxillaCorrect answer: (b) Parietal | |
| Which of the following bones is part of the rib cage and connects to the ribs?(a) Humerus(b) Vertebra(c) Sternum(d) FemurCorrect answer: (c) Sternum | |
| Which of the following bones forms the upper part of the vertebral column and the neck?(a) Cervical vertebra(b) Lumbar vertebra(c) Thoracic vertebra(d) SacrumCorrect answer: (a) Cervical vertebra | |
| Which of the following bones forms the lower part of the leg and is known as the “shinbone”?(a) Femur(b) Radius(c) Tibia(d) UlnaCorrect answer: (c) Tibia | |
| What bone is known as the “calf bone” and is located alongside the tibia in the leg?(a) Femur(b) Radius(c) Tibia(d) FibulaCorrect answer: (d) Fibula | |
| Which of the following bones is part of the hand and forms the fingers?(a) Humerus(b) Radius(c) Ulna(d) PhalangesCorrect answer: (d) Phalanges | |
| Which of the following bones is located at the back of the foot and forms the heel?(a) Calcaneus(b) Astragalus(c) Tibia(d) HumerusCorrect answer: (a) Calcaneus |
Satisfaction questionnaire used for students' perception about the exam.
| Question made | Rating (1–5) |
|---|---|
| Overall, I consider that the exam was well-written. | 1. Strongly disagree2. Disagree3. Neutral4. Agree5. Strongly agree |
| The language used in the exam was clear. | |
| The language used in the exam was confusing. | |
| The language used in the exam was concise. | |
| The language used in the exam was coherent. | |
| The language used in the exam was simple. | |
| The questions of the exam were difficult. | |
| I consider that I studied enough for this exam. |
From the 55 questions created, the professor reviewed and selected 50. The remaining questions were eliminated for being either redundant or outside the focus of the course. Regarding the students' performance, the interval of correct answers was between 17 and 44 points out of 50 possible and a mean score of 26/50. Concerning the quality of the test measured using the Likert scale, 89% of the students considered the questions to be concise and the writing comprehensible; 91% mentioned that the language was clear, while on the other hand, 76% of the students described the language as “simple”. Finally, 40% of participants thought they studied enough, and 24% referred to the questions as difficult. For each command, the time needed for ChatGP-3.5 to create the questions was approximately 10–15 s, whereas the time invested by the teacher for question selection and test final design was close to 60 min.
DiscussionOur results are consistent with those of previous studies, in which ChatGPT demonstrated the capability to support teachers in generating MCQ.5 Also, a probable capability to decrease the time and effort required for exam preparation by the professor was observed. This could contribute to decreasing the workload and thereby positively impacting their performance as educators and healthcare workers. Nonetheless, even though we found benefits using this AI in education, our sample is too small to confirm its true utility, and more studies are needed to determine if a significant difference is observed between exams done by professors and ChatGPT and the professors’ perceptions in the usage of AI in academia and their potential advantages related to diminishing workload.
Overall, according to the satisfaction survey in the present study, a generally favorable perception was observed concerning syntactic comprehension of the exam. Despite the discussion about the potential utilities of ChatGPT in education and the known capability of ChatGPT to generate text, understand context, and be versatile.6 Its acceptance in the classrooms as a tool for students for studying or as a supporting tool for their teachers in the development of class material or assessments continues to be debated. Although, according to Tangadulrat et al.,7 students perceive that the use of ChatGPT has a positive effect on their education. Both the acceptance by students and the documented benefits and capabilities of this LLM could give more confidence and motivation to healthcare academics in the usefulness of this AI as a novel support tool for classes and therefore serve as a beneficial aid to alleviate the workload related to academia tasks. Finally, this positive attitude by students and professors could also facilitate the integration of AI-related subjects into the health curricula of universities.
However, it’s important to note that professors remain responsible for verifying the validity of the questions as ChatGPT can create duplicated questions or “AI hallucinations”.5,8 Also, in further studies, the quality of the questions created by this LLM should be measured with a validated instrument, as well as the accuracy of the prompts used by educators for they will impact the quality, complexity, and ambiguity of the generated questions,9 as seen in the present study.
ConclusionOur results describe promising results that ChatGPT can efficiently generate quality evaluation instruments that are understandable and clear to students, saving time and effort for the evaluator. This suggests that LLMs can be a potential resource to enhance education on both sides while at the same time positively impacting healthcare workers' performance by reducing their workload. However, studies with more rigorous methodology and larger sample sizes are needed to know the true scope of such a tool.
Ethics disclosureThe authors declare that no experiments were performed on humans or animals for this study. The authors declare that no patient data appear in this article. The authors declare that no patient data appear in this article.
FundingThis research was funded by the Department of Medicine and Health Sciences from the University of Sonora, Hermosillo.
