An integrated computer system for teaching Psychology

Abstract

PSICO-A is a computer system for teaching Psychology. It is especially indicated for secondary school students and students in first year of a Psychology degree. It is an innovative system as it is the first integrated computational system designed for teaching educational psychology units. PSICO-A integrates various tools and educational influences: it introduces concept maps, free recovery of memory, an effective mechanism of “feedback”, simulations and digital games and looks into the student's metacognitive abilities. Its computational architecture is modular and integrates an analyzer that helps the teacher to collect data. We have evaluated this system, confirming a significant improvement in a lot of learning variables.

Resumen

PSICO-A es un sistema informático para la enseñanza de la psicología. Está especialmente indicado para estudiantes de psicología de secundaria y del primer curso del grado. Es un sistema innovador porque es el primer sistema computacional integrado para la enseñanza de unidades didácticas de psicología. PSICO-A integra varias herramientas e influencias educativas, introduce mapas conceptuales, recuperación libre del recuerdo, un mecanismo efectivo de “feedback”, simulaciones y juegos digitales e indaga en las capacidades metacognitivas de los estudiantes. Su arquitectura computacional es modular e integra un analizador que ayuda a que el profesor obtenga los datos. Hemos evaluado este sistema, confirmándose una considerable mejora en numerosas variables del aprendizaje.

PSICO-A is a computer system designed for teaching Psychology in Secondary Education and first years of University. It is a pioneering system in this area that combines simulations, digital games, and various educational influences. Taking agent-based systems like MetaTutor (Azevedo, Witherspoon, Chauncey, Burkett, & Fike, 2009), Betty's Brain (Davis, Leelawong, Belynne, & Bodenheimer, 2003) and REAL (Bai, Black, & Vitale, 2007) as a reference, it is a modular design system that introduces games and simulations inspired by Black's (1992) “representational theory”, according to which knowledge is best represented through images (Finke, 1989) and mental models (Gentner & Stevens, 1983) that configure virtual learning environments (Jonassen & Land, 2012). PSICO-A in turn introduces teaching tools of proven value for improving learning like concept maps (Novak, 1977), free memory retrieval (Karpicke & Blunt, 2011), a mechanism of self-generated “feedback” (Slamecka & Graf, 1978), and the emphasis on metacognition (Dunlosky & Metcalfe, 2008) through the ideas of the global metacognitive model of Mayor, Suengas, and González Marqués (1993). In general, we can say that its design is founded on the constructivist paradigm of Ausubel's (1968) learning.

The system has been described in González Marqués and Pelta (2013) and presents to virtual agents being involved in simulations and digital games. While simulations “are structured environments, abstracted from some specific real-life activity, with stated levels and goals” (Aldrich, 2009), computer-based learning games are “applications using the characteristics of video and computer games to create engaging and immersive learning experiences for delivering specified learning goals, outcomes, and experiences” (de Freitas, 2006). Computer games are interactive experiences that are fun to engage while educational simulations usually develop skills and capabilities more rigorously. In PSICO-A simulations and games are based on genuine theories (Hull, 1952) and experiments (Crespi, 1945).

Reflection Mode allows the student to verify the accuracy of their learning. The system asks them a series of questions. The student responses are scored in the Reflection Mode and the adjusted score Analyzer of the system back-end are also reflected in the Learning Assessment Module. This module finds the correct response or no response, also from the “feedback” given to the student in Reflection Mode, rephrasing the question in case of failures. In the Concept Maps Design window (Cañas, 2004), concepts will be distributed on the screen. There are three kinds of conceptual relations in form of arrows: a type of causal relationship, a type of hierarchical relationship, and a descriptive class of relationship (Leelawong & Biswas, 2008). As the student chooses the concepts, the boxes containing the same concepts on the screen are drawn out and the student traces the connections between them using the above types of arrows. The screen Concept Map Editor displays the number of concept boxes and arrows of various kinds that the subject has inserted at the end of the task, specifying the names of ones as well as others. In the central part of the main display area is a “start button” which, when activated, generates above a button “Learning” and the Didactic Unit to study.

By clicking on the button “Learning” from the top center, the Didactic Unit window disappears and “Prior Knowledge”, Notepad”, “Confidence Judgments”, “Metacognitive Judgments”, and “Evaluation” buttons come up. In “Prior Knowledge”, the subject has to state whether she knew anything about the topic before studying it and what (background information). The Notepad is a free recovery exercise of what is studied (Karpicke & Blunt, 2011). This is one of the strengths of PSICO-A: comprehensive training conjugation with retentive learning.

The “Confidence Judgments” button displays a window that asks the subject to enter their degree of confidence (as a percentage) in their learning of the Didactic Unit (Dunlosky & Metcalfe, 2008). The Metacognitive Judgments area consists of 10 questions (Mayor et al., 1993), for which the student must choose and check one of four options that are recorded.

The back-end or interface of the teacher is the heart of the computational system. Its core is a student performance Analyzer that allows the interpretation and verification of their texts given in natural language.

In this article our objective is to determine the effectiveness of PSICO-A on learning versus the classical method based on teacher explanations.

Method

Participants and Materials

The participants were 104 students (average age = 17.6) of Psychology as an optional choice in the second year of A-level of two high schools in Madrid. The curricular material used came from the A-level Psychology textbook (Alonso, 2012). The 33 students in the control group (GC) received regular instruction as in a standard class. Students in the experimental groups (EGI and EGII) were randomly selected from two groups of students of Psychology as an optional choice. With the EGI group (36 students) used the computer learning system of Psychology, PSICO-A, with the difference that instead of using the simulation, the students used the game. Meanwhile the EGII group, consisting of 35 students, also used PSICO-A but accessing the simulation, not the game.

Design

As independent variables, the study included three different combinations of education systems, PSICO-A + Game, PSICO-A + Simulation, and Lectures (it is thus treated, as a multi-group design). The dependent variable was the students’ performance in the following parameters related to learning: confidence judgments, metacognitive judgments, number of concepts generated in Notepad, fidelity of the definitions given to these concepts, number of concepts on the concept map, percentage of correct conceptual relationships and percentage of correct answers in the Reflection Mode. All groups carried out a pre-treatment before starting the experiment (phase 1). This pre-treatment was used as a covariate control analysis of individual differences. Once the two experimental groups were treated using PSICO-A and the control group received the standard class of the same content, the three groups carried out the post-treatment (phase 2). Finally, all were subjected to a monitoring test of what they had learnt in the previous phase (phase 3) consisting of 30 multiple-choice questions, each with three options. Before the completion of the pre-treatment, the students of all three groups received an explanation with questions of part 5.3, Experimental Method, of Unit 1 (Alonso, 2012). All the sessions took place on a day when there was no interference with examinations or tests of other subjects. The meetings were performed in parallel, i.e., the same day, the same time, with two teachers attending the sessions in the computer room.

Results

We have used the SPSS 14.0 statistical program and have accepted a degree of significance of .05. A test of homogeneity variances or Levene's test and an One-Way ANOVA were used to determine if there was a significant difference in the pre-treatment received by the groups. After evaluating the results, there were no significant differences between groups in any of the categories; in other words, they were homogeneous groups regarding their performance on the test.

In the post-treatment and in the corresponding follow-up test, the same parameters as in the pre-treatment were evaluated. In each case, an One-Way ANOVA and a Tukey's HSD test for comparison between groups were applied. Besides, Eta-squared, the most common measure of effect size for One-Way ANOVA, was applied. We now analyze the results by different parameters or categories.

In confidence judgments there was no significant difference between groups. In the post-treatment, the group that expressed greater confidence in their learning was the EGII group, with a percentage of 69.14.

In metacognitive judgments there was simply a marginally significant difference between groups. In the post-treatment test the results of the three groups significantly improved regarding the points scored in the initial test. The EGI group achieved the highest average, with 7.70 out of 10 points, compared to 7.52 of the EGII group and 7.18 of the control group (CG). See Figure 1.

Figure 1.

Metacognitive Judgments (post-treatment).

In the section related to the number of concepts in the Notepad, in the post-treatment, the One-Way ANOVA indicated the existence of a significant difference between groups and Tukey's HSD specified such difference between groups, favoring the two experimental groups compared to the control group. Using eta-squared, 69% of the total variance was accounted for the treatment effect (η² = .69). The difference remained on the following up test (see Table 1).

Table 1.

One-Way and Repeated Measures ANOVAs with Post Hoc Tests for the number of concepts in the Notepad.

	Post-treatment							Follow-up test
	SS	df	MS	F(2, 101)	p	η^2	Tukey's HSD	SS	df	MS	F (2,101)	p	η^2	Tukey's HSD
Between	687.001	2	6.802	22.329	.000	.69	CG	518.177	2	5.130	31.690	.000	.61	CG
Within	303.758	101	151.879					325.169	101	162.584
Total	990.760	103						843.346	103

Note. EGI n = 36, EGII n = 35, CG n = 33.

Regarding the concepts’ definitional loyalty or percentage of correct definitions of the concepts carried out by the individuals in the free memory of the Notepad, the One-Way ANOVA, F(2, 101) = 26.320, p < 000, and Tukey's HSD applied to post-treatment showed significant differences between groups in favor of the EGII group with respect to the EGI group (p < .000) and very significant for these two groups regarding the CG group (p < .000 and p < .002, respectively). The degree of association between effect and the dependent variable was η² = .65. The results were maintained in the follow-up test, F(2, 101) = 25.590, p < .000, in ANOVA and p < .001 for the difference between EGII and EGI in the Tukey's HSD (see Table 2).

Table 2.

One-Way and Repeated Measures ANOVAs with Post Hoc Tests for the fidelity of the definitions given in the Notepad.

	Post-treatment							Follow-up test
	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD
Between	440.947	2	4.366	26.320	.000	.65	CG,EGI	396.477	2	3.926	25.590	.000	.66	CG,EGI
Within	229.813	101	114.906					200.908	101	100.454
Total	670.760	103						597.385	103

Note. EGI n = 36, EGII n = 35, CG n = 33.

In the generation of concepts in the concept map, the post-treatment and the follow-up test continued marking differences in favor of the EGII group, as evidenced by the One-Way ANOVA and Tukey's HSD (see Table 3). Effect size was moderated.

Table 3.

One-Way and Repeated Measures ANOVAs with Post Hoc Tests for the number of concepts generated in the Concept map.

	Post-treatment							Follow-up test
	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD
Between	325.956	2	162.978	47.829	.000	.48	CG,EGI	246.951	2	123.475	53.888	.000	.51	CG,EGI
Within	344.159	101	3.408					231.424	101	2.291
Total	670.115	103						478.375	103

Note. EGI n = 36, EGII n = 35, CG n = 33.

In the percentage of correct relations in the concept map, ANOVA, F(2, 101) = 15.453, p < .000, and Tukey's HSD found significant differences in the post-treatment phase between the groups, especially between the experimental groups and the control group, also surpassing EGII over EGI (p < 010 in Tukey's HSD). In the follow-up test this significant difference was maintained, F(2, 101) = 16.725, p < .000 in the ANOVA and p < .005 for the difference between EGII and EGI in the Tukey's HSD. The correlation between an effect and the dependent variable was low (η² = .23 and η² = .24) (see Table 4).

Table 4.

One-Way and Repeated Measures ANOVAs with Post Hoc Tests for the correct relations in the Concept map.

	Post-treatment							Follow-up test
	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD
Between	8295.359	2	4147.679	15.453	.000	.23	CG,EGI	7554.882	2	3777.441	16.725	.000	.24	CG,EGI
Within	27108.141	101	268.397					22811.733	101	225.859
Total	35403.500	103						30366.615	103

Note. EGI n = 36, EGII n = 35, CG n = 33.

Finally, regarding the percentage of correct answers of the Reflection Mode's multiple-choice question test, in the post-treatment phase significant differences between groups were found, as shown in the One-Way ANOVA, F(2, 101) = 79.056, p < .000 (see Table 5), and a significant difference was located through Tukey's HSD (p < .001 and p < .000, respectively; see Table 5) favoring the EGII group against the groups EGI and CG. Effect size was η² = .61. This difference was also given favoring the group EGI, in comparison to the CG. Such differences were endorsed in the follow-up test, F(2, 101) = 65.109, p < .000 in the ANOVA and p < .000 in Tukey's HSD. Effect size was η² = .56.

Table 5.

One-Way and Repeated Measures ANOVAs with Post Hoc Tests for the Reflection Mode.

	Post-treatment							Follow-up test
	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD	SS	df	MS	F (2, 101)	p	η^2	Tukey's HSD
Between	134.991	2	67.495	79.056	.000	.61	CG,EGI	125.585	2	62.792	65.109	.000	.56	CG,EGI
Within	86.231	101	0.854					97.406	101	0.964
Total	221.221	103						222.990	103

Note. EGI n = 36, EGII n = 35, CG n = 33.

Discussion

Using PSICO-A appears to have improved the overall performance of students in the learning of a Psychology teaching unit against classical education system based on lectures.

First, we must refer to the free remembering test. Karpicke and Blunt (2011) emphasize the value that this type of activity has for learning educational materials by students, even over the use of “prior organizers” like concept maps. It is customary to consider that elaborative type of tasks (creating diagrams, concept maps, etc.) contribute more to student learning than simple memory recovery activities. The key lies in that the first type of tests underpins more meaningful learning of students. However, Karpicke and Blunt show that the tasks of free memory recovery catalyze meaningful learning, even more so than the elaborative learning techniques based on concept maps; that is, the students improve understanding and inferential ability and reinforce learning through fundamental mechanisms in recovery rather than in processing. Therefore, the free recovery processes would also be active learning mechanisms because students have to create an organized structure recovery and discriminate individual concepts from that structure. Accordingly, the practice of free memory activates the tracks to recover a particular block of knowledge, excluding as candidates other possible information units. Rather than increasing the amount of coded features of a text, self-recovery allows us to reconstruct what has been studied by restricting the set of possible informational candidates specified by a given track and defining cognitive search space. Even more, it is a valuable tool to rebuild and promote conceptual learning in science, precisely one of the central activities of the implicit teaching in PSICO-A: the generation and understanding of concepts. Hence the free memory parameter is useful to determine the superiority of PSICO-A versus the classical education system, if connected with the generation of concepts in the free memory test (Notepad). In the largest number of concepts generated in the free memory test the structure of modular design may have influenced, which ensures that students have to fill in the Notepad box after mentioning if they have prior knowledge on the subject. Apart from the temporal proximity with the information just studied, perhaps forcing students to recall whether the issue “reminds them” better actives, mnemonic free memory tracks and, in line with the established by Karpicke and Blunt, generates a better conceptual recovery.

A fidelity definition concept refers to the percentage of correct definitions of concepts carried out by the individuals in the activity of free memory. PSICO-A appeared to have not only an activating effect on the retention of concepts but also in its understanding. The reason can be found, as already mentioned, in its way to enable the free memory tracks to distinguish between the various concepts and their meanings.

As regards the number of concepts in the concept map, Karpicke and Blunt (2011) state that the elaboration of concept maps without having the text to study in front of you (as in PSICO-A) is also a practical example of information recovery. Maybe an automatic construction of concept maps (see Cmap Tools, Briggs et al., 2004) could also be superior over the use of traditional paper and pencil. However, PSICO-A would not change the sign of learning between those groups who use it, since it would not sufficiently strengthen the results of the “disadvantaged” groups. Hence the EGI group, which already generated fewer concepts than EGII in the post-treatment, did not recuperate its disadvantage in the follow-up test. In fact, effect size was moderated generating concepts in the concept map and very low regarding the percentage of correct relations. Pupils did not know how to recognize the three kinds of conceptual relations between concept boxes.

In the task of proper recognition of concepts through the Reflection Mode multiple choice questions, perhaps the best elaboration of the concept map–activity that precedes the activity linked to Reflection Mode–among students of the experimental groups, is that it has been an indication that these students have consolidated the acquired knowledge better than in the case of CG group individuals. But also, members of the CG group performed their own concept map, so the difference is to be explained because PSICO-A provides a flexible “feedback” in the Reflection Mode and an integrated module design which sequences well the exercise of each subtask.

Conclusions

PSICO-A is a computer system designed for the Psychology educational unit teaching to students from secondary education and perhaps for first degree courses at University. It is a system influenced by agent-based systems, like MetaTutor, Betty's Brain, and REAL, and it is a modular design that is inspired by the “representational theory” of Black's knowledge. Its originality is that it combines pedagogical tools that are usually applied to students in an isolated way without being accompanied by an experimental support. Among them, we emphasize the free memory recovery, the use of concept maps, a mechanism of “feedback” that combines the advantages of immediate and delayed response, the importance given to Metacognition, and an active vision of teaching, its own constructivist learning paradigm.

In terms of computational design, PSICO-A also introduces interesting contributions. First, its architecture is modular, with modes that interact and serve to sequence the tasks to be performed by the student. Secondly, the teacher's possibilities to collect and analyze data from pupils are very large, due to having a “back-end” which can store and analyze the information of all the tests. Apart from the desktop version, PSICO-A has been implemented on the Internet, with all the advantages that this entails: increased visibility, greater maneuverability avoiding memory capacity problems and easier to redesign, by using a specific programming language for network design, as PHP5 language (Lerdorf, Tatroe, & MacIntyre, 2006).

We have conducted an internal evaluation of our system, comparing the learning performance of three groups of high school students: one control group received a traditional class and the other two experimental groups handled PSICO-A (subdivided, in turn, in a group that interacted with the system game and one that could only have access to the simulation). PSICO-A proved to be a learning tool that seems to overcome the traditional method of teaching and learning based on the conventional type of class exposing and resolving doubts.

In the immediate future, PSICO-A must be able to improve its interface with the student and has to host a proper monitoring to enhance the metacognitive ability of students. It has to become part of the avant-garde in the current educational design lead by technology platforms, learning communities, or the challenges brought about by the Semantic Web.

Conflict of Interest

The authors of this article declare no conflict of interest.