Development of a Pedagogical Content Knowledge test of chemistry language and models

Abstract

In 1986 Shulman developed a typology of teachers’ professional knowledge. Since this, research on teachers’ professional knowledge, especially on the measurement of professional knowledge, has increased. Measuring teachers’ professional knowledge requires tests which focus on specific knowledge types and subjects. However, there are only few professional knowledge tests analysing teachers’ pedagogical content knowledge of using models and chemistry language in chemistry classes. The following article describes the development of a pedagogical content knowledge test for chemistry teachers, which focuses on teachers’ pedagogical content knowledge regarding the handling of models and chemistry language. As a result the test measures the intended construct reliably.

Resumen

En 1986 Shulman desarrolló una tipología del conocimiento profesional de profesores. Desde ese momento, la investigación sobre el mismo se ha incrementado. La medición del conocimiento profesional de profesores requiere de pruebas que se enfoquen sobre tipos y contenidos específicos. Sin embargo, existen solamente unas cuantas pruebas de conocimiento profesional que analicen el conocimiento pedagógico del contenido sobre el uso de modelos y el lenguaje de la química en las clases. El siguiente artículo describe el desarrollo de una prueba sobre conocimiento pedagógico del contenido para profesores de química sobre este tópico en particular. Como resultado, la prueba parece medir el constructo pretendido fidedignamente.

Introduction

Teachers’ professional competence is supposed to have an influence on students’ achievement (Kunter, Kleickmann, Klusmann, and Richter, 2011), and includes for example teachers’ motivation, beliefs and professional knowledge (Baumert and Kunter, 2006). Teachers’ professional knowledge is a substantive precondition for their competent acting in classroom situations. In the last centuries, different national and international studies have already measured and analysed teachers’ professional knowledge (e.g. COACTIV, MT21). In Germany, research on pedagogical content knowledge in chemistry, especially on using models and chemistry language in class, is limited. Analysing teachers’ pedagogical content knowledge focusing on using models and chemistry language, especially in large-scale assessments, requires a valid, reliable, and objective test-instrument.

Teachers’ Professional Knowledge

Shulman (1986, 1987) describes seven types of teachers’ professional knowledge (“curriculum knowledge”, “knowl-edge of educational ends, purposes, and values, and their philosophical and historic grounds”, “general pedagogical knowledge”, “content knowledge”, “pedagogical content knowledge”, “knowledge of learners and their characteristics”, and “knowledge of educational contexts” (Shulman, 1987, p. 8)). Contemporary research mainly focuses on content knowledge (CK), pedagogical knowledge (PK), and pedagogical content knowledge (PCK) (Baumert and Kunter, 2006).

Pedagogical content knowledge is described by Shulman (1987) as a “special amalgam of content and pedagogy that is uniquely the province of teachers, their own special form of professional understanding” (Shulman, 1987, p. 8). At a PCK summit in 2012 a definition of pedagogical content knowledge was devised by a workgroup led by Gess-New-some, Carlson, and Gardner. They defined pedagogical content knowledge as “the knowledge of, reasoning behind, planning for, and enactment of teaching a particular topic in a particular way for a particular reason to particular students for enhanced student outcomes” (Gess-Newsome, 2013; Garritz, 2013). In this article pedagogical content knowledge is defined as the knowledge that enables teachers to structure, link, represent, and explain the content to stu-dents (Schmelzing, Wüsten, Sandmann, and Neuhaus, 2010; Krauss, Neubrand et al., 2008). This includes the knowledge of how to present the content of a subject comprehensibly to learners by using e.g. analogies and demonstrations (Shulman, 1986). In addition, pedagogical content knowledge involves knowledge about students’ conceptions and misconceptions, and how to deal with them (Shulman, 1986; Garritz, 2013).

Shulman (1987) describes pedagogical content knowledge as an amalgam of content knowledge and pedagogical knowledge. Based upon this assumption, a correlation between the two knowledge categories should be expected. The correlation between pedagogical content knowledge and content knowledge of mathematic teachers has been examined by Krauss, Brunner et al. (2008). They concluded that teachers who taught mathematics at Gymnasium¹ (GY) scored higher in content knowledge and pedagogical content knowledge than mathematic teachers of other secondary schools. However, they could not “distinguish the two knowledge categories empirically in the high-expertise group of GY teachers, but that this distinction was clearly visible in the group of NGY² teachers” (p. 724). Differences between types of school could be found in chemistry as well (Tepner and Dollny, 2014). Chemistry teachers teaching at the GY score higher at the content knowledge (CK) test and pedagogical content knowledge (PCK) test than teachers of other secondary schools. However, reported differences in PCK regarding different types of school are smaller if effect of CK variance on PCK variance is controlled by including CK as a covariate (Tepner and Dollny, 2014). Overall, content knowledge seems to be a precondition for developing pedagogical content knowledge (Baumert et al., 2010; Tepner and Dollny, 2014).

The Facets Models and Chemistry Language

Using Models in Class

Models play an important role in the acquirement of knowledge in science and science education (Justi and Gilbert, 2002a). “(…), they function as a bridge between scientific theory and the world-as-experienced (‘reality’)” (Gilbert, 2004, p. 1169). Based on Hodson’s (as cited in Justi and Gilbert, 2002a) purposes for science education (“learning of science”, “learning about science”, “learning to do science”), Justi and Gilbert (2002a, 2003) describe the role of models in science education: Students should know the most important models in science, how they were developed and the limitations of models. They should develop and test their own models and know about the importance of models when scientific findings were disseminated and accepted (Justi and Gilbert, 2002a; 2002b; 2003). In this context Gilbert (2004) speaks of “Learning to Use Models”, “Learning to Revise Models”, and “Learning the Reconstruction of a Model”. These intentions can be found in national and international standards which ask for using models, developing, and testing models and reflecting on models in class (NRC, 1996; KMK, 2005).

Teachers can help students learning about and with models by differentiating between the model and the experience (Mikelskis-Seifert, 2009; Saari and Viiri, 2003). In addition, it is important to discuss the limitations of models (Justi and van Driel, 2005; Saari and Viiri, 2003) and to carve out the change or replacement of models (Maia and Justi, 2009; Mikelskis-Seifert, 2009). It is also necessary for learning to use different models which represent a concept under different aspects or for different purposes (Grosslight, Unger, and Jay, 1991; Harrison and Treagust, 2000; Saari and Viiri, 2003). The colour of a model can lead to students’ mis-conceptions, because of this it is important to discuss the function of the colour in class (Justi and van Driel, 2005). Teachers should involve students in modelling processes, by creating, developing, building, testing, communicating, and reflecting their own models (Gilbert, 2004; Grosslight et al., 1991; Henze, Van Driel, and Verloop, 2007a; Justi and Gilbert, 2003; Maia and Justi, 2009).

In order to do so, it is important for teachers to know how to create learning opportunities which include e.g. adequate teaching models, modelling activities and reflection on models (Gilbert, 2004; Henze, Van Driel, and Verloop, 2007a; Justi and van Driel, 2005). Recent studies indicate a small teachers’ knowledge about models and modelling in science (Henze, Van Driel, and Verloop, 2007b; Justi and Gilbert, 2002a, Justi and van Driel, 2005). Research on teachers’ pedagogical content knowledge on using models and modelling is rare in Germany.

Operationalization of Chemistry Language

Before discussing the importance of language and chemistry language in class, the meaning of language and communication is refected.

Communication is the basis for human interaction. However, defining communication per se is not possible. A linguistic approximation to the construct communication describes it as any form of information processing between humans or data processing machines and humans by using signs and symbols (Bußmann, 2002). Communication comprises the use of body language, non-verbal elements, as well as talking and presenting orally or written in front of an audience. In addition, illustrating and presenting facts with e.g. graphics, as well as intrapersonal (e.g. monolog, affirmation) and interpersonal abilities (e.g. asking, listening, feedback) are part of communication. Transferring information is the basis of every form of communicating (Eunson, 2012). Models of communication describe modelling information transfer. Communicational models always depend on their context of development and scope of application and usually contain a sender and a recipient which are a communicational unit and are related in a certain way (Bußmann, 2002; Eunson, 2012; Grucza, 2012; Kessel and Reimann, 2008).

Language is a characteristic of communication (Eunson, 2012) and is used for exchanging thoughts, beliefs, and knowledge (Bußmann, 2002). From a linguistic point of view, language is the primary medium of communication. Although definitions of language are very different, a general definition describes languages as distinct systems of arbitrary but conventionally conveyed and used signs. Language is central for the exchange of and about knowledge (Bußmann, 2002).

For the exchange on a subject related level, technical languages are used (Grucza, 2012; Rincke, 2007; Roelcke, 2005). Research of different disciplines gives attention to the development and use of technical language as well as finding ways to foster students’ use of technical language (Becker-Mrotzek, 2013; Buhlmann and Fearns, 2000). Although, current research misses a consistent definition of technical language, different approaches emphasize common characteristics of technical language that can be used for forming a definition. The central characteristic of a technical language is its’ functional fixedness: It is used for the effective communication between experts of a specific domain. Technical language is shaped by the use of a subject specific lexis (technical terms) (Fluck, 1996; Grucza, 2012; Rincke, 2007; Wellington and Osborne, 2001) which should be free of everyday connotations (Bußmann, 2002). Besides, technical language is characterized by complex syntactic structures, nominalizations, and compositions (Schmölzer-Eibinger, 2013) as can be found in written communication (Koch and Oesterreicher, 1985, 2007). Based upon these definitions, technical languages are subject and content- specific and are used for communicating about these contents. Especially in class, language is fundamental for teaching and learning (Norris and Phillips, 2003; Wellington and Osborne, 2001; Yore and Treagust, 2006). Language, especially technical language, is deeply connected to the learning in every subject (Merzyn, 2008; Özcan, 2013; Schmölzer-Eibinger, 2013; Sumfleth and Pitton, 1998). Technical language links students’ everyday life and science (Parchmann and Bernholt, 2013). Characteristics of technical languages are e.g. a complex syntax and a formal conception (Koch and Oesterreicher, 2007; Schmölzer-Eibinger, 2013). In contrast to this, subject specific lexis is not generalised across different subjects. In chemistry, learning content is correlated to students’ skills with chemistry language. Poor chemistry language skills are associated with difficulties in learning chemistry (Özcan, 2013). In conclusion, facilitating students’ technical language is desirable for subjects like chemistry and physics (Becker-Mrotzek, 2013; Buhlmann and Fearns, 2000; Busch and Ralle, 2013; Kulgemeyer and Schecker, 2013; Özcan, 2013; Sumfleth, Kobow, Tunali, and Walpuski, 2013; Vollmer and Thürmann, 2013). Furthermore, teachers should use technical language in a sophisticated way, in order to foster students’ use of technical language in class (Linneweber-Lammerskitten, 2013; Schmölzer-Eibinger, 2013; Vollmer and Thürmann, 2013).

Development of the PCK Test

Measuring Teachers’ Professional Knowledge

Several studies use interviews, questionnaires, tests, video analyzes or mixed methods approaches (e.g. Henze, Van Driel, and Verloop, 2007a; Justi and Gilbert, 2003; Krauss, Baumert, and Blum, 2008; MT21; Riese and Reinhold, 2008; Van Driel, de Jong, and Verloop, 2002) to analyze and measure teachers’ professional knowledge.

The subject specific professional knowledge test of Dollny (2011) measures content knowledge (e.g. about the periodic table of elements, acids and bases) and pedagogical content knowledge of chemistry teachers. The content knowledge items are designed in a multiple-choice single-select format. The pedagogical content knowledge items have a closed-item format with a Likert-scale, ranging from 1 (e.g. “very important”, “very meaningful”) to 6 (e.g. “unimportant”, “meaningless”) (Dollny, 2011; Witner and Tepner, 2011). The items were constructed using the “Item Development Model for Assessing Professional Knowledge of Science Teachers” (Tepner et al., 2012) (Fig. 1). The three axes of the model describe the knowledge areas, the PCK-facets and the content. The knowledge axis is divided into procedural, declarative, and conditional knowledge. Conditional knowledge composes the knowledge of when and why an activity or process is executed. Procedural knowledge is the knowledge about how behaviour and processes in class are created. Facts and terms form the declarative knowledge. Originally, the PCK-facets included the knowledge of experiments, students’ conceptions, and models as central elements of PCK (Tepner at al., 2012; cf. Jüttner, Boone, Park, and Neuhaus, 2013; Tepner and Dollny, 2014; Van Driel, de Jong, and Verloop, 2002). Later on, teachers’ handling of chemistry language was added to the facets.

Figure 1.

“Item Development Model for Assessing Professional Knowledge of Science Teachers” adapted of Tepner et al., 2012.

Although the PCK test of Dollny (2011) contains some items referring to conditional and declarative knowledge of models, there were not enough items about models and no items measuring the handling of chemistry language in class. For this reason, new PCK items focusing on using models and chemistry language in class have been developed. The items are included in a new PCK test called FEMo (German short form for chemistry language (Fachsprache), experiments and models) and were developed by the authors. Items which focus on the use of experiments in class are added for the main study and have been developed by Tepner, Backes and Sumfleth (unpublished). Those items are not part of this article.

The PCK items concerning the use of models and chemistry language were evaluated in a pilot study. Based upon an expert-rating a point of reference was developed. Two booklets, each containing 15 items (a set of PCK items of using models and chemistry language) were developed for the pilot study. All items are in a closed item-format, so teachers can rate on a 6-point Likert-scale.

Development of Pedagogical Content Knowledge Model Items

The structure of the pedagogical content knowledge item is in line with the structure of the ones developed by Dollny (2011). Each item has an item stem that describes a classroom situation, e.g. a presentation of a student drawing which represents a model or a model which should be used. Every situation is presented as if the teacher, who fills out the test, helps a novice teacher. An item has four answers which describe a possible reaction or behaviour in this situation. These answer alternatives were developed theoretically and include the research findings and demands how to deal with models in class as described above. The items are subdivided into modelling processes; criticizing models, knowledge of models, and use of models. Items labelled use of models deal—e.g. with the adequacy of a given model for explaining a modelled concept. Items on criticizing models focus on—e.g. how a teacher could create instructions or learning opportunities which deals with e.g. the limitations of a given model (scientific model (Bohr) or real model (sodium chloride cube model)). The items on knowledge of models include e.g. dealing with different models, modelling the same phenomenon or the “upgrade” of models (e.g. Dalton’s atomic theory to Rutherford’s atomic theory).

Initially, more than thirty items were developed while fifteen items were selected for the pilot study. Two of these have no Likert-scale. Teachers have to rank the answers, because one item describes a process and the other one involves the structure of acquiring knowledge by using models. For these two items just one answer is definitely right. The other items are rated on a Likert-scale from 1 (“true”, “very good suitability”, “very important”) to 6 (“not true”, “inadequate suitability”, “not important”).

Development of Pedagogical Content Knowledge Items of using Chemistry Language

The subject specific professional knowledge test of Dollny (2011) implicitly tests the teachers’ knowledge of chemistry language. The knowledge of chemistry language is intrinsically tied to content knowledge (Merzyn, 2008). While knowledge of chemistry language is not a part of pedagogical content knowledge, the knowledge of using and handling of chemistry language is an aspect of pedagogical content knowledge (Riese and Reinhold, 2012). For this reason, it was necessary to develop items for the new test which gather the knowledge of using and handling chemistry language and not the knowledge of chemistry language.

In line with the structure developed by Dollny (2011) is the structure of the items regarding the knowledge of using chemistry language. In the beginning of every item, the situation in class is briefly described. Following the description, a short dialog between a pre-service teacher and one or more students is presented. The dialogs are inspired by videotaped lessons of a previous study in a comparable grade (Pollender, unpublished). Four answer alternatives or statements which focus on teachers’ behaviour in the dialog or possible activities in class are given in the test. Teachers have to rate these answers or statements on a Likert-scale ranging from 1 (“excellent”, “applicable”) to 6 (“inadequately”, “not applicable”). This scale refers to German school marks (1 (“excellent”) to 6 (“inadequately”) and is familiar to teachers.

15 items with four possible activities each were constructed as described above.

The Study

Project Setting

The newly developed PCK test FEMo is part of the project ProwiN (Professional Knowledge in Science (Professionswissen in den Naturwissenschaften) which is founded by the German Ministry of Research and Education. ProwiN is subdivided into two phases. In the first phase, three teachers’ subject specific professional knowledge tests focusing on teachers’ CK and PCK in biology, chemistry, and physics have been developed and evaluated (Dollny, 2011; Jüttner, 2013; Kirschner, 2013). These tests were supplemented by a pedagogical knowledge (PK) test (Thillmann and Wirth, unpublished).

In the second phase, a video study on chemistry teachers’ professional knowledge analyses the relationship between chemistry teachers’ professional knowledge, their behaviour in class, and students’ achievement (Troeger, Struebe, Sumfleth, and Tepner, 2014). Two lessons of the unit atomic structure and periodic table of elements are videotaped in this study (Fig. 2). These lessons will be analysed with coding-manuals focusing on the use of experiments, models, chemistry language, and dealing with students’ conceptions. In a pre/post test design, students’ achievement, interest, and motivation are measured by several paper-pencil-tests (e.g. knowledge about atomic structure and periodic table of the elements). Teachers’ professional knowledge is measured by two paper-pencil-tests (Troeger et al., 2014). At the beginning of the unit, teachers process the professional knowledge test developed by Dollny (2011), and Thillmann and Wirth (unpublished). The FEMo is processed in the end of the unit.

Figure 2.

Design of the video study of the ProwiN project.

Pilot Study and Expert-Rating

In summer 2012 a pilot study and an expert rating were conducted. In the pilot study teachers of an advanced teacher training were asked to fill out the new test. In addition, several inservice teachers did an online survey. The items were distributed via two booklets. Each booklet contained fifteen items. Booklet A contained seven PCK items on models and eight on chemistry language. Booklet B contained seven PCK items on chemistry language and eight items referring to models. Every teacher filled out one booklet (booklet B: N = 23 (14 females, 9 males); age mean value = 36.8; SD = 7.938; booklet A: N = 26 (16 females, 10 males); age mean value = 40.0; SD = 11.79), see also Tab. 1, 2, 3). The testing took 30 minutes. Teachers who used the online survey got a link for one booklet.

Table 1.

Teaching background of the sample (Booklet B).

	M	SD	Min	Max
Years of service	9.38	8.78	1	40
Years of teaching chemistry	9.167	8.89	1	40
Teaching chemistry hours per week	10.0	5.5	0	20

M = Mean value, SD = standard deviation, Min = Minimum, Max = Maximum. Years of service means the years in which a teacher is in service. Years of teaching chemistry could differ from years of service, because teachers can acquire an additional qualification for teaching chemistry in an advanced teacher training.

Table 2.

Teaching background of the sample (Booklet A).

	M	SD	Min	Max
Years of service	13.54	11.26	2	38
Years teaching chemistry	13.92	11.44	2	38
Teaching chemistry hours per week	10.12	5.12	0	18

M = Mean value, SD = standard deviation, Min = Minimum, Max = Maximum. Years of service means the years in which a teacher is in service. Years of teaching chemistry could differ from years of service, because teachers can acquire an additional qualification for teaching chemistry in an advanced teacher training.

Table 3.

Distribution of teachers working at different types of school.

	Realschule	Gymnasium	Comprehensive school
nteachers of booklet A	1	22	3
nteachers of booklet B	2	16	5

Nine German professors of chemistry education at six universities (2 female, 6 males) and one teacher trainer (1 male) conducted the additional expert rating. The two booklets were either sent to the experts or filled out online. The aim of the expert rating is to create a point of reference for appraising teachers’ results. Teachers score if their answers agree with the so-called aggregate expert.

Results

Expert-Rating

Results of the Model Items

For creating a point of reference, the median of classified data for each answer of the model items was calculated with IBM^® SPSS^® Statistic. The median of classified data is used as the so-called aggregate expert of the model items. After this, the four answers of each item were ranked (see Tab. 4 and 5). The rank of an answer describes the position in the order that was done after the ranking. The order was the basis for creating quasi-ranked pairs (Thillmann, 2008, Walpuski et al., 2012) (Tab. 4 and 5). For the fifteen model items fifteen orders and 84 quasi-ranked pairs (relations) were created (booklet A: 36 relations and one item is a true/false item; booklet B: 48 relations).

Table 4.

Order and relations of Item MMp 4.

Answer	MDcd	Rank	Order	Relations
A1	2,29	1	A1 ≤ A2 < A3 ≤ A4	A1 ≤ A2;
A2	2,33	2		A1 < A3; A1 < A4
A3	5,33	3		A2 < A3; A2 < A4;
A4	5,17	4		A3 ≤ A4

MD_cd = Median of classified data. Notice that the quasi-ranked pairs A1≤ A2 and A3 ≤ A4 contain an equal to or less than sign. If teachers rank A1 equal to or less than A2 they get a point.

Table 5.

Order and relations of Item MNM 28.

Answer	MDcd	Rank	Order	Relations
A1	1,50	1	A1 < A4 < A3 < A2	A1 < A2;
A2	4,75	4		A1 < A3; A1 < A4;
A3	4,17	3		A2 > A3; A2 > A4;
A4	3,00	2		A3 > A4

MD_cd = Median of classified data.

Scores for the experts were distributed based on the relations. An expert or test-person will score if he or she estimates the same relation as the aggregate experts (Walpuski et al., 2012) (Tab. 5). If the difference of the median of classified data of two answers is smaller than .50, as shown in Tab. 4, the rank is not distinct. The expert or test person will score, if he or she estimates e.g. A1 lower or the same as A2 on the Likert-scale, but he or she will not score if A1 is rated higher than A2, A3, or A4 (see Tab. 6).

Table 6.

Examples of possible teachers’ ranking and points of item MNM 28.

	A1	A2	A3	A4	Order	Points
Teacher 1	1	5	4	2	A1	6
Teacher 2	2	4	3	1	A4	5

8 relations of booklet A and 8 relations of booklet B show a variance of zero that is why they are not included in the calculations (booklet A: α = .807; n_{relations/oneitem} = 29; n_experts = 9; booklet B: α = .430; n_relations = 40; n_experts = 9). Cronbach’s alpha was calculated by using data, in which missing data were recoded into zero points as it is common for achievement tests.

The frequency distribution of the scores was calculated (see an example in Tab. 7). A relation is not included in the calculations later, e.g. MNM 28 relation 2_3 (Tab. 7), if less than 5 experts score in a relation. At the end booklet A contains 32 relations and booklet B 43 relations.

Table 7.

Frequency distribution of the relations of Item MMp 4 and MNM 28.

Item label and relation	f (1 Point)
Mmp 4 relation 1_2	5
Mmp 4 relation 1_3	9
Mmp 4 relation 1_4	8
Mmp 4 relation 2_3	6
Mmp 4 relation 2_4	8
Mmp 4 relation 3_4	7
MNM 28 relation 1_2	9
MNM 28 relation 1_3	9
MNM 28 relation 1_4	8
MNM 28 relation 2_3	4
MNM 28 relation 2_4	9
MNM 28 relation 3_4	6

FD = Frequency distribution, N_experts = 9. The frequency distribution shows the number of experts who got a point for these relations.

The two items which have no Likert-scale and had to be answered by a ranking were interpreted differently. The item in which the steps of a process have to be ranked has just one right answer based upon the theoretical background. The right answer gets one point. This item is part of booklet A which is why booklet A has 36 relations at the beginning. The point of reference for the other item which involves the structure of acquiring knowledge by using models refers to relations as described above. This item is contained in booklet B that is why booklet B has 48 relations.

Results of the Chemistry Language Items

The expert-rating was used for creating a point of reference for the 60 possible activities (15 items respectively four answer alternatives). For each possible activity, it was analysed if the aggregate expert agrees or refuses the possibility of action. The median of each answer alternative was calculated and used as the aggregated experts’ opinion. On this account, the Likert-scale (1 to 6) was dichotomized (1 to 3 “experts’ agreement”, 4 to 6 “experts’ rejection”). The expert-rating shows a good internal consistency of α = .81.

Pilot Study

Results of the Model Items

32 relations and one item of booklet A are used for teacher’s scoring. 8 relations of this booklet were deleted because of a negative separation effect. One relation had a variance of zero. The mean value score of booklet A is 15.38 (N_teachers = 26; Min = 2, Max = 23, s² = 19.52; SD = 4.42). The reliability of booklet A is satisfying (α = .816, n_{relations/oneitem} = 23; N_teachers = 23 (3 teachers were deleted because of missing data)).

43 relations of booklet B are used for teacher’s scoring. 11 relations of this booklet were deleted because of a negative separation effect. Booklet B has a satisfying Cronbachs alpha (α = .864, n_relations = 32). These calculations are based upon 15 teachers. 8 teachers were excluded because of missing data. The mean value score of booklet B is 22.91 (n_teachers = 23; Min = 12; Max = 31; s² = 27.68; SD = 5.26).

The sum scores of both booklets are distributed normal which was tested by the Kolmogorov-Smirnov-Test.

Even though relations which have a negative separation effect were not included in the calculations, there are relations with a bad separation effect (booklet A: .086 to .601; booklet B: .000 to .607). Relations, which show a separation effect lower than .3 were revised by formulating the alternative answers much more clearly. This means if a relation which has a bad separation effect is A1 < A2, A1 and A2 were revised in a way that the differentiation between the answer alternatives is much more strength. A rating of these answer alternatives should be much more easier which means that one answer should be better than the other one.

Results of the Chemistry Language Items

Teachers’ rated the answers on a Likert-scale (1 to 6), as aforementioned. Afterwards the Likert-scale was dichotomized (1 to 3 “agreement”, 4 to 6 “rejection”). Based upon the mean values of the expert-rating an appraisement of teachers’ rating was done. Teachers’ scoring is distributed normal, which was tested by a Kolmogorov-Smirnov-test. However, neither booklet A (α = .511) nor booklet B (α = .357) have satisfactory reliabilities. The unsatisfactory reliabilities may be accounted for by the small number of items (booklet A: n_item = 32 (2 without any variance), booklet B: n_item = 28 (4 without variance) and a small number of test-persons (booklet A: n_teacher = 26, booklet B: n_teacher = 23) (Wirtz and Caspar, 2002).

In addition to the results of the pilot-study, the results of the expert-rating were taken into account to evaluate the chemistry language items. The items were evaluated by using different criteria originating in the data of the pilot study and the expert-rating. These criteria are the item-separation and the standard deviation of the experts’ answers regarding the answer alternatives. Furthermore, items which have no variance in the pilot study were identified. Assuming that the selected items are not adequate for explaining variance or obtaining a majority for refusing or agreeing to an alternative answer, these items were radically revised. Moreover, qualitative and individual modification proposals of the experts were included in the revision. Statements and answers which were linguistically unclear have been rephrased and items with an unsatisfactory separation value were revised. The dialogs were not changed in the revision process.

In this manner 12 items with 48 answer alternatives were devised. These 12 items, 12 model items and 14 expersiments items were arranged in the new booklet FEMo.

Discussion and Outlook

All in all, pilot study’s results are encouraging. The systematic approach for measuring chemistry teachers’ PCK in terms of dealing with models and chemistry language seems to be helpful and lead to a reliable test instrument that is adequate for large samples. The recent results are used in a constructive manner in order to improve items and answer alternatives. One substantive critical issue could be the small test person sample of each booklet (n_A = 26, n_B = 23) of the pilot study, which was reduced by missing data (Wirtz and Caspar, 2002). Furthermore, the expert sample is very small and consists of nine professors and one teacher trainer, only. A larger expert sample would be better not only for an explicit agreement or rejection of the answer alternatives, but also for distinct relations.

Another problem is the high number of excluded relations of the model items. It reduces scoring for the items in a substantive way. For example only for three relations of item MMp 4 a scoring could be done. This reduces total scoring from 6 to 3 points for this item. In addition, scoring for every item is different. Some items just have a maximum number of 2 points whereas others have a maximum number of points of 6. Nevertheless, analysing model items by using relations is a good alternative to a right and false analysis, because acting in class is influenced by different aspects that is why there is no right or false behaviour in class.

Validity of the FEMo has not been analysed, so far. Vogelsang and Reinhold (2013) emphasize the importance of analysing professional knowledge tests for their ability to measure the knowledge of behaviour in class (validity of action). Expert-ratings and think-aloud interviews can be used for analysing the validity of action, even though these test processes are indirect (Jüttner and Neuhaus, 2013;Vogelsang and Reinhold, 2013). Jüttner and Neuhaus (2013) could analyse the content validity of their test successfully by using think-aloud interviews. Krauss, Baumert, and Blum (2008), and Dollny (2011) used contrast group for validation. According to Dollny (2011) and Jüttner and Neuhaus (2013) it is planned to analyse the validity of the FEMo by interviewing and testing science teachers (biology and physics) and non-science teachers (English, German and/or French) as a contrast group. The teachers will answer the complete FEMo booklet, containing model items, experiment items, and chemistry language items. Biology and physic teachers use models and experiments often in a different way to chemistry teachers. That is why it is interesting to analyse if items focusing on these two facets could differentiate between chemistry specific PCK of using models and experiments in class and PCK of biology and physic teachers. Teachers’ results will be compared to results of chemistry teachers of the main study.

Before starting the validation a new expert-rating will be conducted, because the revision of items could have changed the agreement or rejection of alternative answers. For this purpose it is planned to ask professors of chemistry education to serve as experts, again.

All in all first results of the main study are expected for June 2014 and we are confident of being able to present a reliable and valid PCK test instrument (FEMo) at the end of the study.