Student Outcomes from Innovations in Undergraduate Chemistry Laboratory Learning.

Abstract

Much of the research on implementation of novel curriculum, instruction, and assessment in undergraduate science laboratory courses in the U.S. has been funded by public funding agencies. A review and analysis was conducted of awards made between 2000 and 2008 from the National Science Foundation that have focused on laboratory learning in undergraduate chemistry. The study is concerned with characterizing the types of interventions that occurred, and what was studied and learned related to student learning and associated outcomes. For projects that concluded by August, 2011, an overview is presented of the current ‘state of the art’ of the empirical knowledge base, in the interest of suggesting gaps and opportunities for further study. The findings are also contrasted with a recent summary of the state of chemical education research, largely focused at undergraduate level teaching and learning, that was presented in a review paper commissioned by the National Academies of Science as part of a consensus study on discipline-based education research.

Resumen

Resultados en el estudiante a partir del aprendizaje en el laboratorio de química de la licenciatura. Una revisión de los proyectos subvencionados por la Fundación Nacional de Ciencias de Estados Unidos

Mucha de la investigación acerca de innovación curricular, educación y evaluación de la implementación en la licenciatura de cursos de ciencia con laboratorio ha sido subvencionada por agencias públicas de financiamiento. Se ha realizado la revisión y análisis enfocados a los otorgamientos hechos por la “National Science Foundation” entre 2000 y 2008 dirigidos al aprendizaje de química en la licenciatura. El estudio se dedica a caracterizar los tipos de intervención educativa que se han presentado, lo que fue estudiado y sus resultados relativos al aprendizaje estudiantil y otros logros paralelos. Para los proyectos que se completaron en este periodo de nueve años se presenta una panorámica del ‘estado del arte’ actual de la base empírica de conocimiento, con el interés de detectar brechas y oportunidades para el estudio ulterior. Se contrastan igualmente los hallazgos con un resumen reciente del estado de la investigación educativa de la química, centrado en la enseñanza y el aprendizaje a nivel licenciatura, que fue presentado en un artículo de revisión comisionado por las “National Academies of Science” como parte de un estudio consensual sobre investigación educativa basada en las disciplinas.

Introduction

Chemistry is a laboratory-based science. Thus, learning in the laboratory is critically important to developing knowledge of chemistry, whether a student ultimately seeks to become a professional chemist, to follow a career path that requires some understanding of chemistry, or simply to be a scientifically literate citizen. Laboratory-based courses or course components accompany nearly every chemistry course at the undergraduate level, both for science majors and non-majors. It is, therefore, no surprise that there have been many efforts by faculty in chemistry departments to develop courses and curricula, instructional materials, assessments, and instructional approaches for laboratory learning. Many of these efforts, particularly those for which a faculty member has been awarded external funding, have a concomitant evaluation and assessment to measure intended student outcomes.

Examined and analyzed as a whole, the empirical knowledge base emerging from funded efforts to improve undergraduate laboratory learning should offer guidance about approaches and interventions that are most promising in effecting positive student outcomes, as well as what kinds of student outcomes can result from different kinds of interventions. The present study examines chemistry laboratory education projects recently funded by the U.S. National Science Foundation (NSF) to describe the “state of the art” that is being identified from the projects’ interventions and findings.

Background and literature review

Learning in the laboratory has long had a place in university science education, with a history dating back 200 years in chemistry (Reid & Shah, 2007). As the opening paper (Hofstein & Mamlok-Naaman, 2007) of a special issue of Chemistry Education Research and Practice devoted to laboratory learning in chemistry points out, many researchers have conjectured about its benefits and have studied its outcomes (Hofstein & Lunetta, 1982, 2004; Tobin, 1990; Lazarowitz & Tamir, 1994; Hofstein, 2004; Lunetta et al., 2007). In addressing the purpose of laboratory work in the undergraduate chemistry curriculum, Reid and Shah (2007) point out that there are important tensions to navigate: students learn more when they are engaged and see the relevance of what they are learning, but they may not realize the importance of a particular activity until later in their studies; it is often difficult, however, for instructors to balance the goals of meeting students’ needs with communicating their subject. More recently, in an ACS Award address, and later in a published paper, Abraham (2011) presented a review of three decades of research on laboratory learning. The main findings from this review are: 1) while general chemistry instructors identify concepts as the most important outcome of laboratory learning, the instructional strategies that are used do not align with what is known about how to teach these concepts effectively, 2) students are capable of identifying important differences between verification type laboratory experiments and more inquiry-based approaches, including whether there is an emphasis on laboratory skills and scientific processes, and 3) inquiry-based approaches in the laboratory are more effective in promoting both students’ conceptual understanding and positive attitudes toward science. This latter conclusion is further elaborated in a study by Emenike et al. (2011), which demonstrated that how laboratory experiments are structured affects how students view their learning. In particular, the structure of a lab experiment as classical (verification), discovery-focused (a version of inquiry-based), or focused on understanding the instrument influenced whether cognitive, affective, or psychomotor prior knowledge was seen by the students to be relevant in their own learning.

What laboratory learning actually comprises is not precisely defined, but rather rests in the intentions of the designer of the instructional materials, the availability of resources and other conditions, the role of the instructor as the facilitator of the laboratory learning experience, and the role of the student as the consumer and interpreter of the experience. An answer to the question of whether laboratory learning achieves its intended outcomes depends not only on the experience, but also on the methods of measuring of those outcomes. Thus, the “state of the art” of laboratory learning has many contingencies.

What Faculty Deem Important

Student outcomes that are intended and measured as part of grant-funded research are related to what faculty deem important as goals for laboratory learning. A recent study found identifiable differences between what faculty cite as the goals of laboratory learning, depending on whether the faculty member is currently or has been engaged in externally funded efforts to improve laboratory learning (Bruck, Towns & Bretz, 2010). Bruck et al. interviewed 22 faculty from two-year colleges and four-year colleges, and found that all chemistry faculty who participated in their study shared two goals for undergraduate laboratory learning: mastery of laboratory techniques and skills, and the development of critical thinking skills and experimental design. However, the authors also found that faculty who had received funding differed from faculty who had not received funding (called “regular faculty”) in several ways. For general chemistry, regular faculty cited teamwork skills as an important goal, while grant-funded faculty instead cited connecting lecture and lab. For organic chemistry, both groups of faculty were in agreement on three goals: techniques and lab skills, critical thinking skills, and written communication skills. However, for upper division courses (e.g., analytical chemistry, physical chemistry), there was no common ground: regular faculty focused on laboratory techniques and skills, while grant-funded faculty focused on experimental design and uncertainty in measurement.

In comparison, a consensus study, entitled America's Lab Report, identified in the research literature seven goals for laboratory learning at secondary (grades 9-12) levels (National Research Council [NRC], 2005, p. 3): enhancing mastery of subject matter; developing scientific reasoning; understanding the complexity and ambiguity of empirical work; developing practical skills; understanding the nature of science; cultivating interest in science and interest in learning science; and developing teamwork abilities.

Table 1 presents a comparison of the goals for laboratory learning from the two studies.

Table 1.

Comparison Between Bruck et al. (2010) and America's Lab Report (NRC, 2005) on Goals for Laboratory Learning.

Goal	Post-Secondary Chemistry	Faculty (Bruck et al., 2010)	Research Literature on High School Laboratory (NRC, 2005)
	Regular Faculty	Grant-Funded Faculty
Content knowledge		Connecting lecture and lab^§	Enhancing mastery of subject matter
Scientific reasoning	Critical thinking skills^*	Critical thinking skills	Developing scientific reasoning
Complexity and ambiguity in empirical work		Uncertainty in measurement	Understanding complexity and ambiguity
Practical skills	Techniques and lab skills	Techniques and lab skills^†	Developing practical skills
Nature of science		Experimental design	Understanding the nature of science
Interest	Engagement in science^§	Engagement in science^§	Cultivating interest
Teamwork Communication	Teamwork skills^§ Written communication skills^*	Written communication skills^*	Developing teamwork abilities

*

Goal only for organic chemistry.

†

Goal of grant-funded faculty for general and organic chemistry, but not for upper division courses.

§

Goal only for general chemistry.

The Laboratory Activity: What is Inquiry?

Congruent with Abraham's findings (2011), there is general consensus that laboratory activities that engage students in some inquiry-based activity are more conducive to learning than more traditional verification-type laboratory activities. Hofstein et al. (2005) found that students exposed to more inquiry-based laboratory learning were more capable of asking more and better questions for future laboratory investigations. The views of students about the purpose of laboratory are connected to how learning in the laboratory is structured. In a study in which 13 students were interviewed at length about what they viewed to be the purpose of undergraduate laboratory learning experiences in chemistry and their relation to lecture, Russell and Weaver (2008) found that students did not view there to be a connection, particularly to understanding the theory presented in lecture, but rather they perceived that the purpose of the laboratory was to complete the procedures. This conclusion contributed to the premise of continued work by these researchers: “students often fail to make the technical gains that should be possible during their laboratory work… due to a disconnect between students and instructors in terms of laboratory goals and purposes” (Russell & Weaver, 2011, p. 57). In their continued work on this problem, Russell and Weaver (2011) studied student outcomes associated with understanding the nature of science — “what science is, how science is done, and who does it” (p. 58) — when students were engaged in traditional (verification, following established procedures) vs. inquiry-based (students invent questions and design experiments) vs. research-based (integration of authentic science practice, students contribute to generating new scientific knowledge) laboratory curricula. In this study across courses at five universities, they found that students in the research-based labs developed the deepest understanding of the nature of science, and students in the traditional labs developed the shallowest understanding.

There is a long history of debate on what “inquiry” means. Russell and Weaver took a definition of inquiry according to Abraham and Pavelich (1999) in their study, but measures of how inquiry-based a laboratory activity or course is are important if comparisons are to be made across studies. In response to the need to make comparisons among different interventions in undergraduate science laboratory courses, Bruck et al. (2008) developed a rubric to classify the level of inquiry based on characteristics of the laboratory activity and how much independence students have to make decisions about what to do.

Towns and Kraft (2011) recently conducted a review of ten years of empirical research on teaching and learning in chemistry. They included 29 research studies that focused on the laboratory environment. In the review, they conclude that the “body of research doesn’t point towards one incontrovertible approach to laboratory” (p. 9). However, they call out two structural approaches to laboratory activities that have been shown to result in positive student outcomes: problem-based learning (PBL) formats, and supporting and developing inquiry skills in the laboratory.

Which Outcomes Are Measured

The desire to implement a pedagogical intervention is always preceded by a belief that its implementation will result in positive student outcomes. If the belief can be unpacked, then it informs the determination of outcomes that should be affected by the intervention. For example, if one believes that a critical feature of being a scientist is to be able to design questions that have the potential to advance society in responsible ways, then an intervention might be designed that incorporates values for social responsibility and opportunities to strengthen the ability to pose questions, and outcomes to be measured might include the extent to which the desire to pursue socially responsible science is sharpened (affective), and the ability to ask relevant questions is enhanced (cognitive). The act of developing a proposal for funding to improve laboratory learning requires articulation of a set of purposes for the laboratory, and these reflect assumptions about what is important in laboratory learning.

Reid and Shah (2007) provide an historical review of the aims of laboratory learning in undergraduate chemistry, and synthesize the body of literature to offer a set of four aims for laboratory work:

• •

  Skills relating to learning chemistry: Making chemistry real, illustrating ideas and concepts, exposing theoretical ideas to empirical testing, and teaching new chemistry.

• •

  Practical skills: Learning to handle equipment and chemicals, practicing safety procedures, mastering specific techniques, measuring accurately, observing carefully.

• •

  Scientific skills: Learning how to observe, deduce, and interpret, appreciating the place of empirical study as a source of evidence, learning how to design experiments that offer genuine insight into chemical phenomena.

• •

  General skills: Working collaboratively, reporting, presenting, discussing, managing time, approaching problems.

They point out that not all of these can be achieved easily by experiments that are typically included in undergraduate chemistry laboratory courses or components. For example, the idea of confirming theory was seen strongly in 19^th century chemistry education and is still very present in undergraduate chemistry education today, but it stands in contrast to exposing theoretical ideas to empirical testing.

There is a variety of ways of organizing types of student outcomes that can result from an effort that deliberately targets specific purposes for the laboratory. One approach to examining student outcomes is to recognize that they must exist within the larger scope of what matters in laboratory learning, and then measure broadly and examine what changes occur. Montagut et al., (2002) have devised an organizational scheme for measuring learning in the chemistry laboratory. It includes five aspects: 1) the quality and appropriateness of the instructional materials, 2) the performance of the instructor, 3) the attitudes and motivation of the student, 4) the competencies achieved by the student in conducting laboratory work, and 5) the quality of the work produced by the student. However, when measuring broadly, there is the danger of missing observing a phenomenon whose grain size is small.

When measuring specifically, there is the attendant difficulty that the act of measuring may cause learning that may not otherwise have occurred. To wit, students learn what the instructor deems valuable to learn by what the instructor assesses. Ramírez et al. (2010) identify this circular argument, and assert that assessments can act to promote particular competencies. In particular, in analyzing basic assessments for seven undergraduate chemistry courses at one university in Argentina, these researchers found that when assessment activities were classified into three categories — 1) memorization and calculation, 2) management of theories and concepts, and 3) integration of conceptual, methodological, and information management — most (77%) of activities fell under the second category, the remainder fell in the first category, and none fell in the third. A similar study in biology by Momsen et al. (2010) analyzed assessments used in 77 introductory biology courses at 44 higher education institutions in the U.S. Through a Bloom's taxonomy analysis of all assessment questions, they found that the vast majority of assessment items (93%) target the lowest two of Bloom's six cognitive levels, and that this holds true regardless of the cognitive level of the course or the type of institution.

How Students Receive Laboratory Learning

In addition to cognitive outcomes of laboratory, there are often affective (i.e., emotion-related) outcomes. These refer, generally, to fostering affective responses through the laboratory that will spill over to the course, to the discipline, or to science more generally. There are three affective outcomes that are frequently identified in the field: attitude, interest, and motivation.

Attitudes. Attitudes are the positive or negative evaluations of an object and the related beliefs about that object. The “object” can be an idea (e.g., science), a process (e.g., experimentation), or a specific item (e.g., the science textbook). The object in this case is usually personally relevant, and the individual experiences some sort of positive emotional experience — this experience is not necessarily enjoyment or entertainment, per se, but can be positive through intellectual challenge. Attitudes are relatively stable, but can be changed through interactions with other people or other objects, especially through an emotional stimulus (McGuire, Lindzey, & Aronson, 1985). Until the work of Thurstone (1928), many psychologists considered it impractical or impossible to measure attitudes reliably. More recent advances in psychometrics and assessment have advanced this field further (cf.Andrich, 1978; Roberts, Laughlin, & Wedell, 1999). In STEM education research, “attitude” is often used to relate to the individual's perception of the discipline (e.g., chemistry) or the field (e.g., science; Reid, 2011).

Interest. Interest is willingness to engage in activities with an object. Interest is believed to involve not only a relationship between the individual and the object, but also about the context in which the individual encounters the object (Krapp, 1999). As with attitudes, interest is an internal state that had been considered difficult to measure directly. Compared to attitude, interest is considered less stable and more influenced by immediate context. Krapp (1999; 2004) provides further review of research on interest in educational psychology.

Motivation. Motivation is a desire to continue with an activity, and is typically related to behaviors that derive, at least in part, from an individual's interest in the object itself or in outcomes that derive from the interaction with the object (Weiner, 2010). Motivation research has distinguished two aspects of motivation: intrinsic and extrinsic. Intrinsic motivation is based on one's interests in a topic or to achieve enjoyment, whereas extrinsic motivation pertains to achieving outcomes (cf.Vallerand & Ratelle, 2004). Compared to interest, motivation is more context dependent, meaning that it is dependent on the individuals’ interaction with the object or process for a specific time and in a specific place. Motivation is typically inferred from behavior, such as choice of tasks, effort expended while engaged in such tasks, persistence in these tasks, and associated verbal comments (Pintrich & Schunk, 2002).

How Outcomes Are Measured

A variety of instruments exist for measuring outcomes specific to chemistry, and some are even more specific to laboratory learning in chemistry. In particular, there are two papers that provide overarching principles about evaluating student outcomes in undergraduate chemistry courses. For chemistry courses in which one wishes to evaluate learning in a comprehensive manner, Viera et al. (2007) present dimensions for evaluation based on a constructivist paradigm. For more specific questions probing why particular outcomes occur, a collection of chemical education researchers in the U.S. recently collaborated to provide some guiding principles about assessment of student outcomes in chemistry, and to profile several instruments that can be used to assess the outcomes of curricular reform in chemistry (Holme et al., 2010). The authors argue that, used in conjunction and within the bounds of their validity, these instruments are capable of measuring student problem solving, metacognition, and cognitive development, as well as affective aspects of learning.

Objective of this study

A large fraction of the research and evaluation of innovative approaches to laboratory learning has been funded by public funding agencies. The present study includes an examination of all awards in six programs related to undergraduate chemistry laboratory learning that were awarded from 2000 through 2008 by the National Science Foundation, the federal agency in the U.S. that funds the vast majority of course, curriculum, and laboratory improvements and innovations at the undergraduate level. The research questions under study are: 1) What types of interventions are being funded? 2) What are the intended student outcomes? 3) What are the effects of the interventions on student outcomes? This paper reports on the first two questions for all projects initially funded between 2000 and 2008, and concluded by 2011.

Methods

Data Sources

This study examined undergraduate chemistry laboratoryfocused education projects supported by the U.S. National Science Foundation (NSF) that were funded and completed between the years of 2000 and 2011. The pool of projects was identified by conducting a search of the database of abstracts of NSF-funded projects, which is publicly available at the following URL: http://www.nsf.gov/awardsearch/. One of the options is to “search all fields,” which allows specification of multiple search terms. Four specifications were used: 1) the text string “chemistry” was present in the title or abstract, 2) the award was made between August 1, 2000, and August 1, 2011, 3) only expired awards were included, and 4) projects from six grant programs were included by specifying the program element codes, listed with their respective programs. The six programs included were Course, Curriculum and Laboratory Improvement (CCLI) program Type 1 projects (program element code 7494); CCLI Type 2 projects (code 7492); CCLI Type 3 projects (code 7493); Transforming Undergraduate Education in STEM (TUES) Type 1 projects (code 7513); TUES Type 2 projects (code 7511); and TUES Type 3 projects (code 7512). These programs were included because they focus explicitly on varied implementation and educational research in undergraduate education. The type of project reflects the level of grant support. Type 1 projects are the smallest awards, and typically include developing instructional materials for a particular course or set of courses, piloting a pedagogical or professional development approach with faculty, or integrating new instrumentation or equipment into undergraduate laboratories in a way that is designed to improve student learning. Type 2 projects address several components in a cycle of developing and studying an intervention, perhaps working toward systemic change in science departments at an institution, or scaling an intervention and studying it across several institutions. Type 3 projects support large-scale efforts, building on a significant evidence base and scaling the intervention and study broadly, e.g., nationally.

The original search netted a pool of 99 expired awards. Eliminating duplicates, which occur when separate awards are made to different institutions collaborating on the same project, the number of unique projects was 91. The abstracts of these awards were then read and coded to identify those that focused on undergraduate laboratory learning in chemistry. Projects were excluded if they did not focus explicitly on student learning in the chemistry laboratory. Out of the initial pool of 91 unique projects, 57 projects (or 63%) were found that focused on laboratory learning in chemistry.

Coding Scheme

For each of the 57 undergraduate chemistry laboratory projects, the award documents and any available publications were reviewed and coded. Additionally, other public documents (e.g., Ph.D. theses, conference presentations) were reviewed and coded, if available. All of the documents were coded using an open-coding strategy (Patton, 2002) to capture the variety of interventions and the intended student outcomes. The coding focused on the following areas: the nature of the intervention; the project's focal audience and/or courses; the student learning outcomes identified by the PIs; and, where possible, the project's findings about the effects of the interventions on student outcomes. The coding scheme is summarized in Table 2.

Table 2.

Categories used in coding the proposals, project reports, and published papers.

Category	Code	Information Coded For
Type of intervention	IM-exp	Instructional Materials Development
		• Development of instructional materials for new laboratory experiments
		• Development of a new laboratory course
		• Intervention across the entire curriculum (e.g., use of NMR in all laboratory courses)
	Curric	Curricular Approach
		• Integration of lecture and lab
		• Development of a curricular focus (e.g., sprout growth)
	Instruc	Instructional Approach
		• Comparison of a particular pedagogical approach across various settings
		• Design and testing of new instructional strategies in the lab
	Resource	Resource Development
		• Development of online materials (e.g., virtual labs)
		• Development of a remote instrumentation network
	Research	Research About Instruction in the Lab
		• Development of assessment measures for the laboratory
		• Study of how instruction occurs or goals of instruction in the lab
Courses	Course name(s)	• Specific courses (general chemistry for science majors, general chemistry for nonmajors, organic chemistry, analytical chemistry, etc.)
	Multi	• Multiple courses
	Across	• Across the entire undergraduate chemistry curriculum
	Interdept	• Interdepartmental (e.g., chemistry and art history, horticulture)
Audience	UGs	• Undergraduate students
	TAs	• Graduate students (e.g., teaching assistants)
	Fac	• Faculty
Cognitive outcomes	COG1	• Knowing and comprehending (e.g., being able to explain how a GC/MS instrument works)
	COG2	• Analyzing and applying (e.g., being able to interpret an NMR spectrograph and apply analysis to determining whether a compound belongs to a class of structures)
	COG3	• Evaluating and creating new knowledge or approaches (e.g., being able to design an experiment to determine something not investigated in the laboratory course)
Affective outcomes (includes identification of referent)	Att	• Attitudes toward
	Motiv	• Motivation, engagement in the moment, enthusiasm for
	Percep	• Perceptions of (such as satisfaction with own work in lab)
	Int	• Interests in (looking toward the future)
		Referents: learning, course, discipline, science, career, research
Skills and Activities	URes	• Undergraduate research
	Writing	• Scientific writing ability
	Tchnq	• Technique in using instrument
	Vis	• Spatial ability
	Team	• Teamwork

Type of Intervention. Determination of the type of intervention relied primarily on the answer to the question “for whom?” If the intervention involved the development of instructional materials for students in a course or a program, then the project was coded as IM-exp (see Table 2). If the focus was, instead, to develop a curricular approach that could be (or was being) tested in multiple venues, then the project was coded as Curric. Examples of curricular approaches were the development of The Molecules of Life curriculum (Jordan & Kallenbach, n.d.) that integrates interactive pedagogies, web-based molecular modeling exercises, and inquiry-based laboratory experiments, and a focus on using the growth of sprouts as an organizing feature of the entire chemistry curriculum. Instructional interventions (coded Instruc) included developing and testing pedagogical approaches in the laboratory, such as following the Science Writing Heuristic approach. Resource development included the development of remote instrumentation networks and virtual labs. A final category of intervention was a research study centered on how instruction occurs in the lab.

Courses and Audience. Interventions were targeted either at particular courses (e.g., general chemistry for science majors, analytical chemistry), at a set of courses (e.g., bioanalytical chemistry, toxicology, and environmental chemistry), across the entire undergraduate chemistry curriculum, or across two or more disciplines (e.g., chemistry and art history, chemistry and plant biology). While most interventions targeted undergraduate students (UGs) as the primary target audience for the intervention, some also targeted graduate student teaching assistants (TAs) and/or faculty (Fac).

The four aims of undergraduate chemistry laboratory learning of Reid and Shah (2007) were encompassed in two of the three sets of outcomes coded in this study. In particular, ‘skills relating to learning chemistry’ were collected under cognitive outcomes of the laboratory, while the other three sets of skills are represented under a broader set of skill outcomes. A third set of outcomes recognizes the affective domain.

Cognitive Outcomes. The grouping of cognitive outcomes was informed by the study of Momsen et al. (2010) to code undergraduate biology assessments according to Bloom's taxonomy (Bloom, 1956). Coding in the present study used a rougher level grouping, as assessment items generally were not provided in the documents that were examined, so determination of cognitive outcomes was made based on descriptions of anticipated student outcomes. COG1 encompasses the first two Bloom taxonomy levels, knowing and comprehending. Examples of this include being able to explain how a mass spectrometer works, assigning peaks in an NMR spectrum to particular features of molecules, naming molecules by IUPAC conventions, and relating a 3D molecular structure to its Fisher projection. COG2 encompasses the middle two Bloom taxonomy levels, analyzing and applying. Examples of this include analyzing a data set that is provided, interpreting an NMR spectrum and figuring out whether a compound belongs to a particular class of structures, or determining the rate of a reaction from an energy profile graph. COG3 encompasses the highest two Bloom taxonomy levels, evaluating and creating new knowledge or approaches. Examples of this involve applying what has been learned to novel situations, rather than repeating what was done in the laboratory, for example, designing an experiment to determine something that was not investigated in the laboratory, or critiquing the analysis in a research paper.

Skill and Action Outcomes. Codes initially created mirrored the list of all of the specific laboratory skills identified by Reid and Shah (2007), and then were collapsed into collective codes when tendencies were noticed across projects to group related skills. The remaining collective codes that described all projects that included skills as intended aims were: involvement in (or deepening of) undergraduate research, scientific writing ability, technique in using instruments (e.g., GC/MS, NMR), spatial or visualization ability, and teamwork.

Affective Outcomes: Affective outcomes for the projects were coded according to the distinctions among attitudes, interests, and motivations, as discussed in the literature review above. Projects that referred to affective outcomes that did not use these terms were coded according to the above-mentioned distinctions: attitudes related to positive or negative evaluations; interests related to willingness to engage; motivation referred to behaviors demonstrating engagement. As an example, projects that referred to “satisfaction with” or “perception of” a laboratory or course were coded as attitude. Projects that explicitly used a term, such as “interest,” were coded into the respective category in most cases.

Coding Process

Initially, the public abstracts of the projects were coded, to determine what the intervention was, and to assess, if possible, the intended student outcomes. Based on this initial coding, projects were grouped into the five types of interventions. Other documents for each project were then examined to verify the intervention type. In a few cases, the classification of type of intervention was modified. For example, several projects that were initially classified as instructional materials development were reclassified as curricular approaches when it was recognized that the project was designed around a particular approach to laboratory learning, such as integrating laboratory and lecture, and the assessment of the project involved looking across the entire intervention to determine the extent and quality of this approach.

The materials on each project were then searched to identify intended student outcomes and methods for their measurement. In some cases, assessment instruments were provided in appendices, and in other cases they were described. Some projects reported using published instruments and provided citations. In many cases, documents described the process by which assessments were developed and the expertise of the individuals who contributed to the development, so it was possible to deduce the intended student outcomes that were measured.

Information about every project was sought in publicly available journal articles and other published work. For all 57 projects, searches were run in four journals that are the most common publishing venues of such projects (Journal of Chemical Education, Chemistry Education Research and Practice, Journal of College Science Teaching, and Council on Undergraduate Research Quarterly) to locate published work by the PIs that reported on the projects. NSF's project report template requires including citations of journal articles, which are subsequently automatically drawn from the project reports and appended to the public abstracts available on the NSF award database. Where this occurred, these papers were also included in analysis. The search for papers also included examining the publication sections of CVs of PIs that were available on many faculty members’ own websites. In some cases, this resulted in locating other publications that reported on the work of the projects. For example, for one Type 2 project in which the evaluator was a PI, the Ph.D. thesis of a student in the research group that PI was publicly available (Jiang, 2008), making it possible to learn much more about the intended student outcomes of the project and the methods used to measure them. For journal articles found in the four journals searched, a determination was made about whether the paper was related to the NSF project by application of three criteria: 1) if the date of publication was at least six months after the award was made, 2) if the specific NSF project was acknowledged in the paper (this occurred in nearly all cases), 3) in cases where the specific NSF project was not acknowledged, if the paper was about the intervention described in the NSF abstract of the project. The vast majority of published papers were presentations of new laboratory experiments that had been developed under the NSF funding. Some of these papers also included a presentation of evidence that the laboratory activity resulted in specific student outcomes. The journal articles were coded for two main purposes: 1) to verify the intervention type identified in the coding of the NSF documents, and 2) to verify and provide more specific data for coding the cognitive, affective, and skills outcomes studied by the project.

Most projects were focused on studying outcomes for undergraduate students in the laboratory, so codes were identified as 1 (present) or 0 (absent). A few projects were focused on enhancing the abilities of faculty or TAs in teaching the laboratory, and one project focused on studying the goals of faculty for laboratory learning. Codes for student outcomes intended were marked N/A in these cases. Both authors independently coded all projects, and then discussed results to resolve any minor discrepancies that arose. No major discrepancies occurred.

Results

The data were first examined according to the courses in which the interventions took place. Table 3 presents a summary of the data by courses, contrasted with the findings of Bruck et al. (2010), who interviewed a number of chemistry faculty who engage in these innovations. Across all course types, many projects focused on COG 1 student outcomes. By contrast, relatively lower percentages of projects focused on COG 3 student outcomes, with the exception of general chemistry projects (29%). Regarding affective outcomes, a higher proportion of projects that targeted multiple courses across the chemistry curriculum focused on attitudes (70%).

Table 3.

Comparison by course between grant-funded faculty members’ goals for laboratory learning found by Bruck et al. (2010) and goals identified in projects examined in this study.

Course	Bruck et al. (2010)	(% of ProjCognitive	This Study ects Indicating the StudeAffective	nt Outcome) Skills & Activities
General chemistry (both majors and nonmajors) N=14	• Connecting lecture and lab	COG1: 71 COG2: 43 COG3: 29	Attitudes: 36 Motivation: 7 Interest: 36	Research: 7 Writing: 7 Technique: 14 Teamwork: 0
Organic chemistry N=2^*	• Techniques and lab skills • Critical thinking skills • Written communication skills	COG1: 50 COG2: 0 COG3: 0	Attitudes: 0 Motivation:0 Interest: 0	Research: 50 Writing: 0 Technique: 0 Teamwork: 0
Upper division courses (e.g., physical, analytical) N=10	• Experimental design • Uncertainty in measurement	COG1: 80 COG2: 20 COG3: 0	Attitudes: 40 Motivation: 0 Interest: 40	Research: 30 Writing: 20 Technique: 40 Teamwork: 40
Multiple courses at a variety of levels or across the entire chemistry curriculum N=30	• Mastery of laboratory techniques and skills • Development of critical thinking skills and experimental design	COG1: 83 COG2: 40 COG3: 10	Attitudes: 70 Motivation: 17 Interest: 20	Research: 27 Writing: 0 Technique: 27 Teamwork: 7

*

Data are reported, but the data set is too small to draw any inferences.

The data were also examined according to the interventions that the projects implemented. Table 4 presents the number of projects for the respective intervention type and the project's intended outcomes (cognitive, affective, and skills). The majority of projects, 47, were Type 1 (IM-exp) interventions, so this will be the focus of analysis. Of the 47 IM-exp interventions, 37 (or 79%) focused on Level 1 cognitive skills, 13 (28%) focused on Level 2 cognitive skills, and just 3 (6%) focused on Level 3 cognitive skills. Regarding affective outcomes, 22 (or 47%) focused on attitudes, 14 (30%) addressed interest, and just 5 (11%) focused on motivation. Finally, regarding skill outcomes, 13 (28%) focused on undergraduate research skills, 14 (30%) focused on laboratory technique outcomes, 6 (13%) focused on students’ teamwork abilities, and just 2 (4%) addressed writing skills.

Table 4.

Number of projects identifying specific student outcomes by type of intervention.

Intervention Type
Outcome	Curric (N=4)	IM-exp (N=47)	Instruc (N=3)	Research (N=1)	Resource (N=2)
Cognitive
COG 1	2	37	3	0	2
COG 2	3	13	2	0	2
COG 3	2	3	1	0	1
Affective
Attitude	3	22	3	0	2
Motivation	0	5	0	0	1
Interest	0	14	0	0	1
Skills
Research	0	13	0	0	0
Writing	0	2	1	0	0
Technique	0	14	0	0	0
Teamwork	0	6	0	0	0

Note. The total number of projects coded was 57. A single project could be coded into more than one outcome (e.g., a project could address both Writing and Teamwork).

Discussion

This study offers a review and analysis of a detailed facet of the larger body of work in chemical education. At present, there is a National Research Council consensus study in progress on Discipline-Based Education Research (DBER). The charge of the DBER study includes assessing the “status, contributions, and future direction of discipline-based education research (DBER) in physics, biological sciences, geosciences, and chemistry” (Board on Science Education, 2011). In the consensus study, DBER is defined as combining “knowledge of teaching and learning with deep knowledge of discipline-specific science content. It describes the discipline-specific difficulties learners face and the specialized intellectual and instructional resources that can facilitate student understanding.”

Reporting on Studies Devoted to Laboratory Learning in Undergraduate Chemistry

During its 30-month duration, the DBER study commissioned several papers on the state of research in the various disciplinary areas of undergraduate science education included in the study. One of these papers was charged with reviewing and summarizing the current state of research on teaching and learning in chemistry (Towns & Kraft, 2011). This paper included in its review 379 peer-reviewed research papers in chemical education published across nine journals. The studies were organized into the following categories (number of papers in parentheses): pedagogy (161), misconceptions (70), particulate nature of matter (45), instrument development (13), student achievement (18), and miscellaneous (72). Within the pedagogies category, 29 of the 161 papers focused on the laboratory environment. There were also a few research papers focused on the laboratory environment in the particulate nature of matter and instrument development categories. As noted in the Introduction of this paper, with regard to the papers on laboratory learning, the authors conclude that “the body of research doesn’t point towards one incontrovertible approach to laboratory” (p. 9).

Well over half (63%) of the NSF grants whose projects formed the data in the study presented in this paper were focused on laboratory learning, yet according to the review by Towns and Kraft, less than 10% of the research literature in chemical education is focused on laboratory learning. There are several reasons for this difference; two are discussed here. First, the commissioned paper from the DBER study included research published by authors from many countries, while the data in this study are from the U.S. only. Second, all of the IM-exp projects in this study included an evaluation effort, but most evaluation efforts were small and focused on accountability to the goals of the project and formative evaluation to provide feedback to the PIs on the progress of their work. When only a small part of the project is focused on studying student outcomes, it may not yield sufficient results to produce a research paper. Rather, peer-reviewed publications from these types of projects are usually presentations of novel laboratory experiments and approaches. These publications sometimes present findings about student outcomes as evidence of the value of the laboratory activities, but these findings are not the prime focus of the papers (see Alignment section below). These types of papers were not included in the review paper commissioned by the DBER study. Thus, the present study offers a more detailed review of one facet of the wider body of literature in chemical education, undergraduate laboratory learning, and provides a complement to the paper commissioned by the DBER study.

Faculty Goals for Laboratory Learning in Chemistry

The present study validates some of the findings and offers nuances to other findings of Bruck et al. (2010). As described in the Introduction, Bruck et al. found that mastery of laboratory techniques and skills, and the development of critical thinking skills and experimental design, were the primary goals for laboratory learning deemed important by all faculty who were interviewed. As the present study only examines projects that were funded by NSF, it can be compared to the findings of Bruck et al. about the goals for laboratory learning espoused by grant-funded faculty. Table 3 presents a comparison between grant-funded faculty goals identified in the study by Bruck et al., and goals found to be intended in the funded projects examined in the present study. Several inferences may be drawn from these data.

The largest spread across cognitive outcomes expected occurs in projects focused on the general chemistry level. Projects focusing on upper-level courses and on multiple courses across the curriculum tended to focus less on the highest cognitive levels than general chemistry focused projects did. Motivation as an affective outcome appears to be of greatest importance to projects focusing on multiple courses throughout the curriculum, and was not targeted, or only targeted to a small extent in projects focusing on only one level. There may be an opportunity here to study how different levels, or specific levels, of courses contribute to the motivation of students to do chemistry.

Very few projects focused on general chemistry target involvement in undergraduate research as an explicit outcome. This is surely due in part to the way in which the data were organized in this study. General chemistry serves all science majors (not just chemistry majors), and some projects included in the general chemistry bin in Table 3 were focused on general chemistry for non-science majors, some were on general chemistry for science majors only, and some focused on both types of courses. Other projects funded by NSF, which were not included in this review, have indeed focused on connecting undergraduate research to general chemistry (cf., Russell & Weaver, 2011; Weaver et al., 2006; Russell et al., 2009). However, it appears that targeting undergraduate research in connection with first-year/general chemistry may be an area for opportunity to learn more that could be valuable in advancing understanding of chemical education.

Mastery of technique in using instruments was seen to be most prominent in upper division courses, and least prominent in general chemistry. Bruck et al. (2010) found that grant-funded faculty tended to place a focus on lab skills and techniques at organic chemistry rather than upper division courses. This discrepancy may reflect a difference in definition of the term “technique” between these studies. What Bruck et al. term “technique” reflects the ability to follow detailed procedures and handle laboratory glassware and other equipment carefully. What was meant by technique in the present study was the skill of operating instruments, such as NMR and FT/IR, properly. These are components of understanding the nature of science, as operationalized by Russell and Weaver (2011), who found that students engaging in more research-oriented laboratory activities at the first-year university level resulted in deeper understanding of the nature of science. Nevertheless, few projects focused explicitly on developing students’ skills in experimental design and understanding uncertainty in measurement, so these codes were not retained in the final results. This finding demonstrates a difference between the present study and a conclusion by Bruck et al. (2010) that grant-funded faculty tend to value experimental design and uncertainty in measurement as important goals for upper division courses. This disparity may be due in part to two differences in the data sets. First, the studies had slightly different samples, in that the present study drew only upon information on NSF-funded projects, whereas Bruck et al. (2010) both externally funded and nonexternally funded work. Second, the studies used different data collection methods. Data for Bruck et al. (2010) were obtained from interviews of faculty. The present study drew upon data on NSF-funded projects, and publications about what was accomplished.

Purposes of Laboratory

Both America's Lab Report (NRC, 2005) and Reid and Shah (2007) draw primary conclusions that include the need for greater coherence between laboratory learning and lecture. Reid and Shah state that,

Laboratory work in higher education cannot be seen in isolation. For most students it … has to relate to material taught in lectures and tutorials. However, of greater importance is the need to see the ‘hands-on’ laboratory time as part of a wider process of learning. (p. 172)

The Committee that produced America's Lab Report based its conclusions on a wide review of research on high school (grades 9-12) laboratory learning, but the Committee's findings may also hold relevance to university laboratory learning:

Conclusion 2: Four principles of instructional design can help laboratory experiences achieve their intended learning goals if: (1) they are designed with clear learning outcomes in mind, (2) they are thoughtfully sequenced into the flow of classroom science instruction, (3) they are designed to integrate learning of science content with learning about the processes of science, and (4) they incorporate ongoing student reflection and discussion. (NRC, 2005, p. 6)

Five of the NSF projects cited a primary objective of integrating and increasing the coherence between lecture and laboratory learning. Of these, three were specifically curricular approaches as interventions, one was focused on development of a resource that would facilitate this integration, and only one was a project primarily focused on developing instructional materials that would integrate lecture and lab. This may point to an opportunity for projects that would focus on integrating lecture and lab so that greater coherence is achieved between them. Attendance to how students perceive the coherence between lecture and lab to be relevant to their learning of chemistry has also been shown to be an important issue, as described earlier in summarizing the findings of Russell and Weaver (2008, 2011). Future study might investigate questions around whether students being more keenly aware of the reasons why coherence between lecture and lab is desirable would result in positive student outcomes.

How Laboratory is Anticipated to Change Students

As summarized above, there are often desired affective, cognitive, and skill outcomes for science education (Reid, 2011). These should be seen prominently in the types of interventions that have been studied most. Thus, a comparison is made here between the outcomes studied in the projects included in this study and those observed to have been substantial in the review conducted by Towns and Kraft (2011). In their review of 10 years of research literature in chemical education, Towns and Kraft identified two promising approaches to laboratory learning. First, they identified that problem-based learning has been shown to have a positive effect on students’ attitudes and perceptions in the general chemistry and organic chemistry laboratory. Second, they identified across the literature that when pedagogical structures support students’ development of inquiry skills in the laboratory, students are better able to ask questions, hypothesize, and suggest good questions for further study.

The present study echoes these findings to some extent, in terms of anticipated affective outcomes. Of the 47 projects focused on instructional materials development, eight (17%) specifically included problem-based learning as a strategy. Among this subset, there was a range of affective outcomes anticipated. Six intended to study changes in attitudes toward science, chemistry and/or the laboratory course, two examined interests (one in lab, and the other in solving problems), and one was interested in motivation to engage in laboratory learning.

A number of projects (13, or 23%) also expected to foster greater engagement by students in undergraduate research. Projects that included an explicit focus on undergraduate research were 2.2 times as likely as projects not explicitly focused on undergraduate research to express a goal that students’ technical expertise in using instruments in the laboratory would increase. There were no appreciable relationships between other skills (writing, teamwork, and visualization ability) and undergraduate research as a focus, nor were there any appreciable associations between cognitive or affective outcomes anticipated and whether undergraduate research was a focus. One interpretation of these results is that there was an expectation that, if students use instruments more, they will be more likely to choose to engage in undergraduate research. Another interpretation is that institutions that acquired instruments (e.g., NMR, GC/MS) did so because they desired to provide more opportunities for undergraduate students to engage in research.

Alignment between Goals of the Intervention and Anticipated Outcomes

The projects that were funded represent the best proposals submitted to NSF, as judged through the peer-review process and subsequent decisions by program officers, who take the peer reviews under advisement along with other factors when determining which projects to recommend for funding. The program solicitations (sometimes called program announcements) that called for proposals defined the expectations for what the NSF was interested in funding and how to make a case for why a particular idea should be funded. All versions of the solicitation for this program have in common the following information regarding the use of and contribution to knowledge about science, technology, engineering and mathematics (STEM) education, and expectations for measurable outcomes (NSF, 2011):

Use of and Contribution to Knowledge about STEM Education: Projects should reflect high quality science, technology, engineering, and mathematics. They should have a clear and compelling rationale, use methods derived from existing knowledge concerning undergraduate STEM education, build on existing projects of a similar nature, and present evidence supporting the approach. They also should have an effective approach for adding to this knowledge by disseminating their results.

… Expected Measurable Outcomes: Projects should have goals and objectives that have been translated into a set of expected measurable outcomes that can be monitored using quantitative or qualitative approaches or both. These outcomes should be used to track progress, guide the project, and evaluate its ultimate success. Some of the expected measurable outcomes should pay particular attention to student learning, contributions to the knowledge base, and community building.

These communicate an expectation that, to be successful, proposed interventions should contain an alignment between the need for an intervention, a rationale based on evidence for why a proposed approach should be successful in achieving an expected set of outcomes, and a method for determining whether the outcomes are achieved. Most undergraduate chemistry laboratory projects that have been completed under this funding program (47 out of 57 during 2000-2010) have had primary goals related to developing new instructional materials, curriculum, or approaches to doing experiments in the laboratory. However, they also have all included measurable outcomes related to student learning. Additionally, all projects are expected to contribute to the knowledge base in STEM education. Formal contributions to the literature are, of course, not the only way to contribute to the knowledge base. Emerging from the 57 projects examined in this study (some of which concluded only recently, so it is likely that there are publications in preparation or in press), so far there have been 22 papers published in peer-reviewed journals: 14 of the 22 were papers on course development efforts or laboratory activities (Brown, 2007; Birdwhistell et al., 2008; Cancilla & Albon, 2008; Zovinka & Stock, 2010; Durham et al, 2011; Faraldos et al., 2011; Schurter et al., 2011; Szalay et al., 2011; Lanigan, 2008; deProphetis et al., 2010; Jin & Bierma, 2010; Nivens et al., 2010; Stanford, 2011; VanDorn et al., 2011), with the last 5 of these having reported an evaluation of student outcomes within the paper; and 7 of the 22 were research studies related to how students learn effectively in the laboratory (Fay et al., 2007; Leinhardt et al., 2007; Rudd et al., 2007; Evans et al., 2008; Schroeder & Greenbowe, 2008; Bruck et al., 2010; Jiang et al., 2010; Rauschenberger & Sweeder, 2010). A theorybased rationale for why an intervention should work can inform what the expected outcomes should be, and can also inform what to measure so that a mechanism can be uncovered for how and why the outcomes occur or not. Future study could investigate the relationships among laboratory activity projects’ theory of action (the reason an intervention should work), theory of learning (what the investigators believe to be true about learning), and the resulting student learning outcomes.

Conclusions

An analysis was conducted on all of the projects related to undergraduate chemistry learning that were funded by the NSF's Course, Curriculum and Laboratory Improvement program and concluded between 2000 and 2011. Of 91 unique projects, 57 focused on laboratory learning. Of these, the vast majority (47) were interventions that involved developing new instructional materials for undergraduate chemistry laboratory activities. Assessment of student outcomes generally included both cognitive and affective outcomes, and often also included skills. Approaches employed in many interventions are well aligned with the conclusions of a consensus study of the National Academies of Science and of researchers studying the purposes of laboratory learning in undergraduate science courses.