The University of Arizona
Course Title: A Geological Perspective
Contact: Jessica Kapp
Project Abstract
Final Report (as of 6/1/09)
- Impact on Students
- Impact on Cost Savings
- Lessons Learned
- Pedagogical Improvement Techniques
- Cost Savings Techniques
- Implementation Issues
- Sustainability
The University of Arizona (UA) plans to redesign A Geological Perspective, a general education course for non-science majors currently enrolling ~600 students each semester. The course is offered in three large lecture sections taught by different faculty members and includes optional, weekly study groups taught by graduate teaching assistants (GTAs). Lectures meet three times a week for 50 minutes, and the optional study groups meet once a week for 50 minutes.
Students often feel overwhelmed by the science and associated jargon, and many are prone to skipping lectures altogether. Optional study groups are only attended by ~50% of the students. Faculty members find the traditional course to be a lot of work, mainly due to the grading it generates. Lecture activities are often limited to short multiple-choice or true-false assignments, limiting evaluation of the depth of students' understanding. Many students lack math and critical thinking skills, and lecture time is spent reintroducing concepts students should already know from the text.
The redesign, using the Replacement Model, includes replacing one hour of lecture with break-out sessions run by undergraduate preceptors and graduate teaching assistants (GTAs). In the remaining lecture periods, lectures will be brief with a large portion of the class devoted to active learning in small groups. Once-a-week mandatory break-out sessions will also replace the optional study groups where small groups of students will complete activities that have been introduced in the lectures. Students will submit the majority of their work through the course web site. Software will track student progress, and those who are not succeeding will receive individualized assistance.
The redesign will enhance course quality by providing an active learning experience with individualized assistance. Students will spend more time interacting with their fellow students, undergraduate preceptors, GTAs, and their instructor, as less of their time will be devoted to passively listening to lectures. They will receive one-on-one help in break-out sessions on assignments designed to enhance the lecture material. Completing assignments online will allow students more flexibility in where and when they work on them. It will also provide immediate feedback on their submitted work. Online tutorials, virtual office hours, and a classroom participation system will give students more options for contacting the instructor and also allow instructors to monitor student progress.
The UA Conceptual Astronomy and Physics Education Research (CAPER) Team, who specialize in project evaluation, will lead the assessment effort. The team will compare pre- and post-test learning gains for content and process skills as well as common final exams before and after the redesign. The team will also inductively analyze student interviews or reflective writing that describe student satisfaction with the modified learning environment.
The redesigned course will reduce instructional costs by decreasing the number of GTAs each term from seven to four and replacing many of them with undergraduate preceptors who are not paid but receive academic credit for their teaching service. The number of hours spent by faculty and graduate teaching assistants on preparation, class time and grading will be greatly reduced. The cumulative impact of these efficiencies will reduce the cost-per-student from $437 to $185, a 58% decrease. The savings will be used to allow faculty more time to work on research-related priorities and advising their graduate students.
Improved Learning
The Geosciences Concept Inventory (GCI v1.0) was given as a pre-test/post-test to students in the traditional course and the redesigned course to compare student learning in terms of gain scores. Only 36 of 78 items that most directly related to course content were used for analysis. Students in the traditional course had a pre-test GCI percentage correct of 28.67 (SD=11.45, n=96) which increased to 45.26 (SD=11.76, n=84) on the post-test. After course redesign, students had a pre-test GCI percentage correct of 38.62 (SD=8.42, n=144) which increased a post-test GCI score of 48.73 (SD=7.49, n=132). Although the gains from pre-test to post-test are statistically significant, the difference in post-test GCI scores between the two groups is not. Students’ knowledge levels, insofar as the GCI can measure, were equivalent in both courses. Also, an analysis of common four separate essay-style exam questions was conducted to compare students in both the traditional and the redesigned courses. In all categories, the student-supplied responses from the two courses were indistinguishable.
Based on the qualitative data reported below, the team believes that the selected common assessment instruments had insufficient resolving power to detect differences in student achievement or the student experience, rather than there were no actual differences in the student experience. Indeed, during this study, the GCI 1.0 planners started a major revision to the instrument.
Improved Retention
Student success rates (C or higher in the course) were 90% in the traditional course, 87.2% in the pilot semester, and 89.6% in the full redesign. The differences are statistically insignificant.
Other Impacts on Students
The Likert-style Attitudes Toward Science Survey (ATSI) was given as an end-of-class post-test to students in the pre-modified course and as a pre-test/post-test the redesigned course to compare student attitudes between the courses. There were no statistical differences evident among the survey results. In contrast, interview and focus group transcripts reveal that students find the redesigned course to be relevant to their lives and educationally satisfying, both of which are only infrequently observed among traditional introductory science survey courses.
At the mid-point of the redesigned semester, a team of evaluators from the University of Wyoming ’s Cognition in Astronomy, Physics and Earth sciences Research (CAPER) Team conducted individual and focus group interviews with participating faculty, graduate teaching assistants, undergraduate peer mentor instructors, and students. Four recurring themes were prominent across the extensive qualitative data collected. These notions are 1) structured discussion among students was meaningful; 2) the interactive nature of the course engaged students and instructors; 3) the non-adversarial nature of the course climate was critical; and 4) the purposeful instructional activities directly support meta-cognition. Each of these recurring and dominant themes are consistent with the perspective of creating a learner-centered environment that supports learning and students’ meta-cognition.
The team did not make any significant changes to the planned redesigned course format, which reduced the cost-per-student from ~ $437 for the traditional course, to ~ $185 for the redesigned course. The bulk of the savings came from using twice as many undergraduate preceptors and fewer half-time graduate teaching assistants (GTAs.)
Pedagogical Improvement Techniques
- Making lecture slides available on the web site. Lecture slides were made available to students prior to lectures so they could print them out and bring them to class. This reduced frantic note-taking and encouraged active participation in class activities.
- Low-stakes online quizzes. The team awarded more points for online work before the redesign than after. In the traditional course, students completed weekly quizzes as well as higher stakes assignments online since there were no break-out sessions in which to work on assignments. In the redesigned course, only low-stakes weekly reading quizzes were completed online. Higher-stakes assignments were completed in break-out sessions under the guidance of GTAs and undergraduate preceptors. Students overwhelmingly reported that they preferred this method and that they learned more and earned more points.
- Active learning in the lecture emphasizing problem-solving. To facilitate active learning in large lectures (150+ students), students were given many opportunities to solve problems collaboratively with the students around them in the form of think-pair-share questions as well as graded work. One change made after the pilot semester was to go back to the original format of giving graded in-lecture assignments. Although this slightly increased the amount of time instructors put into grading (which was essentially zero during the pilot semester), the team found that it greatly improved attendance and encouraged active participation in class, as the students were given the opportunity to collaborate with other students before turning in their work for a grade. These assignments were generally easy-to-grade, multiple-choice questions, but they relied heavily on students interpreting information as opposed to memorizing facts. Often they involved data and plots, images and scenarios that the students had to interpret before selecting their answer. When activities were not graded, students provided their answers with color coded cards labeled A, B, C, D. In this way instructors were able to scan the room and get a quick read on the percentage of students getting the material.
- Learning names. Learning the students' names is a technique that two team members used consistently. Student interviews found that students seemed to link instructors learning their names to instructors caring about their learning. It was striking how strongly this technique influenced student perceptions of the course.
Cost Savings Techniques
- Use of undergraduate preceptors vs. GTAs. It is clear that using preceptors is the single best way of saving money in this course. Preceptors cost nothing, while a half-time graduate teaching assistant costs roughly $34,000 per year.
- Reduction in instructor time. Regardless of the amount of assignments administered online vs. in class, or the detail with which the lectures were pre-prepared, instructors found that they inevitably found themselves spending more time than they planned on preparation for teaching. This is because they are all passionate about teaching, enjoy offering their lectures and are therefore constantly trying to improve those lectures. All lectures were completely pre-prepared for instructors and handed to them well in advance of the beginning of the semester, yet they found themselves spending at least a couple of hours a week modifying them. This was done purely by choice, not out of necessity. None of them regretted doing so, and none of them would have had it any other way. The amount of time spent grading also varied slightly from semester to semester, again largely due to instructors choosing to add more rigorous questions to either their in-class assignments or their exams (e.g. essay questions). This was also purely by choice and could have been avoided had they chosen to use all multiple- choice questions or all online assignments.
Implementation issues
- Time required to administer pre- and post-tests and interviews. The team did not anticipate how much class time would have to be sacrificed to administer the Geosciences Concept Inventory and to collect student interview data. In retrospect, the team could have created a highly abbreviated version of the GCI to be given online, out of class, and compared these pre- and post-test results for both the traditional and redesigned course. Although they thought it would be more time effective to use an existing pre- and post-test, the test was so long that it cut into valuable learning time twice per semester for all three semesters. Interviews were conducted with students, undergraduate preceptors, graduate teaching assistants and instructors. The interviews were so informative that focus groups often spent more time than anticipated, and several issues were re-visited with instructors in several interview sessions.
- Lack of preceptor expertise and confidence in running break-out sessions. Two very common concerns of the preceptors were 1) they did not know the material as well as GTAs and instructors and therefore were not able to address all questions posed by the students, and 2) they were not confident enough to walk into a break-out session and take control as the instructor. The first issue was addressed by 1) only inviting students who earned A's in the course to be preceptors, 2) having weekly preparatory meetings in which faculty went over the material the students were working on, and 3) posting all lectures on a web site for the preceptors to access prior to running the corresponding break-out sessions. Taking these actions did not seem sufficient as a common comment from students in interviews was that the preceptors were not as knowledgeable as the instructors and students felt they were given incorrect information. This is somewhat unavoidable, as undergraduate preceptors have not had as much background or training in the subject area. All are non-science majors who took the course as part of a general education requirement. The second issue of confidence as teachers was addressed by a preceptor training workshop in which a teaching expert from the university teaching center gave preceptors valuable examples of ways to better their teaching, how to answer questions without giving away answers, how to cater to various learning styles, and other techniques. Every preceptor who participated said they found this valuable, but still felt less confident than the GTAs because of their lack of geology knowledge.
The sustainability of the project is not in question unless the general education program at the university sees some significant changes in the wake of budget cuts. During the spring 2009 semester, all sections are being run essentially as they were during the fully implemented redesign semester.
The redesign did not produce a significant improvement in student learning, which is still the team’s main goal. The team firmly believes that interaction with a human being is more useful than interaction with a computer. They will continue to look for ways to improve learning and that may require the use of more GTAs in the future. They will also consider whether student learning is affected by having a GTA or a preceptor running break-out sessions. A rough analysis of the data suggests that having a GTA leads to better quiz and exam scores. If this is the case, the team will strive to get more GTA support.
Quick Links:
