The Learning MarketSpace, July 2003
A quarterly electronic newsletter of the Center for Academic Transformation highlighting ongoing examples of redesigned learning environments using technology and examining issues related to their development and implementation.
TABLE OF CONTENTS
Perspectives on issues and developments at the nexus of higher education and information technology.
Build It, But Will They Come?
The need to make Web-based interactive learning resources available to faculty in order to accelerate the benefits of technology-based learning has been well recognized in higher education. Indeed, both private foundations and governmental agencies have funded numerous projects. Most recently, efforts such as MERLOT and the MIT OpenCourseWare (OCW) Project have received widespread attention. Most involved in developing and funding these efforts have defined the problem merely as lack of technology-based materials. They seek to create repositories whose contents are vouched for by institutional reputation (MIT) or by individual faculty contributors (MERLOT). The assumption is that providing access to these materials will cause individual faculty to use them in redesigning their courses.
This approach has several drawbacks. Entries are selected and mounted by interested individuals, but the materials are not tied to improved student learning outcomes. Many of the included learning objects are intended for specific (and possibly unique) upper division courses that are not necessarily part of the curricula at other institutions. Other materials are designed for sophisticated students and may not be relevant to a more diverse student body at other institutions. In addition, these projects tend to assume that more options are always better. MERLOT cites "links to thousands of learning materials" as one of its benefits, yet only a tiny subset has been evaluated by anyone other than the contributors. Most importantly, these projects lack a methodology for transfer to other institutions. Their strategy of hope-for-the-best has been tried many times in the past and failed (e.g., programs supported by Apple and IBM in the 1980's and 1990's, and attempts by national organizations like Educom).
To respond to the need for high quality, interactive learning materials, we advocate a fundamentally different strategy. First, we need to create repositories of materials that are research-based—materials that have been tested with large numbers of students at multiple institutions and have demonstrated statistically significant increases in student learning. Second, establishing repositories of materials that have led to increased learning is a good first step, but it is not enough to ensure that they will be used. One must have a transfer methodology. Our experience in the Program in Course Redesign has shown, for example, that supplementing classroom experience with low-stakes quizzes (the materials) may lead to increased learning, but by giving points for student participation (how to use them), instructors can increase student learning substantially. Faculty members and others with experience in how to use the materials need to be paired with less experienced institutions to ensure that the materials will be used successfully.
Let's consider this example in more detail.
At the University of New Mexico (UNM), General Psychology is the largest and most popular undergraduate "killer" course, enrolling 2,250 students annually. Prior to its redesign, the course was taught in a traditional lecture format with no recitation sections. UNM's primary redesign goal was to improve the course's extraordinarily high 42% drop-failure-withdrawal (DWF) rate, 30% of which were failures and a disproportionate number of which were minority students. With an undergraduate minority student population of approximately 46.4% (31.3% Hispanic, 5.5% Native American and 9.6% other), UNM leads the nation's research universities in student diversity. UNM also has one of the lowest student retention rates among public research universities. UNM students are primarily commuters who work 30 or more hours per week. High failure rates in core curriculum courses such as General Psychology are known to have a strong negative impact on UNM's low overall retention and graduation rates.
UNM's goal of reducing drop and failure rates in General Psychology has been achieved. The failure rate was reduced from previous levels of 30% to 12%, and the DWF rate fell from 42% to 18%. The number of students who received a C or higher rose from 60% to 76.5%, and there were more A (34%) and B (31%) grades than recorded in previous semesters. At the same time, the course was arguably more difficult, requiring students to cover completely a high-level introductory text. Instructors in previous semesters sometimes omitted chapters from the course because they were unable to cover all of the text material.
The redesigned course reduced the number of lectures each week from three to two and incorporated a weekly 50-minute studio session led by undergraduate teaching assistants. These activities were supplemented by interactive Web- or CD-ROM-based activities, quizzes, and programmed self-instruction (PSI) offered on a 24/7 schedule. PSI is a learning technique that provides the individual student a self-paced method of learning new information. Using a branching sequence of interconnected questions, PSI includes repetition, examples, illustrations, and anecdotes to convey important psychology concepts. Students took repeatable quizzes each week requiring a C-level of mastery. Graduate teaching assistants monitored quiz performance, contacting and counseling students who failed to achieve a C-level of performance.
What contributed most to improved student performance? The UNM team believes that the major determinant was the required mastery quizzes. For all quizzes, only the highest scores counted. Students were encouraged to take them as many times as necessary in order to receive a high score and to gain a sense of mastery over the material. Additionally, they were told that questions on the four in-class exams, worth 50 percent of their grade, would be taken from mastery-quiz items. The more often they repeated quizzes, the more likely their chances of seeing actual exam questions.
Students received credit for completing two online mastery quizzes per week for 16 weeks, which represented 25 percent of their grade. Each quiz consisted of 10-20 randomized multiple-choice questions drawn from a pool of 150-200 test bank questions per week; the total pool consisted of approximately 3,000 questions for the semester. An additional 20 percent of their grade was determined by performance on 10 additional quizzes compiled from the self-paced interactive CD-ROM set that accompanied the text, representing another pool of 550 questions. Quizzes were delivered and graded automatically on a 24/7 schedule using WebCT.
An analysis of quiz-taking behavior indicated that the more times students spent taking quizzes and the higher their scores, the better they performed on in-class exams. Students who received grades of C or better took the quizzes on average four to six times. Some students developed strategies to increase exam performance. They would continue taking quizzes even after they had attained perfect scores because doing so would increase their chances of seeing items that might appear on the next exam.
Quizzes closed on a weekly basis corresponding to that week's topic. If students missed a quiz deadline, make-up quizzes, which were identical to the original quizzes, were always available online. To encourage students to take quizzes in a timely manner, however, make-up quizzes counted only 75% of the original quizzes. During the last three weeks of the semester, students were allowed to take "amnesty" quizzes, which were identical to the original quizzes and for which students received full credit for completing.
UNM implemented a quasi-experimental design for two spring 2002 sections (350-450 students each) where all students had access to all components of the redesigned course. Although students in both sections had access to the same instructor, text, CD-ROM, and curriculum, only students in the redesigned section were required to complete all aspects of the course. To determine whether quizzes that were mandatory (required for course credit) or voluntary (no course credit) would differentially affect exam and grade performance, students in one section received course points for completion of weekly online mastery quizzes. Students in the other section were encouraged to take the mastery quizzes (and were told that taking the quizzes would improve their grades), but received no course points for doing so.
On in-class exams, student's in the section that were required to complete quizzes for credit always outperformed students in the section where taking quizzes was voluntary. Students in the redesigned section received more A's, B's, and C's, in addition to fewer C- or below grades, than students in the voluntary quiz section. Students took more quizzes, scored higher, and spent longer on quizzes when course credit was at stake than students in the section where quizzes were not linked to credit. Moreover, relatively few students successfully completed quizzes when credit was not a consequence, and some students chose not to take quizzes at all.
Other projects in the Program on Course Redesign have had equally dramatic results. As part of its redesign of Understanding the Visual and Performing Arts, Florida Gulf Coast University (FGCU) assesses students' development of content knowledge primarily through three multiple-choice, module exams (one on the Visual Arts, one on the Performing Arts, and one on the contexts of the various arts in the Renaissance through the twentieth century), and also through the application of the content knowledge in short essays analyzing artwork. In the area of content knowledge, students demonstrated a markedly enhanced level of learning in the redesigned course. The average score on standardized exams in the traditional course was 72%; in the redesigned course it was 85%. Further increases in student learning are documented in the grade distribution. The percentage of A's and B's on the standardized exams went from 37% in the traditional course to 77% in the fully implemented redesigned course. The percentage of D's and F's went from 38% in the traditional to 10% in the redesigned course.
The FGCU team attributes increased student performance on the exams directly to students taking required practice tests. In the course redesign pilot, practice tests only counted as part of students' participation grade. Because of this, students only needed to take the practice tests once--regardless of how they scored--to get full credit. After listening to Gordon Hodge, the lead professor at UNM, describe his point system at a grant recipients workshop, FGCU changed its strategy. In the full implementation of the redesign, each course activity that the students must complete is assigned a point value. Each practice test includes 10 questions worth two points each, and the practice tests receive their own grade. The grade that is recorded is the highest score on the test. Students now take the assignments more seriously, banking their points as they progress through the semester. Students take the practice tests repeatedly, with some students taking them as many as 25 times. Questions on the objective portions of the module exams draw questions from the practice test banks. Those students who take the practice tests three or more times regularly score A's on the module exams.
Unlike most of the materials included in MERLOT or MIT's OCW Project, the materials used by UNM and FGCU are research-based; they have been tested with large numbers of students and have led to statistically significant increases in student learning. But simply including those materials in a repository would neither ensure that they would be used by other faculty members nor that they would be used effectively. A transfer methodology that teaches faculty how to use them (give course credit for practice) is needed. UNM and FGCU can now say to other institutions, use these materials as we specify and we guarantee that you will see measurable increases in student learning as well as higher retention rates. Can MERLOT or MIT say that?
--Carol A. Twigg
Featuring progress reports and outcomes achieved by the Program in Course Redesign.
South Dakota's Governor Michael Rounds Funds a Statewide Initiative to Increase Quality and Reduce Costs
In order to encourage the six public higher education institutions in South Dakota to improve quality and reduce costs using technology, Governor Michael Rounds has established the Rounds Grants Program in Course Redesign modeled on the Program in Course Redesign at the Center for Academic Transformation. Eight inter-university teams have been awarded nearly $550,000 to collaborate in the redesign of courses. The South Dakota Board of Regents contracted with the Center for Academic Transformation to provide workshops based on the successful practices used in the Program in Course Redesign and to consult with each project team. More than 100 faculty and administrators from across the state attended three workshops, and plans are underway for future initiatives to expand the Rounds Program. To learn more about how your institution, system or state can replicate the successes achieved in the Program in Course Redesign and the consulting options available from the Center for Academic Transformation, click here.
The Program in Course Redesign Goes to Congress
The results of the Program in Course Redesign were shared with the U.S. Congress on July 10, when Carol Twigg was invited to testify at a Congressional hearing on college affordability. The hearing was conducted by the sub-committee on 21st Century Competitiveness, chaired by Representative Buck McKeon. This sub-committee is part of the House Committee on Education and the Workforce. Titled "We Know There's A Problem; What is the Solution?," the hearing was conducted as part of the higher education reauthorization process. Using a high-speed connection to the Internet, anyone can watch the hearing on the CSpan Web site. Go to http://c-span.org/, choose Video Library and Congress from the list at the top left of the screen, and then click on House Hearing on Affordability in Higher Education. Full testimony is available at http://edworkforce.house.gov/hearings/108th/21st/afford71003/twigg.htm.
Ohio State Redesign Recognized for an Innovative Job Well Done
Nominated by the Microsoft Corporation, "The Statistical Buffet," was recognized in April 2003 as one of 10 finalists in the Education and Academia Category of the Computerworld Annual Honors. Established in 1988, the Honors program is dedicated to identifying the men and women, organizations and institutions, that are leading the global information revolution and to recording the impact of their achievements on society. The Statistical Buffet is described as a case study in which college students, informed by their own learning styles and the testimony of previous learners, are offered a choice of content delivery strategies designed to match their individual strengths. To learn more, see www.cwheroes.org/caa_4.asp.
Two New Articles Reporting Results from the Program in Course Redesign
Two new articles analyzing the results of the Program in Course Redesign are in print. The July/August 2003 issue of Change magazine includes an article titled "Improving Quality and Reducing Cost: Designs for Effective Learning" that discusses successful redesign strategies used by the 30 projects. The September/October 2003 issue of EDUCAUSE Review will include a second article titled "Improving Learning and Reducing Costs: New Models for Online Learning." This article discusses the five redesign models that have emerged from the Program in Course Redesign with examples from the participating institutions.
Final reports for the Round II projects have been posted on the Center's Web site and can be found by following the links at Project Descriptions Sorted by Grant Rounds. The comparison of the projected savings and the actual savings achieved by the project are also available at Projected and Actual Savings. Round II projects planned to reduce costs by about 44 percent, with a range of 20 percent to 84 percent, with a total annual savings of $1,043,821. They actually reduced costs by 38 percent on average, with a range of 25 percent to 74 percent, with a total annual savings of $1,039,112. An overview of the Round II findings and an analysis of what was learned from the perspective of the program staff will be posted in early fall 2003.
On June 19, 2003, Round III participants in the Program in Course Redesign reported their findings to date at a program workshop in Seattle, WA. Here are some previously unreported results:
The redesign of Understanding the Visual and Performing Arts at Florida Gulf Coast University has clearly reduced costs and improved student learning. The average grade on module exams increased from 72% in the traditional model to 85% in the redesigned course. The cost-per-student in the traditional course has decreased from $129 to $69 in the redesigned format. Based on using this scalable, growth model, the university anticipates further savings as the number of students taking the course increases.
The Statistical Buffet at The Ohio State University has increased its retention rate such that 250 students do not need to retake Statistics each year. Grades in the redesigned model have a higher median score than in all forms of the traditional course. In addition, the model requires fewer faculty hours and demonstrates overall lower cost, on the way to a projected savings in the cost-per-student of 31%.
At Portland State University, the course redesign's preliminary results indicate that the cost-per-student has decreased 29%, while the capacity of the course has increased 85%, enrolling more than 580 additional students annually. The redesign has reduced the number of class meetings each week and increased the opportunities for interactive writing, reading and grammar experiences and for acquiring oral proficiency in Introductory Spanish.
The team at Tallahassee Community College has shown an increase in success rates of C grades or better from 60.7% in the traditional format to 68.7% in its redesign of English Composition. Using SMARTHINKING, a company that provides online feedback on essay drafts, students have increased access to tutorial assistance and greater opportunity to revise and improve their work. The redesign has shown a decrease of 43.5% in the total amount of time faculty spend on the course.
At the University of Southern Mississippi, retention has increased from an average of 80% in the traditional model to 87% in the redesign of World Literature. In keeping with the goals of the project, learning has increased in the areas of reading skills (from 70% in the traditional to 85% in the redesign) and writing skills (from 70% to 80%). Learning levels in thinking skills are unchanged. Final cost savings are still being calculated but are on track to the projected level.
Highlighting themes and activities that cut across redesign projects.
Using Commercial Software – A Key Factor in Successful Redesign
Well-selected commercial software is a key ingredient in the Program on Course Redesign. Using purchased learning materials cuts development time substantially and saves significant dollars in course delivery, while making a major contribution to improved student learning. High-quality instructional software provides effective opportunities for increased student interaction with course content by allowing students to practice difficult learning concepts and to receive rapid, automated feedback. Students know what they have mastered and how to find additional resources for further study as needed. Using software reduces faculty and TA grading time, and the need for lecture as a method of conveying information, thus freeing faculty and TAs to provide more individualized assistance to students who need it. Most of the projects in the Program on Course Redesign used some form of interactive software as part of their redesigns. Here are some examples:
In the University of Iowa's redesigned introductory chemistry course, online homework software provides students with problem-solving experiences and immediate feedback until they achieve mastery. After using one commercial package, Mastering Chemistry, for five consecutive semesters, the team switched to a different one, McGraw Hill's ChemSkillBuilder On-line. ChemSkillBuilder is similar in scope and purpose but better at providing tutorials and context sensitive help. During the five semesters Iowa used Mastering Chemistry, student participation was at a level between 50% and 65% of homework credit. During the two semesters of using ChemSkillBuilder, participation was at the 80% level. Student evaluations of the electronic homework as a factor that helped them learn (with 1 low and 10 high) went from a class average of 3.6 in fall 2000 (Mastering Chemistry) to 7.6 in fall 2002 (ChemSkillBuilder).
At Rio Salado Community College, mathematics software by Academic Systems allowed four math courses to be taught online simultaneously by one instructor and one course assistant. Because most of the math questions are answered through the interactive characteristics of the software, 90% of the email students send to the instructor is not content-related and can be answered by the course assistant. The combination of high-quality software and assistance from a support person allows large numbers of students to be taught effectively by one faculty member. Student achievement was equivalent to that of the traditional course, retention was improved, and costs for delivering the course were reduced. At the University of Central Florida (UCF), 25% of 60 college algebra sections are being taught with reduced seat time using the Academic Systems software. The course includes scheduled labs for tutoring small groups of students. Reduced seat time means a reduction in classroom space, which is extremely beneficial to the rapidly growing UCF.
MyMathLab by Prentice Hall is being used at three institutions participating in the Program in Course Redesign: the University of Alabama, the University of Idaho and Northern Arizona University. Each institution independently reviewed multiple software programs and concluded that MyMathLab is the most useful for students, citing its flexibility and user-friendliness. Using MyMathLab in a modified emporium (computer lab) model has led to equivalent or increased learning among students with widely varying levels of preparation at all three institutions.
In redesigning Spanish, the University of Tennessee, Knoxville (UTK) and Portland State University (PSU) are both using commercial software that accompanies their selected texts. UTK has redesigned Spanish Transition to reduce class time by one hour per week, eliminate lab time, and substitute online instruction using the grammar and vocabulary exercises from Sabias que? by McGraw Hill, which includes links to the publisher's Web site. UTK has increased the number of students served by 500 per year and reduced the cost per student from $109 to $28. PSU uses ?Como? by Houghton Mifflin, a multimedia version of a comprehensive, proficiency-oriented, introductory Spanish program. Students interact with the software to strengthen and reinforce important concepts. PSU's redesign will enable the university to almost double the number of students served annually at the same total cost.
A key component of the fine arts redesign at Florida Gulf Coast University (FGCU) is the Intelligent Essay Assessor (IEA), a computer program designed to grade well-structured essays. Students are asked to critique a painting or other image of a work of art in short essays, which are then evaluated by the IEA. The IEA, once programmed, assesses 100-500-word student essays based on content, grammar, punctuation, mechanics, and spelling. In order to program the IEA, the FGCU team entered a digitized version of the textbook and 200 essays scored holistically by the design team. Once programmed, the software was able to grade the short essay questions and provide a score. The team was able to demonstrate a high level of inter-rater reliability that exceeds that of two faculty readers. Using the IEA means that faculty grading time for this part of the course falls to zero, grading consistency is greater, and evaluations are returned to students more quickly.
Details about how each redesign project has used these commercial software packages can be found by following the links at Program in Course Redesign.
Reporting on initiatives that share the Center's goals and objectives.
Kansas State Redesigns Introductory Psychology without a Grant
Kansas State University was a finalist in Round II of the Program in Course Redesign but was not selected to receive a grant. That didn't stop Dr. Kelly Welch's desire to apply what she learned at a program workshop and redesign her Human Growth and Development course. In the traditional format, Dr. Welch lectured to nearly 1,000 students twice a day/twice a week and was supported by a team of seven graduate assistants. The redesigned course meets once a week and includes an interactive CD ROM and online assignments, exams, quizzes, and streaming video "lectures." The support team has decreased from seven graduate assistants to one. Copying costs have dropped from approximately $10,000 per academic year to less than $1,000. This course redesign has succeeded in reducing cost as well as improving learning, and student evaluations to date have been extremely positive. As Dr. Lynch wrote to us, "This [redesign] experience has been one of the most rewarding experiences in my career. I wanted you to know that my burning desire to create a course for the 21st century learner came to fruition"-and without a grant! To learn more, contact Dr. Kelly Welch at firstname.lastname@example.org.
Redesigning Undergraduate Teacher Education
To foster greater student interaction with course content as well as with each other, the Faculty of Education at the University of Western Ontario (UWO) replaced nine large group lectures of 200-400+ students with nine online content modules and online discussion. The objective was to move away from the simple transmission of information through face-to-face lectures to active student engagement with interactive content modules, links to Web-based resources, and online discussion. The UWO faculty found students quite receptive to the change and better prepared for required, face-to-face workshops. The redesign enabled UWO to save classroom space. For more information, see "From Large Lecture to Online Modules and Discussion: Issues in the Development of Online Teacher Education" by George Gadanidis and Sharon Rich in the July/August issue of The Technology Source at http://technologysource.org/article/from_large_lectures_to_online_modules_and_discussion/.
Scaling Up or Down Depending on Student Demand
The University of North Texas (UNT) School of Library and Information Sciences offers a certification program and an option in the Master's Degree in Library and Information Sciences that allow core courses to be scaled up or down as enrollments fluctuate. While enrollment has grown 170% in UNT's school librarian program, there is a severe shortage of faculty in this area. The redesign uses teams of full-time faculty, clinical faculty, regular TAs, "super" TA's, and practitioner mentors, all with specialized responsibilities. Clinical faculty are outstanding practitioners, mostly from Texas but also from out of state. Students attend short institutes on campus, available in several formats, and the remainder of the program is offered online. UNT has licensed the courseware to another university and is in discussion with a second that wants to start a new school library certification program. Here UNT will license the entire suite of courses which would allow the new program to offer a high-quality program with minimum startup time and costs and use clinical faculty in its home state. For more information, contact Dr. Philip Turner at PTurner@unt.edu.
The Center for Academic Transformation serves as a source of expertise and support for those in higher education who wish to take advantage of the capabilities of information technology to transform their academic practices.
Copyright 2003, The Center for Academic Transformation