Return on Investment: How do we know if we are preparing teachers who are ready for today’s classrooms?

By: Elizabeth A. Lewis, Ph.D., Associate Professor, Science Education, University of Nebraska-Lincoln
Stacy Duffield, Ph.D., Professor, North Dakota State University, School of Education

Teacher preparation is a heavily resourced venture with investments from multiple stakeholders. Teacher candidates invest greatly of their effort and money, faculty time and expertise are allocated by teacher preparation programs, funds are provided from the institutions, contributions are made by partner P-12 schools, and we cannot forget the high stakes of effectively educating our youth.  Therefore, it is important to know if what we are doing in teacher preparation is working.  This blog will explore the investment of a teacher preparation program in collecting high quality data about candidate and completer preparedness for today’s classrooms.

Developing Quality Measures
North Dakota State University (NDSU) is a land grant, research institution that serves nearly 15,000 students in bachelor through doctoral degrees. In 2010, NDSU was awarded a grant from the Archibald Bush Foundation that provided an opportunity to partner with 13 other institutions in Minnesota, North Dakota, and South Dakota to improve our teacher preparation programs and increase teacher effectiveness. This group of 14 institutions became the Network for Excellence in Teaching, NExT.

One of the first actions of the partnership was the establishment of a Common Metrics workgroup with the charge of creating measures of program effectiveness. The Common Metrics group produced four highly valid and reliable surveys. At the time NExT took on this challenge, valid and reliable measurement instruments were rare in the teacher preparation profession. The initiative required extensive investment from the institutions beyond the funds provided by the Bush Foundation including countless hours of faculty time; expertise within and beyond the institutions; and intense collaboration across institutions, with P-12 stakeholders and with external constituents. NExT members were aware of the importance of these instruments being trustworthy in order to produce data that could be used for program improvement and evaluation. The three surveys of interest to this blog include the following:

• Exit Survey: administered to student teachers near the end of their experience. They rate their preparedness for student teaching.
• Transition to Teaching Survey: administered to graduates in the academic year after their program completion. If teaching, they rate their preparedness for their first year as well as the resources available in their schools. If not teaching, they identify what they are doing and why.
• Supervisor Survey: administered to the supervisors of first year teachers. Supervisors rate the preparedness of the teachers for their first year.

The surveys are aligned with each other and the InTASC standards for professional teachers. This alignment enables comparison with other program measures and among the surveys. The surveys are administered by more than 60 institutions in seven states, which also allows for the creation of super-aggregate reports that can be used not only for benchmarking but also for identification of common areas for improvement that can be addressed collaboratively through Communities of Practice that cross institutions.

Disaggregating Data to Improve Outcomes
The surveys are administered each year at NDSU. Resources are allocated for faculty time to enable high response rates for representative, robust data. Reports are prepared for the overall NDSU aggregate and data are disaggregated for individual program areas. We also disaggregate the data for our Noyce scholarship recipients, so we can see their results and compare them to peers who are not in these elite programs.

NDSU was awarded a Noyce grant in 2012 to recruit career changers into the teaching profession in mathematics and science titled, “Pipeline to Excellent Rural Teachers,” or PERT. With a no-cost extension, we are in the process of concluding the grant at the end of 2018. We are serving 14 fellows through this grant; the last fellow completes student teaching fall 2018.

Although the fellows complete the same professional education classes as their math and science education peers, fellows complete a full major in their content area and participate in additional experiences intended to enhance and add value to their preparation. For example, we have held multiple Noyce Learning Nights with local experts who provide interactive learning experiences aimed at helping the fellows become stronger classroom teachers.  We have also held other professional development events such as a day-and-a-half mini-conference focused on educational technologies and practices in math and science education. The fellows are also offered numerous opportunities to attend Noyce regional and national events and other national STEM conferences.

The patterns in the survey results for the PERT fellows are similar to their peers in the NDSU teacher preparation program. For example, using educational technology is the lowest rated item in the sampling below for both the Noyce fellows and other teacher candidates. The figure below presents items from the Exit Survey completed by student teachers and compares PERT fellows to other NDSU teacher candidates. The sample size for the PERT fellows is 13 (100%), with one still in the preparation program.  The sample size for all NDSU teacher candidates is 250 for the three-year span of 2015-16 through 2017-18 academic years. The response rate for all teacher candidates ranged from 86-100% for these three administrations.

As expected, PERT fellows rated their preparedness in their content areas slightly higher than their peers, who do not complete a full content area major. The fellows complete 4-5 additional classes depending on the math or science major. The difference is slight, however, which may tell us that the coursework completed by those who major in math education and science education is sufficient for teaching in a secondary classroom.  Instead of the additional content coursework required for Noyce programs, the future teachers may be better served with additional pedagogical coursework.

Putting Data to Work for Future Teachers and Their Students
Because we are preparing our Noyce fellows and all of our teacher candidates for high-need classrooms, we look in particular at their preparation for working with diverse learners. There is a section on the surveys that focuses on different types of learners including English learners, students with mental health needs, students on IEPs and 504 plans, and students in poverty.  We have found over time, this area is the lowest rated by our student teachers (Exit Survey), new teachers (Transition to Teaching Survey), and supervisors of first year teachers (Supervisor Survey).  See the figure below for a representation of the scale for diverse learners in comparison with other scale means for the 2018 administration of the surveys.

The disaggregated results for diverse learners scale for the PERT fellows on the Exit Survey are lower than the overall program results with a scale mean of 2.94. When analyzing this data, we noted that PERT fellows teach in high-need schools at a higher rate than the overall population of student teachers.  For example, one of the 14 PERT fellows student taught at a school on an American Indian reservation where there are extremely high rates of poverty, students identified as English Language Learners, and students with mental health needs.  Another fellow student taught at an alternative school for youth who are in the juvenile justice system.  Yet another student teacher was placed in an urban school in a large city where poverty and diversity are high.

The NDSU teacher preparation faculty use these results to identify areas for improvement. For example, to address the trend of feeling unprepared for working with diverse learners, we added a 15-hour field experience in an English language learner classroom, increased the content related to instruction for English learners, and added a series of units related to working with students with mental health needs.  To enact the mental health curriculum, several faculty members participated in extensive training for Trauma Sensitive Schools to become trainers for our faculty and teacher candidates.  We are continuing to track candidate, completer, and supervisor ratings of preparedness for working with diverse learners using the three surveys.  We are also tracking other program measures such as key assessments and field experience evaluations embedded within the classes where the curriculum changes were made.

Although most of us do not characterize data as fun, it is rewarding and valuable to use good data to inform program improvement and then celebrate when program changes result in positive outcomes for our future teachers. An example of a successful data-driven initiative is our classroom management course revision.  A pattern emerged in the surveys that student teachers and new teachers reported feeling unprepared in classroom management.  In response, the faculty examined the curriculum and identified a promising program being offered through Vanderbilt. In 2014, the Vanderbilt curriculum was adopted for our class: Classroom Management for Diverse Learners.  As noted on subsequent surveys, preparedness for classroom management has risen to one of the higher ratings on the surveys.

Stakeholder Implications
Valid and reliable data upon which to make programmatic decisions has implications for a variety of stakeholders including researchers, teacher educators, P-12 educators, and policy makers.

• Researchers can use data from valid and reliable instruments such as the NExT surveys to learn more about best practices in teacher preparation. These data can be especially valuable when used in conjunction with other high quality data to create robust data sets for comparative and predictive analyses.
• Teacher educators can study current practices and new initiatives to ensure investments are made for best practices that enable well-prepared teachers. These studies need to be on-going in nature to support practices that evolve with the needs of students and communities.
• P-12 schools benefit from a well-prepared work force and can reduce investments needed to fill in gaps that exist when new teachers are not well prepared for their classrooms. In addition, P-12 educators can use the research produced by teacher preparation to identify areas for professional development to maintain a well-prepared teaching force.
• Policy makers can use the data produced about teacher preparation to determine budget allocations and initiatives needed to support high quality teacher preparation, supporting a collaborative approach to education across P-12, higher education, and state and local government.

For additional information, publications, and presentations related to the NExT surveys, see the Common Metrics tab at www.nexteachers.org.