This story is the second in a two-part series. Read Part 1 here.
The Wake County Public School System, like other districts in North Carolina, uses the Education Value-Added Assessment System (EVAAS) to rate its teachers.
But EVAAS has drawn fire from education academics, teachers and teacher organizations in the state and around the country.
Despite these criticisms, Wake Superintendent Tony Tata is already attempting to use EVAAS to staff struggling schools and classrooms with teachers deemed effective by the program.
Chief among EVAAS’s critics is Audrey Amrein-Beardsley, assistant professor at Arizona State University, who has published the most comprehensive studies articulating problems with the program.
The North Carolina Association of Educators (NCAE), the state’s largest teacher organization, has also heard complaints from educators across the state and documented abuses of EVAAS.
Critics say EVAAS:
- Does not take into account a student’s background
- Has been used as a punishment
- Has not been sufficiently assessed in peer-reviewed scholarly journals
- Is confusing for many teachers
- Encourages teaching to the test
Student Background
Just before the last school board election in 2009, a report surfaced that criticized WCPSS’s in-house system for rating its teachers – the Effectiveness Index.
William Sanders, developer of EVAAS and senior director of the EVAAS K-12 division at SAS, helped author the report.
SAS criticized the Effectiveness Index for adjusting student performance data by considering socioeconomic factors.
“For a rich white kid and a poor black kid we should expect the same level of academic achievement and want the same expectation for both kids next year,” Sanders said in a phone interview with The Record.
The report seemed to lend some credence to conservative school board contenders — who went on to win that October — looking to do away with the policy of assigning students to schools based on socioeconomic status.
Of course, the new conservative board majority accomplished this goal in 2010. But many education academics believe socioeconomic status is essential to any study of teacher effectiveness.
Beardsley points to a preponderance of data showing socioeconomic background predicts a student’s performance in school. Poorer kids tend to do worse academically than their more affluent counterparts.
“The real question here is … do students from, let’s say, low socioeconomic backgrounds learn as fast as students who have two parents, who travel, who have tutors outside of the classroom perhaps, who have homework priorities, who read as part of their leisure time, etc.?” Beardsley wrote in an e-mail to The Record.
If so, she said, then EVAAS — “the only sophisticated analytical model for measuring student achievement that does not account for student background factors” — will show inherent biases against teachers of low-income students and in favor of teachers of affluent students.
The current school year marks the third full year that WCPSS has used EVAAS. Wake County Superintendent Tony Tata has repeatedly said that he wants to use EVAAS data to pair high-performing teachers with low-performing students.
William Sanders, developer of EVAAS, claims that he has evidence that teachers who move from one environment — affluent and high-achieving — to a radically different environment — low-income and underachieving — generally continue to do just as well in EVAAS terms.
[pullquote]“The real question here is … do students from, let’s say, low socioeconomic backgrounds learn as fast as students who have two parents, who travel, who have tutors outside of the classroom perhaps, who have homework priorities, who read as part of their leisure time, etc.?”~Audrey Amrein-Beardsley[/pullquote]
But Sanders offers no proof of this claim.
Whether teachers perform well with varying sets of students at new schools may depend on how the matches are made.
“There should be some kind of perk,” said Angela Farthing, manager of the Center for Teaching and Learning of the NCAE. “Just because a teacher is successful at Panther Creek High School and Enloe High School doesn’t mean they’ll be successful at Halifax County Schools. Involuntary transfer is not going to help. It takes all the staff at a school to be successful. If you transfer a teacher to a school, take a high-performing teacher from one school and put them in schools that are not high-performing, that may not be the only piece to help turn the school around.”
“Hammer Instead of a Carrot”
As potential data for teacher evaluations, EVAAS ratings are protected as confidential personnel information under state law. But the law has not always been respected.
Farthing cites Guilford County, where EVAAS teacher ratings were posted in school hallways, as an example of districts using EVAAS to harm instead of help.*
Also under state law, EVAAS ratings can be part of a teacher’s evaluation but cannot be used as the sole basis for termination.
Yet throughout the state, said Farthing, administrators imply or directly state that decisions about termination are linked to testing data.
“The reason that [EVAAS] has gotten a negative connotation is that principals say that if your scores are bad, you’re gone,” she said.
Using EVAAS data alone — apart from observations, artifacts and other effectiveness measures — to make decisions about a teacher’s future goes against scientific standards, Beardsley said.
“Given this standard [of multiple measures] was developed by the American Educational Research Association, American Psychological Association and the National Council on Measurement in Education, those arguing, not evidencing, that these models are so strong that they can indeed operate alone and in isolation are essentially giving the best measurement specialists and researchers in the field an ideological middle finger,” Beardsley wrote.
Sanders said EVAAS was never designed to be used in isolation.
Wake County uses EVAAS data as part of the teacher evaluation process, but Assistant Superintendent for Evaluation and Research David Holdzkom denied the district uses the program to punish teachers.
Farthing knows of no instances of misuse of EVAAS data in Wake County. But Tata has expressed his willingness to publish the effectiveness ratings of individual teachers.
“I know if my kid was in school and you were a highly effective teacher, I would want to know,” said Tata, after a press briefing in April.
“It’s being used as a hammer instead of a carrot,” Farthing said. “Students are more than test scores. We’re focusing more on the test scores than on helping students be learners and thinkers.”
Some education academics question the assumptions programs like EVAAS make about teachers.
“I’m not sure the discussion about good teachers is even a useful one,” said Jason Osborne, associate professor of educational psychology at North Carolina State University. “I’m sure the vast majority of teachers are doing their darnedest to do what’s right for students. It may be that it’s not the right environment. It may be that the students aren’t coming to the table and allowing teachers to help them.
“We need to get away from seeing teachers as heroes and villains,” he added. “Teachers are only half the story, or less than half.”
EVAAS is password-protected and accessed online. Other than board members and staff in the offices of the superintendent and the Evaluation and Research Department, the only school personnel that have access to EVAAS are principals and any designees that a principal chooses, such as department chairs. Teacher ratings are not visible to other teachers, but student ratings are, making it simple to pair a student’s rating with his or her teacher.
Peer Review
Multiple reports on EVAAS have noted the shortage of available data on the program in respected academic journals.
Sanders points to the U.S. Department of Education’s approval of Tennessee’s growth model, which uses a version of EVAAS, as evidence the program has been fully vetted and validated. But Beardsley and other researchers do not have findings that this evidence adequate.
“Because these [value-added models] certainly make sense, and nearly everyone agrees that these models are improving the current accountability system, our federal and state policymakers … have essentially gone ‘all in’ without nearly enough evidence that they work in the ways intended, and for which they are being oversold,” Beardsley wrote.
Moreover, researchers have questioned the reliability of EVAAS, complaining that in some cases teachers are rated effective one year and ineffective the next.
Sanders acknowledges that with less data EVAAS’s ratings become less reliable. He says that with only one or two years of data available for a teacher, the margin of error can be as high as 50 percent.
“This simply isn’t good enough to make consequential decisions,” Beardsley wrote. Even with three years of data, the district can only be 80 to 90 percent certain that the ratings are accurate.”
Teacher Confusion
Farthing believes that to use EVAAS correctly and to its full potential, teachers and principals have to completely understand it. Impossible, she said, as long as SAS maintains its algorithms as proprietary secrets.
“I have seen presentation after presentation [about EVAAS] and I couldn’t tell you how it all works,” Farthing said. “It’s something that statisticians understand, but the average educator and the average parent does not understand.”
Farthing believes EVAAS works best when principals allow access to all of their teachers.
Sanders counters that the complexity of EVAAS is its strength.
“The No. 1 huge disadvantage of more simple value-added models is that they throw away data that they don’t have for kids,” he said.
EVAAS compensates for missing data due to students entering the district late or missing tests, problems that disproportionately affect poorer children.
“I shudder to think how many places have adopted that simple approach [of other value-added models] without realizing how grossly inaccurate they are,” Sanders said.
Teaching to the Test
Since EVAAS uses end-of-grade and end-of-course test data, some argue the program pushes teachers to focus only on content and skills featured on those exams.
“Some policy makers say that’s what we should be doing,” said Farthing, who used to teach social studies. “But for a long time the curriculum was a mile wide and an inch deep.”
The North Carolina Standard Course of Study is the state’s curriculum for most subjects. This curriculum has been improving, Farthing said, but the standardized tests are still lackluster.
“The U.S. History EOC is an awful test,” she said. “Recall a constitutional amendment or a paper from The Federalist Papers. It’s just a recall test.”
A focus on testing makes for uninspired teachings, Farthing believes.
“To teach to the test means you have to practice the test,” Farthing said. “Students get tired and bored.”
Some teachers — those in grades K-2 and those not teaching core subjects — do not have to deal with the standardized tests. This puts unfair pressure on teachers of higher grades and subjects like reading and math, Farthing said.
She acknowledges that EVAAS has proved useful in student math placement.
“One thing I’ve heard [from teachers] is that it does help them if they know a student is going to need help in algebra,” she said.
bleclaire@raleighpublicrecord.org
Read more about EVAAS pros and cons in these related studies and published papers:
