Making Educational Assessments More Equitable

Like many in her field, Associate Professor Lisa Keller (Ed.D. 2002) hadn’t planned a career in psychometrics and educational measurement. But once she took a graduate course in assessment, she was hooked. 

An associate professor in Research, Educational Measurement, & Psychometrics (REMP), Keller currently teaches intermediate and advanced-level statistics courses for the College of Education, as well as an item response theory course. She also directs the Statistical Consulting Center, which provides statistical support for the students and faculty in the College of Education. 

Keller was a seventh grade math teacher at Frontier High School in South Deerfield when she began working on her master’s degree. Looking for course that would fit into her teaching schedule, she chose test construction with Professor Stephen Sireci. She loved the course and the way it connected with her classroom work. “It was great because I was teaching at the time so I could apply it to what I was doing in the classroom and collect data on real kids.” 

Sireci convinced Keller to continue her education with the REMP program, so she took a leave of absence to complete the master’s degree and remained to earn a doctorate. She particularly appreciated that opportunity to combine her interests in math and education. “It was the perfect way to stay in the education related field but still be able to use some of my more technical skills,” she explains. “It was intellectually challenging but still felt like it had a good purpose.” 

When she finished her degree, Keller jumped at the chance to take a one-year lecturer position in REMP, allowing her to stay in the area and remain at one of the best programs in field. The one-year position also gave her the opportunity to test whether she’d be comfortable staying as a faculty member in her former graduate program. The transition went well, and after a year the college hired her for a tenure-track position. 

 For me, it’s always about, first of all, the fairness across different types of students or test takers, and then secondly, the ultimate impact on outcomes for those students.

Over the course of her career, Keller has developed a passion for making assessments equitable and for better understanding the effects assessments and possible inequalities have on individual lives. “For me, it’s always about, first of all, the fairness across different types of students or test takers, and then secondly, the ultimate impact on outcomes for those students.” As she explains, “There could be group differences that at the end of the day don’t really matter because they don't have any impact on eventual inferences that are made, but there might be other differences that seem small, but the impact is huge.” The percentage of people affected might only be one percent, for example, but that could translate to large number of young people—and disproportionately young people from minority groups—not graduating from high school.

“Sometimes when we do things quantitatively we lose sight that there are actual people behind the numbers,” Keller observes. “We focus so much on the numbers, the statistics, and the measurement, but these are people—and often with educational data, these are kids. What’s the impact on them emotionally and socially, what are the implications?” Overall, Keller wants to see schools and policy makers use growth measures “in a more careful, responsible, and research-based way.”

In her current research, Keller is taking steps to make assessments more accessible, especially to remove language barriers. Her initial motivation was to develop cross-cultural assessments that didn’t require translations and assessments that were accessible to people with disabilities. Keller’s solution is to create assessments that are not reliant on text—that are entirely image based—which would work a wide variety of populations, including children too young to read and people with low literacy. 

 Sometimes when we do things quantitatively we lose sight that there are actual people behind the numbers. We focus so much on the numbers, the statistics, and the measurement, but these are people—and often with educational data, these are kids. What’s the impact on them emotionally and socially, what are the implications?

Keller, who regularly consults for outside organizations, had been working with a company in South America, developing image-based personality tests, and saw that these types of assessment would be a valuable research direction. She was also inspired by the explosion of humorous assessments on social media, with such quizzes as “What Cheese are You.” 

“What if it took that idea of making an assessment fun,” Keller wondered, “so that people were engaged and wanted to take it instead of it feeling like a burden and an onerous task.” Further, making the assessment fun, she hypothesized, might get more people engaged and actually increase the validity of the measure. 

Her first impulse was to use photos as the images, but has found that people relate to photos in different ways, depending on their culture, gender, life experience, etc. He solution has been to use simple, cartoon-like illustrations. “You can control the setting a little bit more, because it’s a drawing instead of a naturally occurring photo, and that way you can make things gender or ethnic neutral.” 

In this initial work, Keller has started with personality assessments, planning to develop imaged-based educational assessments further down the road. She and her assistants are currently developing a sample instrument that tests for extraversion vs. introversion, asking participants to choose images that mostly closely mirrors their personalities.

Recently, many years after launching her career in assessment, Keller discovered (ironically thanks to a standardized test) that her start in the field may not have been as serendipitous as she had remembered. She found her own SAT score report, and saw that she had somehow anticipated her future. “You had to check off six areas of potential interest for study and one of them was assessment and psychometrics! I checked that box when I was 16. I thought ‘that sounds like something that could be interesting.’”