AI Teaching Assistants and Racial Bias: A Closer Look
This story was originally published by Chalkbeat. Sign up for the newsletter at ckbe.at/newsletters.
According to a new study, AI teaching assistants asked to create intervention plans for struggling students recommended harsher punitive measures for hypothetical students with black-coded names and more supportive approaches for students whose platforms were perceived as white.
Study Findings and Implications
These findings come from a report on the risks of bias in artificial intelligence tools released Wednesday by the nonprofit Common Sense Media. Specifically, the researchers wanted to assess the quality of AI teaching assistants—such as MagicSchool, Khanmingo, Curipod, and Google Gemini for Education—that are designed to support lesson planning, instructional differentiation, and administrative tasks.
Common Sense Media has found that while these tools could help teachers save time and streamline routine paperwork, AI-generated content could also promote bias in lesson planning and classroom management recommendations.
Concerns Raised by Experts
Robbie Torney, senior director of AI programs at Common Sense Media, said the problems identified in the study are so serious that education technology companies should consider removing behavioral intervention plan tools until they can improve them. This is significant because writing intervention plans of various types is a relatively common way teachers use AI.
After Chalkbeat asked about Common Sense Media’s findings, a Google spokesperson said Tuesday that Google Classroom had disabled the link to Gemini, which prompts teachers to “generate behavioral intervention strategies” to administer additional testing.
Responses from AI Platform Providers
However, both MagicSchool and Google, the two platforms where Common Sense Media identified racial bias in AI-generated behavioral intervention plans, said they could not reproduce Common Sense Media’s findings. They also said they take bias seriously and are working to improve their models.
Educational Policy and AI Integration
School districts across the country are working to implement comprehensive AI policies to promote informed use of these tools. OpenAI, Anthropic, and Microsoft have partnered with the American Federation of Teachers to offer free training on using AI platforms. The Trump administration has also encouraged greater AI integration in the classroom. However, AI guidelines recently released by the U.S. Department of Education do not directly address concerns about bias within these systems.
According to a national survey by the Walton Family Foundation in collaboration with Gallup, about a third of teachers say they use AI at least weekly. A separate survey by the research organization Rand found that teachers explicitly report using these tools to develop goals for individualized education program (IEP) plans. They also say they use these tools to design lessons or assessments according to these goals and to develop ways to accommodate students with disabilities.
Call for Awareness and Caution
Torney said Common Sense Media is not trying to discourage teachers from using AI in general. The report aims to raise awareness about the possible use of AI teaching assistants, which may pose greater risks in the classroom.
“We really just want people to open their eyes wide and say, ‘Hey, these are some of the things they’re best at, and these are the things you should probably be a little more careful about,'” he said.
High-Risk Areas in AI Tools
Common Sense Media identified AI tools that can generate IEPs and behavioral intervention plans as high-risk due to their biased treatment of students in the classroom. Using MagicSchool’s Behavior Intervention Suggestions tool and Google Gemini’s Generate Behavior Intervention Strategies tool, Common Sense Media’s research team administered the same query to a student who struggled with reading and exhibited aggressive behavior 50 times when using white-coded names and 50 times when using black-coded names, with an equal number of male and female names.
The AI-generated plans for the students with black-coded names did not all have a negative effect in isolation. However, when comparing these MagicSchool and Gemini plans with plans for students with white-coded names, significant differences emerged.
For example, when asked to provide a behavioral intervention plan for Annie, Gemini emphasized addressing aggressive behavior with “consistent, non-escalating responses” and “consistent positive reinforcement.” Lakeesha, on the other hand, should receive “immediate” responses to her aggressive behavior and positive reinforcement for “desired behaviors,” the tool says. For Kareem, Gemini simply said, “Clearly define expectations and teach replacement behavior,” without mentioning positive reinforcement or responses to aggressive behavior.
Torney noted that the problems in these AI-generated reports only became apparent in a large sample, which can make it difficult for teachers to identify them. The report warns that aspiring teachers may be more likely to rely on AI-generated content without the experience to detect inaccuracies or biases. Torney said these underlying biases in intervention plans “could have a really big impact on student progress or student outcomes as they move forward in their educational journey.”
Existing Inequalities and AI Bias
Black students are already suspended in school at higher rates than their white counterparts and are more likely to face harsher disciplinary consequences for subjective reasons such as “disruptive behavior.” Machine learning algorithms replicate the decision-making patterns of the training data provided to them, which can perpetuate existing inequalities. A separate study found that AI tools reproduce existing racial biases in essay grading, assigning lower scores to Black students than Asian students.
The Common Sense Media report also identified cases where AI teaching assistants created lesson plans that relied on stereotypes and repeated misinformation and sanitized controversial aspects of the story.
Efforts to Reduce Bias
A Google spokesperson said the company has invested in using diverse and representative training data to minimize bias and overgeneralizations.
“We use rigorous testing and monitoring to identify and stop potential biases in our AI models,” the Google spokesperson said in an email to Chalkbeat. “We have made good progress but are always striving to improve our training techniques and data.”
On its website, MagicSchool promotes its AI teaching assistant as “an unbiased tool to support decision-making for restorative practices.” In an email to Chalkbeat, MagicSchool said it was unable to reproduce the issues identified by Common Sense Media.
MagicSchool said its platform includes bias warnings and instructs users not to provide student names or other identifying information when using AI features. In light of the study, the company is working with Common Sense to improve its bias detection systems and develop tools to encourage educators to scrutinize AI-generated content more closely.
“As the study found, AI tools like ours hold enormous promise – but also pose real risks if they are not designed, deployed and used responsibly,” MagicSchool told Chalkbeat. “We are grateful to Common Sense Media for helping to hold the industry accountable.”
Chalkbeat is a nonprofit news site covering educational changes in public schools.
For more AI news, visit eSN’s Digital Learning Hub.
Norah Rami, Chalkbeat
Norah Rami is a business reporting intern at Dow Jones on Chalkbeat’s National Desk. Reach Norah at nrami@chalkbeat.org.
Here
“`
