Chatbot tracking California students fails as questions mount about kids’ privacy

Weeks before the collapse of an artificial intelligence tool tracking California students academically, a whistleblower was raising privacy and other concerns, according to multiple media reports.

The AI chatbot was touted by a California district superintendent as recently as April for its ability to “transform education,” even as it was clear the project was in trouble, according to the New York Times.  

Less than two months later, the Los Angeles Unified School District (LAUSD) shuttered the project when the contractor, technology company AllHere, laid off most of its staff, citing financial problems, reports EdSurge.

The project nonetheless went forward, in part as a progressive answer to conservative efforts to reform education, judging by public comments made by the LAUSD superintendent hyping the AI plan.

According to EdSurge, the company raised $12 million in venture capital and has already been paid half of the $6 million LAUSD agreed to pay in a five-year contract.  

Reports have surfaced that a company whistleblower has been warning district officials, the LAUSD independent inspector general’s office and California state education officials that the chatbot violated district privacy rules, and that the company was unqualified to complete the job.  

“When AllHere started doing the work for LAUSD, that’s when, to me, all of the data privacy issues started popping up,” Chris Whiteley, the former senior director of software engineering at AllHere, told The74 in an interview.  

Whiteley charged that the chatbot, named “Ed,” stored the personally identifiable data of students in ways that made the data vulnerable to breaches and violated privacy laws and best practices for data security.  

Those violations included sharing data with third parties, and the use of foreign servers to store and process the data, reported The74.  

That’s because the scope of the project was too ambitious to satisfy the goals of the district, while also maintaining students’ privacy, said experts.  

It’s also because the project was, in part, influenced by the politics of K-12 education being employed in districts from coast-to-coast.   

“Imagine the power of artificial intelligence and comprehensive data working together to personalize an action plan for the benefit of our teachers, our students, our parents,” LAUSD Superintendent Alberto Carvalho pitched last year to an audience at Walt Disney Concert Hall, according to the Los Angeles Times. 

He made the comments during a chatbot presentation the LA Times characterized as a “pep rally” that included a Korean fan dance and an American Idol-style National Anthem.  

Imagine, said Carvalho, parents getting “real-time updates on grades, test results and attendance — empowering them to monitor and support progress and immediately address the concerns.” 

Carvalho promised the use of technology would cut the time spent on “standardized testing” up to 45%, a favored goal of teachers’ unions, which generally oppose standardized testing.   

Carvalho then contrasted an AI-powered future in Los Angeles schools with other approaches to improving academic performance, such as reforms instituted in Texas and Florida that have advocated going back to basic education models that have proven effective in the past.   

He accused Florida, under Republican Gov. Ron DeSantis, of wanting to ban “a book, every book.” 

In a further stab at Republicans, Carvalho added that “deliberate ignorance shall not influence who we are or what we stand for,” reported the Los Angeles Times.  

Experts have noted some educators and technology enthusiasts want AI to solve problems in a way that overestimates what AI can do in the real world. 

“There’s a dream that AI is just more or less automatically going to solve all or many problems [of K-12]. It’s overhyped. That is not how learning and education works,” Ashok Goel, a professor of computer science and human-centered computing in the School of Interactive Computing at Georgia Institute of Technology, told EdWeek. 

Now, Carvalho’s grandiloquent political vision appears to have led to engineering shortcuts in the AI chatbot that allegedly violate the data privacy of kids and families. Whiteley told The74 that personally identifiable information was included in each chatbot prompt, even when the data wasn’t relevant to the task.  

He said the chatbot shared unnecessary data about students with third parties, making that data vulnerable to hackers.   

Carvalho’s comments on the project have highlighted the dangers and the difficulties of turning over kids’ education – and data – to AI.  

In April, Carvalho acknowledged to a group of venture capitalists at an AI conference hosted by Arizona State University that the chatbot would have access to student data on test scores, mental health, physical health and family socioeconomic status, said the New York Times.  

Whiteley told The74 that needing that much data on demand meant “cheating” privacy rules.  

“The issue is, we’re sending data overseas, we’re sending too much data, and then the data were being logged by third parties. The product worked, but it worked by cheating. It cheated by not doing things right the first time.” 

And that exposes students and their families to danger.  

Carvalho bragged in April that one 7^th-grade girl who tested the chatbot said, “I think Ed likes me,” reported the New York Times. In fact, the 7^th–grader may be buoyed by the feeling that the chatbot “likes” her.  

Critics warn technology is good at seducing kids that way.  

But neither Superintendent Carvalho, nor the LAUSD, nor the tech company AllHere, nor the state of California can assure the young girl or her parents the data she shared with “Ed” won’t be used against her.  

The74 notes this is the third data breach for LAUSD in the last month, including what may be “one of the largest [data breaches] in history.”  

In 2022, the district was also subject to a cyberattack that ensnared up to 2,000 students.