Dive Brief:
- Over half of New York City’s council members called for a two-year artificial intelligence moratorium for the nation’s largest school system in a letter sent Tuesday to Mayor Zohran Mamdani and New York City Public Schools Chancellor Kamar Samuels.
- The 29 council members behind the letter described the New York City Department of Education’s drafted AI guidance as “flawed,” saying that the guidance shares no proposals to strengthen student data privacy protections involving AI companies.
- Demand for AI moratoriums in NYC and beyond are growing alongside a broader push to roll back districts’ 1:1 device programs and screen time in classrooms.
Dive Insight:
On a national scale, a coalition led by the nonprofit Fairplay announced in April that it is calling for a five-year pause on all student-facing generative AI products in pre-K-12 schools. The coalition stated that AI products threaten students’ and educators’ privacy, among other issues.
Concerns over the safety and security of student data are increasingly being cited in arguments by those opposed to AI in schools.
Keeping student data safe from breaches and cyberattacks has been an ongoing challenge for schools since ed tech began proliferating in classrooms, particularly in the last decade. As schools have embraced technology, they have also struggled to keep up with the resources needed to protect the swaths of sensitive student data stored in their systems.
Until there are “rigorous guardrails” in place with public and expert input, the letter said, a pause on AI use in schools is needed “to prevent serious damage to NYC children and their future.”
NYC’s AI guidance for schools, released in March, does require educators and leaders to get approval for an AI tool before it can be used in any school. Specifically, the approval must go through the NYCPS Data Privacy and Security Compliance Process.
The school system said in the guidance that it implemented additional standards in December 2024, that mandate vendors be transparent about their AI capabilities and prohibit the training of AI models with student data.
While the vetting process reviews AI tools for data privacy and security, NYCDOE said in its guidance that it has yet to evaluate for “algorithmic bias, equity impact, or instructional effectiveness.”
The council members’ letter, however, cited an audit released in April by the New York State’s comptroller that found NYCPS’ policy “does not fully align” with the National Institute of Standards and Technology Cybersecurity Framework. The state audit added that NYCPS’ policies did not cover “certain fundamental areas related to data privacy and security.”
The New York City Department of Education did not immediately respond to a request for comment.
In addition to student data privacy concerns, the council members’ letter said the department’s AI guidance does not include ways to mitigate the risks of AI for students, such as negative impacts to students’ cognitive development, creativity, mental health and the environment.
That concern has been echoed by some educators, as seen in a recent poll by NPR and Ipsos in which more than half of teachers said AI is making it more challenging for students to develop critical thinking skills.
Meanwhile, American Federation of Teachers President Randi Weingarten recently called for a ban on student-facing AI tools in elementary schools nationwide in addition to prohibiting screen use among students in pre-K through 2nd grade. AFT reasoned the AI ban is necessary to prevent harm to younger students and to help them develop skills to build relationships and resilience.
Student mental health concerns involving AI have also surfaced in recent years after multiple lawsuits against several tech companies have been filed by the parents of children and teens who allegedly harmed themselves after using AI chatbots.
In May, the Guidelines for User Age-verification and Responsible Dialogue Act — or the GUARD Act — advanced out of the Senate Judiciary Committee and now awaits a vote on the Senate floor. During a markup hearing for the legislation, which would ban AI companions from interacting with children and teens, lawmakers cited a teen who died by suicide after allegedly interacting with the AI companion tool Character.AI.
