Introduction 

Recently, the National Human Rights Commission (NHRC) raised concerns about possible risks to children’s privacy arising from an artificial intelligence collaboration between the US-based AI company Anthropic  and the Indian education NGO Pratham. The system in question, known as the “Anytime Testing Machine (ATM)”, processes children’s handwritten responses and academic data using AI tools. 

A complaint filed by the  NAMO Foundation  alleged that the system may expose children’s personal data to vulnerabilities such as improper storage, processing risks, and potential cross-border data transfers. These concerns have been raised particularly in the context of India’s Digital Personal Data Protection Act, 2023 (DPDP Act). 

Taking cognisance of the issue, the NHRC issued notices to all states and union territories, as well as to relevant ministries, including the Ministry of Electronics and Information Technology and the education departments. Authorities have been asked to review agreements with NGOs handling children’s data and submit an “action taken” report within two weeks. The incident has renewed attention on the responsibilities of NGOs when they collect and process personal data, especially data relating to children. 

This development highlights a broader regulatory question: how should NGOs ensure compliance with data protection laws while adopting emerging technologies such as artificial intelligence? 

Emerging Data Privacy Concerns in AI-Enabled Educational Systems 

AI systems often require large datasets to function effectively. This creates a situation where educational institutions or NGOs may unintentionally accumulate more personal data than necessary. Without clear safeguards, such systems could expose children’s information to risks such as data breaches or unauthorised access. 

The NHRC’s intervention, therefore, reflects a broader concern about whether organisations deploying AI technologies are adequately considering privacy and legal compliance obligations, particularly when dealing with vulnerable groups such as children. 

Why the Issue Has Gained Attention 

The matter has gained significant attention primarily because it sits at the intersection of  three fast-moving sectors: artificial intelligence, children’s rights, and data protection law. 

First, the use of AI in education has expanded considerably in recent years. AI-based assessment tools, digital learning platforms, and automated feedback systems are increasingly used by schools and NGOs to improve educational outcomes. These systems often require the processing of large amounts of student data, which creates new governance challenges. In the Indian context, this shift is particularly evident through large-scale implementations that aim to bridge the gap between overstretched teaching resources and diverse student needs. 

Under the DPDP Act 2023, the definition of a "Child" is strictly any individual who has not completed eighteen years of age. Unlike international frameworks like the GDPR, which allow for a lower age of digital consent, the Indian framework maintains a high threshold, necessitating "verifiable parental consent" for almost all AI-driven educational interactions. This creates a significant operational hurdle for NGOs using automated systems like the Anytime Testing Machine, as the verification process must itself be privacy-preserving and robust. 

The following examples illustrate how these technologies are being deployed and the governance frameworks emerging to manage them: 

1. Mindspark (Adaptive Learning) 

Used extensively across Rajasthan and Haryana through partnerships with organisations like Educational Initiatives, this AI engine identifies specific learning gaps. 

• How it works: If a Year 7 student struggles with Year 3 maths, the AI pivots to foundational lessons until they achieve mastery. 

  
  

• The Result: Research by J-PAL shows that students using this platform experience nearly double the learning gains of those in traditional settings. 

2. Embibe (AI Teaching Assistants) 

In states like Goa and Nagaland, the Embibe platform acts as a digital co-teacher, providing high-fidelity resources to schools that might have limited infrastructure. 

• Assessment: Teachers use AI to generate syllabus-aligned exams in minutes, while students receive automated feedback on "behavioural" mistakes, essentially identifying if an incorrect answer was due to "carelessness" or a genuine "lack of concept". 

  
  

• Visualisation: Through Embibe Lens, students can use a smartphone to turn flat 2D textbook diagrams into interactive 3D models. 

3. Governance and Data Challenges 

The processing of large datasets has prompted a shift toward formalised Data & AI Governance. Recent 2026 roundtables in Mumbai have highlighted the move from simple "tracking" to using AI for "system strengthening". 

• Predictive Analytics: Systems are now being used to predict dropout risks by analysing attendance and performance patterns, allowing for early intervention. 

  
  

• Ethical Oversight: With the 2026-27 mandatory AI curriculum rollout, there is an increased focus on "safe & trusted AI", ensuring that the processing of student data remains transparent and adheres to the NISHTHA upskilling framework for teachers. 

Second, the issue comes at a time when India has recently enacted the Digital Personal Data Protection Act, 2023, a comprehensive law regulating how organisations collect, store, and process personal data. The Act establishes clear responsibilities for entities that determine the purpose and means of processing personal data. 

Third, the involvement of children has heightened regulatory scrutiny. Data relating to minors is treated with greater caution under most privacy frameworks because children may be less capable of understanding consent mechanisms or potential risks associated with data sharing. 

Finally, concerns regarding cross-border data flows have also contributed to the significance of this issue. When AI systems involve collaboration between domestic organisations and foreign technology companies, questions arise regarding where the data is stored and whether it is transferred outside India. 

These combined factors explain why the NHRC has sought explanations from multiple government departments and state authorities. 

Legal Framework & Key Compliances under the DPDP Act 

NGOs that decide the purpose of collection and then collect and process data will usually be treated as data fiduciaries under the Digital Personal Data Protection Act, 2023 (hereinafter “DPDP Act”) [Section 2(i)]. Therefore, all the applicable obligations of data fiduciaries need to be kept in mind, which include the following:

• Process data for a specific purpose: NGOs acting as data fiduciaries are allowed to process personal data only for the specific purpose for which consent was obtained [Section 4(1)(a)]. Data may also be processed for certain “legitimate uses” permitted under the Act [Section 4(1)(b)]. For instance, if an NGO collects personal data for a health awareness programme, it cannot subsequently use that data for unrelated activities such as fundraising. In this case, the student data processed by the "Anytime Testing Machine" must be used strictly for educational assessment and not for broader AI training or data harvesting without fresh consent. 

  
  

• Notice before collecting personal data: The obligation to provide notice before collecting personal data is particularly significant for NGOs operating at the grassroots level. NGOs must give the data principals a clear and simple notice explaining what personal data is being collected and why [Section 5]. This notice must be available in English or any language listed in the Eighth Schedule of the Constitution. For the pilot project in question, Pratham must ensure that the notice regarding digitising handwritten responses is provided in the local vernacular to ensure total transparency for the students and parents involved. 

  
  

• Explicit, informed and free consent: Organisations must ensure that the consent request is presented in clear and understandable language [Section 6]. The Data Principal has the right to withdraw consent at any time, and the process must be as simple as giving it. This carries heightened importance for NGOs because they often operate where beneficiaries may depend on essential aid. In the Anthropic-Pratham collaboration, active steps must be taken to ensure parents genuinely understand the involvement of a third-party US firm so that consent is not merely implied or coerced by the need for educational support. 

  
  

• Store data for only the necessary time: Personal data should be stored only for as long as it is necessary to fulfil the purpose of collection [Section 8(6)]. Once the project is complete or consent is withdrawn, the organisation must delete the data. Under the Act, once the Anytime Testing Machine has finished grading a student’s response, the sensitive images of handwritten work should be deleted rather than retained indefinitely, pushing the NGO toward proper data lifecycle management. 

  
  

• Reasonable Security Safeguards: NGOs should adopt security measures such as encryption, masking, or obfuscation of personal data [Rule 6, DPDP Rules 2025]. They must also maintain logs of data access and ensure proper backup mechanisms. This is especially relevant here, as Pratham relies on foreign technical infrastructure. Contractual safeguards with Anthropic are essential to ensure that student records remain secure during cross-border data transfers. 

  
  

• Grievance Redressal: NGOs must establish grievance redressal mechanisms so that data principals can raise complaints [Section 8(10)]. They must remain accountable and provide clear communication channels. For this collaboration, this means setting up accessible helplines or designating officers to address any queries from parents or the NAMO Foundation regarding how their children’s data is handled. 

  
  

• Dealing with Children: Verifiable parental consent must be obtained before processing the personal data of children. Verifiable parental consent must be obtained before processing the personal data of children. Additionally, processing must not involve behavioural monitoring or tracking [Section 9(3)]. It is critical to note that Section 9(3) of the Act explicitly prohibits any processing of personal data that is likely to cause a "detrimental effect" on the well-being of a child. In the context of AI-based assessments, if an algorithm incorrectly categorises a student’s learning disability or "behavioural mistakes" in a way that limits their future educational opportunities, the NGO could be held liable for a breach of duty, regardless of whether parental consent was obtained. 

  
  

• If your NGO deals with research work:  Under [Section 17(2)(b)], personal data may be processed for research without fully complying with certain provisions, provided it is not used to make decisions about specific individuals. If the Anytime Testing Machine is used to provide individual grades or tailored learning recommendations, this exemption will not apply, and full compliance with the DPDP Act is mandatory. 

  
  

• Significant Data Fiduciary (SDF): The Central Government may classify certain organisations as SDFs based on the volume or sensitivity of data handled [Section 10]. If this pilot project expands to reach a large portion of the student population, Pratham may be required to appoint an India-based Data Protection Officer (DPO) and conduct annual Data Protection Impact Assessments (DPIA) to evaluate the risks of the AI software. 

Regulatory Gaps and Emerging Challenges 

Despite the framework provided by the DPDP Act, several challenges remain in the context of AI-driven data processing. One challenge relates to technological complexity. AI systems often process large datasets in ways that are not always easily explainable. 

The 2025 Rules have also clarified the "Right to Erasure." For educational AI, this implies that once a student exits a specific learning programme, the NGO has a proactive duty to ensure that not only the raw data but also any "derived data" or "fine-tuned weights" specifically linked to that child’s identity are purged from the AI firm's cloud servers, unless retention is mandated by a specific Indian law. 

Another challenge is the growing use of cross-border technological collaborations. When NGOs partner with international technology companies, ensuring compliance with domestic data protection rules can become more complicated. Additionally, many NGOs operate with limited financial and technical resources, which may make it difficult to implement sophisticated security safeguards.  

There is also an ongoing policy discussion about how traditional consent-based frameworks apply to AI systems that continuously learn from large datasets. 

Strengthening Data Protection Practices for NGOs 

To address these challenges, NGOs adopting AI tools may need to adopt stronger governance practices. 

• First, organisations should conduct regular risk assessments before deploying AI technologies that process personal data. This can help identify potential vulnerabilities early. 

  
  

• Second, transparency measures should be strengthened. Clear communication with parents, students, and other stakeholders about how data is used can improve trust and accountability. 

  
  

• Third, collaboration between NGOs, government regulators, and technology providers may help establish shared standards for privacy-protective AI systems, particularly in educational contexts. 

Finally, capacity building within NGOs, such as training staff in data protection practices, can help ensure that legal obligations are properly implemented. 

Conclusion 

The NHRC’s inquiry into the Pratham Anthropic collaboration highlights a broader shift in the governance of digital technologies in India. As NGOs increasingly adopt artificial intelligence tools to support education and development programmes, questions about data protection, transparency, and accountability are becoming more prominent. The Digital Personal Data Protection Act, 2023, provides a bedrock legal framework to regulate how organisations collect and process personal data. However, effective implementation of these rules requires careful attention to compliance practices, particularly when children’s data is involved. 

Want to know more?  

Learn more about India's data protection environment, compliance frameworks, and in-depth analyses of privacy policies at  Tsaaro.com.