Introduction
India’s Digital Personal Data Protection Act, 2023 (“DPDP Act”) represents a monumental statutory assertion of digital sovereignty for 1.4 billion citizens. However, the simultaneous emergence of Generative AI (“GenAI”), trained on vast, opaque datasets and deployed over borderless global cloud infrastructure, creates a profound operational collision.
Where the DPDP Act establishes a territorially rooted, consent-driven legal regime, GenAI operates on general-purpose, purpose-agnostic models that exist everywhere and nowhere at once. This article evaluates the core structural frictions between the DPDP Act and frontier GenAI systems, examining the consent paradox, erasure impossibility, black-box explainability barriers, national security inference risks, and enterprise liability under Shadow AI.
The Consent Paradox in a GenAI World
Under Section 6 of the DPDP Act, consent must be free, specific, informed, unconditional, and unambiguous, accompanied by a clear notice detailing the specific purpose of processing. While manageable in static workflows such as banking KYC, GenAI breaks this paradigm due to the general-purpose nature of foundation models.
A data principal might consent to an AI tool summarizing emails, but that same data could subsequently refine a model that generates pitch decks, drafts legal opinions, or crafts political messaging, purposes completely unknown to the data fiduciary at collection. Furthermore, once personal data is ingested into pre-training sets, it ceases to be a temporary service input and becomes permanently embedded within mathematical model weights, influencing indefinite downstream outputs across multi-modal applications.
Erasure Rights Versus Immutable Model Weights
The DPDP Act confers upon Data Principals the right to seek correction, completion, and erasure of personal data, alongside seamless consent withdrawal. In traditional database architectures, erasure is achieved through a straightforward row-deletion command. In GenAI systems, personal data is baked into billions of numerical parameters. Surgically extracting an individual’s data requires complex machine unlearning or cost-prohibitive model retraining. Where the law presumes data can be retrieved like a document from a file cabinet, GenAI operates like dye dissolved in an ocean, creating a fundamental gap between statutory rights on paper and technical feasibility that will likely trigger extensive regulatory litigation.
Transparency, Explainability, and Black-Box Models
The statutory architecture of the DPDP Act relies on transparency, requiring data protection impact assessments, processing logs, and explicit notices to ensure accountability. However, frontier GenAI models are non-linear black boxes whose outputs emerge from complex interactions across billions of parameters. Tracing a specific output back to particular training data is technically infeasible. As a result, even diligent Data Fiduciaries may find themselves unable to provide case-specific, legally defensible explanations to regulators or affected individuals regarding why a specific algorithmic output was generated.
Strategic Autonomy, National Security, and Inference Risk
The intersection of GenAI and privacy extends beyond individual rights into national security and strategic autonomy. Within the Union government, significant concern exists regarding inference risk, wherein foreign GenAI platforms infer sensitive state priorities or confidential policies from prompt patterns and uploaded documents. This prompted the Union Finance Ministry in early 2025 to prohibit public LLM tools on official workstations.
To counteract this dependency, the India AI Mission is advancing swadeshi GenAI stacks and indigenous LLMs, establishing technological self-reliance where AI sovereignty requires owning both the data and the underlying algorithms.
Cross-Border Data Flows and Sovereignty Trade-Offs
The DPDP Act adopts a negative list approach allowing cross-border data transfers except to restricted jurisdictions notified by the Central Government. For GenAI deployments, this creates a delicate balancing act between protection and participation. While restricting transfers to low-protection or adversarial jurisdictions safeguards sensitive data, an overly defensive posture risks isolating Indian enterprises and startups from cutting-edge global cloud APIs and high-performing foundation models, thereby hindering innovation and escalating operational costs.
Enforcement Gaps: Vulnerable Groups and Shadow AI
The DPDP Act mandates heightened protections for children and persons with disabilities, enforcing verifiable parental consent and severe statutory penalties up to INR 250 crore per instance of non-compliance. However, a stark sovereignty gap exists, as domestic platforms face strict enforcement while foreign-headquartered GenAI platforms operating without a physical presence in India process children’s data with relative impunity. Similarly, in corporate environments, Shadow AI, employees using unapproved personal GenAI
accounts, severely compromises compliance. Pasting proprietary code, contracts, or customer data into public LLMs violates security safeguard duties, exposes confidential information, and leaves minimal traditional access logs for incident response. To operationalize compliance in automated pipelines, the DPDP Act introduces Consent Managers, registered intermediary entities managing consent artifacts and rights requests on behalf of data principals. Although not originally designed for AI, Consent Managers, alongside Significant Data Fiduciaries subject to mandatory impact assessments and periodic audits, will form the primary interface bridging human rights mandates with automated machine learning architectures.
Practical Implications for Data Fiduciaries Deploying AI
Entities integrating or deploying GenAI tools must proactively re-align their operational and compliance frameworks. Ingestion protocols and purpose notices should explicitly address algorithmic model refinement where feasible. Enterprise organizations must enforce strict technical controls, including endpoint monitoring and API proxy gates, to eliminate Shadow AI risks. Furthermore, vendor due diligence must mandate contractual guarantees regarding cross-border compliance, data isolation, and statutory breach notifications, while technical teams establish documented protocols defining the boundaries of erasure feasibility and auditable prompt logging.
AMLEGALS Remarks
The DPDP Act is a general-purpose privacy statute arriving at a GenAI-dominated inflection point. While its principles of accountability and citizen rights provide a robust normative foundation, GenAI fundamentally challenges the law’s underlying assumptions regarding data traceability, reversibility, and purpose foreseeability.
The true test of India’s AI governance regime will depend on Data Protection Board adjudications, cohesive sectoral guidelines from regulators such as the RBI, SEBI, and IRDAI, and the rapid maturation of privacy-enhancing technologies like machine unlearning and synthetic data. India has asserted a firm normative stake that technological innovation must not supersede citizen privacy or national sovereignty, and the imperative now is converting that principle into pragmatic operational standards.
For any queries or feedback, feel free to connect with mridusha.guha@amlegals.com or Khilansha.mukhija@amlegals.com