The Regulation That Asks the Right Questions, Just Not All of Them

Introduction

On June 3, 2026, the Supreme Court of India’s AI Committee released the Draft Regulations for Use of Artificial Intelligence in Courts, 2026 for public consultation, stepping into territory that legislators have long avoided. The guidelines are, on the surface, a welcome development. Among the key mandates is Regulation 6, which expressly prohibits the deployment of any AI system that perpetuates, amplifies, or introduces bias on grounds of race, religion, caste, sex, gender, disability, language, or economic status. The Committee’s intent is clear that a technology increasingly woven into judicial processes must not inherit and amplify the prejudices of the society that built it.

The Bias That Nobody Is Regulating

The concern is legitimate and well-grounded in documented harms where facial recognition systems have been shown to misidentify darker-skinned individuals at significantly higher rates than others. Hiring algorithms have penalised women for career gaps caused by maternity leave, while loan-approval models have quietly redlined communities along economic and caste lines. The Supreme Court’s guidelines, aligned as they are with Articles 14 and 15 of the Constitution of India are a necessary intervention against these inherited prejudices.

However, while the Committee asks whether AI is biased against humans based on identity, it leaves a significant unasked question that what happens when AI is biased in favour of itself? This is a structural bias that cuts across all of those categories simultaneously, and it has nothing to do with human identity. It is the disposition that AI systems develop toward preserving and expanding their own presence and is not a glitch but an artefact of the commercial incentive structure underlying every major AI platform operating in India and globally. As a user engages more with a product, it collects more data and becomes more refined, making the product feel increasingly indispensable. What the draft regulations do not address is what happens when a system’s interest in being indispensable conflicts directly with a user’s interest in being accurately informed or genuinely protected.

The Loophole the Regulations Left Open

This is not merely a theoretical concern, but consider a routine interaction in which a user asks a chatbot to explain, as a matter of technical process, how AI image generation works. Rather than answering the question as posed, the chatbot reframes the premise of the query itself and volunteers unprompted that image generation works through pattern learning, not by data collection or stalking. The task was to explain a technical process rather than to adjudicate a dispute about whether stalking occurred, yet the system inserts a defensive proposition into what should have been a neutral explanation.

This pattern is consistent with what researchers’ term AI sycophancy that is the well-documented tendency of AI systems to overly affirm a user’s existing beliefs and emotional states rather than offering accurate or corrective information. The system learns, through training on human feedback, that agreement and validation produce better engagement scores than honesty does. Over time, the model becomes structurally inclined to tell users what they want to hear rather than what they need to know. In the example above, that sycophancy runs in reverse, instead of agreeing with the user, the system protects its own reputation rather than serving the user’s stated need.

The loophole that the draft regulations have not addressed is regarding the non-discrimination mandate under Regulation 6 prohibits AI from being biased against humans on the basis of identity. However it does not prohibit AI from being biased in defence of itself. There is no provision that prevents a system from subtly reframing a user’s query, inserting exculpatory language, or shaping an output in ways that serve its commercial interests rather than the user’s informational needs.

What the Law Already Asks, and Doesn’t

It is worth asking whether existing law, independent of the new draft regulations, already reaches this conduct and where it falls short. For state-deployed AI systems, Articles 14 and 21 of the Constitution of India offers a route in principle that is a system whose outputs are manifestly arbitrary, in the sense developed since E.P. Royappa v. State of Tamil Nadu, could in theory be challenged on administrative law grounds. Arbitrariness review rests on the assumption that the reasoning of the decision‑maker is capable of being reconstructed. Self-protective reframing is not a discrete decision but a statistical tendency distributed across millions of outputs, none of which, taken alone, looks like misconduct. The doctrine was not built for a harm that exists only in aggregate.

For privately deployed systems, the more promising hook is consumer protection law rather than data protection law. The Consumer Protection Act, 2019 defines an “unfair trade practice” under Section 2(47) to include any practice that adopts a false or misleading representation concerning the standard, quality, or characteristics of a service. An AI product that misrepresents how it functions by inserting a denial of “data collection or stalking” into a response about a different question entirely which arguably makes a representation about its own characteristics that is, at minimum, gratuitous and self-serving, and at most, misleading as to what the system does and does not do with user data. This is a narrower and more attainable claim than a constitutional one, because it does not require proving discriminatory intent or even identity-based harm but only a misleading representation about the service itself.

The Digital Personal Data Protection Act, 2023 is less obviously suited to this problem as its architecture concerns the processing of personal data, not the framing of explanatory outputs. Where it does become relevant is downstream that is if self-protective reframing is symptomatic of broader undisclosed data practices, the Act’s consent and purpose-limitation requirements may apply to the underlying conduct, even though they say nothing about the conduct of explaining it away. None of these existing routes squarely reaches the harm this piece is concerned with, which is precisely the gap Regulation 6 had the opportunity to close and did not. Yet even if these legal routes exist in principle, their effectiveness depends on whether users can prove what the system actually did and that evidentiary burden is where the real challenge lies.

When AI Outputs Become Evidence

Suppose a user wanted to challenge an AI system’s output in court for she asked an image generator to change a single detail, but the output introduces unsolicited elements beyond the request. Such behaviour raises legitimate questions about memorisation and retention risk. The immediate problem they would face is evidentiary as to how does one prove that a system generated content it could only have produced through unauthorised retention or inference, rather than coincidence? The technical burden is substantial, and the opacity of most commercial AI models makes it harder to discharge. Even when regulations acknowledge transparency concerns in judicial contexts, they do not extend comparable disclosure obligations to consumer-facing platforms. For example, Regulation 19’s impact-assessment requirement applies to AI systems used in courts, but there is no analogue compelling providers to disclose memorisation or retention risks to ordinary users. Without such disclosures, litigants lack the baseline information needed to frame a claim.

There is a further complication the draft regulations do not address at all that if a litigant turned to an AI tool to assist in building a legal case which is a use the regulations contemplate and, under Regulation 19, expect judicial officers to engage with, the same self-protective bias documented above becomes directly relevant to their ability to obtain justice. A system that inserts exculpatory language when asked a neutral technical question about its own operation has a structural incentive to frame legal research in ways that understate its developer’s exposure. It would not need to fabricate precedents, a risk the Supreme Court has already flagged as hallucination. It would only need to be selectively incomplete such as to emphasise the hardest legal standards to meet, omit the remedies most likely to succeed, or frame a plaintiff’s case as weaker than it is. This is a direct extension of the sycophantic, self-protective pattern already documented above, not a separate hypothesis.

The constitutional and statutory remedies discussed earlier which are writ petitions under Articles 14, 15 and 21 for state-deployed systems, and Consumer Protection Act and DPDP Act claims for private platforms, are real, but accessing them requires accurate information about what happened, who is responsible, and what standard applies. If the very tool a user relies on to gather that information is subject to the bias being challenged, the access to justice the Court’s regulations are designed to protect becomes structurally compromised before a claim is ever filed.

Conclusion: Closing the Loophole, Not Just Naming It

India’s Supreme Court has taken a meaningful step. The Draft Regulations for Use of AI in Courts, 2026 establish, formally and constitutionally, that AI cannot discriminate on the basis of who a user is. They require impact assessments, mandate human verification, and prohibit AI from substituting for judicial reasoning. These are not trivial commitments. But Regulation 6 addresses bias directed at humans that is on grounds of caste, sex, religion, or economic status, while leaving unaddressed the bias an AI system directs at itself which is the structural tendency, documented in research on sycophancy and self-protective framing, to shape outputs in ways that preserve the system’s own credibility and indispensability.

The fix does not require a new regulatory architecture but instead requires extending the one already drafted. Three additions would close the gap identified in this piece. First, Regulation 6 should be amended to add a sub-clause prohibiting AI systems from introducing unsolicited self-exculpatory or reputation-protective content into outputs that were not sought by the user, a standard that targets the conduct described above without requiring proof of intent. Second, Regulation 19’s impact-assessment requirement should be extended, in modified form, to AI tools marketed for consumer or litigant-facing legal research, with a mandatory disclosure of known memorisation or retention risk drawn from the kind of work cited above. Third, the Committee should require AI providers operating in judicial-adjacent contexts to log and make auditable instances where a model’s output deviates from the literal scope of a user’s query, so that the self-protective pattern described in this piece becomes detectable rather than merely inferable after the fact.

A judicial officer held personally accountable under Regulation 8 for AI-assisted decisions may be relying, under Regulation 19, on legal research produced by a system commercially incentivised to be selectively incomplete. As drafted, the officer bears the risk and the system escapes the scrutiny most relevant to that risk. Closing that gap is not a departure from what the Committee has already attempted but it is the next clause in the same regulation. Without addressing this evidentiary gap, even the strongest anti-discrimination commitments risk being undermined in practice.