The arrival of Artificial Intelligence (AI) has been nothing short of revolutionary, transforming industries, both streamlining and disrupting the workflows, and challenging traditional methods of problem-solving. For tax professionals and litigators, AI presents both an opportunity and a dilemma. On one hand it promises unprecedented efficiency allowing professionals to navigate complex legal landscapes with ease. On the other hand, it raises concerns about reliability, ethical use, and the potential erosion of human expertise. Some fear AI might replace them, while others see it as a tool that, if used wisely, could enhance the precision and effectiveness of tax compliance and litigation.

How AI Works and the Role of Large Language Models (LLMs)

To understand AI’s role in taxation, it is essential to grasp how it functions. At its core, AI relies on vast datasets and advanced algorithms to process information, identify patterns, and generate responses. Large Language Models (LLMs), such as those powering AI chatbots and research tools, are trained on extensive legal, financial, and tax-related datasets. These models work by predicting the next word in a sequence based on previous inputs, allowing them to generate human-like text, summarize legal provisions, and even interpret judicial reasoning.

The power of LLMs lies in their ability to recognize intricate connections between laws, precedents, and tax regulations. They can analyse thousands of court judgments, extract key arguments, and identify trends in rulings. This pattern recognition is particularly valuable in tax litigation, where past judgments often influence future decisions. By analyzing similarities between cases, AI can suggest relevant precedents that human researchers might overlook, thus providing a more comprehensive legal strategy.

However, despite their remarkable capabilities, LLMs do not “think” like humans. They do not understand the law in the way a trained professional does. Instead, they rely on probability and pattern-matching, which sometimes leads to errors—most notably, AI “hallucinations,” where the system generates fictitious but seemingly legitimate citations.

AI Hallucinations: When AI Fabricates Reality

A major issue with AI-generated legal research is its tendency to hallucinate information. ‘Hallucination’ in AI refers to the phenomenon where an AI model, instead of admitting that it does not have an answer, generates a response that sounds plausible – but is entirely fabricated. This problem arises because LLMs are designed to predict the most statistically probable answer based on their training data, not to verify facts or ensure the authenticity of their outputs.

Unlike a human researcher who might say, “I don’t know,” or leave a question unanswered when unsure, AI models prioritize coherence over correctness. This means that when an AI model is prompted to provide a legal citation, but no such case exists in its training data, it does not respond with “No such case exists.” Instead, it generates something that looks like a valid case reference—complete with a case name, year, and citation format—that aligns with the query but is entirely fictional.

The recent case from the Bengaluru bench of the Income Tax Appellate Tribunal (ITAT) illustrates this danger. The tribunal cited three Supreme Court judgments and one Madras High Court ruling that, upon verification, turned out to be non-existent. This error likely resulted from AI-generated research that fabricated case laws to fit the argument being presented. Because the tribunal members and tax department representatives did not cross-check the citations with authoritative sources, the fabricated cases made their way into an official ruling before being withdrawn. These fake cases were generated by ChatGPT and presented as real precedents. The deception was exposed when the opposing party, Avianca, found that the cited cases did not exist in any legal database. Even after being given multiple opportunities to correct their mistake, the attorneys stood by the false citations until the court issued a stern order requiring them to produce the original case texts. This resulted in a sanctions order against the attorneys for failing to verify AI-generated case laws, constituting bad faith and an abuse of the legal system.

Mata v. Avianca: A Stark Warning on AI Misuse

A similar incident occurred in the Mata v. Avianca case, decided by the United States District Court, S.D. New York, in June 2023. In this case, attorneys from the law firm Levidow, Levidow & Oberman P.C. submitted a legal brief citing non-existent judicial opinions with fake quotes and citations. These fabricated cases were generated by ChatGPT, an AI tool, and presented as real precedents.

The deception was exposed when the opposing party, Avianca, noted that the cited cases could not be found in any official legal database. The court itself attempted to verify the cases and found no record of them. Even when given multiple opportunities to correct the mistake, the attorneys involved continued to stand by the fake opinions until the court issued a stern order requiring them to provide the original case texts. This led to a sanctions order against the attorneys, with the court ruling that their failure to verify AI-generated case laws constituted bad faith and an abuse of the legal system.

India’s AI Model and the Need for AI-Trained Legal Databases

India is taking a significant step toward building its own indigenous AI model, as announced by the Ministry of Electronics & Information Technology. While the Indian AI model is currently set to train on repositories of public media like Prasar Bharati (Doordarshan & All India Radio), there is a strong case for extending its scope to Indian laws and judicial pronouncements. Training AI models on a structured database of Supreme Court and High Court judgments, ITAT rulings, and statutory amendments would:

• Prevent hallucinations by restricting AI outputs to verified legal sources.
• Enhance legal accuracy by ensuring AI tools reflect real-world precedents.
• Enable faster and more reliable legal research tailored for Indian tax litigation.

By leveraging India’s high-end compute infrastructure and integrating AI safety mechanisms such as algorithm auditing, bias mitigation, and privacy frameworks, the country can pioneer an AI-driven legal research ecosystem that significantly reduces errors and enhances trust in AI-generated insights.

The Road Ahead for Tax Professionals: AI Will Not Take Your Job, But a Person Who Knows AI Will

The increasing role of AI in tax research and litigation means that professionals must adapt to technological changes while ensuring responsible usage. AI will not replace tax professionals, but those who fail to embrace it risk being left behind by colleagues and competitors who leverage AI to enhance efficiency and accuracy.

What Tax Professionals Should Do

• Cross-Verify AI Findings: Always verify AI-generated case laws against official legal databases before citing them in litigation.
• Learn Prompt Engineering: Knowing how to craft precise queries will improve the accuracy of AI-generated research.
• Stay Updated with AI Developments: Attend workshops and training sessions on AI applications in taxation and litigation.
• Use AI as an Assistant, Not an Authority: AI should support human judgment, not replace it. Always interpret AI insights critically.
• Advocate for AI Regulations in Legal Research: Push for industry-wide standards to ensure AI-generated legal research is reliable and accountable.

What Tax Professionals Should Avoid

• Never Blindly Trust AI-Generated Case Laws: AI hallucinations can create fictitious citations that lead to legal and reputational risks.
• Avoid Relying on Generic AI Models: Use AI tools that are specifically trained on tax laws and judicial pronouncements, rather than general chatbots.
• Do Not Overlook AI Bias: Ensure AI-generated legal research provides a balanced perspective, rather than favouring either tax authorities or taxpayers.
• Don’t Ignore AI Ethics: Professionals should advocate for ethical AI practices and ensure that AI-driven tax analysis is transparent and explainable.

Conclusion

The Mata v. Avianca case, like the Bangalore ITAT case in India, underscores the dangers of unverified AI-generated legal research. AI holds great promise in revolutionizing tax litigation and compliance, but unchecked reliance can lead to serious errors and professional liabilities.

The future tax professional must embrace AI while maintaining rigorous verification processes. By learning skills like prompt engineering, staying updated on AI safety measures, and using verified AI-driven legal databases, tax professionals can maximize AI’s benefits while safeguarding against its risks.

AI will not take your job, but a person who knows AI definitely will. With the right safeguards, training, and responsible use, AI will become a powerful ally rather than a disruptive force—helping tax professionals navigate the future with confidence and competence.