Human vs. Machine?

Ciarb Brazil Branch holds the first-ever Vis Moot demonstration round with ChatGPT.

On 8 March 2023, ChatGPT (Chat Generative Pre-Trained Transformer) was put to the test by Ciarb Brazil Branch and the Arbitration Channel in the first-ever Human vs. Machine Vis Moot Demonstration. The two organisations got together to stage a moot hearing in the spirit of the Willem C. Vis International Arbitration Moot.

A team of law undergraduates was pitted against a team comprised of ChatGPT, a student actor and a budding political scientist, neither of whom had any legal training and were instructed only by ChatGPT. The tribunal members consisted of Amb.(r.) David Huebner C.Arb FCIArb, Sophie Nappert and Louise Barrington C.Arb FCIArb.

Fernanda Romero G. Pereira (lawyer and mootie), and Douglas Amorim de Oliveira (data scientist), provided ChatGPT with input over the course of the fortnight preceding the event, training and preparing it for the Vis Moot. Given the technical limitations of the current version of ChatGPT, the event team designed a format to break the Vis Moot’s fictional case into subsections to enable ChatGPT to produce a coherent initial pleading and be ready to answer questions and provide opposing arguments within the scope of each subsection.

Read on to learn about some of the team’s findings and their views on the practical implications of using Chat GPT in arbitral proceedings. Report by Fernanda Romero G. Pereira and Douglas Amorim de Oliveira, the legal and tech law ChatGPT coaches from the event, and by Cesar Pereira C.Arb FCIArb.

How did you use ChatGPT in the Vis Moot demonstration?

Our aim in the demonstration was to test the performance of ChatGPT as a contestant in the Vis Moot. For this purpose, we used ChatGPT for the preparation of almost everything you would expect of a contestant: sound arguments, structured pleadings, rebuttals and surrebuttals to the opposing party's arguments, and answers to the questions of the arbitral tribunal. The only point left to the humans was the oral presentation – which we left to an actor and a political scientist without any legal training to avoid any interference.

What were the results?

The results of the pleadings were fascinating. It is worth mentioning that, in the process of getting ChatGPT to prepare the pleadings, we encountered a few technical difficulties. Among these difficulties were the limits of ChatGPT's trained database, the limits of ChatGPT's acceptable inputs (such as the size and format of each message), and the limits of its storage capabilities. Since ChatGPT is not able to browse the internet, its knowledge is limited to its trained database and to the information provided in each message. However, once we managed to provide ChatGPT with the main facts of the Vis Moot case, the final pleadings were clear, grounded, and very well structured.

How did ChatGPT perform in the Vis Moot?

In the Vis Moot round, besides the pleadings which were prepared beforehand, ChatGPT had to answer questions and prepare rebuttals and surrebuttals on the spot. This added an element of uncertainty. After all, while the pleadings took several tries until we reached the final versions, the responses and rebuttals would be the first ones prepared on the spot by ChatGPT. To our surprise, in general, ChatGPT still provided convincing answers and rebuttals.

Why is that interesting for arbitration?

The demonstration showed:

(i) That certain human aspects, such as flexibility and sensitivity to the environment, are still essential in the context of arbitration; and

(ii) That there is a lot of potential for the use of artificial intelligence tools as a way to support the activities of lawyers in arbitration, for example, in the brainstorming, preparation, structuring, and critical analysis of arguments. We have been involved with the Vis Moot in various capacities, including as mooties. Although getting ChatGPT to provide useful pleadings and acceptable answers required some work from us, it was only a fraction of all the work that human teams put into their preparation for Vis Moot.

What are the core things in arbitration that can be done by ChatGPT, and to what extent?

From our experience, ChatGPT performed very well at structuring texts and arguments clearly and concisely. In addition, provided with the necessary guidelines, ChatGPT also performed well in the search for legal grounds and judicial precedents. In fact, with the rapid advance of AI, ChatGPT or similar AI tools could be a valuable resource for legal research in the future.

Nonetheless, one should consider a few caveats. First, as of now, ChatGPT is not trained on specific legal datasets, and it is not able to browse the internet. As a result, it may provide inaccurate, outdated, or incomplete information. Second, ChatGPT, as an AI language model, may “hallucinate”; i.e. it may generate outputs that sound correct but are not factually so. Therefore, we believe all responses provided by ChatGPT must be carefully reviewed by trained legal practitioners.

With the advent of AI and tools such as ChatGPT, what is the question we should be asking ourselves as arbitrators?

Among the several possible questions that arise with the advent of AI, it is fundamental for arbitrators to keep in mind the presence of artificial intelligence and its shortcomings, which will require constant questioning as regards the veracity and reliability of the information provided. We should also ask ourselves what the acceptable trade-off between quality and efficiency in advocacy is. Can we accept a machine-generated argument in a format we are not used to but which requires much less cost and effort to produce? How can human effort in arbitration be more productively used if we can rely on artificial intelligence to support it?

Watch Human or Machine? March 2023 Vis Moot Demonstration

About the authors:

Fernanda Romero G. Pereira is a lawyer with experience in corporate law and arbitration. She is an undergraduate student at the School of Economics of Fundação Getulio Vargas (FGV) in São Paulo, Brazil and a former mootie, having attended the 2018 edition of the Willem C. Vis Moot.

Douglas Amorim de Oliveira is pursuing a master’s degree in Artificial Intelligence at the University of São Paulo (USP). Douglas also holds a Bachelor's degree in Computer Science and an MBA in Data Science from the same institution. He serves as Quanto's Data Products and Data Science Manager, a Brazilian fintech focused on Open Finance. In addition, he is a postgraduate professor at FIAP (Faculdade de Informática e Administração Paulista).

Cesar Pereira C.Arb FCIArb is the immediate past chair of Ciarb Brazil Branch. He is a partner at Justen, Pereira, Oliveira & Talamini (Brazil), co-heading the firm’s infrastructure and arbitration practice. His work as counsel, arbitrator, or legal expert focuses on infrastructure projects, regulated industries, public procurement and international contracts, and related dispute resolution. In addition, he has authored or edited numerous articles, chapters, and books in these fields. Cesar holds a doctorate in public law from PUC-SP (Brazil) and has been a visiting scholar at Columbia University, University of Nottingham and George Washington University.

Keen to discuss and learn more about the impact of AI on arbitration? Join Ciarb's Let's Discuss the Impact of AI in Arbitration on 1 June 2023 at 16:00 BST, when Philippa Charles, Stephen Dowling, Anna Förstel-Cherng, Michael Greenop and Amb. (r.) David Huebner C.Arb FCIArb will be discussing the future of AI in dispute resolution. The series is free for Ciarb members.