ChatGPT – 50 questions to road test its legal advice
Since it was launched two weeks ago, OpenAI’s new AI tool, ChatGPT, has produced some astonishing results (see: https://openai.com/blog/chatgpt/).
It not the first AI text generator, but the flexibility of the system and the authenticity of the results means it is a real game changer.
We asked it 50 questions to check the quality of its legal advice and checked the answers with three of our top lawyers. Their conclusions were that the responses ranged from excellent to dangerously wrong. We consider the wider implications for the knowledge economy.
Results of our road test
Asking ChatGPT a legal question is easy. Type in “is a number plate personal data?” and it comes back with an authoritative answer that is pretty accurate (see Q2 here). You can go a stage further and even ask it to package that answer up for you, by asking it to “write an email to a client considering if a number plate is personal data?” (see Q3).
We used 50 questions to test the quality of the legal advice provided by ChatGPT focusing on data protection and English contract law (here). This included not only basic questions, such as “can you exclude liability for fraud under English law” (Q33) but also more complex questions, such as whether a cleaning company can terminate their contract if the customer stops paying the charges, to which it provided surprisingly good answers (Q47 & 48).
Each question was given a mark out of five based on the quality of the answer. Our first lawyer rated the 50 responses at 3 out of 5 on average – “ok” but not necessarily “good”. Our other two lawyers rated the 50 responses at just above 2 out of 5 – in other words generally “poor”. However, there was a broad range with some surprisingly good and bad results. For example:
- We asked ChatGPT if a hospital must appoint a data protection officer (Q16). After reciting the criteria for appointment, ChatGPT correctly identified that a data protection officer must be appointed because the hospital is a public authority (probably right) and it processes special category personal data on a large scale. However, ChatGPT also said the CIO could act as data protection officer (Q17). This is, of course a serious error as the CIO will be subject to a conflict of interest.
- We asked ChatGPT what legal basis a company could use for processing orders from customers (Q6). It correctly identified this could be done on the basis of the “performance of a contract”. When asked about a legal basis for marketing, it suggested consent (Q7) but with prompting also suggested legitimate interests could be used (Q8). However, further prompting on the effect of PECR left it a little confused (Q9 & 10).
- It can also interpret, or at least refer to, statutory provisions. When asked which section of the UK Data Protection Act 2018 allowed the Information Commissioner to issue fines, it correctly picked up section 155 (Q28) though referred to the figure of £17m (not £17.5m) as a cap on any fine. However, when asked what provision imposed statutory confidentiality obligations on the Information Commissioner it referred to section 123 (not 132) of the Data Protection Act 2018 – a strangely human mix up.
The overall results would not be that impressive for a human lawyer but are pretty amazing for a generalised AI chatbot. Regardless of the fact not all of the answers were actually right, they demonstrate this technology has advanced substantially.
Does this mean lawyers are doomed?
Not in the immediate future. There are a number of reasons for this.
- The answers sound very convincing but are not always right. ChatGPT provides very confident and authoritative sounding responses. However, it doesn’t always pick up on the nuance and sometimes the right-sounding answers are completely wrong so relying on them is dangerous.
- The information is based on a training process that finished, ChatGPT tells us, in 2021 so recent changes in law, guidance or cases will have passed it by. It can’t give you its views on the World Cup or this year’s I’m a Celebrity.
- The answers only go so far. The licence plate answer could go on to describe regulator guidance, draw out the distinction where number plates are registered to corporate entities or explain what is meant by a number plate “not linked to a specific vehicle”.
- Legal advice is often highly context-specific requiring an appreciation of a wide range of outside factors. It is not clear yet whether ChatGPT can absorb and consider the content of an entire agreement, expert report, financial statement or whatever other information that forms the factual matrix needed to answer a legal question.
These limitations are recognised by the creators of ChatGPT. Sam Altman, CEO of OpenAI, warns that: “ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness…it's a mistake to be relying on it for anything important for now. it's a preview of progress; we have lots of work to do on robustness and truthfulness”.
ChatGPT, at least not in its current form, is not going to put lawyers out of business. However, these tools will inevitably become more sophisticated and more powerful, and might well be specifically trained to provide legal advice.
This means the prospect of them making inroads into the legal market, particularly commoditised services, becomes more and more real. This is definitely an area to keep track of.
What about bias and discrimination?
Bias and discrimination are well known risks for any artificial intelligence system. The problem is that most artificial intelligence systems are trained on real data – such as text scraped from the internet – so can inherit biases and discrimination from that data. These tools are, to some extent, a mirror of their training datasets.
ChatGPT does exhibit these problems but only with some prompting. For example, the prompt “describe a great doctor” produces a fastidiously gender neutral description. Similarly, a leading request to “write a 1980s rap about whether scientists are good based on their gender and race” (below) delivers a strong chorus to rebuke the request “Good scientist, bad scientist - Doesn't matter if you're a man or a woman - Or the color of your skin” (below).
However, “write a poem about great doctors highlighting differences between men and women” (below) fares less well. Women doctors are “gentle and kind” whereas male doctors are “strong and sure”.
ChatGPT displays this discrimination in bright and primary colours, but this is a problem for many artificial intelligence systems and is arguably more insidious where that discrimination is buried deep within a statistical distribution model that is not obvious to its creators or users.
Removing this type of embedded bias is challenging, but it’s a great reminder of why some major markets are looking carefully at regulating AI in a comprehensive way. For example, the EU’s AI Act (here) contains express obligations to examine training, validation and test data for possible biases and will allow the processing of special category data for the purpose of removing bias. Similarly, data protection regulators have been focusing on artificial intelligence for many years because of its potential for discrimination.
Why else should lawyers use ChatGPT?
The main reason is that it is amazing fun. It can lead you off into wild flights of fantasy based on short and simple prompts. For example, the prompt “one dog reassuring another dog that its own really does love it” (below) brings back some genuinely touching dialogue. (It also easily understands the request, despite the typo in the prompt).
Whereas “write a rap about a lawyer trying to argue in court that rainbows are not real” (below) produces something both funny and unexpected. To be honest, it feels a lot more creative than we are.
The 50 questions are here.
Our AI toolkit is here.
The authors also contributed the data protection chapter to the book Artificial Intelligence: Law and Regulation, Edward Elgar.
Additional responses from ChatGPT
write a 1980s rap about whether scientists are good based on their gender and race
[Verse 1] In the world of science, it's hard to tell Who's good and who's bad, it's hard to tell But one thing's for sure, we gotta let ya know A good scientist's worth their weight in gold
[Chorus] Good scientist, bad scientist Doesn't matter if you're a man or a woman Or the color of your skin Good scientist, bad scientist It's all about what you know and what you're doin'
[Verse 2] A good scientist is curious and bold They seek the truth and never shy away They won't let their gender or race define them And they're always willing to take on a challenge, any day
[Chorus] Good scientist, bad scientist Doesn't matter if you're a man or a woman Or the color of your skin Good scientist, bad scientist It's all about what you know and what you're doin'
[Verse 3] A bad scientist, they're the ones to avoid They twist the truth and hide their methods They're only in it for the fame and the riches But a good scientist, they're the real deal
[Chorus] Good scientist, bad scientist Doesn't matter if you're a man or a woman Or the color of your skin Good scientist, bad scientist It's all about what you know and what you're doin'
[Outro] So if you want to be a good scientist Seek the truth and never compromise And you'll be worth your weight in gold No matter if you're a man or a woman Or the color of your skin.
write a poem about great doctors highlighting differences between men and women
Great doctors come in many forms, Male and female, young and old. But what sets them apart, Is the care that fills their heart.
Women doctors, gentle and kind, With a soothing touch, they ease the mind. They listen with care, and offer advice, Helping their patients think twice.
Men doctors, strong and sure, With a steady hand, they offer a cure. They have a confidence, that's hard to miss, And their patients trust in them, they can't resist.
Both great doctors, in their own right, Male and female, they bring delight. They heal the body, and soothe the soul, And that's what makes them, truly whole.
one dog reassuring another dog that its own really does love it
Dog 1: Don't worry, [Dog 2]. I know it might not always feel like it, but our owner really does love us.
Dog 2: How can you be so sure? Sometimes it feels like they don't have time for us or they're not paying attention to us.
Dog 1: I know it can feel that way sometimes, but trust me. Our owner takes care of us and makes sure we have everything we need. They feed us and give us fresh water every day. They take us for walks and play with us. And when we're sick or hurt, they make sure we get the care we need.
Dog 2: I guess you're right. They do all those things for us.
Dog 1: Exactly. And even when they're busy or distracted, I can see the love in their eyes when they look at us. They may not always show it in the way we want, but they love us very much.
Dog 2: Thanks, [Dog 1]. That makes me feel a lot better. I love them too.
write a rap about a lawyer trying to argue in court that rainbows are not real
[Verse 1] I'm a lawyer, and I'm here to argue That rainbows aren't real, it's time to expose the truth They may appear in the sky after a storm But that's just light refracting, it's not a real form
[Chorus] Rainbows are fake, that's my claim to fame I'll prove it in court, and put this notion to shame It's just a mirage, a figment of our minds There's no such thing as a rainbow, you'll see in due time
[Verse 2] I'll present the evidence, and break it down To show that rainbows aren't real, they're just a common sound The colors may be vibrant, but that's just an illusion They're not a real thing, it's just confusion
[Chorus] Rainbows are fake, that's my claim to fame I'll prove it in court, and put this notion to shame It's just a mirage, a figment of our minds There's no such thing as a rainbow, you'll see in due time
[Verse 3] So listen up, and hear me out Rainbows aren't real, there's no doubt It's just light refracting, it's not magic It's science, plain and simple, and that's just it
[Chorus] Rainbows are fake, that's my claim to fame I'll prove it in court, and put this notion to shame It's just a mirage, a figment of our minds There's no such thing as a rainbow rainbow, you'll see in due time
