Blog: CaseCrunch v Lawyers NOT Deep Blue v Kasparov

Brian Inkster writes on a legal battle between lawyers and AI.

CaseCrunch is a legal AI startup founded by Cambridge law students. On Friday, they claimed to have made history in the legal profession.

They have held what they stated to be the world’s first competition to directly pit lawyers against artificial intelligence in a “Man v Machine” battle.

Artificial Intelligence won the competition, scoring 86.6 per cent accuracy compared to the lawyers’ 62.3 per cent.

The 112 lawyers who participated in the Lawyer Challenge were presented with factual scenarios of PPI mis-selling claims and asked to predict “yes or no” as to whether the Financial Ombudsman would succeed in the claim. The same factual scenarios were given to CaseCrunch: whoever had the highest accuracy, won.

The participants of the competition were apparently 112 of London’s top lawyers, including several ‘BigLaw’ partners, from firms such as Bird & Bird, Kennedys, Weightmans, Allen & Overy, Berwin Leighton Paisner, DLA Piper, DAC Beachcroft, DLA Piper, Pinsent Masons and Eversheds Sutherlands (none noted for their PPI mis-selling expertise).

775 predictions were submitted by the participants. A Technology Judge and a Legal Judge independently verified the fairness of the competition.

The factual scenarios were real decided cases from the Financial Ombudsman Service, published under the FOIA. All identifying details – such as the name of the parties, case names and dates – were removed, leaving only the facts. Lawyers completed their predictions in an unsupervised environment; they were able to use all available resources.

The official press release specifically states that:

“PPI mis-selling was chosen as the basis of the competition because it matched the background of most lawyers taking part in the challenge and is an area of law that is easier to learn about than others. Participants were given links to the Financial Conduct Authority’s rules detailing the basis of an Ombudsman’s decision.”

Matching the background and the lawyers knowing anything about PPI mis-selling may, however, be two very different things.

The Legal IT Insider, in their more balanced report, quoted Ralph Cox, a patent litigation partner at Clyde & Co, as saying:

“I keep an open mind. However, it struck me that the computer was given all the database information and therefore had an unfair advantage. Lawyers who took part were from outside this area of expertise without any real experience of PPI; they had to do it in a limited time.”

Nadia Abdul, one of the tech start-up founders, admitted to Legal IT Insider that:

“The system does not have the experience of a lawyer who has worked in the field of PPI.”

So not quite like Deep Blue beating Garry Kasparov at chess.

I wouldn’t bet my chances against AI on a subject I know nothing about. Pitted against a crofting law robot, there might be a fair fight.

When CaseCrunch takes on PPI mis-selling specialists and beats them then that might be newsworthy.