{"id":22396,"date":"2024-01-17T09:45:24","date_gmt":"2024-01-17T09:45:24","guid":{"rendered":"https:\/\/www.imprima.com\/?p=22396"},"modified":"2024-02-28T18:59:50","modified_gmt":"2024-02-28T18:59:50","slug":"llm-based-ai-how-accurate-is-it","status":"publish","type":"post","link":"https:\/\/www.imprima.com\/blog\/llm-based-ai-how-accurate-is-it","title":{"rendered":"LLM-based AI – how accurate is it"},"content":{"rendered":"\n

In our previous post \u201cShould we blindly trust AI?<\/a><\/strong>\u201d, we discussed the main benefit of an accurate AI tool for information extraction. And that is that it saves the user time, and not that it takes the human out of the loop.<\/p>\n\n\n\n

At Imprima we like to be open about how accurate AI can be. Here we share the results from our latest tests.<\/p>\n\n\n\n

For this experiment, we had a test set of legal documents, which contained 40 different data points (By \u201cdata points\u201d we mean pieces of key information such as clauses, names and dates, such as: effective date, expiration date, anti-assignment, governing law, exclusivity, non-compete, change of control etc.).<\/p>\n\n\n\n

We extracted these datapoints with Imprima\u2019s Smart Summaries<\/a> tool.<\/strong><\/p>\n\n\n\n

Here we show the test results for a selection of data points:<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"Smart
In this experiment we selected the data points for which the most data was available for verification (at least 75 data points for verification for each unique datapoint), in order to optimise the statistical significance of the experiment. Recall is a measure of how many of the data points are found, where 0% means none were found, and 100% means all were found. A False Positive means a data point that was incorrectly identified as a valid result.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

The main take aways from these test results are:<\/span><\/p>\n\n\n\n