{"id":21654,"date":"2023-01-17T12:39:28","date_gmt":"2023-01-17T12:39:28","guid":{"rendered":"https:\/\/www.imprima.com\/?p=21654"},"modified":"2023-01-19T16:46:08","modified_gmt":"2023-01-19T16:46:08","slug":"smart-vdrs-it-is-all-about-accuracy","status":"publish","type":"post","link":"https:\/\/www.imprima.com\/blog\/smart-vdrs-accuracy","title":{"rendered":"Smart VDRs \u2013 It is all about Accuracy"},"content":{"rendered":"\n

Last week we shared our blog on Smart VDRs with you. It included some accuracy stats on Smart Summaries, the key Machine Learning tool to extract data points from transaction documentation (such as dates, locations, etc.). Before that we shared the accuracy stats of our Smart Redaction tool.<\/p>\n\n\n\n

Today we would like to zoom in on these results and discuss what they mean for you.<\/p>\n\n\n\n

AI powered redaction sounds good. So does fully automated redaction\u2026<\/p>\n\n\n\n

But none of that is of value if the results are not accurate: it needs to actually work. That is why, at Imprima, we share our test results with you. We seem to be unique in that. You may ask yourself why other vendors are not doing that. As one of our customers, who used Smart Redaction to redact sensitive data in their client\u2019s data room, said: \u201cThe accuracy rates are high which was something we didn\u2019t experience before when using other redaction tools \u2013 a welcome change indeed.\u201d <\/p>\n\n\n\n

Why is accuracy important? And what do we mean by \u201caccuracy\u201d?<\/p>\n\n\n\n

First, let us reiterate the results we obtained:<\/p>\n\n\n\n

<\/div>\n\n\n\n

Smart Redaction test results<\/strong><\/p>\n\n\n\n

\"\"
Test results of Imprima Automatic Redaction on a combination of English, German, Italian and Dutch documents, randomly picked from Imprima VDRs, where the objective was to automatically redact: person name, address etc., see table. Imprima\u2019s machine-learning algorithm (a native tool in the Imprima VDR) was applied. As can be seen from the results in the table, a very high recall is achieved for all entities, in all languages, where overall recall is higher than 90% for all items redacted. So, what does this mean? Recall is a measure of\u202freliability, in particular how many of the items to be redacted are actually found. Not shown in the table, but this was also achieved with very high precision (a measure for how few words were redacted that did not need to be redacted): only a very tiny fraction of all words in the document were \u201cover redacted\u201d. If you would like more information on this experiment, please contact Imprima<\/a>.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Smart Summaries test results<\/strong><\/p>\n\n\n\n

\"\"
Test results on a Real Estate transaction with Dutch documents. As can be seen both precision (a measure of the absence of False Positives) and Recall (a measure of the absence of False Negatives) are high (F1 is a measure of overall accuracy, combining Recall and Precision). Average recall is 86%, meaning that 86% are automatically found. This was achieved after training on less than 100 examples per data point. When combined with manual review of the AI results (Technology Assisted Review), it will result in both very significant time saving, and higher reliability.<\/em><\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Accuracy – what is it, and why is it important – Saving time and increased reliability<\/strong><\/p>\n\n\n\n

In essence it is all about how many data points can be automatically (and correctly!) retrieved.  In other words: \u201cRecall\u201d1<\/sup>.<\/p>\n\n\n\n

So how does this all help you? Obviously, if – as in the above cases – around 90% of the data points are automatically extracted or redacted, then you then only have to verify and add the missing ones.<\/p>\n\n\n\n

Obviously, that means that a lot of time is saved. Quoting yet another customer: \u201dThe AI-based redaction option is an excellent tool and saved us a tremendous amount of time.\u201d In addition, the reliability of the end results is significantly improved. Humans are not infallible, as research has shown the ability of humans to retrieve key data from documentations results in recall of only 50%-85% (according to a 2011 scientific study by Grossmann and Cormack).<\/p>\n\n\n\n

<\/div>\n\n\n\n

Technology<\/strong><\/p>\n\n\n\n

The reason that we can achieve such high accuracy is that we use a Neural Net that:<\/p>\n\n\n\n