Putting continuous active learning to the test

Wednesday, 6 February 2019

By Vishka Peiris

What is CAL?

As the name suggests CAL is an application embedded in Relativity designed to actively learn what documents are considered relevant in large unstructured data sets, based on reviewers’ input. The key advantage of CAL is the real-time intelligence our experts can derive from each coding decision made by a reviewer. From this intelligence our experts can identify if reviewers are in sync in terms of correctly identifying documents as Relevant or Not Relevant or if certain reviewers are coding documents inconsistently. Real time performance monitoring of the review allows the experts to effectively address any issues with the review which in turn causes a drastic reduction in review time. Once the reviewers have a firm understanding of relevance CAL becomes increasingly more effective at accurately identifying Relevant documents.

CAL in the real world – case study

The mission

Our client received a notice from the Australian Competition and Consumer Commission (ACCC) to produce documentation in relation to certain insurance products as part of an inquiry into the insurance sector. As with most inquiries from the regulators the scope of what was to be provided was extensive. This resulted in a substantial number of documents requiring review for relevance and privilege before being handed over to the regulator. The mission – to provide the client with an end to end eDiscovery solution utilising CAL to minimise the financial impact caused by staff down time due to onerous document review, and to complete this process within the stringent time frames set by the regulator.

The data and deadline – mission impossible

The total data set amounted to 360 GB which was processed using Relativity’s processing engine in less than two days. The results of the processing left us and the client with a staggering 1.8 million documents (1.2 million after de-duplication) to review with less than two weeks to review and respond to the regulator.

The hard yards

Our client had 1.2 million documents to review with a team of nine reviewers and less than two weeks to complete their entire review, allow for QA and produce a final set of data to comply with the notice meant the client needed to utilise Relativity’s analytics technology to complete the task. To ensure that the analytics would work effectively our Relativity specialists setup a custom CAL Project in Relativity. This provided the reviewers with documents to immediately start coding (marking documents as Relevant or Not Relevant). Coding decisions are ingested by the CAL model which becomes better at serving relevant documents to reviewers after each individual document is coded by a reviewer. This is the key differentiator between CAL and other TAR methods. One week and two days into the review and our client had reviewed a total of 20,000 documents. At this point our experts reviewed the CAL results to ensure the CAL model was accurately identifying what was Relevant and what was Not Relevant. More importantly the experts were able to compare the reviewers coding against the CAL results to confirm the reviewer’s determination of relevance was aligned with the CAL results. These results were validated by performing an Elusion Test. This test required a review of a random sample of documents based on how the CAL process had ranked them. It established for us and the client that the CAL model was correctly differentiating between what was relevant and not relevant.

The win

Before TAR, a review similar to this, with a team of nine reviewers would equate to at least 6 months’ worth of review time and tens of thousands in cost to the client. By effective utilisation of Relativity’s CAL model, the client reviewed a total of just over 20,000 out of 1.2 million, this resulted in the system identifying 255,000 items as relevant out of the entire data set.

By utilising CAL we were able to efficiently assist the client with responding to the regulators stringent deadlines, avoiding costly review time and fines for non-compliance.