Thu. Mar 28th, 2024
ARCHIVED CONTENT
You are viewing ARCHIVED CONTENT released online between 1 April 2010 and 24 August 2018 or content that has been selectively archived and is no longer active. Content in this archive is NOT UPDATED, and links may not function.
 

Impression

A Challenge with Technology-Assisted Review

Provided for your review is a short but important comment from a recent LinkedIn Group discussion on the evaluation of machine learning protocols for technology-assisted review.  The discussion originated with the availability of a new paper published by well known technology-assisted review experts, Attorney Maura Grossman and Professor Gordon Cormack.  The paper, “Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, was prepared for The 37th Annual ACM SIGIR Conference on Research and Development in Information Retrieval.

The comment shared (see below) addressed key points in the paper and advanced the discussion beyond the evaluation of machine learning techniques into the questioning of a greater issue with technology-assisted review. That issue, as stated in the comment, is the fact that current technology-assisted review systems are limited to text.

This comment is worthy of careful consideration as it:

  1. Highlights the fact that this limitation calls into question the true effectiveness of text-based technology-assisted review tools.
  2. Begs the question of what classification tools are the best choice for technology-assisted review if almost all current tools are in fact limited to text.

 

Comment by Joe Howie of BeyondRecognition*

Interesting – Cormack and Grossman define the “TAR Problem” as not knowing about a data set at the outset (p. 2); point out that simple key term searching to select the seed set improves the performance of all TAR protocols (p. 7); and indicate that some protocols have difficulty with low prevalence or low richness collections (p.9). Visual classification impacts those issues plus addresses the implicit issue that TAR is text-restricted (i.e., should actually be called TR-TAR).

Visual classification classifies visually similar documents whether or not text is contained in them, and reviewing one document per classification provides an awareness of a collection’s content that overcomes a possible initial lack of knowledge. The classifications also provide a far more granular and meaningful tool for selecting documents than simple text searching. By eliminating plainly unresponsive document types, reviewers can increase the richness of the remaining documents, and may in fact be able to identify which specific attributes which make documents responsive, thereby short cutting the need to “predict” which documents are responsive.

Since the last century, text analysis has been the primary tool used to classify documents, but its durability as the tool of choice doesn’t mean that it remains the best choice.

* The comment was used by express permission of its contributor, Joe Howie of BeyondRecognition.

 


Future Consideration

Like many, I find the discussion and commentary on this issue quite interesting and certainly expect that the issue raised in the comment will spark relevant discussion among providers, consumers and commentators on the topic of technology-assisted review.

 

ComplexD QR Code

 

 

Generative Artificial Intelligence and Large Language Model Use

ComplexDiscovery OÜ recognizes the value of GAI and LLM tools in streamlining content creation processes and enhancing the overall quality of its research, writing, and editing efforts. To this end, ComplexDiscovery OÜ regularly employs GAI tools, including ChatGPT, Claude, Midjourney, and DALL-E, to assist, augment, and accelerate the development and publication of both new and revised content in posts and pages published (initiated in late 2022).

ComplexDiscovery also provides a ChatGPT-powered AI article assistant for its users. This feature leverages LLM capabilities to generate relevant and valuable insights related to specific page and post content published on ComplexDiscovery.com. By offering this AI-driven service, ComplexDiscovery OÜ aims to create a more interactive and engaging experience for its users, while highlighting the importance of responsible and ethical use of GAI and LLM technologies.

 

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know, and we will make our response to you a priority.

ComplexDiscovery OÜ is a highly recognized digital publication focused on providing detailed insights into the fields of cybersecurity, information governance, and eDiscovery. Based in Estonia, a hub for digital innovation, ComplexDiscovery OÜ upholds rigorous standards in journalistic integrity, delivering nuanced analyses of global trends, technology advancements, and the eDiscovery sector. The publication expertly connects intricate legal technology issues with the broader narrative of international business and current events, offering its readership invaluable insights for informed decision-making.

For the latest in law, technology, and business, visit ComplexDiscovery.com.