Defending Concept Driven Analytics

Defending Concept-Driven Analytics

CAAT uses mathematical algorithms to determine how words and documents relate. Its Latent Semantic Indexing (LSI) algorithms are well-understood and well-documented in academic literature, and its results are repeatable: the same query against the same data will always yield the same results. CAAT even highlights words and sentences that it considers the most relevant, i.e., the most like what is being queried.

In the two main ways CAAT is used, there is no question of defensibility for an organization strategy. Organization along the lines of content and concepts is merely a way to facilitate better, more efficient review.

Predictive or suggestive coding solutions are an extension of this notion of organization: they take user actions on a sample subset of the documents and apply them across the whole collection, allowing iterative refinement along the way. The software is mimicking what the attorneys did who "trained" the coding solution, so defending the results is about defending the original decisions and understanding how representative the sample was against the whole collection. Recent court decisions like Da Silva Moore support these solutions as defensible.

Concept Search

If concept search didn't contain the term "search," the question of defensibility might not exist the way it does for keyword search. This is because concept search is very different from the more-familiar keyword search in several dimensions.

In keyword search, a relevant document will not be returned, and ultimately not included, if keywords aren't present. In contrast, concept search will return all relevant documents, even if they don't contain the actual keywords, because the decisions that it makes are relative.

Consider a simplistic example of a search using "bank" in the financial institution sense. Concept search will return documents containing that word, but also return every document that is relevant to the query about financial institutions even if it does not contain the word "bank." Keyword search will not do this, and in fact, is unable to. This issue with keyword search, called under-inclusion, can result in e-discovery errors. Concept search has no such issue; an important distinction.

In addition, a keyword search including "bank" will return documents about rivers as well as financial institutions. Concept search understands that the query about "bank" was in reference to money, so it will rank documents about financial institutions as very relevant and ones about riverbanks as irrelevant. It will also return the documents ranked in relevant order based on the sense of the query.

The question of defensibility actually comes down to workflow. If an attorney crafts poor searches, they may obtain poor results. While there is a better likelihood CAAT will return relevant results than keyword when the search strategy is poor, the ability to defend the results lies in the thinking behind what those searches were meant to find. Thus, it is a workflow and strategy consideration, not a software one.