Review Dark Matter: Most Big Data Will Be Invisible to eDiscovery

Nice GigaOm blog post on what what big data is, and what it's for.  And what it's for is analytics.
From IBM's website:


Everyday, we create 2.5 quintillion bytes of data–so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals to name a few. 


Big data spans three dimensions: Volume, Velocity and Variety.

• Volume – Big data comes in one size: large. Enterprises are awash with data, easily amassing terabytes and even petabytes of information.
• Velocity – Often time-sensitive, big data must be used as it is streaming in to the enterprise in order to maximize its value to the business.
• Variety – Big data extends beyond structured data, including unstructured data of all varieties: text, audio, video, click streams, log files and more.

Big data is more than a challenge; it is an opportunity to find insight in new and emerging types of data, to make your business more agile, and to answer questions that, in the past, were beyond reach. Until now, there was no practical way to harvest this opportunity. Today, IBM’s platform for big data uses technologies like Hadoop to open the door to a world of possibilities.



In the industry where big data is discussed, involving companies like GigaOm, Opera Solutions and IBM, the goal is to take very very large data sets and conduct complex analytics to extract insight that would otherwise remain obscured.  Extracting insight takes a lot of analytics.


Anyone who has worked on cases knows that any data that requires expert assessment runs along a separate track with a pretty direct route to consulting experts.  Attorneys look at roll-up reports and emails.  Experts handle anything that requires interpretation.  These types of data are certainly discovery, but they do not comprise the stuff about which one thinks when contemplating "e-discovery".


Because of this, it is unlikely, despite the hoopla, that e-discovery will be directly impacted by big data.   Unstructured data repositories will undoubtedly get bigger; there will be challenges to address them in e-discovery.  And e-discovery will have to identify relevant documents that discuss the insights and information gleaned by big data processes.


But the vast ocean of data for which "big data" was coined will likely in the main pass through the litigation process without registering on e-discovery processes.