Swift Data

Swift Data

 In Blog

Big Data to Swift Data

While most of us are still coming to terms with big data – often described by the 4 Vs (volume, velocity, variety, and veracity; some add value to this list and make it 5 Vs) – the conversation has already started to move on to the next topic: what’s after big data? 

This is natural because once you have figured out a good way to store and manage your big data sets, you then want to be able to retrieve meaningful information from those data sets, on demand. And while the big data paradigm has more or less solved the issue of managing massive data volumes, the speed at which the information is retrieved is still, more often than not, limited by the same old concept of issuing queries to aggregate tables (read data marts).

A paradigm shift in this field is happening: swift and accelerated data access will soon become one of its most discussed and debated topics. A new breed of analytic solutions will bridge this gap quickly. We believe the transition to this new paradigm will be driven by three main factors:

  1. acceleration due to in-memory storage and high throughput transactional databases;
  2. the advent of hardware acceleration using massive parallel processing capabilities; and
  3. the upcoming alternatives to map-reduce such as Impala, Shark/Spark, Drill (based on Google BigQuery).

Incidentally, if you have been following the news of Google abandoning the Map Reduce paradigm for its search indexing, you would notice some early warning signs of what I am talking about. According to Eisar Lipkovitz, a senior director of engineering at Google, Google has moved back-end indexing system away from MapReduce and onto BigTable. MapReduce is a sequence of batch operations. It suffers from “stragglers,” he says. “You can’t do anything that takes a relatively short amount of time,” Lipkovitz says, “so we got rid of it.”

We foresee that the next generation of big data analytics solutions will be polyglot implementations based on this latest wave of new technologies. We will be posting more on this in our coming blogs.