One of the buzzwords in the IT industry over the past few years has been Big Data. This refers to a movement within the industry that has developed distributed computer systems that can process information on massive scales to power the internet mega-sites such as Google, Facebook and Amazon.

The challenge of Big Data processing has seen the development of new data storage and processing mechanisms quite different from the mainframe relational-database driven models used 20 years ago. In the file storage and processing space this has seen technologies such as Hadoop. And in databases, the NoSQL movement, using data models such as key-value stores, bigtable-inspired databases, document databases, and graph databases. 

But why is Big Data necessary?

Well it turns out that some of the most computational challenges, such as understanding language semantics, are very difficult to solve using algorithms, but can be (effectively) solved using brute-force data analysis on very large data sets.

