[caption width="1920" align="alignnone"]<img src="http://dorkutopia.com/wp-content/uploads/2013/06/Servers-Server-Farm-Engine-Room.jpg" width="1920" height="1200" class /> Source: Google[/caption]

One of the buzzwords in the IT industry over the past few years has been Big Data. This refers to a trend toward distributed computer systems able to process information on massive scales. This technology powers the Internet mega-sites such as Google, Facebook and Amazon.

The challenge of Big Data processing has seen the development of new data storage and processing mechanisms quite different from the mainframe relational-database driven models prevalent over the past 30 years. 

In the file storage and processing space this has seen the development of technologies such as Hadoop - which achieves massive data throughput by splitting large problems down and solving them on vast server-farms that use thousands of commodity PCs. 

And in databases there has been the development of non-relational data models that better handle semi-structured data. This has seen the rise of the so-called NoSQL movement, which uses diverse data models such as key-value stores, bigtable-inspired databases, document databases, and graph databases. 

But why is Big Data necessary?

Well it turns out that some of the most computational challenges (such as understanding language semantics, and predictive modelling) are very difficult to solve using algorithms, but can be effectively solved using brute-force data analysis on very large data sets.

<blockquote>Data beats Algorithms</blockquote>

For an excellent introduction to the power of big data, take a look at the Google paper <em>"The Unreasonable Effectiveness of Data"</em>, downloadable <a href="http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf">here</a>.