<img src="http://ucttraining.files.wordpress.com/2013/10/java_logo.jpg" width="651" height="400" class="alignnone" />

For the last few weeks I've been reading and watching everything I can about Big Data and NoSQL. I've been learning all about Hadoop, Hive, Hama, BigTable, HBase, Hypertable, Cassandra, MongoDB, Neo4j, OrientDB, Giraph, Mahout, Riak, Redis, CouchDB and the list goes on.

<blockquote>But nowhere in all this reading and listening has 
anyone said the word ... Microsoft.</blockquote>

There are a couple of exceptions (such as CouchDB, which is written in Erlang), but the overwhelming majority of the innovation in this space is being done in <strong>Java</strong>.

What is also evident, is that unlike the previous trends in IT, innovation is not being driven by the vendors (IBM, Microsoft, Oracle), but by the service providers (Google, Facebook, Amazon, Linked-In) whom are solving problems not addressed by the offerings of the entrenched vendors.

This says a lot about the direction of the enterprise IT industry over the next 10 years or so. Despite Java having been technologically usurped by other languages with significant user-bases (such as C#/.NET, Python and Ruby), the cutting edge for the next decade is clear ... and it is Java, for better or for worse.

Another given is that while Google & Co already have access to the best Big Data technology available, there is a lot of catch-up for the rest of the commercial sector and for a plethora of government agencies. In terms of market penetration, even 10 years in to this trend Big Data has probably tapped less than 5% of its potential market. The opportunities are huge.    

The ubiquity of the Internet has certainly changed the landscape, but so too has the growth of mature Open Source counterparts to the proprietary innovations being made Google, et al. The service providers have allowed the blueprints of their novel solutions into the public-domain, and in somecases have even given away their IP altogether (such as Apache Cassandra, which was released to the open-source community by Facebook in 2008).

Much of the public disclosures have been made in journal articles. Here are a collection of some of the best:
<a href="https://www.usenix.org/legacy/events/osdi04/tech/full_papers/dean/dean.pdf" title="Google MapReduce paper" target="_blank"></a>
<a href="http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf" title="Google BigTable paper" target="_blank"></a>
<a href="http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf" title="Amazon Dynamo paper" target="_blank"></a>
<a href="http://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf" title="Facbook Cassandra paper" target="_blank"></a>
<a href="http://kowshik.github.io/JPregel/pregel_paper.pdf" title="Google Pregel paper" target="_blank"></a>