Warning: This post has all kinds of unconfirmed rumors, but is indicative of where the future of high performance computing is going, at least according to IBM.

I had the pleasure of getting to see the Blue Gene installation at IBM's TJ Watson Research Center, which gives us a glimpse at the future of High Performance Computing (HPC).  While IBM's webpage about Blue Gene is a bit out of date, it gives you a glimpse into what's running there.  Installed in 2005, it has been upgraded a number of times since then with new HPC technology.  Wikipedia has some more technical insight into the evolution of Blue Gene.

One of the things about Blue Gene is how much power it consumes, how much computing power is packed into a small space, and how it is cooled.  The current Blue Gene models (the big black row boxes with a slant) make up the majority of the Blue Gene data center.  This new "Blue Gene/P configuration" hosts 4096 processors per rack.  The rack uses contained cooling with water-filled radiators cooling the hot air exhaust.  The entire system fits in the alien-pod-shaped row container.

In total, there are 20 racks in the Watson Blue Gene installation, consuming "about 1MW" according to my friends at IBM.  The overall square footage appears somewhere around 2,000 square feet.  That results in 50kW per rack (versus today's average of 6.6kW per rack) or 500 Watts per square foot (versus today's average of 150 Watts per square foot). Online estimates for power consumed by the Blue Gene's IT load varies between 140kW and 400kW, so I can't guarantee the 1MW number is correct for the full data center power profile.

While this is impressive (and daunting from a density point of view!), what seems even more interesting was a new row that had been added to the facility that looked different than the big black out-of-this-planet row enclosures.  Instead it was made of stainless steel panels and pipes, and had water flowing into it.  I was told that this is an experiment with water-cooling (liquid-cooling) direct to the CPU cores, to be able to run at a much higher computing density (beyond today's 4.5 teraflops per rack).  While the specifications were not disclosed to me, the design makes sense (although the dangers of water in the data center are high, maybe they'll use this stuff in Slide 12 or this rack flipped on its side).  I was told that the plumbers aren't completely excited with the new water-flow system, as it took a few iterations to get all the leaks worked out.

What I also liked is the fact that the HPC world is benchmarking efficiency -- useful work per unit of energy.  We're not seeing this yet in the data center world; for some reason it went away with the mainframe.  In the case of Blue Gene, they're looking at flops per Watt.  This metric works well for sustained workloads (such as protein folding, the primary use of Blue Gene), but doesn't work as well for bursty workloads like we find in a typical data center.  In that case, we need to look at the application units performed per Watt, such as sessions hosted or transactions completed.  The Blue Gene cluster at the Watson Center ranks 9th on the Green 500 list, which computes the flops per Watt of the world's 500 most powerful supercomputing clusters.

Do you remember when Deep Blue beat Kasperov in a chess match in 1997?  (Watch the match here) Well IBM is at it again; this time building a cluster to take on Jeopardy contestants called DeepQA.  A working prototype was built in Hawthorne that you can use for $100 per minute, and it looks like a new version is being rolled out at Yorktown.  And while I'm sure it will be more powerful (both in computing performance as well as power consumption), I don't yet know how much.  I did get to see the outside of the doors (pretty and white) where they're building this new system, but nothing more to report at this point except a 5MW rumor for full facility power load.  I wonder if IBM plans to also build their own nuclear power station outside the Watson Center in Yorktown?

Also check out new Blue Gene clusters coming online: JUGENE at Jülich Supercomputing Centre and 20 petaflop Blue Gene/Q at Lawrence Livermore, perhaps using some of the chip-based water-cooling.