I thought I'd share some thoughts about my recent visit to #facebook. I don't like facebook as a company at all, but I do build my own servers and datacenter so it is always interesting to look a those bits for me. The datacenter we visited is a newly built second datacenter in luleå on the same grounds as the old one. We were about 60 people with free food and drinks. No photographing was allowed (except for the picture below).

They claimed a PUE number of 1.07 which is a lie because about 5% is lost at each transformer stage, and they were running high voltage directly into their datacenter. They probably had 10 kV -> 230 V-> 12V. For the frontend servers they had a novel power delivery with 230 V busbars above the racks, pluggable 32A 3 phase breakouts to feed each rack. Each rack had 3 psu's 230->12Vdc. Those psu's was also fed 48V DC from an lead acid UPS. The UPSes were designed to run for 90 seconds during the dual main power feed switchover (they had dual high voltage feeds). This means that on a power outage they will sacrifice the frontend servers and move that to another datacenter. The database servers and network where fed via other UPSes.

Their network was whitebox switch based with their own firmware that ran bgp directly to the top of rack switches. They had their own dark fibre connections with dwdm.

Their cooling technique is described everywhere else so I won't go into that. Overall there where lots of cable porn, talented people and nice clean designs.