Subject: Free online Data Center Summit – case studies & best practices

Join myself, Graeme Hay and Jan Wiersma of Data Center Pulse as we present a roundtable presentation at BrightTALK’s free Next Generation Data Center Summit on July 28.   The presentation will be part of an all-day event in which experts discuss innovative tips and best practices through a series of interactive webcasts that you can view from the convenience of your computer. 

View the full program and sign up to attend your favorite presentations at: #0000ff">http://www.brighttalk.com/r/Bbf

Presentation:
“The State of the Data Center: What's Next?”
Mark Thiele, Jan Wiersma & Graeme Hay, Data Center Pulse

You will be able to view any or all webcasts live or after on-demand.  If you are able to tune in live, you will be able to submit real-time questions to presenters and take part in presenter-led polls.  I hope you will be able to join us.

Today we added Eddie Schutter from AT&T to the Data Center Pulse board of the directors.  You can see the press release here.  But for this blog entry, I wanted to give a bit more information and perspective on Eddie and our expansion into local chapters.

I have worked closely with Eddie on emerging technology opportunities and industry efforts for almost three years. The most recent was with the Green Grid where we both sit on the Advisory Council with a great group of End Users.  In February, 2010 Eddie and I co-presented at the Green Grid Technical Conference in San Jose, California.  The presentation was focused on the GG birds of a feather session and the DCP top 10. This was a result of the collaboration agreement between DCP and the Green Grid established in early 2010.

in April, 2010 the DCP Board decided that it was time for us to harness the interest and capabilities of local end users by establishing DCP chapter groups. We started informal chapters in Utah and Arizona because of the activities and interest in those two states. The first meetings were great, but we realized very quickly that we needed someone to focus on developing this program from the ground up.  The Utah end users that we knew of represented over 200MW of current future consumption in the state.  The Arizona end users represented over 50MW and that was just scratching the surface.  Eddie came to mind immediately to help lead this effort.  As mentioned in the press release, Eddie has been active in many different industry groups including the Green Grid, Uptime Institute and 7x24 Exchange in which he helped start the Lone Star Chapter in Austin, Texas.  Eddie's day job is the Sr Technical Director of Data Center Architecture and Planning for AT&T.  Over the years we have shared best practices, debated on technological approaches and generally brainstormed on what is possible. One of the most memorable was the Data Center Pulse working session at the 7x24 Exchange in Phoenix, Arizona in November of 2009.  Eddie, along with the majority of the DCP board, Olivier Sanche and other end user leaders met following the session and decided on a strategy to push for next generation cooling/compute solutions.  Stay tuned for more blog updates on recent updates that were spawned from that great session.

We have great expectations from Eddie and the development of the local chapters.  We believe that this is a very productive and effective way to have end users participate in work that will directly benefit their companies and the DCP efforts. The majority of the DCP work has been done over linkedin and through conference partnerships over the last few years.  This approach will allow a much larger group of individuals have direct input in helping us to shape our efforts to influence the industry.  The voice of the customer will get bigger but also smaller - more focused on their own backyard.

With that, I wanted to address a few concerns that people have raised over the last few weeks.

First, we are not competing with any of the industry groups.  Quite the contrary. We plan to partner up with other industry groups to optimize peoples time and increase participation of end users.  We plan to approach this very much like the conference partnerships we have established.  There will be DCP end user only meetings to allow for open dialog and focus to synthesize content and requests before we meet with the industry.  There will then be additional sessions with the industry to engage and share the learning and discussion/debate from those sessions.

Secondly, we will be establishing additional chapters, but we want to make sure that we do this in a controlled manner.  Eddie will be establishing the bylaws and governance to ensure that we can scale and optimize the local work of our members.

Lastly, this is not just a North American effort. We plan to establish local chapters globally.  For example, the activity in China, India, Europe and other parts of Asia (to name a few) is amazing.  There are tens of billions of dollars being spent globally in new and retrofit data centers.

Eddie has a full plate, but we have every confidence that he will be able to deliver a great program. If you are interested in helping or have ideas please email info@datacenterpulse.org or contact Eddie directly at eddie.schutter@datacenterpulse.org.

Thanks, Dean

Cloud Computing and Data Centers are killing our planet.  Drive to the hardware store, buy a hammer, chisel and piece of stone and begin writing about it. Put away your computer, turn off your internet connection, un-plug the video game console, iPod and the TV. We must do everything we can to get rid of these giant, energy sucking, pollution generating, and planet killing warehouses of death immediately.  

All kidding aside, it is true that many of our older data centers are in serious need of improvements in their power efficiency.  However, it's also true that data centers contain much of our work effort and play environments. If this work and play was to be distributed in small chunks throughout the business or our households instead of concentrated in data centers they would be considerably more wasteful of our planets resources.

In my 20 plus year career in IT I've always been proud of my ability to bring efficiency to IT, and the business. When Data Center Pulse was founded the driving motivation was to push for the development of power sipping IT equipment designs, combined with more efficient data centers. In parallel we're actively working to persuade owners to implement those new solutions more quickly. We strongly believed that the IT/Data Center industry had a need to focus more attention on effective use of energy. The DCP Leadership team was made up of like minded individuals that each have work history examples of a focus on reducing energy consumption.  So why would I be writing an article about data centers getting a bad rap? It seems like I should be agreeing with those articles, as it seems like I'm contradicting myself.  Well, that couldn't be farther from the truth.

There is no doubt that there are good ways and bad ways to build data centers, from site selection, to air flow dynamics and especially on how the IT equipment is put to use. This effort to build an efficient space to meet the ever growing need for compute capacity doesn't seem to be enough. At least it isn't if you read some of the green oriented blogs that hound data center builders, cloud providers and perceived "GreenWashing" in the technology industry regarding them. 

What I always find missing in these stories criticizing large data centers and cloud computing is a comparison of the alternatives and a fair assessment of market and political drivers affecting data center build decisions. I'm very much in favor of conservation and doing the best we can for the environment, but we also need to find a way for society to keep moving forward. In a capitalist system, profit is the only way we can maintain forward momentum. So I believe we have to first make a consideration, if you put free Twinkies in front of someone with limited funds, but charge them $20 for a similar amount of Broccoli, generally speaking which one will the person pick to eat? In other words, why would a company pay 15 cents a kilo Watt hour for clean energy, when federal and state governments are using taxpayer funds to practically give away fossil fuel generated energy? Our Federal government continues to subsidize the coal and fossil fuel industry to the tune of billions of dollars every year.  And unless I'm mistaken, these energy companies seem to be enjoying fairly strong profit margins.  

The following are some examples borrowed from sourcewatch.org;

"Examples of U.S. Treasury Department Funding

Examples of new or proposed coal-fired power plants that are funded in part by tax-exempt debt include the following:

• The Prairie State Energy Campus Project in Illinois is a mine-mouth 1600 MW supercritical steam turbine power plant without carbon capture technology. The more than $4 billion plant has several participating partners, with one partner, the Northern Illinois Municipal Power Agency (NIMPA), buying 120 MW of the 800 MW plant with $303 of its $318 million investment portion financed with tax-exempt debt.


• The Longleaf Energy Station in Georgia is a proposed 1200 MW pulverized coal fired power plant supported by the Early County (Georgia) Development Authority with federally backed local development bonds.


• The Two Elk coal plant in Wyoming is a proposed coal plant that purports to use so-called "waste coal" and has received hundreds of millions of dollars in tax-exempt debt authority since it was classified as a solid waste recycling facility. Approval for the tax-exempt financing is currently being audited by the Internal Revenue Service.

Global Support for the fossil fuel industry is as much as 500 billion US dollars annually

Build America Bonds: Additional Source of Funding

A new program under the American Recovery and Reinvestment Act, Build American Bonds (BABs), expands the U.S. Treasury's use of financing tools to subsidize coal-fired power plants. Under the program, issuers of the taxable bonds are provided a 35% direct pay interest subsidy to reduce the costs of borrowing. Power companies are eligible for these federally subsidized taxable bonds funding under BABs: American Municipal Power Ohio used the tax-exempt bond market to finance the construction of the Prairie State Energy Campus in Illinois and, after the 2009 financial crisis began, issued through the BABs program nearly $500 million dollars of federally subsidized taxable bonds to finance the last phases of construction. The bonds have also been used for scrubbers at existing plants."

It's not all bad news though. There are many states with aggressive clean energy plans in place with subsidies for developing renewable energy supplies. Also, the federal government is working with the G20 to eliminate government subsidies for fossil fuel development. President Obama has already removed the subsidies from this year's Federal budget.

Another positive trend is the number of US states that are pushing energy providers to supply a significant portion of their power through renewable sources over the next 10-15 years.

Using the information in figure 1 below, try to find an activity other than reading a book (not a digital one) or sleeping that uses less energy than 10 Google searches. This should be a wakeup call that the concentration of compute resources and their effective utilization are much more efficient than everyone doing the same work through independent personally owned computer, network, storage resource and or their car.

Figure 1

We're not Providing Real Incentives for the use of Renewable Energy to Companies Building Data Centers

How can we expect our companies who only survive if they stay competitive to ignore the opportunity of something cheaper, with more incentives to boot? The fact is the state is making the site more cost effective at tax payer expense, while they help subsidize the energy provider and then take more tax money to clean up the mess created. If we're going to go after someone for perpetuating this bad behavior we should be going after the federal and state governments. If we as a nation can provide reasonable incentives to corporations to make the right choices, then we drive up the use of alternative energy and drive down the incentive to do otherwise. Technology and data centers can be implemented poorly, but our planet stands little chance of surviving the forward march of the human race without them.

The Complexity of Data Center Energy Sourcing

Data center owners are faced with a myriad of choices to select the "perfect" site for their next Data Center.  When finding an energy source the Data Center owner will face both business and government politics often incentivizing poor choices.  Combine unfocused sustainability efforts and missing or backwards incentives from the government and it puts the selection of clean or renewable energy at the bottom of priority list.

It's no wonder companies like Digital Realty Trust are doing so well these days. The complexity and work effort associated with making a decision around your data center's site selection is extremely high. There are a number of factors (Geography, Network, Water, Staff, etc) that need to be considered, most of which can be accommodated by an experienced Facilities or IT person with data center knowledge. However, as you can see from figure 2, the layers of concern on the question of energy alone can be a back breaker.

Often times the Facilities or IT person that is making Data Center decisions is not a political or energy sector expert and is lucky to see beyond their immediate layer of concern and rarely gets past the corporate layer.  Who then is accountable for corporate energy decisions?  Unfortunately the answer all to often is "no one" the multiple layers often keep the "right" decision out of reach despite the best intentions.

Figure 2

Using Lots of Energy Makes the Data Center a Target of Opportunity and of Protest

Yes, it's bad to have to use coal to power a large data center, but isn't it worse that the tax payer is actually supporting the effort? I'm not suggesting that we should reduce our focus on data centers either, as evidenced by this Data Center Pulse proposal to the federal government. I am suggesting that data centers, especially modern efficient ones aren't the real problem, in the majority of cases they are creating huge environmental benefits and energy savings versus the alternative of everyone running their own IT equipment. We can't forget that all of us use technology every day. If everyone was to provide their own equipment in their homes, imagine the impact to power draw? We would have billions of systems, all of which are running at or below 5% average utilization, and don't forget that we'd have even less control over where the power came from for all these home based servers, network and storage devices. The energy used to build all this extra IT gear, along with the waste generated would quickly outstrip the world's ability to support it.

I think our next target should be those Planet Killing Busses!

Maybe we should yell at bus manufacturers for making vehicles that use so much energy? After all the average bus carries 10X the number of passengers that a car does, but uses at least 5X the energy.  I guess I should go back to driving.  It turns out it's not data centers that are going to kill the planet, it's busses!

My next blog might take a while because I'm going to have to ride my bike to the hardware store and buy that hammer, chisel and stone. I need to hurry because I'm worried there will be a rush on iStones.

A big THANKS to Jeremy Rodriguez for his contributions to this article!

EBay’s flagship data center is open for business in Utah

When I joined EBay in September of 2009, I had the privilege of taking on the responsibility of delivering the single largest infrastructure project that the company has ever undertaken. A new data center code named project Topaz. At $287 million, it is also the single largest infrastructure investment the company has ever made.  It is also the most complex construction project we have ever undertaken.  Over 1.2 million man hours of work in just 14 months.  But, the most important component of Topaz isn't the project itself, it is what it will be used for.  It will house EBay’s core businesses - ebay.com, the world’s largest online marketplace with over 90 million buyers and sellers in 32 countries and PayPal.com, one of the leading ways to pay online with 81 million registered accounts available in 190 markets and 24 currencies. In 2009, the total worth of goods sold on eBay was over $60 billion: that’s over $2000 a second. Topaz isn't just a datacenter, it is the home of our business.

Ok, no pressure there...

As you can see, we live and die by the performance of our datacenters. Our buyers and sellers depend on its reliability. Project Topaz is a critical part of the EBay engine. It is the foundation for our business and must be solid, stable, and secure. In a nut shell, it needs to be bullet proof.

On May 4, 2010 we completed project Topaz on-time and under budget. A monumental task considering the shear volume of work that had to be completed in the short time frame. The data center, located outside Salt Lake City, Utah was designed to be concurrently maintainable and fault tolerant. What that means is we can sustain major impacts to any part of the data center and it will continue to operate. We can isolate and fix any component in the datacenter without disrupting the engine. Picture that everything has a backup - even the backups have backup. Now keep in mind that nothing is really 100% bulletproof, but in terms of a resilient data center, we have built the highest level possible.

Now, many think that when you build a data center with this much redundancy, it will be extremely expensive to operate and very inefficient. Quite the contrary. Besides running the data center operations for the company, I’m also responsible to pay the power bill.    So, the datacenter must be built like a tank, be able to brush off major faults, lower our operating costs and be extremely efficient. Did I mention that these are goals in my annual performance review?  Ok, no pressure there either...

So, put yourself in my shoes for a minute.  With these rather challenging deliverables, who would you want to drive the design and delivery of a bullet proof data center?  Why two former tank commanders from West Point of course!   I was blessed to have two very solid guys running with the Topaz ball.  Mike Lewis, my Distinguished Architect who owns Data Center design and Mission Critical Engineering standards and Greg Fennewald, the local Utah Data Center Manager responsible for bringing Topaz on-line and operating it going forward. And yes, they are former tank commanders. When I saw some pictures of what they did for fun at West Point, I knew they could roll over any barrier that was in front of them to get the job done.  That car had no chance. 

I digress.  Now Mike and Greg didn’t do this alone of course, but they coordinated the logistics and collaborated with our partners to deliver.  Data Center resiliency was priority one but an almost equal requirement was efficiency. Remember, inefficiencies affect my budget directly.  So it was top of mind for everyone involved.  We ha to achieve both. The only way to deliver on that is to form true partnerships with the vendors involved in the project. And we had some stellar partners on this project. Skanska led the construction, RTKL did the design, and over forty additional companies worked tirelessly on this project for more than a year.  At its peak we had over 700 hundred workers on two shifts to deliver this data center. On top of partners, there was a multitude of eBay internal employees from almost every aspect of the business working on the success of Topaz. Tax, Legal, Risk, Finance, Procurement, Product Development, and over 100 people in Operations.  It was an incredible effort and impossible to mention and thank everyone involved.

What I am very proud to announce is that we have delivered on all of the challenges in this project.  We have built a fault tolerant Tier IV level data center that is 50% less expensive to operate than the average of all other data centers we lease today. It is also 30% more efficient than the most efficient data center in our portfolio. At a designed PUE of 1.4, it lowers both our economical and ecological costs. We only consume the energy we need, when we need it.  Now, I don’t want to go into the religious debate of who has the lowest PUE, but I do want to point one thing out.  In the business of on-line commerce, we do not have a choice but to build a highly available data center to support our customers.  From my perspective, achieving a 1.4 PUE with a hard requirement to meet this level of redundancy is quite an accomplishment.  The point is you can be resilient, efficient and cost effective if you set your mind to it from the beginning.

Now, the juicy details. ☺

Phase one is a 240,000 square foot two story building with three, 20,000 square foot rooms to house IT equipment.  These three rooms deliver 7.2 Megawatts of total server load.  We have our own substation capable of delivering up to 30 Megawatts of total power.  The 60 acre site master plan consists of four phases.  This facility is part of Technology Operations four year data center strategy. It is the next step in consolidating our leased data centers spread over three states. This not only helps us reduce our operating costs, it increases reliability. In essence, that means every server we consolidate into this new data center will lower costs and increase efficiency. On top of that, the site is built to scale so we are enabling the company to grow as we consolidate!  We get the best of both worlds.

Now lets talk about some of the efficiencies. 

First, everything is running at 400V. This means we lose an entire level of transformers and deliver 230V to the servers.  That’s a 2% efficient gain through the entire electrical system and the modular busway system (starline) allows us to change receptacles in minutes rather than days. We have a 400,000 gallon cistern which collects rain water and will be used as a our primary cooling source. We are using a water side economizer, which allows us to use the outside air to cool the data center for more than half the year instead of running expensive chillers. In addition, we are using technology that will dynamically match the power used by the pump and fan motors to the cooling loads ensuring we only consume the energy needed to support the compute load.  We also have a fully contained hot aisle design that isolates the heated air from the cold air. In addition. we have closely coupled cooling units (in-row) to add additional capacity where it is needed.  That means we can put anything anywhere and still ensure it meets its optimum cooling efficiency even with a mixed workload.  We can support racks that are less than 1,000 watts to dense racks that generate more than 30,000 watts of heat (more that an industrial pizza oven).  We have created a flexible, scalable and efficient infrastructure that increases the agility of the company.  Also, we expect Topaz to achieve LEED Gold certification status from the US Green Building Council.  A great validation of the effectiveness of our efficiency commitment in this project.

As I mentioned earlier, there are three IT rooms in the data center.  Room one is for eBay Marketplace (shown in the picture above), room two is PayPal.  Room three is planned for further consolidation work to support our strategy.

But, on May 4th, room three became something complete different - It became the coolest looking data center in the world!

Welcome to CLUB EBAY!

Every great project deservers a great party!

We converted the datacenter raised floor into a Club 51 atmosphere that the Hollywood A-list would feel right at home in - complete with a red carpet walkway. The attendees had no idea we planned to treat them like royalty.  But they were!  They were responsible for making this project a success.  Needless to say, it was one of the most unique company parties in eBay’s history. 

We started off the festivities with a video summarizing the magnitude and impact of this project on eBay and its employees.

Once we finished the video, Mazen Rawashdeh, VP of Technology Operations, James Barrese, VP of Architecture and Mark Carges, our CTO, gave their perspective of the importance of project Topaz to the success of eBay, PayPal and our adjacencies. We also had Jakob Carnemark, the Sr VP of Mission Critical from Skanska explain the complexity of the project and how crucial it was to have a full partnership in a construction project of this magnitude and complexity. Finally, I was able to bring up Mike and Greg and express our thanks for leading the construction teams to deliver this project.

Then we brought the house down!  A crew of local hip hop and break dancers proceeded to interpret the key words of our project -  Transformation, Innovation & Leadership.  They also helped us become "cool" as you can see in the picture.  And finally after years of planning, approval, construction, commissioning and verification, we plugged this puppy in and fired it up!

After the power-on ceremony.  We conducted tours of the site for all who attended.  Ironically, the majority of people that worked on this project, had done it remotely.  It was the first time that many were able to see the massive infrastructure that was built to house their technology.  The tours were complete with iPads showing full 3D models of the data center (BIM) and an interface into the infrastructure applications. 

We ended the day with a party at The Depot in downtown Salt Lake City.  It was a night filled with games, a live band and even some performances from the eBay employees themselves.  I had a chance to sing a rendition of YMCA with a motley crew of eBay employees. Our version was “Shop At eBay”! As you can see from the photo, everyone got involved!  It was quite a performance!

The Topaz launch party was a tribute to those that put their blood, sweat and tears into this project for more than two years.  Topaz is the epitome of collaborative teamwork and innovation as hundreds of people in all areas of the business came together to plan, design, validate, and execute on this incredibly complicated project. It is a testament to the eBay culture and the ability of our people to innovate, lead, execute and deliver truly exemplarily performance.  Ultimately, this data center is allowing us to deliver a better and more reliable experience for our buyers and sellers while keeping our infrastructure costs under control.

Here are some fun statistics about this incredible project:

•  
  • Over 1.2 Million man hours worked with no loss time due to injury
  • 30 Megawatt power substation
  • 60,000 sq ft of usable IT space
  • 57 Miles of underground electrical conduit
  • 2 Million pounds of copper for underground critical power
  • 295 miles of Copper Cable Network
  • 20,000 Fiber Optic Cable 176,000 feet, Approximately (43,000) strands of fiber over (33 miles)
  • 2006 tons of Steel
  • $10 million in savings through 3-D computer modeling (BIM)
  • Over 200,000 measured points – all the way down to the individual server power plugs.

Needless to say, we are very proud of Topaz and the people that made it possible. Now, stay tuned for even more innovation and efficiency from the eBay engine.  We’re just getting  started …