Data Center Pulse Blogs


Bullet Proof!

Posted Fri, 05/21/2010 - 15:22 by geekism

EBay’s flagship data center is open for business in Utah


When I joined EBay in September of 2009, I had the privilege of taking on the responsibility of delivering the single largest infrastructure project that the company has ever undertaken. A new data center code named project Topaz. At $287 million, it is also the single largest infrastructure investment the company has ever made.  It is also the most complex construction project we have ever undertaken.  Over 1.2 million man hours of work in just 14 months.  But, the most important component of Topaz isn't the project itself, it is what it will be used for.  It will house EBay’s core businesses - ebay.com, the world’s largest online marketplace with over 90 million buyers and sellers in 32 countries and PayPal.com, one of the leading ways to pay online with 81 million registered accounts available in 190 markets and 24 currencies. In 2009, the total worth of goods sold on eBay was over $60 billion: that’s over $2000 a second. Topaz isn't just a datacenter, it is the home of our business.

Ok, no pressure there...

As you can see, we live and die by the performance of our datacenters. Our buyers and sellers depend on its reliability. Project Topaz is a critical part of the EBay engine. It is the foundation for our business and must be solid, stable, and secure. In a nut shell, it needs to be bullet proof.

On May 4, 2010 we completed project Topaz on-time and under budget. A monumental task considering the shear volume of work that had to be completed in the short time frame. The data center, located outside Salt Lake City, Utah was designed to be concurrently maintainable and fault tolerant. What that means is we can sustain major impacts to any part of the data center and it will continue to operate. We can isolate and fix any component in the datacenter without disrupting the engine. Picture that everything has a backup - even the backups have backup. Now keep in mind that nothing is really 100% bulletproof, but in terms of a resilient data center, we have built the highest level possible.

Now, many think that when you build a data center with this much redundancy, it will be extremely expensive to operate and very inefficient. Quite the contrary. Besides running the data center operations for the company, I’m also responsible to pay the power bill.    So, the datacenter must be built like a tank, be able to brush off major faults, lower our operating costs and be extremely efficient. Did I mention that these are goals in my annual performance review?  Ok, no pressure there either...

So, put yourself in my shoes for a minute.  With these rather challenging deliverables, who would you want to drive the design and delivery of a bullet proof data center?  Why two former tank commanders from West Point of course!   I was blessed to have two very solid guys running with the Topaz ball.  Mike Lewis, my Distinguished Architect who owns Data Center design and Mission Critical Engineering standards and Greg Fennewald, the local Utah Data Center Manager responsible for bringing Topaz on-line and operating it going forward. And yes, they are former tank commanders. When I saw some pictures of what they did for fun at West Point, I knew they could roll over any barrier that was in front of them to get the job done.  That car had no chance. 

I digress.  Now Mike and Greg didn’t do this alone of course, but they coordinated the logistics and collaborated with our partners to deliver.  Data Center resiliency was priority one but an almost equal requirement was efficiency. Remember, inefficiencies affect my budget directly.  So it was top of mind for everyone involved.  We ha to achieve both. The only way to deliver on that is to form true partnerships with the vendors involved in the project. And we had some stellar partners on this project. Skanska led the construction, RTKL did the design, and over forty additional companies worked tirelessly on this project for more than a year.  At its peak we had over 700 hundred workers on two shifts to deliver this data center. On top of partners, there was a multitude of eBay internal employees from almost every aspect of the business working on the success of Topaz. Tax, Legal, Risk, Finance, Procurement, Product Development, and over 100 people in Operations.  It was an incredible effort and impossible to mention and thank everyone involved.

What I am very proud to announce is that we have delivered on all of the challenges in this project.  We have built a fault tolerant Tier IV level data center that is 50% less expensive to operate than the average of all other data centers we lease today. It is also 30% more efficient than the most efficient data center in our portfolio. At a designed PUE of 1.4, it lowers both our economical and ecological costs. We only consume the energy we need, when we need it.  Now, I don’t want to go into the religious debate of who has the lowest PUE, but I do want to point one thing out.  In the business of on-line commerce, we do not have a choice but to build a highly available data center to support our customers.  From my perspective, achieving a 1.4 PUE with a hard requirement to meet this level of redundancy is quite an accomplishment.  The point is you can be resilient, efficient and cost effective if you set your mind to it from the beginning.

Now, the juicy details. ☺

Phase one is a 240,000 square foot two story building with three, 20,000 square foot rooms to house IT equipment.  These three rooms deliver 7.2 Megawatts of total server load.  We have our own substation capable of delivering up to 30 Megawatts of total power.  The 60 acre site master plan consists of four phases.  This facility is part of Technology Operations four year data center strategy. It is the next step in consolidating our leased data centers spread over three states. This not only helps us reduce our operating costs, it increases reliability. In essence, that means every server we consolidate into this new data center will lower costs and increase efficiency. On top of that, the site is built to scale so we are enabling the company to grow as we consolidate!  We get the best of both worlds.

Now lets talk about some of the efficiencies. 

First, everything is running at 400V. This means we lose an entire level of transformers and deliver 230V to the servers.  That’s a 2% efficient gain through the entire electrical system and the modular busway system (starline) allows us to change receptacles in minutes rather than days. We have a 400,000 gallon cistern which collects rain water and will be used as a our primary cooling source. We are using a water side economizer, which allows us to use the outside air to cool the data center for more than half the year instead of running expensive chillers. In addition, we are using technology that will dynamically match the power used by the pump and fan motors to the cooling loads ensuring we only consume the energy needed to support the compute load.  We also have a fully contained hot aisle design that isolates the heated air from the cold air. In addition. we have closely coupled cooling units (in-row) to add additional capacity where it is needed.  That means we can put anything anywhere and still ensure it meets its optimum cooling efficiency even with a mixed workload.  We can support racks that are less than 1,000 watts to dense racks that generate more than 30,000 watts of heat (more that an industrial pizza oven).  We have created a flexible, scalable and efficient infrastructure that increases the agility of the company.  Also, we expect Topaz to achieve LEED Gold certification status from the US Green Building Council.  A great validation of the effectiveness of our efficiency commitment in this project.

As I mentioned earlier, there are three IT rooms in the data center.  Room one is for eBay Marketplace (shown in the picture above), room two is PayPal.  Room three is planned for further consolidation work to support our strategy.

But, on May 4th, room three became something complete different - It became the coolest looking data center in the world!

 

Welcome to CLUB EBAY!

 

Every great project deservers a great party!

We converted the datacenter raised floor into a Club 51 atmosphere that the Hollywood A-list would feel right at home in - complete with a red carpet walkway. The attendees had no idea we planned to treat them like royalty.  But they were!  They were responsible for making this project a success.  Needless to say, it was one of the most unique company parties in eBay’s history. 

We started off the festivities with a video summarizing the magnitude and impact of this project on eBay and its employees.


Once we finished the video, Mazen Rawashdeh, VP of Technology Operations, James Barrese, VP of Architecture and Mark Carges, our CTO, gave their perspective of the importance of project Topaz to the success of eBay, PayPal and our adjacencies. We also had Jakob Carnemark, the Sr VP of Mission Critical from Skanska explain the complexity of the project and how crucial it was to have a full partnership in a construction project of this magnitude and complexity. Finally, I was able to bring up Mike and Greg and express our thanks for leading the construction teams to deliver this project.

Then we brought the house down!  A crew of local hip hop and break dancers proceeded to interpret the key words of our project -  Transformation, Innovation & Leadership.  They also helped us become "cool" as you can see in the picture.  And finally after years of planning, approval, construction, commissioning and verification, we plugged this puppy in and fired it up!

After the power-on ceremony.  We conducted tours of the site for all who attended.  Ironically, the majority of people that worked on this project, had done it remotely.  It was the first time that many were able to see the massive infrastructure that was built to house their technology.  The tours were complete with iPads showing full 3D models of the data center (BIM) and an interface into the infrastructure applications. 

We ended the day with a party at The Depot in downtown Salt Lake City.  It was a night filled with games, a live band and even some performances from the eBay employees themselves.  I had a chance to sing a rendition of YMCA with a motley crew of eBay employees. Our version was “Shop At eBay”! As you can see from the photo, everyone got involved!  It was quite a performance!

The Topaz launch party was a tribute to those that put their blood, sweat and tears into this project for more than two years.  Topaz is the epitome of collaborative teamwork and innovation as hundreds of people in all areas of the business came together to plan, design, validate, and execute on this incredibly complicated project. It is a testament to the eBay culture and the ability of our people to innovate, lead, execute and deliver truly exemplarily performance.  Ultimately, this data center is allowing us to deliver a better and more reliable experience for our buyers and sellers while keeping our infrastructure costs under control.

Here are some fun statistics about this incredible project:

  •  
    • Over 1.2 Million man hours worked with no loss time due to injury
    • 30 Megawatt power substation
    • 60,000 sq ft of usable IT space
    • 57 Miles of underground electrical conduit
    • 2 Million pounds of copper for underground critical power
    • 295 miles of Copper Cable Network
    • 20,000 Fiber Optic Cable 176,000 feet, Approximately (43,000) strands of fiber over (33 miles)
    • 2006 tons of Steel
    • $10 million in savings through 3-D computer modeling (BIM)
    • Over 200,000 measured points – all the way down to the individual server power plugs.

Needless to say, we are very proud of Topaz and the people that made it possible. Now, stay tuned for even more innovation and efficiency from the eBay engine.  We’re just getting  started …


 

 

What Should a CEO Look for in a CIO?

Posted Mon, 04/12/2010 - 10:00 by mark.thiele

What to look forWhat Should a CEO look for in a CIO

You should use the same decision criteria you would with proper "architecture" principles
- How will this new addition fit with the current team?
- How will their experience fit into corporate culture?
- How will their execution capabilities fit into corporate goals and objectives?
- Will this person be willing to stand up in front of a large crowd of executives and tell them what's needed and provide reasonable options for getting there?

As I've mentioned in previous posts I believe that in order for a CIO to be outstanding they have to be much like a CEO. The IT function is often times a mirror of the larger organization. There are teams or application groups that are focused on lines of business or certain business functions (I.e., Engineering or Marketing). Because IT is like a small version of the enterprise it's critical for the CIO to be Common Visionable to tie these groups together in a common vision. Only by tying the groups together can the CIO expect to deliver on broader cross functional corporate opportunities.

Building a vision for the entire IT organization that helps everyone understand how they fit into the bigger enterprise puzzle is crucial to team moral and to limiting intergroup competition or infighting. Building a vision isn't easy, but the CIO is more likely to be successful if s/he makes it a team effort. Getting your reports and their reports involved in the effort will help to ensure buy in and it's an automatic way to contribute to the communication of the plan. The vision should demonstrate how each vertical when working together contributes to making a successful enterprise objective. An opposite example might be something like the Marketing group building a great lead candidate DB, but not realizing that they could be pulling information from the call center and or providing information directly to the CRM solution. IT must be in a position to bridge that gap and no individual IT function can do that it takes the entire team. 

Key things to remember here from a CEO's perspective:

  • Include the CIO in ALL exec functions
  • Expect your CIO to act as an enterprise exec, not an IT exec (must still have a solid IT background)
  • Hire a CIO with the same due diligence that the board used to hire you
  • Solicit the CIO to help cross boundaries of functions to ensure a common vision for enterprise IT deliverables can be communicated to the rest of IT

Key things to remember from a CIO's perspective:

  • You're not a technologist first, but you must be strong enough in technology to deal with the highest level partners and to intelligently argue direction with direct staff
  • Lead by example. The CEO's job is to help set and communicate company vision, and to enable his/her team to get their job done, the CIO must do the same. The CEO also has significant customer communication requirements, you're role is no different.
  • Don't hesitate to use your team. You don't have to have all the ideas, you just have to show that you can effectively implement them.
  • AlCommon Vision 2l of our team (VPs to Helpdesk) need to be on the same page. If a customer stops anyone from IT in the hall and asks "what's IT working on", they should all have the same answer.

There's no way to guarantee that you'll hire the right person for the job, but at least you'll have a fighting chance if you treat to objective with the strategic vision it deserves

Is "Good" Enough? - How Should You Apply The 80/20 Rule?

Posted Mon, 04/05/2010 - 10:09 by mark.thiele

There's 20K in your budget for software and you had intended on using it for a real time data collection system for data center management, but the best solution would cost more than 20K just in professional services. Maybe it's not the money at all, but the concern over potential disruption to your DC production environment. You've got 30K for a monitoring tool, but you really want the comprehensive capabilities of an enterprise solution that starts at 150K. I guess you should just not spend the money or use it for more of the same old thing, because you can't buy the "perfect" or 100% solution.80/20 Rule


Sitting on your hands saying "I don't have time" or avoiding decisions, because of the search for perfection isn't the answer!  Don't waste your opportunity to get something instead of nothing.


When is "Good" enough? That's a tougher question than I thought it was, even though I consider myself an 80/20 rule evangelist.  I use the 80/20 rule on a daily basis for every decision from whether to file a document, to determining whether 2% milk is acceptable when 1% isn't available.  However, as a data center owner I've historically looked for absolutes and have always attempted to get the "perfect" environment.  Unfortunately there rarely are any absolutes and perfection is a momentary condition at best, unless we're talking about Halle Berry (my wife would say Harrison Ford).  Good enough isn't enough in many areas of data center construction and management.  I would never assume my electrical designs were "OK".  I would examine the plans for hours and have peers provide additional review. This process also applies to calculations around airflow, or the commissioning process and much more.  What we need to recognize is that regardless of how "perfect" a specific solution is, it could always be better, but when do you sacrifice more time in an effort to get more perfect?  A critical factor in determining where and when to use the 80/20 rule is the risk associated with getting it wrong. If you're calculating the power capacity of a cabinet, the 80/20 rule probably isn't a good idea. On the other hand if you have a need for improved visibility into your DC metrics, then any visibility is better than none.  I picked DC metrics on purpose, because most of us hesitate in this area of operations because we assume a solution will either cost too much or be too disruptive to implement.  This concern over covering everything in your metrics and or efficiency management is a roadblock for many because of the DC environment impact associated with putting metering and monitoring in a legacy space. However, any light in a dark spot is still better than no light at all, which is why I believe this area to be a perfect example of a place to use the 80/20 rule.


The impact of the search for perfection in IT projects


There are countless stories of IT projects gone wild because of scope creep. Generally speaking scope creep is a search for "new" perfection. New perfection is generated from hind sight and additional time to review technical options.  However, scope creep as we all recognize comes with risks to cost, delivery schedule and even overall project success.  Does risAnalysis Paralysisk to success mean we should never allow scope creep, absolutely not. However, new requirements should be weighed and an ROI review completed before the change (scope creep) is officially adopted. So how does scope creep and a search for perfection apply to "good enough"?  It applies in several ways, including improving project execution and avoiding project paralysis. In the context of the article, I'm mostly covering "project paralysis" or "decision paralysis".  Where and how does project paralysis come in to play? Project paralysis can be manifested in the simplest ways, like the struggle to select a food item on a menu that has dozens of items to choose from. It can also be demonstrated in IT solutions that are never successfully implemented because we're attempting to implement the perfect or best option.  


The failure to choose and implement an 80% solution, because you couldn't find what you believe will be the "perfect" solution is not success. In other words no decision is still a decision, and it's often the worst option.  Considering the struggles each of us face to find resources (human, hardware, or dollars), we can't afford to stay locked in a downward spiral or flat line because we don't have time to find the perfect option.  A great example of how we all use the 80/20 rule is the "low hanging fruit" approach; "how can we get the most bang with the lowest investment of time and money?"


Assuming we all generally agree with the 80/20 rule, why then are we constantly struggling with fire fighting or ignoring efficiency improvements because we don't have the time to spend on selecting and implementing the Good Enough"perfect" tool or process.   We need to carefully consider our priorities for the year and determine the best way to spend what little we have in budget and human resources, but this should be a process for generating opportunity, not for doing the same old thing.  When you look at a solution keep in mind your ability to execute quickly and the flexibility of your commitments.  I know this strategy of looking at point fixes flies in the face of what many of us grew up with. All of us have worried about implementing things that will increase environment complexity and make future integration activities more costly.  You should continue to worry about commitments that become a burden or technologies that add complexity, but the good news is that Cloud/SaaS offerings can help alleviate many of the risks.


Quick Tips on using the 80/20 rule in IT solution acquisition:


- Consider what your long term ownership of the solution will be


- Are there other systems that the new solution needs to integrate with


- Can you get out from under the new solution quickly when needed


- Will it bring immediate monetary or process improvement


- Is it a bridge solution that will support a piece part acquisition strategy


- Is it a near term gap filler


- Does it create an opportunity that might otherwise be missed for years while you wait for the perfect time/solution


The next time you see a solution that has the potential to positively impact your ability to support enterprise initiatives, don't just pass it up because it's not the tool or process you really want.  Take a moment to consider your rules of ownership. Also, be sure to apply things like Cloud & SaaS to your ownership model as this creates a new perspective on the difficulty or ease of acquisition. After all, many of us want a Ferrari, but we still end up with a Ford or Toyota because we can't wait to have a car indefinetly.

Your Next Data Center - Can You Say "Cookie Cutter"

Posted Mon, 03/01/2010 - 09:35 by mark.thiele

"Cookie Cutter Data Center", blasphemy I say, "I can build a better data center than anyone else, I'll build it myself"! All of us who have grown up as IT folks harbor that feeling of "we can do it better ourselves". The truth is that feeling of being able to do it better is what makes us good IT folks, but it comes with a serious risk. That risk is that you’re spending valuable time and money making something unique, when the off the shelf option might have been “good enough”.


I'm certainly not an advocate for eliminating the spirit of invention inherent in IT folks, but I do believe that it needs to be managed effectively. The risk of proliferating new solution options whose only benefit is that they are different from your neighbors is a big deal, especially as it relates to data centers.


In just a few short years we've gone from most data centers having a PUE of 2.0 or higher to most new data centers having a PUE of 1.5 or lower, a 50% improvement in roughly three years is nothing to sneeze at. Now we are rapidly moving towards a new standard for PUE of being 1.2. I'm great with this achievement and wouldn't even consider minimizing it or the associated benefit to enterprises and the environment. My guess is that collectively data center improvements in the last year have provided more than 5X the benefit to the environment than the famous Cash for Clunkers program did.


Now for the contrarian point of view


How much do we spend making a data center unique vs. how much benefit do we gain? In other words, if I were to build a data center from scratch and obtain a PUE of 1.18, how much did that unique effort cost as compared to just using another design that would have guaranteed a PUE of 1.2? Unique is expensive, generally speaking, and unless you have the scale of a Yahoo, Google or Microsoft, you probably can't really afford it now that the efficiency improvement opportunity has shrunk so much. So what can we do? Collaboration! What a concept, if we aren't actually using our data centers to compete against each other, why don't we collaborate on building them to a common if not standard design? Remember that there's more to efficiency that the PUE number. What is the ROI of your .02 improvement in PUE? If you spend 2 million to save 2.2 million over 10 years, you'll find that most CFO's won't be too happy with that decision. There's another important point to consider; if we moved to a standard model for building data centers we could drive down the cost of building them and the cost of the MEP gear used to run them. My guess is the cost benefits of standard implementations probably outweigh the benefit of dropping your PUE by 2 one hundredths.


I know this sounds easier than it is


This is a more difficult problem than it might seem. There are a number of perceptions about what's important to a specific data center’s design, and while I won’t try to cover all of them the following are a few of the more obvious;


- Location


- Equipment Density


- Infrastructure (virtualization, physical, cloud or a combination)


- Cost of Power


- Environmental Conditions


- Size


- Etc., etc..


All of the above can be real reasons for thinking you have a unique need and you might be right, but. Data Center's are still designed around the inefficiency of the equipment they house and they are designed to provide additional redundancy for our systems to avoid business interruption. I believe the negative drivers (inefficient IT gear and redundancy or DR) are being minimized through improvement in server design and the introduction of virtualization and cloud.


How are Cloud, Virtualization & Server design affecting Data Center design criteria?


A large part of the world is in the temperate zone, this means we have temperatures that are cool enough to support data center operations without additional cooling from HVAC units all year. In fact new server designs can handle inlet temps of up to 35C or 95F. Now, consider the ability to keep the data center much hotter with the benefit of using a virtual or cloud solution for your entire environment. With a cloud solution your redundancy and system protections are inherit in the platform, you no longer need (in most cases) to build a Fort Knox to protect your systems, the software takes care of it. Another key benefit of having your infrastructure in a cloud is that you can move towards treating your hardware as commodity and allow it to fail in place.  Now if you sum up what I've just said it adds up to a data center that can run hotter, and needs fewer people to operate it. This new data center environment would be incredibly more efficient that any of today's typical DCs would be because you reduce the space required for staff and you can run hotter in a smaller space. Less space, smaller staff, less power, less equipment, etc., etc..


A Few Prognostications about What's Next for the Data Center?


I'm not going to try and answer that question completely, I'm just going to hit the highlights:


- Modularity is a key opportunity area for the DC - Whether that's Containers or within a larger structure


- Rapid Deployment - Like any IT solution getting it into production faster is a huge benefit vs. getting it 100% perfect (that doesn't mean you don't do commissioning. There's a big difference between best possible efficiency and a functionally perfect data center)


- Efficiency is measured in more ways than PUE - PUE is an important measure, but it needs to be compared against overall project delivery and ownership costs


- Remote Management - Data Centers will be added to your environment as "compute capacity" so it makes sense that they should be remotely managed and therefore can be placed almost anywhere


- Failure in Place -  hardware will/should move more towards commodity, which allows the owner to make ROI decisions based on factors like; failure rate, power to compute ratio, ease of acquisition, etc..


- Flexibility - Can be deployed with a mix of densitand tier levels


None of the above happens by accident and if we aren't trying to get there on purpose it will take much longer to happen. So, look around you and see who else might benefit from sharing your next data center build so you can implement the best solution to protect your business going forward and to a positive ROI, not just a “good” PUE.


 

Death To The Datacenter!

Posted Thu, 02/25/2010 - 21:59 by geekism


Last month, we killed our first eBay data center. Don’t worry, it had it coming...

 

I arrived early at the eBay San Jose campus as the rain continued to drench northern California. I joined a group of lively eBay employees from technology operations, product development and IT on a bus headed to Sacramento. We were on a journey to put our oldest data center to rest. This journey had started over a year and a half earlier, long before I had joined eBay. At that time, an aggressive plan was put into motion. The goal was to consolidate the data center portfolio to decrease costs, increase our availability and take eBay to the next level of Operational agility. It was a lofty goal.

eBay, like many of the rising star Silicon Valley companies, had been in constant react mode to keep up with demand. They had amassed a data center portfolio that spanned three states and in twelve different data center sites. Eight years earlier, the Sacramento data center (SMF) was the first to be brought online as a disaster recovery location and it was supposed to be temporary. It quickly expanded to become much more than that. When the idea of shutting it down was raised, the feeling was it was too big a task, too complex and too costly to execute. It would be like rebuilding the engine of a jumbo jet while you were in flight.

So much for that! The operations team had created a four year strategy to consolidate the data centers from colos into owned sites. The board had approved the construction of a large, owned site in Utah and that would take two years to bring online. More about this in future blog entries. But, I digress…

I’m now on my fifth month at eBay. I joined this project as it was nearing completion, so I did not have the full context. On the bus ride up to Sacramento I heard from the team just how difficult this project had been. As I listened to them explain the huge effort, both logistically and technically that it took to line this up, I started to see the real power of the team I had joined. When the eBay collective gets behind a problem, no matter how difficult, they swarm and solve it. The collaboration between architecture, product development, domain, site speed, network, the NOC, logistics, Asset Management, corp IT and the business units was simply amazing. Hundreds of people collaborated, planned, tested and executed on this highly complex project for over 18 months and got the job done on time and with zero impact to the site availability.

As the bus continued to roll north on I-80, I saw the pride on the faces of the team as they reflected back on the project. They should be proud! This is one of the hardest problems to solve in IT. They had successfully decoupled the app and DB tiers, landed them in a temporary location, moved live databases and applications – all without causing a single outage to the site.

When the bus pulled up to the data center, the rain was still falling and everyone was eager to get our first glimpse of the emptied floor space that they had all worked so hard clear. Once the rest of the group arrived we realized it was the first time that (nearly) all of the key players in this project had literally been in the same room. After a quick meet and greet, everyone was able to put a face to the name they had worked so closely with over the last 18 months. Following a complete tour of the emptied facility, we were all able to convene for lunch. A nice spread, complete with a variety of celebratory beverages, was provided and we were able to enjoy lunch on the raised floor of what, only days before, had been a live and completely functioning data center.


Paul Santana, the longest tenured eBay operations employee (11 years) and the Sacramento region data center manager had lead the team on shutting down the physical site. As you can see from the pictures it rapidly went from order to disarray as they neared the end.

Then came the really fun part! At least for me. :-)

After some final words were spoken by Paul, Bala (the project lead), myself and even Olivier Sanche, my predecessor who kept this crazy thing going before I arrived, we got down to the business of truly putting this data center to rest. After 8 years of serving as a core to eBay’s complex portfolio of data centers, all of the 3400 assets had found a new home, except one…and it was not going to get off easy. This old x335 was sitting timidly in the last rack left on the floor, waiting for its time to go, not knowing that we had something else planned for it. So much time, effort, collaboration, planning and precise execution had been put in to bring this facility to a close, but one more action was needed to officially signify the death of this data center. And with that…WHAP! I reared back and drove an axe through the final remnant of this once complex, powerful and archaic beast of a data center. A symbolic, but appropriate end to the life of this aging server…and its home… (don’t worry the asset was recycled. :-)

With the final server properly dealt with, the only thing left to do was cut the power. The honor was given to Bala Meduri, the data center closure project lead who spent countless hours preparing the data center for this moment. He donned a pair of heavy electrical gloves and pulled the switch. The switch went down, the lights went out and the data center was dead…RIP. Some final photo ops were taken, a few celebratory toasts and then it was gone, just like that.


Now, on to the next kill...another colo in California...I feel like Dexter :-)

More shutdown pictures of the SMF DC death event are included on shutterfly: http://eBayDC.shutterfly.com

(shutdown photos courtesy of Steven Stafford, Stafford Photography - imagemaker747@gmail.com)

A Session on the DCP Stack with BrightTalk Virtual Events

Posted Tue, 02/23/2010 - 18:56 by mark.thiele

I'm going to be doing a talk on the DCP Stack for Brighttalk virtual events on March 17th. The topic is certainly timely and the goal is to help demonstrate how the Stack can be used to develop a more holistic approach to managing your data center environment.

Please pass the word and join in if you have the chance.

Mark

Warming up to my new Gig at ServiceMesh & a few notes about DCP

Posted Fri, 02/12/2010 - 17:46 by mark.thiele

It’s been two weeks now and I’m still trying to get my head wrapped around my role, with a new company.  The product we offer creates so many opportunities in the infrastructure & cloud space that my head is constantly spinning with thoughts on more ways the tools can be put to use.  In my new role as VP of Data Center Strategy I’ll have several responsibilities, not the least of which is worrying about our own DC infrastructure going forward. In the near term though, but job will be to help ensure we’re partnering with the right people and that the best ideas from the real world of data centers are incorporated in our product. I’ll also be working with our customers to get them the best possible advice on how to plan their own operations as they move into the cloud.


A little about ServiceMesh


ServiceMesh is a small startup with a big (huge) opportunity in the cloud orchestration space. We’re working with some of the biggest customers helping them to solve their issues of moving to the cloud while maintaining standards, common interface, governance and policy, all while increasing execution efficiency. The offices are in Santa Monica & Austin, but I’ll be working out of my home in the Bay Area.


This new role is a significant departure for me. I’m going to be doing much more of the strategic visioning, and customer communication. I’ll also be more directly involved in product direction.  The beauty of this new arrangement is that I’ve found a role that uses my experience as a strategic advantage, not as an operational necessity. 


A Few Notes about DCP


I expect to keep doing my night job, which is working on Data Center Pulse. This group has taken on a life of it’s own and requires much more attention (for all the right reasons) than Dean and I would have ever dreamed in September of 2008 when the group was founded.  On an almost daily basis now folks are reaching out to us with opportunities to speak, contribute to product design or partner.  While we’re gratified at the value this group has created for the industry, we’re also worried about keeping up with demand. I can tell you that without the selfless (payless) contributions of our board of directors, our technical advisory board and some of our general members, we just wouldn’t be able to keep up. A few quick facts; DCP now has 1390 members, in over 45 countries, from 700+ companies.  We have signed partnerships or working agreements with several key industry organizations, like The Green Grid & 7X24 Exchange and we’re working on others.  


It’s a humbling experience to work with and learn from so many smart folks, all of whom are just trying to help do what’s best for their companies and the industry as a whole.  My hat is off to the membership of Data Center Pulse. I hope you’re getting as much from this deal as Dean and I are.

Upgrading an existing Data Centre Part 3

Posted Fri, 01/29/2010 - 04:09 by jack.mcardle

Well its been a hard shift. Data Room 7 is nearly finished and ready for handover. It has been one of these projects where the hard bits have been easy and the easy bits have proven to be hard.The upgrade of the main incoming electrical supply which on the face of it was fraught with disaster actually went very well and gave us no issues at all, other than the fact that the day we planned it coincided with the heavy snow falls just prior to Xmas.The easy bit, which should have been converting an office to a Data Room fell apart a bit because the building firm went burst in the middle of the project.But the rasps this time went to GEA Denco who managed to allow 6 air-side dry coolers to freeze up and shatter their coils over the Xmas period....What Glycol, I hear you say. EXACTLYAh well, onwards and upwards and at last GEA Denco are coming to the end of their commissioning process.Next week sees final clean up and Fire Suppression connections and its given to the sales guys to sell.Nothing can go wrong now....Can it?????

The Pulse = Collaboration

Posted Fri, 01/15/2010 - 02:34 by geekism
TGG_DCP

When I joined eBay in September of 2009, I had the pleasure of taking a seat on the Green Grid Advisory Council (AC).  The AC was formed in late 2008 to provide input and guidance on the general direction of the consortiums strategies and drive a greater awareness of the Green Grid with end users.  The AC is made up of executives from AT&T, ADP, eBay, Nationwide Insurance, Strato, The Walt Disney Company, Tokyo Electric Power Company, and Verizon. My first meeting was at the New York Stock Exchange where the GG held a call for action event.  The NYSE hosted the GreenGrid event and later allowed the BoD and the AC to go onto the trading floor.  The GG Board of Directors were also able to ring the closing bell. Too bad the platform wasn't bigger - we all couldn't fit. :-) But, we were able to watch from the trading floor.  In the picture shown here, we were right below the platform. Denis, in the brown jacket to the right of me, runs all of Disney's Data Centers. We happened to run into the guy who does all of the trading for Disney (in the blue jacket). Very cool. I just wish I could have found the eBay trader, but we're not listed on the NYSE.  The NYSE trading floor isn't what you see in movies (chaotic, with people yelling and exchanging paper). It is filled with computers, and thousands of screens with tools providing real time reports that the traders use to make decisions. After the closing bell, we were able to take a few photos (note 4:12:59 on the clock).  Being a uber-geek certainly has it's perks.  :-)
This was the first Green Grid event I had ever attended. I had seen the great work that they had been doing on defining and documenting PUE/DCiE and the way to measure PUE (quickly becoming the industry standard), free cooling maps for North America, Europe and Japan, measuring data center productivity and their work with the US government (DOE/EPA/others) on efficiency.  When I was at Sun, Mark Monroe, Director of Sustainable Computing joined the GG BoD. He  participated in the Data Center Pulse summit on behalf of the Green Grid to obtain feedback on the metrics work they were doing.  I was watching this organization grow - from a distance.  And grow they have.

 

As I attended the NYSE event, I started seeing the potential for our two organizations to work together. The GG Advisory Council seemed like a perfect conduit for DCP members to share information. We had similar charters. It also seemed like a great avenue to expand the work we had been doing on the DCP Stack Framework and the Chill Off efficiency experiements we had been championing.  In addition, the Green Grid has over 175 global companies represented in their members list including all of the Chill Off participants.  At this session I offered up the possibilities of collaborating with the AC and BoD.  At the next AC meeting we started laying out the framework of how this might work. In December of 2009 I attended the GG board of directors meeting along with other AC members. We laid out the approach and the benefits of working together.

 

Today, I am pleased to announce that this collaborative agreement between Data Center Pulse and The Green Grid has been formalized. We are excited about the opportunity to provide direct and timely input and feedback.  The Green Grid is very interested in being able to get "the pulse" of this large group of end users.  Our goal has always been to influence the industry through the eyes of the end user.  This is exactly how we do it.  I want to ensure our members that we are staying true to our original vision of protecting their anonymity. We established the relationship through the GG Advisory Council to do just that.  Over the next few months we will be working through the mechanics of this relationship.

 

I quickly scanned the headlines today and a number of the industry media groups have picked up the story.  Great perspectives from Data Center Knowledge and downstream blog.

 

Stay tuned for more information. If you have any thoughts or ideas, please share them by emailing dean.nelson@datacenterpulse.org or feedback@datacenterpulse.org.

Data Center Monitoring is a Converging Space

Posted Wed, 11/18/2009 - 18:28 by jeremy.rodriguez

Data centers increase in complexity as redundancy and higher power per rack designs are introduced.  Managing the modern data center environment is spawning a new type of business critical application which I often refer to as a “critical facilities application”. 
The modern day BMS is a new type of application that supplies information about the operating data center. The BMS and its architecture match traditional IT systems with application servers and databases.

With a focus on efficiency, electrical and mechanical monitoring are becoming more complex with trending features, real-time PUE monitoring, and more points in more systems.

This convergence in application sophistication mirrors the converging disciplines of IT and Facilities professionals.  The IT workers are becoming interested in BMS data and Facilities professionals are looking more at IT server based monitoring points.

With overlapping needs for data the two organizations are at most need of collaborative efforts. In other words, Facilities professionals are typically not router/server/storage/database experts and IT professionals are typically not electrical/mechanical/plumbing experts! BMS applications are spanning the two disciplines.

New needs are also surfacing for network administrators who may be caught in the middle of the resource management struggle.  Being asked to do more is apropos for network professionals who may find themselves explaining that vlans are effective at isolating BMS traffic and physical device separation may not needed.

 

 

What is driving this change and convergence?  Simply put Moore’s Law.  Electrical/mechanical components are becoming more instrumented for monitoring and control.  IT hardware is, in turn, demanding the more granular facility management to further the use of new technology (multi-core processors, higher RAM quantities, and more storage).

The Stack Framework can be applied to understand the changing needs of BMS and monitoring solutions for technology coordination and for determining metrics important to you and your business for each stack layer in your data center. Take a look at the stack and use it to have a conversation with counterparts interested in BMS, server, and network data.  Use the stack to pin-point areas of responsibility overlap and work out who does what.

The future will continue the commoditization and convergence of data center technologies.  Professionals working in the data center space should be prepared to collaborate and become more multi-focused.  Technology doesn't solve problems, people do. Efficiency is not a single person's or department’s problem to solve and collaboration with data sharing is a great step into the future of data centers!