Cisco UCS M3 and Intel Romley

Cisco UCS M3 and Intel Romley

 by nashwj

Today Cisco officially unveils the next generation of UCS servers.  These go along with the new announcement by Intel this week of their “Romley” platform that use the E5-2600 family of CPUs.  Let’s take a look at what’s new, changed, and where that puts us.

First, Let’s Look at Intel’s Announcement

These new blades are based on the new E5-2600 series of CPUs.  The codename for this whole platform is/was Romley.  The E5-2600 CPUs were code named Sandy Bridge.  If you’ve seen those names floating around now you know how it all comes together.  These CPUs replace the Xeon 5600 line that has been the basis for the B200 and C200 series of servers from Cisco.  Where as the 5600 line topped out at 6 physical cores per CPU the new E5-2600 line goes to 8 cores.  Just like the 5600 line they have HyperThreading so they can execute up to 2 threads per clock cycle.  This means we’ll now have dual-socket servers with 16 physical cores capable of executing 32 threads.

Memory is usually the constraint in a virtual environment, and that’s where we play most often.  These new E5s can do 24 DIMM slots in a single dual-socket server.  Combine those slots with the newly available 32GB DIMMs and you have a real small powerhouse that can do 16 cores and 768GB of RAM.  Along with this you’re going to see much higher memory throughput, as well.  Maximum memory speed goes up to 1600MHz with up to two DIMMs per bank.  When you go to 3 the speed drops to 1066MHz.  The QPI (QuickPath Interconnect), the direct path between CPUs, is now faster and there are now two of them.  Previously it was 6.4GT/s and now it is 8GT/s.

Along with fast RAM you also get more maximum cache.  The higher end chips go to 20MB of onboard cache.  You can see how these are really starting to blur lines with the E7 line of processors.

Another enhancement is what is referred to as Turbo 2.0.  The previous generation of chips would raise the clock speed of the CPU up when it could.  That was when the thermal load allowed for faster speed and usually when not all cores were utilized.  If you had a 6 core CPU but only 2 cores were active and the temperature of the chip had some headroom the system would “turbo” up the CPU in 133MHz increments.  These new CPUs are more aggressive and will actually go above the TDP (Thermal Design Power) for a brief period of time.  Basically, you should see turbo mode more often which leads to faster processing.

I was going to put a table here of the new CPUs but I’ll let the fine people at Anandtech do that for me here.  They also have a more in-depth overview of the new offerings.

The bottom line is that these CPUs are fast…of course they are faster than the 5600s they replace but it’s not just more cores and faster clockspeed.  In fact, clock speed is stagnant or getting a little slower yet these CPUs still get more done in a cycle than their predecessors.  When looking and comparing you need to go find good benchmarks or CPU leveling scales to make accurate estimations for sizing.  Some benchmarks I’ve seen show the same clockspeed CPU on the 5600 and E5 lines and the E5 almost 50% faster.  That’s impressive.

Oh yeah..one more thing…  Eventually Intel will have a line of E5 CPUs that are quad-socket capable.  Consider that a less expensive quad-socket option than the E7s.

What is Cisco Doing With These?

First, Cisco is not releasing M3 versions of all servers.  The new E5-2600s do not replace the E7 line of CPUs so the B230M2 will continue to be offered in its current form for a while.  They are releasing a B200M3, C220M3, and C240M3.

Now let’s start with the B-series enhancements.

So along with the new CPU and chipset, what else is new for the M3 series of blades?  A number of things, especially around I/O.  Recently we saw the introduction of new second generation interconnects (6248UP and soon the 6296UP) along with the new FEX (Fabric Extenders), the 2208XPs with 8 ports each.  To help drive that sort of I/O from the blades Cisco released…well..announced…the VIC1280 mezzanine card.  This card gives a blade eight 10Gb ports for a total I/o throughput of 80Gb.  Along with the new M3 blades is the introduction of the VIC1240.

This new module provides 40Gb of throughput using two 10Gb ports to each fabric and lets you create up to 256 virtual devices (vNICs and vHBAs).  This will basically replace the M81KR (Palo) adapter that has been our good friend for a while.  But wait!  There’s more.  Along with the VIC1240 you can add an optional module, the Port Expander Card.  This takes the VIC1240 up to 80Gb of throughput, much like a VIC1280…but you can add this optional expansion at any time.

These modules will be known as MLoM, Modular LAN on Motherboard.  Why not just LoM?  Because they are optional.  There was talk earlier about building the VIC1240 on the board but that’s not the case in the final version.  These new MLoM adapters fit on a connector on the blade.

The older generation mezzanine adapters will NOT fit these new blades.  That also means the other adapters, like Qlogic and Emulex CNAs, will be revised to a new version as well.  Yes, they are still available just like they were before.  But….I see two slots there.  What’s the other one for?  3rd party modules.  What sort of modules?  Well…I get asked all the time for other things on UCS blades like GPUs for rendering…Fusion I/O type caching cards…VFCache from EMC was recently announced and needs a PCIe type card.  I suspect will see announcements about these other modules soon.  And yes, you absolutely can do the VIC card and a 3rd party card at the same time.

Note:  If you add the Port Expansion to the VIC1240 that means you can’t use a 3rd party module as the Port Expansion uses that mezzanine card slot.

What else about the new B200M3?

We already talked about the mezzanine adapters.  There is an internal USB 2.0 port.  You can see the 24 DIMM slots in the picture.  It still retains the two hot swap SAS drives in the front but now you have the option for SSDs on the B200.  The M3 blades now also have two internal SD card slots.  Eventually you’ll be able to boot from these but that won’t be available right away.  My guess is that we’re waiting on an updated UCS Manager since there is no boot option for that currently.  This new feature is called “Cisco Flexible Flash”.  Pretty exciting name for two SD slots.

A Word About I/O

Along with these new blades another FEX for the UCS chassis was released, the 2204XP.  It is the second generation 4-port FEX.  To take advantage of all features of the VIC1240 and the Port Expander you’ll need to upgrade from the 2104XP to the 2204XP or 2208XP.   I suspect we’ll see some good upgrade offers for these, but I don’t know for sure.  That does explain why the 2208XP has been the FEX included in the UCS bundles lately.  Cisco should be releasing a support and features matrix showing the VICs and the FEX options and how they work together.

 And  Now the C-Series Offerings

I won’t lie…the vast majority of UCS business that I’m involved in is B-series but Cisco’s C-series rackmount servers continue to evolve.  Along with the new B200M3 Cisco is releasing the C220M3 and C240M3.

The picture above is the new C220M3 and I have to say, the servers are at least starting to look nicer!  This is your standard 1U server that can do two CPUs for a total of 16 physical cores and 768GB of RAM in 24 DIMM slots.  It has two PCIe slots.

That is the C240M3.  Can you guess what its claim to fame is?  If you said disk density you were right!  Since it’s based on the E5-2600 line it is also dual socket and can go up to 768GB of RAM.  It has 5 PCIe slots and can hold 24 SAS drives in 2U.

Cisco has recently changed the architecture for how you can connect and manage the C-series servers under UCS Manager.  I’ll do another post on that soon, but the good news here is that now all C-series will support being used in that type of configuration.  It’s not something that has been popular in the past due to the complexity but I’m interested to see how the new offering does.  It use a lot less ports and makes much more sense.

Dual BIOS Support

This falls under both B and C-series and is worth mentioning.  All the new M3 systems have dual BIOS capability so that they can recover from a corrupt BIOS image.  Previously this would require RMAing the system, but that’s no longer the case.

Jason’s Opinion

You can probably stop reading here…  My thoughts are simple.  The only real surprise here is the updated I/O capability and it’s not totally unexpected.  Cisco tipped their hand a bit with the 2208XP FEX a while back.  These are all very good evolutions, especially the option for 3rd party mezzanine cards.  That’s something that customers are starting to ask about and with EMC’s announcement of VFCache and their involvement with VCE there had to be a solution for that.  You’ll find I’m a big fan of options and these new offerings give you options.  Intel has done a great job building a platform that Cisco then takes to the next level.

As Intel expands out the E5 line you’ll see Cisco revise the blade offerings but for now we get a good taste of what they are thinking.  I am looking forward to the quad-socket E5 CPUs as well as the price of RAM continuing to drop so that 16GB DIMMs get even cheaper and one day those new 32GB DIMMs become affordable.  768GB in the standard workhorse B200M3 blade!  CRAZY!

One advantage that Cisco has had for a while now is their extended memory support using custom developed silicone.  They could do more memory density than others while also keeping the memory speed high at 1333MHz or 1066MHz.  It’s been a serious differentiator for them.  With the new Romley platform the playing field is now level.  If you look at the B200M3 you’ll see there just isn’t room for more DIMM slots and you can’t go bigger on DIMMs than 32GB (at least for now).

But what about the B230 and the E7 line of CPUs?  Aren’t these encroaching on that territory?  Yes and no.  The B230 still offers 10-core E7 CPUs to give you 20 physical cores on a single half-width blade.  Memory support on those blades is still very good and enough for almost everyone.  The E7 also has other features that the E5 line does not.  One example is RAS (Reliability, Availability, Serviceability).  I need to do a good writeup on RAS as many people have no idea it’s in there but it’s a key feature on those CPUs.  For example, if the CPU detects an unrecoverable memory error it can tell vSphere to kill the VM using that RAM but leave the others untouched.  In a system without RAS that would cause a vSphere host to purple screen and take all VMs down.  Very important for business critical applications.  That’s why the B230 and B440 are still very relevant today and I’d have no problem deploying more.

These new offerings can be ordered but probably won’t ship for at least a month.  Work with us (preferably!) or your partner for pricing.