Tieto’s EMC VNX5700 array sparked 5-day disarray – new claim

Post from Chris Mellor (thank you) over at El Reg on the outage at Tieto in Sweden. See the first part here

Tieto’s five-day outage disaster started with multiple failures of its EMC VNX5700 array’s FAST Cache, according to a Finnish source close to the matter.

Tieto is a major IT services organisation across Scandinavia and the Nordic region – although it also provides services globally – and pulls in net sales of SEK17bn (£1.59bn). Its large customer base in Sweden means that when it had a five-day outage in November, it caused chaos to IT services across that country. The stoppage was caused by failures in an EMC storage array and compounded by an inadequate disaster recovery plan involving Networker tape backup files which could not be read. The circumstances are not clear and seemed to involve a VNX array with an upgrade to an NS480 (Celerra) system for flash, which is a logical nonsense.

El Reg has been sent a Tieto slide deck (PDF) describing why the service provider migrated from its Celerra NS4800 to a VNX5700 and the resulting performance improvements: namely lower latency and more IOPS. This deck is in Swedish but Google Translate gets around that little problem.

Based on the translated slide deck text, the story goes like this: in the 2010/2011 period, with a EMC Celerra NS480 array, Tieto saw its storage challenges as performance, response time, scalability and capacity. So it migrated from RAID (4 + 1) groups to Thick Pools composed of 60 disks and began to segment data types into Fibre Channel and NAS. The next step was to install EMC’s FAST Cache with four 200GB SSDs and the cache license, which was beneficial as response times were more than halved to less than 20ms. However the NS480 CPUs were maxed out. Read on here

How To Get Hardware Information On Linux Using dmidecode Command

dmidecode command reads the system DMI table to display hardware and BIOS information of the server. Apart from getting current configuration of the system, you can also get information about maximum supported configuration of the system using dmidecode. For example, dmidecode gives both the current RAM on the system and the maximum RAM supported by the system.

This article provides an overview of the dmidecode and few practical examples on how to use dmidecode command.

1. Overview of dmidecode

Distributed Management Task Force maintains the DMI specification and SMBIOS specification. The output of the dmidecode contains several records from the DMI (Desktop Management interface) table.

Following is the record format of the dmidecode output of the DMI table.

Record Header: Handle {record id}, DMI type {dmi type id}, {record size} bytes
Record Value: {multi line record value}

• record id: Unique identifier for every record in the DMI table.
• dmi type id: Type of the record. i.e BIOS, Memory etc.,
• record size: Size of the record in the DMI table.
• multi line record values: Multi line record value for that specific DMI type.

Sample output of dmidecode command:

# dmidecode | head -15

# dmidecode 2.9
SMBIOS 2.3 present.
56 structures occupying 1977 bytes.
Table at 0x000FB320.

Handle 0xDA00, DMI type 218, 11 bytes
OEM-specific Type
        Header and Data:
                DA 0B 00 DA B0 00 17 03 08 28 00

Handle 0x0000, DMI type 0, 20 bytes
BIOS Information
        Vendor: Dell Computer Corporation
        Version: A07
        Release Date: 01/13/2004

Get the total number of records in the DMI table as shown below:

# dmidecode | grep ^Handle | wc -l
56

(or)

# dmidecode | grep structures
56 structures occupying 1977 bytes.

2. DMI Types

DMI Type id will give information about a particular hardware component of your system. Following command with type id 4 will get the information about CPU of the system.

# dmidecode -t 4
# dmidecode 2.9
SMBIOS 2.3 present.

Handle 0x0400, DMI type 4, 35 bytes
Processor Information
        Socket Designation: Processor 1
        Type: Central Processor
        Family: Xeon
        Manufacturer: Intel
        ID: 29 0F 00 00 FF FB EB BF
        Signature: Type 0, Family 15, Model 2, Stepping 9
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)

Following are the different DMI types available.

Type   Information
       ----------------------------------------
          0   BIOS
          1   System
          2   Base Board
          3   Chassis
          4   Processor
          5   Memory Controller
          6   Memory Module
          7   Cache
          8   Port Connector
          9   System Slots
         10   On Board Devices
         11   OEM Strings
         12   System Configuration Options
         13   BIOS Language
         14   Group Associations
         15   System Event Log
         16   Physical Memory Array
         17   Memory Device
         18   32-bit Memory Error
         19   Memory Array Mapped Address
         20   Memory Device Mapped Address
         21   Built-in Pointing Device
         22   Portable Battery
         23   System Reset
         24   Hardware Security
         25   System Power Controls
         26   Voltage Probe
         27   Cooling Device
         28   Temperature Probe
         29   Electrical Current Probe
         30   Out-of-band Remote Access
         31   Boot Integrity Services
         32   System Boot
         33   64-bit Memory Error
         34   Management Device
         35   Management Device Component
         36   Management Device Threshold Data
         37   Memory Channel
         38   IPMI Device
         39   Power Supply

Instead of type_id, you can also pass the keyword to the -t option of the dmidecode command. Following are the available keywords.

Troubleshooting Using dmesg Command in Unix and Linux

During system bootup process, kernel gets loaded into the memory and it controls the entire system.

When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process.

These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten. You could see all those messages after the system bootup using the dmesg command.

1. View the Boot Messages

By executing the dmesg command, you can view the hardwares that are detected during bootup process and it’s configuration details. There are lot of useful information displayed in dmesg. Just browse through them line by line and try to understand what it means. Once you have an idea of the kind of messages it displays, you might find it helpful for troubleshooting, when you encounter an issue.

# dmesg | more
Bluetooth: L2CAP ver 2.8
eth0: no IPv6 routers present
bnx2: eth0 NIC Copper Link is Down
usb 1-5.2: USB disconnect, address 5
bnx2: eth0 NIC Copper Link is Up, 100 Mbps full duplex

2. View Available System Memory

You can also view the available memory from the dmesg messages as shown below.

# dmesg | grep Memory
Memory: 57703772k/60817408k available (2011k kernel code, 1004928k reserved, 915k data, 208k init)

3. View Ethernet Link Status (UP/DOWN)

In the example below, dmesg indicates that the eth0 link is in active state during the boot itself.

# dmesg  | grep eth
eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem 96000000, IRQ 169, node addr e4:1f:13:62:ff:58
eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem 98000000, IRQ 114, node addr e4:1f:13:62:ff:5a
eth0: Link up

4. Change the dmesg Buffer Size in /boot/config- file

Linux allows to you change the default size of the dmesg buffer. The CONFIG_LOG_BUF_SHIFT parameter in the /boot/config-2.6.18-194.el5 file (or similar file on your system) can be changed to modify the dmesg buffer.

The below value is in the power of 2. So, the buffer size in this example would be 262144 bytes. You can modify the buffer size based on your need (SUSE / REDHAT).

#  grep CONFIG_LOG_BUF_SHIFT  /boot/config-`uname -r`
CONFIG_LOG_BUF_SHIFT=18

5. Clear Messages in dmesg Buffer

Sometimes you might want to clear the dmesg messages before your next reboot. You can clear the dmesg buffer as shown below.

# dmesg -c

# dmesg

6. dmesg timestamp: Date and Time of Each Boot Message in dmesg

By default the dmesg don’t have the timestamp associated with them. However Linux provides a way to see the date and time for each boot messages in dmesg in the /var/log/kern.log file as shown below.

klogd service should be enabled and configured properly to log the messages in /var/log/kern.log file.

# dmesg | grep "L2 cache"
[    0.014681] CPU: L2 cache: 2048K

# grep "L2 cache" kern.log.1
Oct 18 23:55:40 ubuntu kernel: [    0.014681] CPU: L2 cache: 2048K

Introducing Target Driven Zoning (TDZ)

Post by Erik Smith (thank you) on Target Driven Zoning

At 9:28 on Tuesday 12/6/2011, the T11 FC-GS-7 working group approved a motion to incorporate the text for “Peer Zoning” that was prepared by Claudio DeSanti (Cisco).  The actual text can be found in the T11 document 11-411v2.  To me this moment marked the culmination of a four year journey to take a concept, get it through a standards body and into a standards document.

Target Driven Zoning (TDZ) is a proposed application that utilizes Peer Zoningto reduce the number of steps required to provision new storage by 50%.  Since many customers have been complaining about the task of zoning for years, I’m proposing to achieve this reduction by eliminating the manual task of FC zoning. 

Just to be clear, while I view introduction of the Peer Zoning functionality as a game changer for FC, I don’t view the completion of this effort as some kind of extraordinary accomplishment, actually there’s nothing extraordinary about it.  These sorts of efforts are started all the time in standards bodies.  Sometimes they result in useful protocols that are implemented by many (e.g., FCoE and FIP).  Other results are implemented by few but they provide a critical requirement for the industry (e.g., FC-SP) and yet others turn out to be interesting academic exercises but are never implemented by anyone (e.g., FC-SCM).  I have hopes that Peer Zoning will fall into the category of “useful technology that is implemented (and deployed) by many” but it’s too early to tell as this point. 

This post is intended to give you an overview of the technology and give you enough information to decide if TDZ is something that you’d like to use in your environment.  I also feel it’s important to give credit where it’s due and with that in mind without the help of Mark Lippitt, David Black, Claudio DeSanti, Bob Nixon and Ralph Weber; this entire effort would still be hopelessly stalled.      

Background

The story starts back in mid-2007.  I was on a conference call with David Black and Mark Lippitt (both with EMC) discussing what would happen if zoning wasn’t used in an FC SAN (as was being proposed in the T11 FC-SCM working group).  In case you don’t know, EMC has a fairly strict best practice called “Single Initiator Zoning” that was created based on lab testing results.  As the name suggests, the best practice states “zones should only contain a single initiator as well as the storage targets it needs to access.”  Because of the work that I had done with fabric scalability, I was pulled into the meeting (somewhat last minute if memory serves) and asked to provide my opinion on what was being proposed.  Now, as most people who work with me will tell you, I typically call things as I see them and the intensity of my voice will vary with the level of conviction that I feel about a given topic.  As a result, my reaction to eliminating zoning was something along the lines of “ARE YOU <bleep> KIDDING ME?” and I proceeded to provide specific reasons why I thought that this was the *SILLIEST* idea I had ever heard of.  Perhaps it was the intensity of my reaction (or the simple fact that David simply didn’t have sufficient bandwidth to create a presentation that contained all of my points), in either case, he eventually asked me to present my list of concerns to the T11 FC-SCM working group at the August 2007 meeting in Seattle.

Read more on here

Hard zoning versus soft zoning in a FC/FCoE SAN

Post by Erik Smith (thank you) on Zoning in SAN’s

Over the past few weeks, I’ve answered a bunch of questions that have asked something along the lines of “Should I use Hard Zoning or Soft Zoning?”  And no, I have no idea why these questions are coming up now…

The short answer

If you’re familiar with FC Zoning as well as the FC protocol and don’t require a detailed explanation, the short answer is; “You no longer have a choice.  As of today, all of the FC/FCoE switches supported by EMC provide hardware enforcement for both “WWPN” and “Domain, Port” based zones”.  For the sake of completeness, there are some situations where hardware enforcement is effectively disabled but as long as your fan-in and fan-out ratios remain below 1:64 or 64:1 and/or you’re not using NPIV, you won’t need to worry about this.

Read on if you’d like a bit more detail.

The long answer…

As I started writing this section it quickly became apparent that I would need to provide an introduction to zoning and this leads to a discussion about FC Login, discovery, etc.  So I’ll start with the basics first and then get a bit more detailed as I go.  To start with, let’s assume a configuration similar to the following.

Read more on here