Tieto’s EMC VNX5700 array sparked 5-day disarray – new claim

Post from Chris Mellor (thank you) over at El Reg on the outage at Tieto in Sweden. See the first part here

Tieto’s five-day outage disaster started with multiple failures of its EMC VNX5700 array’s FAST Cache, according to a Finnish source close to the matter.

Tieto is a major IT services organisation across Scandinavia and the Nordic region – although it also provides services globally – and pulls in net sales of SEK17bn (£1.59bn). Its large customer base in Sweden means that when it had a five-day outage in November, it caused chaos to IT services across that country. The stoppage was caused by failures in an EMC storage array and compounded by an inadequate disaster recovery plan involving Networker tape backup files which could not be read. The circumstances are not clear and seemed to involve a VNX array with an upgrade to an NS480 (Celerra) system for flash, which is a logical nonsense.

El Reg has been sent a Tieto slide deck (PDF) describing why the service provider migrated from its Celerra NS4800 to a VNX5700 and the resulting performance improvements: namely lower latency and more IOPS. This deck is in Swedish but Google Translate gets around that little problem.

Based on the translated slide deck text, the story goes like this: in the 2010/2011 period, with a EMC Celerra NS480 array, Tieto saw its storage challenges as performance, response time, scalability and capacity. So it migrated from RAID (4 + 1) groups to Thick Pools composed of 60 disks and began to segment data types into Fibre Channel and NAS. The next step was to install EMC’s FAST Cache with four 200GB SSDs and the cache license, which was beneficial as response times were more than halved to less than 20ms. However the NS480 CPUs were maxed out. Read on here