On Call Welcome to On Call, The Register’s weekly dip into the big bag of woe unleashed by users on those responsible for picking up the telephone.
Today’s remembrance takes us back to 2003, the year of Concorde’s final flight and a time when Reg reader “Chris” was working as a CAD engineer at a semiconductor corporation.
It wasn’t one of the giants, but was instead a small design outfit. As is so often the case, Chris inevitably found himself pressed into the role of supporting the CAD infrastructure and maintaining the IT services. His job also entailed dealing with the very special designers.
He ended up supporting a few desktop Linux boxes, SPARC machinery and some Sun Ultra 60 and 80 gear that lurked in the server room. There was also netApp hardware.
“The company,” he told us, “had a strong culture of shooting [the] messenger and also finger pointing with a peculiar brand of root cause corrective action.”
Chris mused: “The compute servers/netApps were nicely racked in an appropriately air-conditioned room. The actual design of the building predated his arrival and the air-conditioning had the test lab and computer room interlocked on one controller.
“What,” asked Chris, “could go wrong…”
There is, alas, a certain inevitability to what happened next.
“One Sunday afternoon one of our brightest and best arrived in for some extracurricular activity,” explained Chris. The boffin had managed to dampen his personal laptop and had come up with a cracking solution – bring the laptop into work, crank up the heating in the test lab and remove the moisture.
Problem solved! “He dried his laptop out and decided to go home,” said Chris, who was enjoying a well-earned Sunday off until, of course, at 9pm he got The Call.
Another of the company’s brightest and best called to complain that his computer had frozen. A bit of checking showed that it wasn’t user error, so Chris climbed into his car for the 50-minute drive to the office.
It wasn’t a pretty sight.
“The server room,” he told us, “was by now cooking nicely. The air con was still set on heating mode!”
“The SPARC servers and netAPP had gone into thermal shutdown,” he recalled, “but the 3Com racks had cooked and were as dead as a dodo.”
Suspecting that someone would try to blame him for the cockup, Chris set about bringing as much back online as he could. He eventually nursed a hub back to life by changing a board-mounted fuse, but the other was a total loss.
He finally managed to bring up a bare-bones level of service, including all the SPARC hardware. There were, however, still a lot of unhappy users; the PCs had to wait until a replacement switch arrived later in the week.
“As for the ‘air-con assassin’, being a member of the ‘golden circle’, nothing was ever mentioned ever again!” remembered Chris.
“He went on to become a chief cheese in the same corporation.”
For Chris, this wasn’t quite the last straw, but he did move on within a year to another semi and another set of oh-so-special engineers. From the over-heated server room to the fire, as the saying doesn’t go.
Ever cranked up the heat and yelled “burn, baby, burn” in the server room? Or had to deal with the smoking wreckage left over from someone else’s thermostat twiddling? If you answered the call, send an email to On Call and let us know how it went. ®
Harnessing the value of data