Changing the mindset

I started in my new role with a preconception of what my job was to be.

I was hired as Systems Operations Manager for a Telecoms company, and as many SysAdmins think, assumed it was my responsibility to keep the systems running, whatever the weather. – Patch systems when vulnerabilities are found, delete files when your running out of space, spec the network for armageddon, and then backup-verify-test-prove that the DR plans stand up.

But despite all the preparations, the fail-safes and the redundancy you build in, something will always goes wrong, and the SysAdmin has to fix it.

Or do you?

What if…

You didn’t care when things go wrong?

I’m not suggesting a blase “meh..” but what if your infrastructure didn’t matter any more?

HB or not 2B…
When I was at school, I had a pencil. I used it all day every day.  I had a pencil case to put it in, so that I could bring it home along with the sort of things many of us have not used since leaving school like a set of compasses, protractors, and those funny wedge rubbers you put on the end of your pencil (which always seemed to smudge more of a mess than they erased), and of course a pencil sharpener.

When your pencil snapped, or had simply worn to such a dull point it wouldn’t write anymore you had to hunch over the waste bin, and sharpen it back into a precise point. The best sharpener was always one of the metal ones with the spare blade screwed to the side (see!…redundant components, even at primary school) But in striving to get that perfect point invariably you’d overcook it from time to time, and snap it before taking it out of the sharpener. But if you were careful, and took your time, a pencil would last you a lifetime (or at least most of the term)

Not to tug on your heartstrings too much (and for the sake of my story) this was how I used to operate throughout school and university, but once my student days were over, flush with the trappings of employment, I was able to afford replacement pencils. Now when one breaks, I throw it away (and I can never find a pencil sharpener anymore)

Plan to Fail…
When i started looking at cloud services, I found dozens of articles which warn that cloud instances fail and often.  A phrase that kept cropping up was “Plan to fail”

What the authors were getting at, is that you should have a plan in place to cope with an instance going wrong. Use load-balancers and multiple instances so that if one went wrong you have time to fix it, and get it back into service.

Turn it all off, and back on again…
My though process is slightly different. If you could start each day with a new pencil its unlikely you’d wear it out before the end of the day, and if it breaks just grab a new one out of the box.

If you could do this with your infrastructure you don’t ever need to worry about it going wrong again. If it does ‘go bang’, just drop a new one in and carry on. In fact, why not nuke the whole lot at 23:59 each day and start fresh at midnight with brand new servers.

Once you come round to this way of thinking, you’ll wonder why you ever need to try and fix a server problem again….

How hard could it be?