DaveWentzel.com            All Things Data

August 2013

The Paradoxes and Ironies of Automation

I have some side work where I help to automate some tedious, time consuming operational DBA tasks.  If I strung these processes out serially it would take about 18 hours to do everything.  Management's goal is to get this to a totally lights-out process, which serially would take about 6 hours (there are certain tasks that can't be sped up with automation, such as copying TB-sized files across a WAN).  Total automation is a noble, albeit unattainable goal.  Management does not totally agree with me.  Total automation is rarely a good idea.  I'm totally for the concept of as much automation as possible, but there are serious side effects to automating EVERYTHING, or even attempting it.
Costs of Automation

Full automation does not make sense from a business perspective.  At the beginning of an automation project, the cost of automating the most egregious things is very cheap and the benefit is very high.  When you near the end of an automation project the cost is very high and the benefit is very low.  This is something akin to the 80/20 rule. The last 20% of the automation tasks will take 80% of your budget.  
If the software you are trying to automate needs frequent maintenance and enhancements then it's very likely that your automation work will need the same, or more, maintenance.  This costs money.  So how much do you automate?  As much as you can, and can afford, and nothing else.
Paradox of Automation
The Paradox of Automation is defined as:  
" the more efficient the automated system is, the more essential the human contribution that is needed to run the Automation System.  Humans are less involved in heavily automated systems, but their involvement becomes more critical."
If an automated system experiences an ABEND, the system is often completely unavailable until the error can be identified and fixed.  Finding the error in an automated system is a much more labor-intensive, specialized operation.  Entry level programmers probably won't be able to find the abend.  It will require senior people to find the errors, and senior people will need to know more domain knowledge.  Let's not even mentionethe fact that many automated systems run but fail to properly trap errors.  In these cases we need to look for the conditions that are not right and try to determine why the automated system is failing.  This is really, really difficult to do.  Just because the system is running at this moment doesn't mean it's running correctly either.

During an automation project you will often find that the manual operators are hostile to what you are doing.  "You are automating me out of a job."  That's rarely the case (unfortunately).  If the automated system experiences an error it will be multiplied until fixed or shut down.  This makes the operator even more important and valuable.  Automation requires more specialized skills.  The operator also needs to be able to spot "the canary in the coalmine" and be proactive to avoid errors.  
Irony of Automation
There is a certain "irony" to a successful automation project.  The more reliable the automation project is, the less opportunity there will be for the operator to practice direct intervention, and the more difficult will be the demands of the remaining tasks requiring operator intervention.
I am a firm believer in the DevOps movement.  A core component of DevOps is automation.  However, you can't just discard your Human Monitors.  Otherwise the result is deskilled and novice operators who cannot fix a failed system.  Any automated system failure is used as evidence of human failure (whether the human operator or programmer) resulting in more effort being expended to automate the system even further!  
More fail-safes and redundancies are built into the system.  The increased automation now hides even more of the small errors that would've occurred and could've been solved easily with "canary in the coalmine" monitoring.  Now latent errors are being hidden until the failure becomes even more catastrophic and difficult to resolve. 
The "Three MIle Island" incident in the 1970's is often used as an example of the Irony of Automation.  So much was automated that the reactor was at a critical mass until alerting began.  At that point there was no operator experienced enough to know exactly what to do.  It is said that an inexperienced engineer began turning knobs and made the problem even worse.  It has been speculated that if the operator just walked away and got some coffee that the system would've resolved itself.  In this case the problem wasn't over-automation, it was a lack of qualified people who understood the complex system.  

Making Ubuntu Faster on Junk Hardware

sudo apt-get install preloadsudo apt-get install preloadIf you 
If you are like me you probably have a lot of old, junk hardware and laptops lying around your office and home.  

Sanitize those stored procedure parameters and don't split your pants!

The Level 6 Architect needed to pass some XML into a stored procedure that would be the payload for a Service Broker message.  

"Piece of cake," I said, "Here you go!" and passed him the following procedure...edited for brevity and to protect the innocent.  

"Oh no no no no no.  That will never do.  We have a strict requirement that we cannot pass XML parameters into a procedure, we must pass in a varchar because the XML schema will change dynamically with the...".  My eyes and ears started to glaze over as I heard a litany of ridiculous reasons why an XML datatype could not be used.  

"Let me get this straight...does our JDBC driver support SQL Server's XML datatype?"


"Then we should be passing in the XML data to a strongly typed parameter, a la one declared as XML."  

"No!  We cannot do that.  Use a varchar(500) instead.  Just do as you are told."  

"No problem-o Mr. Level 6.  Here you go.  But I suggest you sanitize your data in the Java tier to be sure you will never overflow a varchar(500).  Are you sure that is big enough?  XML tends to be obese." I was trying not to lose my cool.  

"Just do as you are told."  

"OK, just remember that when this blows up in your face...I told you so."  








Fast forward 12 months when this code is in production for mission critical, life-or-death data situations...

Level6:  "Your gd stored proc is throwing errors everywhere."
Me:  "Can you show me the error please?"

Msg 9400, Level 16, State 1, Procedure FooBar, Line 8
XML parsing: line 1, character 500, unexpected end of input
Me: "That's interesting that it fails exactly at character 500 every time, don't you think?  Do you remember when I told you this would blow up in your face if you didn't sanitize your inputs?"
Level6:  "Yeah, so what?"
Me:  "So, you ignored me and you are passing in more than 500 characters and the XML parser in SQL Server cannot parse the string as well-formed XML.  I told you to sanitize your parameters before calling the procedure.  You chose to ignore me.  This is your problem."
Level6:  "Prove it. SQL Server has to report an error when we overflow the datatype."
Me:  "Here you go."
Level6:  "There is no way we are possibly passing in more than 500 characters of data.  Impossible.  500 characters should be at least four times more than we could possibly ever send."
Me:  "Well, apparently you are wrong.  I bought my pants two sizes too big last year and now my butt doesn't fit in 'em.  I figured two sizes bigger should be more than enough.  It wasn't.  The difference is, my pants don't silently truncate the extra girth like SQL Server does.  My pant seams scream loudly, "hold together men!"  Maybe what you need to do is make sure your Java code doesn't overflow the datatype just like I need to make sure my butt doesn't overflow the pants."
Level6:  "Well, SQL Server should not silently truncate input data.  That's just crazy."
Me:  "It's only been doing that since at least 1990.  Go fix your code.  And remember, a good architect uses the most specific datatype possible and always sanitizes the input."