DevOps and the Concept of Work

This is another post in my series on DevOps.  In this post I'm going to focus on the finer details of DevOps and how it relates to Kanban.  I've found that as organizations implement DevOps they are also implementing Kanban at the same time.  In the [[DevOps|last post]] I wrote about the common precepts of DevOps and how DevOps gets conflated with other, unrelated ideas.  Kanban has similar issues.  People hear some Kanban buzz phrases and think that they know Kanban.  They don't.  Instead of this being a post on Kanban I'm going to make this a post more about "advanced DevOps" and it will borrow heavily from Kanban.  Don't worry if you've never worked with Kanban, that isn't required to understand the basics for this post.  

What is "Work"?

DevOps is so much more than bringing developers and operations people together.  DevOps is REALLY about understanding work.  If you understand "work" then you'll understand how to make work better.  

Categories of Work

1. Business Projects:  this is what makes us profits.  This should ALWAYS be a company's primary category of work.  If it isn't then...you've got a problem. 
2. Internal Projects:  this makes our operations better.  Automation, code refactoring, etc.  This work category doesn't make us top-line money, but it saves us costs.  Controlling costs is another way to increase profits.  
3. Change:  This type of work occurs whenever we need to alter a system.  Maybe fix a bug, or upgrade an OS, or apply patches.  DevOps always questions EVERY change to ensure it is really needed.  You should be questioning every request for a change too.  In many cases the change really isn't needed or can be delayed and rolled into some other change.  
4. Unplanned Work:  The first 3 categories above are "planned work."  Unplanned work is anything that interrupts your planned work.  But it is more than just unexpected emergencies and outages.  It is sometimes the direct result of technical debt.  

Technical Debt

"Technical debt" is a metaphor for the eventual consequences of poor software or infrastructure within your organization.  It is "debt" because it is really work that needs to be done before a particular project can be considered complete.

Technical debt isn't always bad, but it must be recognized for what it is.  In some cases it is better to recognize some technical debt and decide to fix it later so your code can get out the door and make you money.  But you MUST recognize when unplanned work is the direct result of technical debt and then prioritize it so it stops being unplanned.  

If you don't pay down technical debt then your unplanned work will continue to increase.  Left unchecked, technical debt will ensure that only the work that gets done is unplanned work.  

Why is technical debt and unplanned work so important to DevOps?  Here's an example...often your developers do not know about operational pain that development decisions are causing the ops folks.  If the teams work together the technical debt (and unplanned work) can be paid down.  

And this is why I really love DevOps.  It recognizes that development decisions that do not take operational concerns into account often result in lots of unplanned work.  Likewise, if Ops is making a change to a system that should be run past Dev.  As an example, just because a new version of SQL Server has been released doesn't mean our software needs to be updated to support it.  If there is no new feature that will make us money, then it can be postponed.  Not forever of course, but this is a case where pushing back and denying change is a good thing.  

Two more examples

I worked at a company where a developer decided to introduce Service Broker into his application without any operational input.  EVERY customer upgrade failed because SB was not enabled.  The developer was eventually instructed to put together operational documentation that the DBAs would carry out for every upgrade.  

The document was 3 pages long.  Our upgrades already averaged 19 hours and I determined that this "step" of the upgrade averaged 90 minutes because the DBAs did not understand what each step was supposed to do.  The DBAs didn't understand Service Broker.  

At the same time I was working on a different project in the company where I also needed to use SB.  I created an automated script that ensure SB was enabled and all of the plumbing was in place.  If anything was wrong it would attempt to fix the problem without operational intervention.  It added ZERO time to my project's upgrades.  And it worked flawlessly.  

I was unaware of the first project with its cumbersome SB instructions.  Eventually an Ops person asked me if my process could also be used to automate that process.  We plugged in my process with slight tweaking and solved all of those operational issues.  

In my project I realized that operationalizing something like Service Broker was non-trivial.  You can't just throw some new software onto your stack, you have to understand how that is operationalized.  I knew that the unplanned work around upgrade events and SB would be too much technical debt for us to bear.  So, with some forethought we ensured that operationalizing SB tasks was not de-scoped by a project manager trying to cut corners.  

"The Best Way to Do Work is to Not Do Work"

That is a saying I (think) I created that I use almost daily.  When I hear our people saying that they are doing repetitive tasks then I know that it is time to consider paying down that technical debt.  The best way is through automation.  If a developer can spend a 80-hour sprint automating some task that takes ops people 80-hours per quarter in effort, then it is a no-brainer that we need to pay down that technical debt.  The goal is not to do that work AT ALL.  

In the next post I'll cover some additional DevOps concepts you likely won't hear from many other DevOps authorities.

You have just read "[[DevOps and the Concept of Work]]" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.