DevOps: WIP and final thoughts

Another DevOps post today...this one discusses WIP.  WIP is an abbreviation for "Work In Progress".  Limiting the amount of WIP you have (in other words, the amount of work you have started but is not yet completed) is an excellent way to increase throughput in your software development pipeline.  

Work In Progress

The theory is that people and processes cannot multi-task as well as they think they can.  You can't really code two modules at the same time, or test two modules at the same time, or work on two sets of documentation at the same time.  When you work on two tasks at the same time you aren't really multi-tasking, you are context switching.  When you reduce the amount of WIP in the system you will reduce "inventory" (think of it as the amount of unsold work that is sitting in the warehouse half-assembled).  By reducing inventory you are limiting the number of bottlenecks that can occur, meaning your cycle time (the amount of wall clock time needed to move a piece of work from "started" to "done") should improve.  

DevOps improves upon Kanban's notion of WIP by also removing needless work out of the system.  In DevOps you should always ask the question, "Is this work really needed?"  It might have been needed last week but not this week, so we should eliminate that work.  Outcomes are what matters, not the process, controls, or the work you complete.   We need to protect the company by not putting meaningless work into the system.   To use the manufacturing analogy again, if you have partially completed work as inventory then you'll find your inventory depreciates as it gets older and is no longer desirable by your customers.  In this case the money you spent partially assembling that inventory is wasted.  This is sometimes considered "just-in-time manufacturing" and the equivalent in our industry is "just-in-time" software engineering.  

Here is a classic example of work that was needed but isn't anymore.  A new security patch is released for your DBMS version.  You add it to the backlog but a few months pass before you have bandwidth to work on regression testing the patch.  In the interim a new DBMS version is released that you want to upgrade to and it does not require the patch.  If you eliminate the work required to test the patch and simply apply that time to the DBMS version upgrade, then you've eliminated work.  It's even worse if you started regression testing the patch but didn't have time to fully complete it.  That time would've been wasted.  

DevOps is particularly concerned with this because we know we can eliminate a lot of operational work if we spend some time automating upfront by assigning dev resources to work with ops guys.  In this case we've eliminated work that isn't needed from the system.  

"Stop starting and start finishing"

...is an aphorism that sums up wip limits quite well.  Obviously there are times when you may have a lot of WIP due to factors outside of your control.  For instance, you begin work on a task and determine that the requirements are not clear enough to continue.  You need to get a BA to refine the reqs.  In this case hard-core Kanban-ers would say that you cannot begin work on another task because if you do you'll never get a BA to refine the requirements properly.  Instead, you should simply wait and force the "steel thread" to the BA resource to get a better requirement.  In some cases that may work but the problem is that this doesn't take into consideration that the BA resource may have other, more pressing issues.  Basically, your team's emergency may not be another team's emergency.  I handle these situations by moving the "card" to the BA resource so it affects that resource's WIP limits and then have my blocked resource work on something else.  I don't like to make WIP limit adherence a religious dogma.  Rather, it is a guide to get us the best throughput.  Something as simple as denoting the steel threads with a special colored card is enough to get management to prioritize better.  

Most importantly WIP limits simply ensure that we don't have a bunch of tasks forever sitting at 80% complete.  It's very easy for a developer that needs requirement classification to simply sit the task aside and move on to something else.  WIP limits ensure there is accountability for task completion.  In the real world, in my experience, WIP limits break down when you have "steel threads" from one team to another.  This is by far the Number One Reason for WIP in large software engineering organizations.  As I said before, just because Team A is blocked by Team B doesn't mean the blocked task is a priority for Team B.  In organizations that struggle with steel thread management, such as large monolithic ISVs, WIP limits will probably not work because there are cultural and political reasons why teams don't collaborate.  Only when the culture is forced to change will WIP limits work.  

The Need for Idle Time 

Everyone (or every resource) needs idle time or slack time.  It should be built in to the sprint schedule.  If no one has slack time then WIP gets stuck in the system in queues.  We've introduced unneeded bottlenecks and constraints to the system.  Therefore it is best to always allow for slack time in your development processes.  

There are other ways we can simultaneously reduce work and increase slack time.  Every process has constraints that we can identify and wish to protect.  I discussed this in a previous DevOps post.   These resources, once identified, can benefit from a "keeper".  A keeper is basically a helper that can move tasks around from work center to work center without wasting the constraint's time.  We need the constraint to be working on the important work, not on tasks that another resource could handle (checking-in/merging code, filling out paperwork, attending useless meetings).  The keeper can be an intern or a junior member of your team.  

Minimum Marketable Feature

Limiting WIP is one of the most important tenants of DevOps.  By limiting WIP we find that cycle times are reduced and throughput is improved.   This means that we can get the software to the customer faster.  This generates an invaluable feedback loop.  This is why one of the premises of agile is to release early and often.  It is also why Lean software development preaches understanding and delivering only the "minimum marketable feature" (MMF).  This is also sometimes called the Minimum Viable Product.  We only want to do enough to get sufficient feedback and make profits.  Anything else is just polishing the apple.  

"The Best Way to Get Work Done is to Not Do Work"

I'm certain I didn't invent this saying but I use it daily.  If you work in an ISV shop like I do then you are probably innundated every day with sales guys and BAs that want more and better functionality.  Many times the requirements are half-baked, at best.  Before we start doing work on these items I like to give these tasks some soak time.  I find that after some amount of time that there is no real demand for these tasks and they die before anyone has wasted any time on them.  Or I find that work that was deemed necessary a month ago is no longer necessary (see my patching example above).  In these cases, don't do the work.  I know this sounds a little waterfall-ish (we only start work when it is fully documented) but this is why I never adhere to development practices as if they were dogma.  Every organization is different and every situation is unique.  But if you suffer from half-baked requirements that never get to a customer then consider a more waterfall-ish approach.  You are wasting time creating inventory that is never sold and WIP that never gets to done.  

How do you move to a DevOps Reference Architecture?

Here are my final thoughts on DevOps.  Obviously I think this is a software engineering method that shows some promise and isn't loaded with a bunch of useless platitudes and metrics gathering that offer no hope of getting better software to customers.  This is how I've approached selling and implementing a DevOps approach at organizations I've worked with.  

  1. Read up.  There is no DevOps Manifesto like the Agile Manifesto.  There's a lot of disjoint stuff out there.  My theories on DevOps tend to radically deviate from those who think DevOps is simply getting your developers to manage your operations.  
  2. DevOps is more culture change than process change.  That makes it different from, say, implementing scrum.  You really can begin to adopt DevOps from the bottom-up if you have to.  Start by having some Dev and Ops teams collaborate.  
  3. When you get some small wins using bottom-up then advertise that fact and try to get the top to buy-in and make the cultural changes to the entire organization.  The best way to get small wins is by having the dev guys help to automate some of the ops tasks that bottleneck the release process.  Management loves to see that stuff.  
  4. Remember that DevOps is not the end, rather the means to the end (which is profit).  Too many people adopt DevOps to claim that they are DevOps.  Meanwhile, their software sucks.  This is the same as people who say "We are an agile shop and don't do comprehensive documentation".  That's just agile for the sake of agile.  
  5. Don't rely too much on metrics.  How do you measure whether your team is collaborating more with DevOps?  You can't.  The metrics must be softer.  Less-sucky software is a good metric, but difficult to measure.  Or fewer meetings where Ops complains about the dev folks.  
  6. Not everyone in your organization needs to do DevOps.  A few good developers can start collaborating with a few senior ops guys.  The more mature IT professionals will see the value immediately.  The ones who incessantly complain because they feel Ops work is beneath them will only be a distraction.  Prove that DevOps works in your organization before shoving it down everyone's throats.  It may not work for you.  (If that is the case, then, of course, nothing has probably ever worked for you either).  
  7. Focus on your CM team first.  At every large ISV that I've worked for the CM team (change review board or whatever else you want to call it) was the life-sucking entity that everyone hated the most.  It was the bottleneck.  The CM process is viewed as a process with negative value in the value stream by the guys in the trenches.  And management hates the CM team too because too many bugs are still getting through.  The Review Board really has an important job to do but is relegated to merely being a rubber-stamp organization.  Change that perception and you've solved a lot.  
  8. Don't fall for the NoOps trap unless you are already a NoOps shop.  

Good luck with your journey to DevOps.  I hope this blog series helps you in your quest for greater profits and better software.  

 

You have just read "DevOps:  WIP and Final Thoughts" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.