DaveWentzel.com            All Things Data

DevOps: Theory of Constraints

Another DevOps post today.  This one is on the Theory of Constraints and how it applies to DevOps.  None of this post is specific to DevOps so even if you don't practice DevOps you might learn something here to apply to your workplace.  The Theory of Constraints applies just as equally to DevOps as it does to improving the performance of a SQL Server.  If you understand the concepts and learn to recognize the symptoms that constraints impose on a system you'll find you are a much better IT professional.  

In most companies there are a very small number of resources (men, material, or machines) that dictate the output of the entire system.  These things are called "the constraints."  The company needs a trusted system to manage the flow of work to the constraints.  The constraints are constantly being wasted which means that the constraint is likely being dramatically underutilized.  

That is an extremely strong paragraph and you should re-read it and think about your workplace.  Odds are your shop has a few go-to people who, if they quit or were run over by a bus, would be next-to-impossible to replace.  These people are always busy solving problems and consulting with others.  These people are the constraints.  These people waste most of their important time dealing with tasks that do nothing to better the business or relieve the constraint.  Think again about those go-to people at your workplace.  Do you see them being assigned menial tasks?  Are they constantly being hounded to get to meetings on time?  To do their timesheet on time?  These people are not working to maximum capacity even though they are the busiest people at your shop.  That is why they are actually under-utilized.  

What to do with constraints

Follow these steps to alleviate constraints.  Use DevOps concepts I've already outlined in other post to help you.  

Step 1:  Identify the Constraint.  Surprisingly most managers and scrum masters do not know where the most egregious constraints are on their team.  They'll point to those people that are complaining the most that they are overworked.  Sometimes it isn't them.  The constraint will be the person(s) where the work piles up.  Kanban boards can help to show you this.  Once identified, protect the constraint.  Do not allow the constraint to be sidetracked by time-wasting activities.  The constraint is likely the bottleneck because it is being mal-utilitized.  Make sure the constraint is never wasted.  

Step 2:  Throttle release of work to the constraint.  Your constraint likely works on numerous projects or wears many hats.  Reduce the number of work centers where the constraint is required.  Note that this flies directly in the face of those DevOps proponents who claim that DevOps is merging your teams together.  That will make things worse!  Is your constraint really needed on all of those projects?  Generate documentation or automate the constraint to relieve the burden.  Or train.  

Even if you hire or get more units of the "constraint" you will never be able to actually increase throughput.  This is what Fred Brooks tried to teach us 40 years ago in the Mythical Man Month.  If it takes one painter an hour to paint a room then it doesn't follow that 20 painters can paint the room in one hour.

Just as important as throttling the release of work is managing the handoffs.  This is waittime that must be eliminated.  Example:  you've fast-tracked a bug fix through development but now you have to wait for a QA resource to free up.  You should be looking ahead to determine what work is coming soon.  

Step 3:  Subordinate the Constraint.  At this point the constraint should be less constraining on your system.  You need to keep it that way.  Remember that forcing more work through the system will not increase throughput, it will merely cause the constraint to start bottlenecking again.  Instead, build buffers or slack into the system to ensure that the unforeseen spikes in work will not cause bottlenecks.  

Do whatever it takes to ensure that maintenance occurs to increase availability of the constraint.  Improving daily work is more important that doing daily work. The whole goal of this step is to make the constraint less constraining.  

Step 4:  Automation/Elimination of the Constraint.  After Step 3 the constraint can still be a constraint under poor circumstances.  We don't want that.  One way to eliminate the constraint is via automation.  If your constraint is certain team members that are overworked/poorly-utilized then you can eliminate by cross-training.  For Ops people the best approach is always automation.  Automate as much as you can and then those constraints should only need to be responsible for maintaining and enhancing the automated system.  

Step 5:  Rinse and Repeat.  By now you've done as much with that given constraint as you can.  It's best now to observe your improvements and spot the next constraint and begin again with Step 1.  

Summary

Constraint Theory works well with DevOps.  Understanding which resource is mal-utilized is the first step in improving your IT organization.  You need to protect any constrained resource to ensure it isn't performing any work that another, unconstrained resource could handle.  After that you need to other resources that can assume some of the work of the constraint.  Lastly, automation is an excellent way to finally eliminate the constraint.  


In most companies there are a very small number of resources (men, material, and machines) that dictates the output of the entire system.  These things are called the constraints.  The company needs a trusted system to manage the flow of work to the constraint.  The constraint is constantly being wasted which means that the constraint is likely being dramatically underutilized.  

Step 1:  identify the constraint.  Once identified, protect it.  Make sure the constraint is never wasted.  

Step 2:  throttle release of work to the constraint.  Reduce the number of work centers where the constraint is required.  Generate documentation or automate the constraint.  Even if you hire or get more units of the "constraint" you will never be able to actually increase throughput.  Just as important as throttling the release of work is managing the handoffs.  This is waittime that must be eliminated.

Step 3:  subordinate the constraint.  

This can be nicely summarized in the concept of Drum-Buffer-Rope (DBR) which is a scheduling solution to the ToC.  The "drum" is the scarse resource in a plant that sets the pace for all other resources (think of a drummer in a marching band that can't keep up with the group).  The "buffer" is added to protect the performance of the drum.  It resolves the peak load and provides some slack for the constraint.  Once the drum is working at optimal capacity we then synchronize and subordinate all other resources and decisions to the activity of the drum.  Essentially, a "rope" is stretched from the drum both backwards and forwards.  We can then estimate when work can actually be started and completed based on the drum availability.  We use the rope to pull the work through the system.  

 

Do whatever it takes to ensure that maintenance occurs to increase availability of the constraint.  Improving daily work is more important than doing daily work.  

 

Work In Progress

Kanban is concerned heavily with limiting WIP.  The theory is that people and processes cannot multi-task as well as they think they can.  You can't really code two modules at the same time.  It's not really multi-tasking, it's context switching.  We want to reduce the amount of WIP in the system, which reduces inventory and improves cycle time.  With DevOps we want to take needless work out of the system too.  We should always be asking the question, "Is this work really needed?"  It might have been needed last week but not this week, so we should eliminate that work.  Outcomes are what matters, not the process, controls, or the work you complete.   We need to protect the company by not putting meaningless work into the system.   

Everyone (or every resource) needs idle time or slack time.  If no one has slack time then WIP gets stuck in the system in queues.  The resources that are constraints can benefit from a "keeper", basically a helper that can move tasks around from work center to work center without wasting the constraint's time.  We need the constraint to be working on the important work, not on tasks that another resource could handle (checking in code, filling out paperwork, attending useless meetings).  

By limiting WIP we find that cycle times are reduced (basically, how long it takes a story to get from inception to done).   This means that we can get the software to customer faster.  This generates an invaluable feedback loop.  This is why one of the premises of agile is to release early and often.  It is also why Lean software development preaches understanding and delivering only the "minimum marketable feature" (MMF).  

 

the best way to get work done is to not do work.  

Unfortunately, operations is considered to
be successful if the metrics are stable and unchanging, whereas development is only applauded if
many things change. Because conflict is baked into this system, intensive collaboration is unlikely.
 
Development teams strive for change, whereas operations teams strive for stability (the
definitions of change and stability will be discussed in Chapter 2). The conflict between devel-
opment and operations is caused by a combination of conflicting motivations, processes, and
tooling. As a result, isolated silos evolv
 
 
 
5 Steps in the Theory of Constraints
1.  identify the system's constraints
2.  Decide how to exploit the constraints
3. Subordinate everything else to that decision
4. Elevate the system's constraints (automation)
5. Find the next constraint and repeat the process.  
 
 
With Agile, testers and programmers
became developers, and with DevOps, developers and operations become DevOps. 
 
How do you move to a DevOps Reference Architecture?
  1. Read up.  There is no DevOps Manifesto like the Agile Manifesto.  There's a lot of disjoint stuff out there.  
  2. DevOps is more culture change than process change.  That makes it different from, say, implementing scrum.  You really can begin to adopt DevOps from the bottom-up if you have to.  Start by having some teams collaborate.  
  3. When you get some small wins using bottom-up then advertise that fact and try to get the top to buy-in and make the cultural changes to the entire organization.  
  4. Remember that DevOps is not the end, rather the means to the end (which is profit).  Too many people adopt DevOps to claim that they are DevOps.  Meanwhile, their software sucks.  
  5. Don't rely too much on metrics.  How do you measure whether your team is collaborating more with DevOps?  You can't.  The metrics must be softer.  Less-sucky software.  Or fewer meetings where Ops complains about the dev folks.  
  6. Not everyone needs to do DevOps.  A few good developers can start collaborating with a few senior ops guys.  The more mature IT professionals will see the value immediately.  The ones who incessantly complain because they feel Ops work is beneath them will only be a distraction.  Prove that DevOps works in your organization before shoving it down everyone's throats.  It may not work for you.  (If that is the case, then, of course, nothing has probably ever worked for you either).  
  7. Focus on your CM team first.  At every large ISV that I've worked for the CM team (change review board or whatever else you want to call it) was the life-sucking entity that everyone hated the most.  The CM process is viewed as a process with negative value in the value stream.  The Review Board really has an important job to do but is relegated to merely being a rubber-stamp organization.  Change that perception and you've solved a lot.  
  8. Don't fall for the NoOps trap unless you are already a NoOps shop.  

You have just read "One Last Git Gotcha:  The Merge Pattern" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

You have just read "Theory of Constraints" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

Tags: 

1 comments

Comment: 
Short, Simple & Sound article... :)