DaveWentzel.com            All Things Data

May 2014

DevOps in the real world: Who should be responsible for index maintenance?

Larger ISVs usually employ at least two types of data professionals, the data developer and the database administrator.  In some situations, especially if DevOps is practiced, the lines may be a bit blurred regarding responsibilities.  I find it interesting in my consulting engagements which data professional is responsible for index maintenance.  Who is responsible for indexes at your company?  Who decides what indexes should exist?  Who creates them?  Who ensures proper index hygiene occurs (defragmentation, manual update stats)?  Which group should be responsible for these tasks?  

Informally I've found at my engagements that when both groups exist that it is the DBAs that are responsible for all aspects of indexing.  I think this is just flat-out wrong.  And I'm probably in the minority on this though.  Here's my reasoning:  

  • DBAs probably don't know all of the data access paths, and the most important ones we wish to optimize in the application.  Unfortunately your developers may not either, but that's a different issue.  
  • If you practice some variant of "database change review", where any and all schema changes to a database go through a review system to ensure they follow standards are are modeled sensibly, then you should have a good feel for what indexes should exist prior to those tables hitting your production systems.  
  • If you are doing performance testing then your developers should further understand the data access patterns.  Performance testing won't find every issue but the most egregious issues should be caught.  
  • Too many shops allow production DBAs to alter indexing schemes without consulting the data developers at all.  In EVERY case I've seen disaster because of this.  Here's a few examples:
    • we implemented queue tables as heaps and the DBAs didn't like that so they added clustered indexes and our queues experienced odd locking and concurrency behavior.  Contrary to google, sometimes heaps are good.  
    • DBAs applied indexes based on recommendations in the missing index DMVs and the plan caches.  The net result was cases where the same column in a table was the leading index column for 5 or more indexes, differing only by INCLUDED columns.  These DMVs must be countered with common sense and knowledge of your data.  
    • DBAs removed indexes because the cardinality at the time when they checked the index was LOW.  It's common for some tables to have a ProcessedStatusCode column that is binary (processed/not processed).  Throughout most of the day every row is in the processed state (1) until overnight batch processing starts.  Indexes on a ProcessedStatusCode column are critical, albeit counter-intuitive.  Many people believe that indexes on low cardinality tables are useless, but in situations like this they are required.  
    • DBAs removed indexes because sys.dm_db_index_usage_stats showed that data access was heavily skewed to updates and very little to seeks/scans/lookups.  The problem is, the indexes were needed during month-end processing and were suddenly gone when they were required.  

Contrary to what you are probably thinking, this blog post is not an "us vs them", anti-DBA rant.  "But you said above that it is flat-out wrong that DBAs are responsible for indexing."  


The resolution to all of the issues above is simple.  Collaboration.  Take the last example where important indexes were missing during month-end processing.  Clearly the DBAs had some valid reasons to remove the indexes and the data devs had valid reasons for needing them.  After our month-end processing went from 12 hours to 7 days due to the missing indexes, and after everyone stopped playing the blame game and hootin and hollerin, we collaborated and determined that we could actually build those indexes a day or so before month-end processing and then remove them later.  These indexes are huge and the chargeback for the SAN storage was not insignificant.  

This was a non-zero-sum event.  We managed to save on storage, our system was faster during critical month-end processing (those indexes were less fragmented), and the system was marginally faster during standard daily processing where the update overhead to those indexes could be eliminated. So, here's a case where a bit of collaboration led to one of those truly rare "win-win" situations.  And that's what we all want during conflict resolution.  We don't want to hurt anyone's feelings.  

(I found this graphic on an EXTREMELY popular DevOps site.  Maybe I'm being picky but I still see a separation of Dev and Ops.  Where is the collaboration?  I don't understand how this graphic couldn't be titled "Waterfall")

This is just another example where DevOps works.  My last few blog posts I pointed out that there is a misconception that DevOps means combining developers and ops guys.  It isn't.  It is about about situations like this.  Nobody is perfect.  I've worked in just as many shops where EVERY column in a table had an index because the developer figured that if a few indexes are good then more must be better.  Indexing is one of those cases where it really is difficult to determine which group should have ultimate responsibility, the developers or the DBAs.  Simple collaboration and following DevOps is the key.  

You have just read another DevOps post on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

DevOps: WIP and final thoughts

Another DevOps post today...this one discusses WIP.  WIP is an abbreviation for "Work In Progress".  Limiting the amount of WIP you have (in other words, the amount of work you have started but is not yet completed) is an excellent way to increase throughput in your software development pipeline.  

Work In Progress

The theory is that people and processes cannot multi-task as well as they think they can.  You can't really code two modules at the same time, or test two modules at the same time, or work on two sets of documentation at the same time.  When you work on two tasks at the same time you aren't really multi-tasking, you are context switching.  When you reduce the amount of WIP in the system you will reduce "inventory" (think of it as the amount of unsold work that is sitting in the warehouse half-assembled).  By reducing inventory you are limiting the number of bottlenecks that can occur, meaning your cycle time (the amount of wall clock time needed to move a piece of work from "started" to "done") should improve.  

DevOps improves upon Kanban's notion of WIP by also removing needless work out of the system.  In DevOps you should always ask the question, "Is this work really needed?"  It might have been needed last week but not this week, so we should eliminate that work.  Outcomes are what matters, not the process, controls, or the work you complete.   We need to protect the company by not putting meaningless work into the system.   To use the manufacturing analogy again, if you have partially completed work as inventory then you'll find your inventory depreciates as it gets older and is no longer desirable by your customers.  In this case the money you spent partially assembling that inventory is wasted.  This is sometimes considered "just-in-time manufacturing" and the equivalent in our industry is "just-in-time" software engineering.  

Here is a classic example of work that was needed but isn't anymore.  A new security patch is released for your DBMS version.  You add it to the backlog but a few months pass before you have bandwidth to work on regression testing the patch.  In the interim a new DBMS version is released that you want to upgrade to and it does not require the patch.  If you eliminate the work required to test the patch and simply apply that time to the DBMS version upgrade, then you've eliminated work.  It's even worse if you started regression testing the patch but didn't have time to fully complete it.  That time would've been wasted.  

DevOps is particularly concerned with this because we know we can eliminate a lot of operational work if we spend some time automating upfront by assigning dev resources to work with ops guys.  In this case we've eliminated work that isn't needed from the system.  

"Stop starting and start finishing"

...is an aphorism that sums up wip limits quite well.  Obviously there are times when you may have a lot of WIP due to factors outside of your control.  For instance, you begin work on a task and determine that the requirements are not clear enough to continue.  You need to get a BA to refine the reqs.  In this case hard-core Kanban-ers would say that you cannot begin work on another task because if you do you'll never get a BA to refine the requirements properly.  Instead, you should simply wait and force the "steel thread" to the BA resource to get a better requirement.  In some cases that may work but the problem is that this doesn't take into consideration that the BA resource may have other, more pressing issues.  Basically, your team's emergency may not be another team's emergency.  I handle these situations by moving the "card" to the BA resource so it affects that resource's WIP limits and then have my blocked resource work on something else.  I don't like to make WIP limit adherence a religious dogma.  Rather, it is a guide to get us the best throughput.  Something as simple as denoting the steel threads with a special colored card is enough to get management to prioritize better.  

Most importantly WIP limits simply ensure that we don't have a bunch of tasks forever sitting at 80% complete.  It's very easy for a developer that needs requirement classification to simply sit the task aside and move on to something else.  WIP limits ensure there is accountability for task completion.  In the real world, in my experience, WIP limits break down when you have "steel threads" from one team to another.  This is by far the Number One Reason for WIP in large software engineering organizations.  As I said before, just because Team A is blocked by Team B doesn't mean the blocked task is a priority for Team B.  In organizations that struggle with steel thread management, such as large monolithic ISVs, WIP limits will probably not work because there are cultural and political reasons why teams don't collaborate.  Only when the culture is forced to change will WIP limits work.  

The Need for Idle Time 

Everyone (or every resource) needs idle time or slack time.  It should be built in to the sprint schedule.  If no one has slack time then WIP gets stuck in the system in queues.  We've introduced unneeded bottlenecks and constraints to the system.  Therefore it is best to always allow for slack time in your development processes.  

There are other ways we can simultaneously reduce work and increase slack time.  Every process has constraints that we can identify and wish to protect.  I discussed this in a previous DevOps post.   These resources, once identified, can benefit from a "keeper".  A keeper is basically a helper that can move tasks around from work center to work center without wasting the constraint's time.  We need the constraint to be working on the important work, not on tasks that another resource could handle (checking-in/merging code, filling out paperwork, attending useless meetings).  The keeper can be an intern or a junior member of your team.  

Minimum Marketable Feature

Limiting WIP is one of the most important tenants of DevOps.  By limiting WIP we find that cycle times are reduced and throughput is improved.   This means that we can get the software to the customer faster.  This generates an invaluable feedback loop.  This is why one of the premises of agile is to release early and often.  It is also why Lean software development preaches understanding and delivering only the "minimum marketable feature" (MMF).  This is also sometimes called the Minimum Viable Product.  We only want to do enough to get sufficient feedback and make profits.  Anything else is just polishing the apple.  

"The Best Way to Get Work Done is to Not Do Work"

I'm certain I didn't invent this saying but I use it daily.  If you work in an ISV shop like I do then you are probably innundated every day with sales guys and BAs that want more and better functionality.  Many times the requirements are half-baked, at best.  Before we start doing work on these items I like to give these tasks some soak time.  I find that after some amount of time that there is no real demand for these tasks and they die before anyone has wasted any time on them.  Or I find that work that was deemed necessary a month ago is no longer necessary (see my patching example above).  In these cases, don't do the work.  I know this sounds a little waterfall-ish (we only start work when it is fully documented) but this is why I never adhere to development practices as if they were dogma.  Every organization is different and every situation is unique.  But if you suffer from half-baked requirements that never get to a customer then consider a more waterfall-ish approach.  You are wasting time creating inventory that is never sold and WIP that never gets to done.  

How do you move to a DevOps Reference Architecture?

Here are my final thoughts on DevOps.  Obviously I think this is a software engineering method that shows some promise and isn't loaded with a bunch of useless platitudes and metrics gathering that offer no hope of getting better software to customers.  This is how I've approached selling and implementing a DevOps approach at organizations I've worked with.  

  1. Read up.  There is no DevOps Manifesto like the Agile Manifesto.  There's a lot of disjoint stuff out there.  My theories on DevOps tend to radically deviate from those who think DevOps is simply getting your developers to manage your operations.  
  2. DevOps is more culture change than process change.  That makes it different from, say, implementing scrum.  You really can begin to adopt DevOps from the bottom-up if you have to.  Start by having some Dev and Ops teams collaborate.  
  3. When you get some small wins using bottom-up then advertise that fact and try to get the top to buy-in and make the cultural changes to the entire organization.  The best way to get small wins is by having the dev guys help to automate some of the ops tasks that bottleneck the release process.  Management loves to see that stuff.  
  4. Remember that DevOps is not the end, rather the means to the end (which is profit).  Too many people adopt DevOps to claim that they are DevOps.  Meanwhile, their software sucks.  This is the same as people who say "We are an agile shop and don't do comprehensive documentation".  That's just agile for the sake of agile.  
  5. Don't rely too much on metrics.  How do you measure whether your team is collaborating more with DevOps?  You can't.  The metrics must be softer.  Less-sucky software is a good metric, but difficult to measure.  Or fewer meetings where Ops complains about the dev folks.  
  6. Not everyone in your organization needs to do DevOps.  A few good developers can start collaborating with a few senior ops guys.  The more mature IT professionals will see the value immediately.  The ones who incessantly complain because they feel Ops work is beneath them will only be a distraction.  Prove that DevOps works in your organization before shoving it down everyone's throats.  It may not work for you.  (If that is the case, then, of course, nothing has probably ever worked for you either).  
  7. Focus on your CM team first.  At every large ISV that I've worked for the CM team (change review board or whatever else you want to call it) was the life-sucking entity that everyone hated the most.  It was the bottleneck.  The CM process is viewed as a process with negative value in the value stream by the guys in the trenches.  And management hates the CM team too because too many bugs are still getting through.  The Review Board really has an important job to do but is relegated to merely being a rubber-stamp organization.  Change that perception and you've solved a lot.  
  8. Don't fall for the NoOps trap unless you are already a NoOps shop.  

Good luck with your journey to DevOps.  I hope this blog series helps you in your quest for greater profits and better software.  


You have just read "DevOps:  WIP and Final Thoughts" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  


DevOps: Theory of Constraints

Another DevOps post today.  This one is on the Theory of Constraints and how it applies to DevOps.  None of this post is specific to DevOps so even if you don't practice DevOps you might learn something here to apply to your workplace.  The Theory of Constraints applies just as equally to DevOps as it does to improving the performance of a SQL Server.  If you understand the concepts and learn to recognize the symptoms that constraints impose on a system you'll find you are a much better IT professional.  

In most companies there are a very small number of resources (men, material, or machines) that dictate the output of the entire system.  These things are called "the constraints."  The company needs a trusted system to manage the flow of work to the constraints.  The constraints are constantly being wasted which means that the constraint is likely being dramatically underutilized.  

That is an extremely strong paragraph and you should re-read it and think about your workplace.  Odds are your shop has a few go-to people who, if they quit or were run over by a bus, would be next-to-impossible to replace.  These people are always busy solving problems and consulting with others.  These people are the constraints.  These people waste most of their important time dealing with tasks that do nothing to better the business or relieve the constraint.  Think again about those go-to people at your workplace.  Do you see them being assigned menial tasks?  Are they constantly being hounded to get to meetings on time?  To do their timesheet on time?  These people are not working to maximum capacity even though they are the busiest people at your shop.  That is why they are actually under-utilized.  

What to do with constraints

Follow these steps to alleviate constraints.  Use DevOps concepts I've already outlined in other post to help you.  

Step 1:  Identify the Constraint.  Surprisingly most managers and scrum masters do not know where the most egregious constraints are on their team.  They'll point to those people that are complaining the most that they are overworked.  Sometimes it isn't them.  The constraint will be the person(s) where the work piles up.  Kanban boards can help to show you this.  Once identified, protect the constraint.  Do not allow the constraint to be sidetracked by time-wasting activities.  The constraint is likely the bottleneck because it is being mal-utilitized.  Make sure the constraint is never wasted.  

Step 2:  Throttle release of work to the constraint.  Your constraint likely works on numerous projects or wears many hats.  Reduce the number of work centers where the constraint is required.  Note that this flies directly in the face of those DevOps proponents who claim that DevOps is merging your teams together.  That will make things worse!  Is your constraint really needed on all of those projects?  Generate documentation or automate the constraint to relieve the burden.  Or train.  

Even if you hire or get more units of the "constraint" you will never be able to actually increase throughput.  This is what Fred Brooks tried to teach us 40 years ago in the Mythical Man Month.  If it takes one painter an hour to paint a room then it doesn't follow that 20 painters can paint the room in one hour.

Just as important as throttling the release of work is managing the handoffs.  This is waittime that must be eliminated.  Example:  you've fast-tracked a bug fix through development but now you have to wait for a QA resource to free up.  You should be looking ahead to determine what work is coming soon.  

Step 3:  Subordinate the Constraint.  At this point the constraint should be less constraining on your system.  You need to keep it that way.  Remember that forcing more work through the system will not increase throughput, it will merely cause the constraint to start bottlenecking again.  Instead, build buffers or slack into the system to ensure that the unforeseen spikes in work will not cause bottlenecks.  

Do whatever it takes to ensure that maintenance occurs to increase availability of the constraint.  Improving daily work is more important that doing daily work. The whole goal of this step is to make the constraint less constraining.  

Step 4:  Automation/Elimination of the Constraint.  After Step 3 the constraint can still be a constraint under poor circumstances.  We don't want that.  One way to eliminate the constraint is via automation.  If your constraint is certain team members that are overworked/poorly-utilized then you can eliminate by cross-training.  For Ops people the best approach is always automation.  Automate as much as you can and then those constraints should only need to be responsible for maintaining and enhancing the automated system.  

Step 5:  Rinse and Repeat.  By now you've done as much with that given constraint as you can.  It's best now to observe your improvements and spot the next constraint and begin again with Step 1.  


Constraint Theory works well with DevOps.  Understanding which resource is mal-utilized is the first step in improving your IT organization.  You need to protect any constrained resource to ensure it isn't performing any work that another, unconstrained resource could handle.  After that you need to other resources that can assume some of the work of the constraint.  Lastly, automation is an excellent way to finally eliminate the constraint.  

In most companies there are a very small number of resources (men, material, and machines) that dictates the output of the entire system.  These things are called the constraints.  The company needs a trusted system to manage the flow of work to the constraint.  The constraint is constantly being wasted which means that the constraint is likely being dramatically underutilized.  

Step 1:  identify the constraint.  Once identified, protect it.  Make sure the constraint is never wasted.  

Step 2:  throttle release of work to the constraint.  Reduce the number of work centers where the constraint is required.  Generate documentation or automate the constraint.  Even if you hire or get more units of the "constraint" you will never be able to actually increase throughput.  Just as important as throttling the release of work is managing the handoffs.  This is waittime that must be eliminated.

Step 3:  subordinate the constraint.  

This can be nicely summarized in the concept of Drum-Buffer-Rope (DBR) which is a scheduling solution to the ToC.  The "drum" is the scarse resource in a plant that sets the pace for all other resources (think of a drummer in a marching band that can't keep up with the group).  The "buffer" is added to protect the performance of the drum.  It resolves the peak load and provides some slack for the constraint.  Once the drum is working at optimal capacity we then synchronize and subordinate all other resources and decisions to the activity of the drum.  Essentially, a "rope" is stretched from the drum both backwards and forwards.  We can then estimate when work can actually be started and completed based on the drum availability.  We use the rope to pull the work through the system.  


Do whatever it takes to ensure that maintenance occurs to increase availability of the constraint.  Improving daily work is more important than doing daily work.  


Work In Progress

Kanban is concerned heavily with limiting WIP.  The theory is that people and processes cannot multi-task as well as they think they can.  You can't really code two modules at the same time.  It's not really multi-tasking, it's context switching.  We want to reduce the amount of WIP in the system, which reduces inventory and improves cycle time.  With DevOps we want to take needless work out of the system too.  We should always be asking the question, "Is this work really needed?"  It might have been needed last week but not this week, so we should eliminate that work.  Outcomes are what matters, not the process, controls, or the work you complete.   We need to protect the company by not putting meaningless work into the system.   

Everyone (or every resource) needs idle time or slack time.  If no one has slack time then WIP gets stuck in the system in queues.  The resources that are constraints can benefit from a "keeper", basically a helper that can move tasks around from work center to work center without wasting the constraint's time.  We need the constraint to be working on the important work, not on tasks that another resource could handle (checking in code, filling out paperwork, attending useless meetings).  

By limiting WIP we find that cycle times are reduced (basically, how long it takes a story to get from inception to done).   This means that we can get the software to customer faster.  This generates an invaluable feedback loop.  This is why one of the premises of agile is to release early and often.  It is also why Lean software development preaches understanding and delivering only the "minimum marketable feature" (MMF).  


the best way to get work done is to not do work.  

Unfortunately, operations is considered to
be successful if the metrics are stable and unchanging, whereas development is only applauded if
many things change. Because conflict is baked into this system, intensive collaboration is unlikely.
Development teams strive for change, whereas operations teams strive for stability (the
definitions of change and stability will be discussed in Chapter 2). The conflict between devel-
opment and operations is caused by a combination of conflicting motivations, processes, and
tooling. As a result, isolated silos evolv
5 Steps in the Theory of Constraints
1.  identify the system's constraints
2.  Decide how to exploit the constraints
3. Subordinate everything else to that decision
4. Elevate the system's constraints (automation)
5. Find the next constraint and repeat the process.  
With Agile, testers and programmers
became developers, and with DevOps, developers and operations become DevOps. 
How do you move to a DevOps Reference Architecture?
  1. Read up.  There is no DevOps Manifesto like the Agile Manifesto.  There's a lot of disjoint stuff out there.  
  2. DevOps is more culture change than process change.  That makes it different from, say, implementing scrum.  You really can begin to adopt DevOps from the bottom-up if you have to.  Start by having some teams collaborate.  
  3. When you get some small wins using bottom-up then advertise that fact and try to get the top to buy-in and make the cultural changes to the entire organization.  
  4. Remember that DevOps is not the end, rather the means to the end (which is profit).  Too many people adopt DevOps to claim that they are DevOps.  Meanwhile, their software sucks.  
  5. Don't rely too much on metrics.  How do you measure whether your team is collaborating more with DevOps?  You can't.  The metrics must be softer.  Less-sucky software.  Or fewer meetings where Ops complains about the dev folks.  
  6. Not everyone needs to do DevOps.  A few good developers can start collaborating with a few senior ops guys.  The more mature IT professionals will see the value immediately.  The ones who incessantly complain because they feel Ops work is beneath them will only be a distraction.  Prove that DevOps works in your organization before shoving it down everyone's throats.  It may not work for you.  (If that is the case, then, of course, nothing has probably ever worked for you either).  
  7. Focus on your CM team first.  At every large ISV that I've worked for the CM team (change review board or whatever else you want to call it) was the life-sucking entity that everyone hated the most.  The CM process is viewed as a process with negative value in the value stream.  The Review Board really has an important job to do but is relegated to merely being a rubber-stamp organization.  Change that perception and you've solved a lot.  
  8. Don't fall for the NoOps trap unless you are already a NoOps shop.  

You have just read "One Last Git Gotcha:  The Merge Pattern" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  


Guacamole: A clientless RDP gateway

(Update:  Upgrading to guac 0.9.4)

(UPDATE:  Guac 0.9.2 is out and I'm running it and very satisfied.  These instructions work equally well with 0.9.2 you simply need to change "0.9.0" to "0.9.2" in the requisite spots in the instructions below.  I also wrote Upgrading Guacamole which covers how to upgrade an existing guac instance to 0.9.2.)


Guacamole is an open source (MIT licensed, so it's "free", see my recent Open Source Licensing post on this), clientless RDP solution that runs in any HTML5 browser (which is any browser from the last few years).  Why do you need this or should you care?  If you need to remotely connect over the internet to Windows servers (you deal with small clients or your home network) and you don't want to open up Port 3389 (the default RDP port) then guacamole is an excellent choice.  If you currently have Port 3389 open I suggest closing it.  I can tell you that every script kiddie out there has password crackers that attempt to brute force attack against open, internet-facing RDP ports.  

Guacamole runs on Java/tomcat/Linux on your internal network that then serves up the RDP session via Port 80/443.  So, it's a gateway.  Then why not use /tsweb and IIS that comes with Windows Server?  Because /tsweb still uses Port 3389 to connect, /tsweb merely allows the client to be downloaded from a website.  

With guac no new ports need to be open so it should be much more secure.  Since it doesn't need a client (other than a HTML5 browser) you should be able to connect to your Windows machines from anywhere on the planet including iPhones and Android devices.  You can certainly use something like LogMeIn (which costs a few bucks) or OpenVPN (free, but takes some time to setup and still requires a client) to accomplish secure remote access.  Guacamole is just another tool in your toolbelt.  Guacamole is my solution of choice because it is fast, requires about 256 MB of RAM in a Hyper-V and takes only a few minutes to set up.  

The guacamole install manual, like most OSS, is not user-friendly if you are not a Linux wizpert.  In fact, if you follow the manual you will end up installing an ancient version which does not handle logging in to "modern" Window boxes (Windows 2012/WIN8 has new NLA security features with RDP).  So, this post will show how to setup a VM on your Hyper-V server to host Guacamole.  I assume you understand the absolute basics of VMs and ubuntu.  You need no knowledge of tomcat or Java.  In fact, you really don't even need to know ubuntu if you can type text without too many typos.  

The Process

  1. Install Ubuntu (14.04 is best as of this writing) in your VM.  Patch it (sudo apt-get update;sudo apt-get upgrade), give it a static IP address.
  2. sudo apt-get install make libcairo2-dev libpng12-dev freerdp-x11 libssh2-1 libfreerdp-dev libvorbis-dev libssl0.9.8 gcc libssh-dev libpulse-dev tomcat7 tomcat7-admin tomcat7-docs   These are the standard packages you'll need.  

  3. sudo apt-get install libpango1.0-dev libssh2-1-dev
  4. wget -O guacamole-server-0.9.0.tar.gz http://sourceforge.net/projects/guacamole/files/current/source/guacamole-server-0.9.0.tar.gz/download.  We can't use apt-get because it is currently serving up Guacamole 0.6.0 which doesn't work well with Windows 2012 or Win8.  Instead, we get the source code from sourceforge directly (you may want to see if there is something newer than 0.9.0 since improvements are being made to guacamole weekly). 
  5. wget -O guacamole-0.9.0.war http://sourceforge.net/projects/guacamole/files/current/binary/guacamole-0.9.0.war/download
  6. sudo tar -xzf guacamole-server-0.9.0.tar.gz
  7. cd guacamole-server-0.9.0
  8. ./configure --with-init-dir=/etc/init.d  This will configure autostart on reboot for the necessary services.  
  9. make
  10. sudo make install
  11. sudo update-rc.d guacd defaults  This sets the autostart for the default runlevels.  
  12. sudo ldconfig (that is ell dee config.  It essentially loads your config changes)
  13. sudo mkdir /etc/guacamole
  14. sudo nano /etc/guacamole/guacamole.properties  //enter the items from the screenshot at the right
  15. Ctl+O  (writes out the changes)
  16. Ctl+X  (exits nano, the text editor)
  17. sudo nano /etc/guacamole/user-mapping.xml (a sample is to the right).  There are so many options for rdp that you can use that you should probably eventually read the user-mapping documentation.  You can start apps directly using remote-app and you can autologin using username/password.  I use guacamole to remotely connect to my Linux boxes using ssh too, without opening ssh ports (22) or having an ssh client on my android devices.  You can also use guacamole as a clientless VNC gateway too.  
  18. Ctl+O  (writes out the buffered changes)
  19. Ctl+X (exits the editor)
  20. sudo mkdir /usr/share/tomcat7/.guacamole
  21. sudo ln -s /etc/guacamole/guacamole.properties /usr/share/tomcat7/.guacamole
  22. sudo cp guacamole.0.9.0.war /var/lib/tomcat7/webapps/guacamole.war
  23. sudo service guacd start
  24. sudo service tomcat7 restart
  25. Open a browser on your local network and navigate to http://<ipaddress>:8080/guacamole.  Login with your credentials from user-mapping.xml.  At this point you've got a working guacamole rdp gateway server.  

Any errors you may get will be "Invalid login" at this point.  This is the generic error message.  The errors will likely be a typo in either the .properties file or the user-mapping.xml file.  You can find the logs at sudo nano /var/log/tomcat7/catalina.<today>.log.  There isn't much that can go wrong other than typos.  Doublecheck the two hand-crafted files we created above.  

At this point guacamole is working but it is listening in on port 8080. If you are OK with that then poke a hole in your firewall/router and you can begin using guacamole. Read on for some additional improvements you can make to guacamole...

Backing up your guacamole server

 There's not much to backup on your guacamole server, other than config files, so I like to use git for things like this. If it's easier you can just take a snasphot of your vm too and save it off somewhere. For Git:

sudo apt-get install git
cd /etc/guacamole //this contains your .props file and user-mapping.xml
sudo git init
sudo git add guacamole.properties
sudo git add user-mapping.xml
sudo git config --global user.email ""
sudo git config --global user.name ""
sudo git commit -m 'initial configuration'
sudo git remote add origin ...
//not necessary unless you want to push to your git repo server
sudo git push -u origin master //again, not necessary

I want guacamole to be available on <main web server>/guacamole, not on Port 8080
In other words, you want guacamole to be served up by your standard webserver in a folder there, not on a separate tomcat webserver running guacamole.  This is a fairly simple mod. On your main webserver you need to create a folder that is used to proxy to your guacamole server.  I assume your main webserver is also Apache, if not you'll have to do some research on your webserver to determine how to proxy to another webserver under the same namespace (ARR in IIS for instance).  In your Apache http.conf (or equiv) file you'll need these entries

 # /guacamole settings
    ProxyPass /guacamole
    ProxyPassReverse /guacamole
    <Location /guacamole>
       Order allow,deny
       Allow from all
 # /guacamole settings
 ProxyPass /guacamole http://<guac ip>:8080/guacamole
 ProxyPassReverse /guacamole http://<guac ip>:8080/guacamole
 <Location /guacamole>
      Order allow,deny
      Allow from all
Restart Apache and you should be able to use guacamole on your main webserver by browsing to /guacamole, using standard Port 80.  No changes needed to the guac installation for this.  The ProxyPass simply tells apache that when requests come to /guacamole that they should be rerouted to the guac server on a different port. ProxyPassReverse handles rewriting the headers on the response from guacamole so the client doesn't get confused.  Remember, guac, in this configuration, does not know it is directly attached to the internet because we did not set a hostname anywhere.  
At this point it is also advisable to set up SSL encryption if you deem it necessary.  SSL only needs to be setup on your main webserver, not on the guac server.  So if you have SSL working already the proxying will work transparently with nothing else to do.  You can definitely run guac without https just be aware that you are taking a bit of a risk.  


I do a lot of consulting for small mom-and-pop shops that can't afford expensive IT infrastructure.  When I need to connect to their internal resources the easiest way to do this is to set up a VM with guacamole on it.  Formerly I used LogMeIn but that isn't free anymore.  OpenVPN is an excellent choice but it requires a client and a VM that acts as the gateway, plus a hole in the firewall for OpenVPN.  Guac needs none of this.  Once you setup a guac VM you can save the vhdx file (or equiv if you are using another virtualization technology) and then you have a quick guac server for your next client.  Big time saver...no cost.  

You have just read "Guacamole: A clientless RDP gateway" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

DevOps: The Ways

This post in my DevOps series will cover "The Ways".  

You'll almost never hear The Ways mentioned with DevOps.  That's ashame because when you hit the Third Way, then DevOps is working for you...your development and operations teams are thinking as one.  The Ways is often associated with Kanban.  The Ways sounds like some kind of Kool-Aid cult blathering, but if you understand it, it works.  






(in this case it should be cleear that the bottleneck is the middle column...whatever that is)

The First Way is the visual management of work and pulling work through the system.  A good Kanban board can do this.  We want to see the fast flow of work through the system.  An outsider can look at it and immediately see where the bottlenecks are in your process without any prior knowledge of your business or Kanban.  If your Kanban board isn't doing this then you need to study Kanban some more.  You can think of the First Way as "systems thinking".  How do we get work out the door?  It's kind of very waterfallish.  Things move from left to right.  At this phase of DevOps there is still clear delineation between the developers and the Ops guys.  In this phase the bottlenecks tend to be after development and generally falls with the Ops guys.  This is because we still have developers who are rolling out code without much thought as to how it will be operationalized and who will support it.  

The Second Way is to make waittimes visible.  You need to know when the work takes days sitting in someone's queue.  This may not necessarily be a bad thing but it should lead you to ask some difficult questions.  Do we have enough resources at that phase?  Are those resources being utilized at the right "workstation"?  Or is that resource getting pulled in too many directions.  When your manager complains that it takes too long to do a certain task, perhaps it is due to waittimes?   

The Second Way is sometimes thought of as moving DevOps into the "feedback loop" phase.  The Ops guys should be providing feedback to the devs regarding what worked and what didn't so that the next release is smoother and there are fewer bottlenecks.  In other words, your "DevOps maturity" is improving at this phase and DevOps should begin to show you dividends.  

I like my definition of the Second Way better (make waittimes visible).  In larger organizations where inertia makes dev and ops collaboration politically impossible it is better to think of the Second Way as figuring out what is wrong with the First Way.  Queuing Theory is very concerned with understanding why service times are long and generally you'll find the biggest culprit is waiting on some unavailable resource.  This holds equally true whether you are examining the performance of a SQL Server that is waiting on IO subsystems or development teams waiting on QA resources.  

The Third Way is to ensure that we are continually putting tension into the system, so that we are continually reinforcing habits and improving something.  It doesn't matter what you improve, as long as something is being improved.  If you are not improving then entropy guarantees that your competitors will beat you.  I've heard IT people tout this as the equivalent of simple DR fire drills.  I think that is too simplistic.  The fact is, a DR fire drill tends to involve only Ops guys.  I don't know too many developers getting involved in these.  And remember that DevOps is NOT a simple melding of your developers and Ops guys.  It is collaboration, not job redefining.  

You'll know when you are ready for the Third Way when your people start to get bored.  That means they understand DevOps and Kanban and are not being challenged by it anymore.  Here's a simple way to move into the Third Way.  When you see entropy set in then make an announcement that some simple feature request will be handled as though it were a P0 ops emergency.  The request will be fast-tracked through the system to see how long it takes to move a request from a BA through to operationalization.  It may take 5 days.  Wait a few weeks and take a similarly-scoped request and repeat.  See if you can get your team to reduce the cycle time to 3 days.  Then figure out what the bottlenecks were that you removed.  


There is so much that can be written about The Ways.  To be succinct, think of your IT team as working on a factory floor and you as the manager are overseeing the activity from the catwalk.  You want to see no bottlenecks, no unplanned work, and no excessive waittimes as the work moves along toward doneness.  Then you want to introduce a defect and watch the floor workers swarm to fix the flow of work through the system.  When the defect has been resolved the factory floor should return to its previous state.  Granted, for an IT manager or scrum master it is difficult to think of your IT staff as factory workers, but this method does work when you really need to determine who and what is your bottleneck.  

You have just read "DevOps: The Ways" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  


Open Source Licensing

I recently wrote Git Gotchas for the Savvy Svn'r where I mentioned in passing that some companies like monolithic software repos because it aids in Black Duck auditing.  I then received a couple of emails asking what this was and why it was important.  I also recently had a consulting engagement where the improper use of open source software had the potential to cost them a large portion of their revenue.  At around the same time Heartbleed (the OpenSSL bug) came into the light and suddenly there was a "open source vs closed source" debate.  All of this is important and you should understand it if you want to be a good architect.  

I don't want to get into a religious debate about open vs closed source.  Pick whatever works for you.  Rather, the underlying licensing of any software you are using in your projects is critical.  I consult for various ISVs and all of them srutinize third-party licensing of closed source software if they are going to redistribute any portion of it under their umbrella.  I've mentioned in the past (How to save a *lot* of money with SQL Server Express) how a lawyer for an ISV I worked for really scrutinized how I was using SQL Express.  Eventually all parties (including M$) were satisfied that I was breaking no licensing terms, but my point is that when it comes to redistributing closed source software we, as an industry, tend to scrutinize everything.  

That is good.  But we don't tend to scrutinize open source software that we use.  And we should.  

Black Duck audits essentially look at your software portfolio for areas where your OSS (open source software) is out-of-compliance with the underlying licensing.  If you use OSS, you should be familiar with Black Duck, or something similar.  If you think Open Source = Free Software...you have a BIG problem.  There are tons of free and open source software licensing.  Frankly, I don't understand them all, and I'm not a lawyer so I won't attempt to explain the nuances you'll find in these.  But there are two basic types...these are my words..."free free" (example Apache licensing) and "copyleft" ("free, if your careful"...example is GPL).  

(screenshot of wikipedia's entry for comparison of free and open source licenses.  Note that this is just the licenses at the beginning of the alphabet.)


My Personal Feelings

I have personal reservations about Intellectual Property (IP) rights.  But even though I don't like IP, I have to respect it in my daily dealings.  Open source licensing, in general, disgusts me because if you are a bit lax, even accidentally, your OSS vendor can swoop in and claim large portions of your profits as its own, legally.  In many regards these licenses are far more punitive and Draconian than anything I've ever seen from a closed-source vendor like M$ or Oracle.  Why is this?

OSS vendors have to Make Money.  Choice-of-license is how they do it

Some people believe that OSS vendors chose the open source model because of some kind of altruism.  "We provide our software for free to make the world a better place."  Bullshit!  Every company has to make money.  How a vendor chooses to license their product is how they make it.  There are open source licenses that foster "building a community" prior to building sustaining revenue (example, Apache licensing) and there are licenses meant to generate revenue first, often sneakily (example, GPL).  

Let's take two examples.  Company A develops some software and decides the best way it can make money with it is to build a community of avid users of the software.  "Build it and they will come", if you will.  The Apache License best meets that goal.  As a business-owner and independent consultant for ISVs I look first for software with Apache licensing (or BSD, MIT, or similar).  These licenses are business-friendly.  I'm not an attorney and I bear no responsibility for any misinformation given in this post, but essentially an Apache License gives the licensee full control to do with the software anything it wants to do with it, as long as the copyright notice is preserved.  It's about as close to free as it gets.  No royalties necessary.  In this case Company A believes the Apache License affords it the best method of gaining market momentum.  So, how does Company A make money if its software is "free"?  There are many ways, but in-app advertising and paid services (support) are the two most common.  This example is straightforward and is what most people think of with OSS.  You really have to do something stupid to get in trouble using this variety of OSS.  

GPL, "Copyleft", and Derivatives

Company B would rather sustain its business with licensing revenue (just like Oracle or M$) but it still wants to call itself OSS.  Just my opinion, but this is sneaky and bordering on dishonest.  Here's how it works.  Company B chooses a "copyleft" licensing scheme like GPL or AGPL.  Again, I'm not a lawyer but copyleft works by saying the license is just like an Apache license except that derived works must also be released as open-sourced, copyleft works.  Said differently, the license must be moved forward.  

Huh?  Company C is an ISV that would like to use some open source software to augment its closed source software.  If Company C uses an Apache-licensed piece of software then they may continue to keep their software closed-sourced without paying royalties.  But if they decide to use a GPL license then Company C *must* open source its software or it will be in violation of the GPL.  That means the GPL "moves forward".  

So, how does Company C use GPL'd software in its closed-source product?  Carefully.  All "copyleft" ISVs like Company B will offer both a GPL version of its product and then another version with a non-open-source license that the licensee can then use in its closed-source product without having to open source everything or violate the GPL. The GPL'd version is still "free", the alternatively-licensed version can have whatever royalties and terms that it desires, just like a closed source product.  By "version" I don't mean a separate set of binaries (like Windows Standard vs Enterprise), rather a different "version" of the license. 

A True Horror Story

(I find this logo less-than-truthful)
(Note that "free" means "freedom", which is nebulous.  "Free" does not mean "no cost".)

An ISV I worked for loved using OSS whenever it could.  The OSS always used Apache licensing (or equivalent) whenever we redistributed our code.  If it was software used in-house for administration or development only then nobody really cared about the licensing covenants.  There was a secret push to begin using a NoSQL alternative to SQL Server for some aspects of data persistence for a skunkworks project.  Cost was the reason.  SQL Server was deemed too expensive.  I was not involved in the evaluation or recommendation of the OSS product but a company-that-shall-remain-nameless gave a demo of their "free" distributed document store software to the skunkworks team.  I was not invited.  The product worked well and our developers coded a bunch of stuff against it that was not yet released.  

There are industry horror stories with this particular OSS vendor.  This vendor is known for touting its free OSS and then strong-arming ISVs into huge licensing fees at a later time once your software was coded and released.  You either paid this vendor for the closed-source license, or you open sourced your ENTIRE application.  Even if only a small corner of your application used the vendor's product they threatened that your WHOLE APPLICATION needed to be open sourced.  Most ISVs get scared and do not want to open source their product and rely solely on services revenue.  This would be a dramatic change for most ISV business models.  So these ISVs usually relent and pay the equivalent of shakedown money for a non-free, non-GPL license for the product.  It is either that, or rip out the GPL'd product and start over.  That would be equally unappealing.  

Soon the secret was out that there was a group developing against a new data persistence engine.  This is when I found out.  I knew this OSS package was GPL and its use could radically alter our business model.  The vendor would not give me a straight answer as to exactly how much of our product would need to be open-sourced if we took on their product.  At this point I could've gotten our legal department involved but that tends to be slow.  Much better to have the vendor speak to our management team directly.  

I organized another presentation with the NoSQL vendor and they started out touting the free, open source nature of their product, case studies of its success, etc.  This time the audience was the senior architects and the management team.  About 10 minutes into it I raised my hand and asked very pointedly, "How much will your product cost us in licensing if we choose NOT to open source our product under the terms of the GPL?"  The vendor attempted to deflect but I kept pushing.  Eventually the vendor opened a different Prezi that discussed alternative closed source licensing arrangements.  Clearly they had different presentations based on where the customer was in its implementation of their product.  The vendor did not realize that I did my homework and knew this was going to happen.  Neither did our management.  Or our architects.  "Deer in the headlights" does not do justice to the looks on everyone's faces.  

The alternative licensing was MORE EXPENSIVE per node than SQL Server, not to mention it was a distributed document store which would have meant LOTS of small, little licenses for small, little nodes.  It was eye-opening for management.  There would be no cost savings.  We quickly had our developers pick a document store that was Apache licensed and we delayed our release for a few months until we could rewrite.  

From then on there were policies on which OSS licenses we could use and rules regarding what could be redistributed.  Our Legal Department was now required to sign off on ANY third party tool where the code was checked in and integrated with our source code.  This event scared management so much that they brought in Black Duck Software to make sure our exisitng software portfolio wasn't hiding any other potential OSS timebombs.  (It was).  

Since then customers of our software have also performed the equivalent of Black Duck audits on our software to make sure we were being thorough and honest.  This is mission-critical software.  With GPL software it isn't just the licensee that can get caught up in a licensing battle, it is the "licensee or assigns", which means anyone who purchases our software.  The ramifications of this is really scary.  You could purchase a piece of software and through no fault of your own find yourself on the receiving end of a lawsuit because your vendor was improperly using GPL'd software.  This is another reason why IP rights are so dangerous.  

Choosing Licensing Carefully...Example: Postgresql vs MySQL

When I work with OSS in my personal life I pay ZERO attention to the licensing.  I use what is best for the task at hand.  In my professional life, working for ISVs, I'm much more careful.  Take Postgresql vs MySQL.  If you ask 10 experts you'll likely find that 50% prefer Postgres and 50% prefer MySQL.  No big difference regarding performance or feature-completeness.  They are both very awesome DBMSs.  Usually the choice comes down to comfort and prior experience or specific feature requirements.  That's totally anecdotal of course.  YMMV.  

However, if I needed to propose which one to use for an ISV project I'm ALWAYS proposing Postgres.  Every time.  Why?  Licensing.  Postgres is MIT licensed (similar to BSD and Apache...basically just don't alter the copyright notice and you are free to do whatever you want with it).  MySQL is GPLd and Oracle (who owns MySQL) does have some goofy rules about when a commercial license is required for redistribution.  I'm hesitant to risk another GPL nightmare and choose MySQL unless there was some compelling feature requirement.  

This is just my opinion but I believe that over time Postgres will have a higher installed base than MySQL, merely due to licensing differences.  Most OSS that needs data persistence will offer a choice in backends...usually MySQL, Postgres, or SQLite.  In these cases I choose Postgres because I believe the licensing will eventually cause it to pull away from MySQL and be the preferred data persistence engine.  SQLite is totally public domain (no license AT ALL) so it may even fare better.  

Apache licensing is always preferred to GPL-style licensing?

Not so fast.  If you aren't redistributing or selling your software then it probably doesn't matter what you use.  Check with an expert or a lawyer though.  

Apache licensing has problems too, IMHO, that you should be aware of.  The business model for Apache licensed software brings its sustainability into question, always.  An example is OpenSSL.  It is Apache licensed and is supported by the equivalent of a handful of full-time developers.  It has almost ZERO revenue to sustain the software.  Something like Heartbleed was bound to happen under this situation.  OpenSSL has a large, avid following.  Cisco and M$ use it...heck, it's embedded in EVERYTHING.  And very few users have given ANY money to the project to sustain it because it is Apache licensed (IMHO) and therefore viewed as totally free (without cost).  "Let someone else pay for it".  The very backbone of security on the internet is in the hands of a few OpenSSL developers because no one wants to contribute money.  It is an 800 lb gorilla.  In this case maybe GPL software is a bit better.  If the price isn't too steep.  And since Heartbleed there are some new GPL'd alternatives to OpenSSL that may be safer and more sustainable simply because the revenue-generation model is better.  Only time will tell of course.  Others have pointed out that something as critical as OpenSSL should be closed source anyway.  Again, time will tell.  Perhaps the best answer is a consortium of vendors that can take over OpenSSL and give it the care and feeding it needs, without resorting to GPL.  I'm certainly no expert in any of this.  But it is fascinating.  

Why is this so gd complicated?  Another story

Now you see why I think IP rights cause more grief than they solve and why EVERYTHING I do on this website is public domain with ZERO licensing, not even Apache licensing.  (I always appreciate an attribution, but I'm not going to sue you if you don't).  It's just not worth it to me to deal with all of this licensing.  I don't even have a copyright notice on this site, although I probably should.  

I do some side work as a Drupal consultant.  Drupal is GPL.  Yet there is a lot of Drupal code out there that is available only for a fee.  Especially themes.  People often wonder how that arrangement is possible.  If GPL forces derivational products to also be GPL'd, then how can a themer charge for a Drupal theme?  Very easily.  The themer must provide the source code for any theme and therefore the customer can do anything with it thereafter, including give it away for free.  

Sounds goofy doesn't it?  

I once bought a theme for $50 because it had everything my customer wanted...cool graphics, the ability to skin minutae, it was "responsive" (the ability to resize dynamically to the new form-factor devices coming to market daily), and had lots of other features not available elsewhere.  After I got the code for the theme I realized I was sold a lemon.  The theme did nothing advertised and had ZERO documentation.  I politely asked for my money back and was denied by the vendor, who spoke little English.  I then threatened to simply follow the terms of the GPL license and modify a few bits and re-release the new theme as my own and give it away for free.  

I had his attention now.  He told that was not legal that his theme was his, not mine.  "Nope, sorry, you supplied me with a Drupal theme.  Drupal is GPL therefore your theme is GPL too.  I can modify it however I choose and re-release it to the world.  I can't wait to give your hard work away under my name."  Needless to say my $50 was promptly refunded.  


There is no right or wrong answer to any of this.  The takeaway is that open source software is not cost-free software, usually.  And these vendors aren't always altruistic just because they are OSS.  You must spend some time and understand what you are getting yourself into when you decide to use OSS.  In most cases you will be fine.  If you are an ISV, be a bit more careful.  If you are an ISV that produces OSS then I implore you to be a little more open and honest about how your licensing really works.  Be forthright that you have multi-licenses for different use cases and note the pricing clearly.  Not every small ISV has a cadre of lawyers that understand open source licensing, it would be nice if GPL and GPL-like product vendors would stop trying to hide behind the "free software" label.  It's akin to bait-and-switch.  

You have just read "nodetitle" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

DevOps and the Concept of Work

This is another post in my series on DevOps.  In this post I'm going to focus on the finer details of DevOps and how it relates to Kanban.  I've found that as organizations implement DevOps they are also implementing Kanban at the same time.  In the last post I wrote about the common precepts of DevOps and how DevOps gets conflated with other, unrelated ideas.  Kanban has similar issues.  People hear some Kanban buzz phrases and think that they know Kanban.  They don't.  Instead of this being a post on Kanban I'm going to make this a post more about "advanced DevOps" and it will borrow heavily from Kanban.  Don't worry if you've never worked with Kanban, that isn't required to understand the basics for this post.  

What is "Work"?

DevOps is so much more than bringing developers and operations people together.  DevOps is REALLY about understanding work.  If you understand "work" then you'll understand how to make work better.  

Categories of Work

  1. Business Projects:  this is what makes us profits.  This should ALWAYS be a company's primary category of work.  If it isn't then...you've got a problem. 
  2. Internal Projects:  this makes our operations better.  Automation, code refactoring, etc.  This work category doesn't make us top-line money, but it saves us costs.  Controlling costs is another way to increase profits.  
  3. Change:  This type of work occurs whenever we need to alter a system.  Maybe fix a bug, or upgrade an OS, or apply patches.  DevOps always questions EVERY change to ensure it is really needed.  You should be questioning every request for a change too.  In many cases the change really isn't needed or can be delayed and rolled into some other change.  
  4. Unplanned Work:  The first 3 categories above are "planned work."  Unplanned work is anything that interrupts your planned work.  But it is more than just unexpected emergencies and outages.  It is sometimes the direct result of technical debt.  

Technical Debt

"Technical debt" is a metaphor for the eventual consequences of poor software or infrastructure within your organization.  It is "debt" because it is really work that needs to be done before a particular project can be considered complete.

Technical debt isn't always bad, but it must be recognized for what it is.  In some cases it is better to recognize some technical debt and decide to fix it later so your code can get out the door and make you money.  But you MUST recognize when unplanned work is the direct result of technical debt and then prioritize it so it stops being unplanned.  

If you don't pay down technical debt then your unplanned work will continue to increase.  Left unchecked, technical debt will ensure that only the work that gets done is unplanned work.  

Why is technical debt and unplanned work so important to DevOps?  Here's an example...often your developers do not know about operational pain that development decisions are causing the ops folks.  If the teams work together the technical debt (and unplanned work) can be paid down.  

And this is why I really love DevOps.  It recognizes that development decisions that do not take operational concerns into account often result in lots of unplanned work.  Likewise, if Ops is making a change to a system that should be run past Dev.  As an example, just because a new version of SQL Server has been released doesn't mean our software needs to be updated to support it.  If there is no new feature that will make us money, then it can be postponed.  Not forever of course, but this is a case where pushing back and denying change is a good thing.  

Two more examples

I worked at a company where a developer decided to introduce Service Broker into his application without any operational input.  EVERY customer upgrade failed because SB was not enabled.  The developer was eventually instructed to put together operational documentation that the DBAs would carry out for every upgrade.  

The document was 3 pages long.  Our upgrades already averaged 19 hours and I determined that this "step" of the upgrade averaged 90 minutes because the DBAs did not understand what each step was supposed to do.  The DBAs didn't understand Service Broker.  

At the same time I was working on a different project in the company where I also needed to use SB.  I created an automated script that ensure SB was enabled and all of the plumbing was in place.  If anything was wrong it would attempt to fix the problem without operational intervention.  It added ZERO time to my project's upgrades.  And it worked flawlessly.  

I was unaware of the first project with its cumbersome SB instructions.  Eventually an Ops person asked me if my process could also be used to automate that process.  We plugged in my process with slight tweaking and solved all of those operational issues.  

In my project I realized that operationalizing something like Service Broker was non-trivial.  You can't just throw some new software onto your stack, you have to understand how that is operationalized.  I knew that the unplanned work around upgrade events and SB would be too much technical debt for us to bear.  So, with some forethought we ensured that operationalizing SB tasks was not de-scoped by a project manager trying to cut corners.  

"The Best Way to Do Work is to Not Do Work"

That is a saying I (think) I created that I use almost daily.  When I hear our people saying that they are doing repetitive tasks then I know that it is time to consider paying down that technical debt.  The best way is through automation.  If a developer can spend a 80-hour sprint automating some task that takes ops people 80-hours per quarter in effort, then it is a no-brainer that we need to pay down that technical debt.  The goal is not to do that work AT ALL.  

In the next post I'll cover some additional DevOps concepts you likely won't hear from many other DevOps authorities.

You have just read "[[DevOps and the Concept of Work]]" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  


Is it Pokemon or BigData?

Is it Pokemon or BigData?

This is pretty funny.  You are given a name, say "Crebase", and you must determine whether that is a Pokemon character or a BigData technology.  

I scored a 60%, which means either I don't know my BigData esoterica, or I don't know Pokemon.



There is a lot being written about DevOps, most of it is not accurate.  The biggest inaccuracy is that DevOps is a simple merging of your development group with your operations group...ie,"dev" and "ops".  Nothing could be further from the truth.  I am a huge proponent of DevOps, and if it is properly understood and implemented it can turn around a faltering IT organization.  I have seen this and experienced it first hand at both large shops (120 developers/50 admins) and small ones (13 devs/3 ops guys).  

DevOps is the melding of Development and Operations

Let's just get this one out of the way first.  I see everywhere on the internet that DevOps is the merging of your developers and ops guys.  This is a dangerous over-simplicatioin of DevOps.  IMHO.  Yes it is true that DevOps takes the best-of-breed approach of both worlds and makes them more compatible, but we aren't just redefining job roles.  

I can think of few things worse than having my development staff also handle operations activities.  That's a recipe for disaster.  This misconception probably started with with small startups (or smaller groups in larger organizations) that didn't have the money or time to hire dedicated ops guys.  The easiest solution is to have your developers eat their own dog food.  But this isn't really DevOps.  It's NoOps.  It works great for NetFlix but might not work great for you.  At small shops NoOps is a reality and a necessity when you don't have the headcount for anything more than getting the product shipped to a happy customer so you can generate revenue.  Nothing wrong with that.  

But at some point the jack-of-all-trades model won't scale anymore.  Your product will become too big (in terms of code volume or customer volume) for everyone to wear every hat.  It is a core premise of DevOps that their is a degree of cross-training and understanding of every role, but that does NOT mean every engineer should be doing every role.  

Further, some people are better suited to be either a developer or a ops guy.  This is basic division of labor.  I don't want my smartest developer patching a system, just like I don't want my CEO taking out the garbage.  It's not that developers are better than ops, or the CEO is more important than the janitor, it's just the reality that we want our people working on what they are best suited for.  

Are devs and ops guys diametrically opposed?

I've worked at large ISVs where the ops guys spend all of their time keeping things running smoothly and the dev guys spend all of their time making changes for the next release.  Change is the enemy of the former and the friend of the latter.  Sales guys need whiz bang features and product management is tasked with having the devs deliver it.  But after every release the ops guys are scrambling because the systems are no longer stable.  Performance probably sucks again and web containers need hourly recycling.  While product management is declaring victory with the release the ops guys are getting no sleep dealing with the new issues.  Who can't forgive an ops guy for wanting fewer releases?  But isn't that anti-agile?  

At these large shops I've seen entire management teams that bicker over whether the next release should even be released, and when it should be released, and what it will do to system stability.  At the last minute risky features are removed which causes more risk because this new configuration was never tested.  All of this friction ensures that no real work gets done.  New features aren't being developed and operational tasks aren't being completed.  That is the core of DevOps...stop the bickering and let's ensure the teams are working together toward the same goal -- profitability.  

Lastly, there is far more to DevOps than just devs and ops guys.  QA is a part of it.  Change review boards need to be integrated into DevOps.  Product management, scrum masters, sales...everyone.  

So, What is DevOps?

For me, in a nutshell, it has each group try to think and behave like the other group and instill best practices in each other.  That begs the question, "what are the best practices?"

DevOps embraces tools, methods, and technologies, that can meld your operations and development staff so they work together better.  Some examples:

Example Behavior How DevOps Helps
Your developers use Visual Studio for stored procedure development but your DBAs use management studio.   Get your Ops people using VS.  
Your developers have a build/deploy process for their code but your DBAs don't follow SCM for their processes.   Your DBAs should have the same build/deploy process.  Visual Studio makes it simple to keep operational scripts under the same source control as everything else, or you could use a third party build/deploy tool like MD3.  Everyone follows the same processes.  
Your developers use Git, your ops people use nothing, and your QA folks use a separate Git repository, or SVN.  The Change Review Board uses a file share with neither versioning nor permissioning.   Everyone uses the same repository and follow the same policies.  
Your developers add indexes based on their dev findings.  Your DBAs drop them and add new ones with included columns or whatever.   DBAs need to educate developers as to why their indexing choices are not working and developers need to LISTEN.  
Develops code against systems that aren't properly patched, have varying versions of the application stack installed, use ancient 32 bit systems that were prod 10 years ago.   As part of a build deployment consider deploying on a new VM with the approved stack.  This ensures that if a developer needs to upgrade WebSphere that it is also upgraded in the VM build.  Nowadays this is easy to script, especially if you use something like Aptitude.  This is called "infrastructure-as-code" or "software-defined environments."  If your ops guys tell you their architecture is too complex to do this then you need to educate them that more things are moving to the cloud because of this mentality.  It's not the 1970s anymore.  
Your developers need to change a piece of infrastructure and there is nowhere to version control their script that an Ops person can use.  For instance, you use SQL Replication but there are no scripts that a developer can alter to change an article and re-snapshot.  Or the developer doesn't even know that replication is installed, therefore a table change they make breaks production.   Everything should be scripted and available to your developers.  Good developers can help the Ops people do this.  Good Ops people use the scripts for everything.  No settings are changed via SSMS GUI.  Everything is scripted.
Your Ops people ask your developers to automate a process and they code it in C# and your Ops people don't know C#.  (Hint:  your dev guys should've used PoSH and your Ops guys better learn it if they don't).   Everyone should use a scripting language like PoSh for these tasks.  PoSh is simple for Ops people to learn to do a modicum of modifications, and it is very similar to C# so it should not be a huge learning curve for the dev people.   
Non-prod deployments are totally automated with TeamCity or Jenkins, but when we move to prod then it is a manual process.   Everyone uses the same processes.  More on this below.  
Your developers use VersionOne for defect management.  Your Ops guys use Jira.   Use the same tools for bug tracking.  
Your developers pair with your BAs on new features.  Since you are Agile you don't document the nitty gritty details of each business process.  Your QA folks struggle to test it because they know nothing of what is not documented.   You could immediately state that all requirements be documented, or you could simple expand your pair to include a QA person that can then understand the decisions and possibly document them as a QA artifact.  

If this seems common-sensical then good for you.  However, if you are a developer who couldn't care less about how your code is operationalized, then maybe you should learn more about DevOps.  But, the above just scratches the surface of what DevOps is.  In the next few blog posts I'll cover the far more important concept of "work" and how to manage it properly in a DevOps environment. 

Even simpler, DevOps is an extension of agile where the ops guys get to participate in agile development processes too.  


How Does DevOps Affect Existing Processes

Whenever a new software management method arrives on the scene it freaks people out.  We all know certain parts of our software organization is broken but we don't want wholesale change either.  Here is a list of items that are important to large ISVs and the impact of DevOps upon them:

  • Change Management: if your people think your cm process is broken, then it probably is.  If it is viewed as just a "process" where people know they are going to be harangued because they weren't consulted prior to the change, then the cm process really shows no value.  Collaboration can fix this.  So can automation.  If you automate the change process then people have more confidence.  
  • Release Management:  at large companies there is usually total automation for non-prod envs.  Click a button in Jenkins and your build is deployed to your env.  When ops guys move the change to prod the process is thrown out the window and there is manual intervention again.  The rationale is that there needs to be backups and swapping of hardware, etc.  The fact is if there are special "prod-only" processes then those should be baked into the automation process.  Get the Ops guys to collaborate with the devs to make this a reality.  



This post was my summary on the basics of DevOps.  I clearly feel DevOps is not just another software management method that we can jam down the throats of our people.  It does contain a lot of new and controversial ideas, like having your Dev and Ops teams work together.  This is tough to do in highly-siloed organizations.  But in my experience DevOps works VERY well.  In the next post I'll cover some additional tenants of DevOps that you don't read about much that can also help you on your DevOps journey.

You have just read "nodetitle" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

The Cargo Cults of Software Development Management

Managing the software development process is broken.  Just my opinion.  I have yet to put my finger on exactly what needs to be fixed, and guess what?  No one else has either.  If they did the software dev process wouldn't be broken.  Yet corporations and management constant search for the elusive fix, like Francisoco de Orellana searching for the Lost City of Gold.  Fred Brooks' classic, The Mythical Man Month, comes the closest to flat-out telling us, "There is no El Dorado."  

This is a lead-in post to a DevOps series I'm going to do. DevOps is the latest software development method out there.  It is one that I truly believe can deliver on its promises of fixing some of the ills of our industry.  

The Cargo Cult

Sometimes people observe a successful outcome and believe that if they replicate the circumstances that they too can reproduce the same outcome.  When success isn't forthcoming it is because the circumstances had absolutely nothing to do with the outcome.  This is a fallacy called, post hoc ergo propter hoc...after this, therefore because of this.  Like any fallacy, understanding when you are susceptible to it is half the battle.  And in my opinion, software architects primary responsibility is understanding when their organizations are falling for fallacious concepts.  

So why is it a Cargo Cult?  World War II brought western wealth to indigenous peoples of once-isolated, tiny Pacific islands.  These people had little exposure to the West and led more primitive lives.  In many cases these islands didn't see the bloodshed of war, instead they were used as supply depots.  Some jungle would be cut down and a landing strip would be made with landing lights to guide the supply planes.  The cargo was often food that wasn't always easy to get on these islands.  After the war these islands were abandoned by the war powers and suddenly the indigenous people didn't see food and cargo any more.  There are documented cases where these people would cut down more trees and build more landing strips and then light bonfires and wave discarded signal flags, yet no planes would land.  This baffled the people who assumed that these circumstances were responsible for their food.  

And it is this cargo cult behavior that I see today when adopting new software development methods.  Management says (or hears) that some process (agile maybe) was involved in some wildly successful software project.  Management believes that therefore if they adopt the process that they too will experience success.  

Other Cargo Cult Examples



Various Software Development Methods

I've worked in, and think I understand, *at least* the following development methods:  

Method Dime Tour Does it work?
Waterfall actually I'm not sure about this one.  I've never had any company actually claim to be waterfall.  I think that is because "waterfall" has a negative connotation.  But some of us know when we've worked in a waterfall environment.  And it isn't all bad.  I've worked in waterfall environments that were wildly profitable and fun.  And I've seen waterfall projects be death marches too.  
Scrum/Agile I lump these together.  They are separate and unique but in every environment I've worked in the practitioners claim they practice both.  Works ok but practitioners sometimes adhere to the tenants like it was a religion.  It isn't, don't drink the Kool-Aid
XP another variant of agile Works for me.  
ICONIX I started out on this.  I guess this is roughly waterfall. Worked for me.  
Rational Unified Process ok, maybe I didn't understand this one when I was forced to use it.  Kinda waterfall-ish I guess.  We shipped software with it and were profitable, and that's the metric I use for a working method.  
Kanban a just-in-time method.  It attempts to focus developers on what is really important and on getting that done.  Runs the software dev process like a manager would from a catwalk on a manufacturing floor.   Works well when people understand more than the simple platitudes like "stop starting and start finishing" and really understand how "work works"
Microsoft Operations Framework I actually worked at a place that took MOF and made it into a software engineering process.  M$ even came in and did a lecture series on how to make it work.   Seemed ok...we made money...I got home at a decent hour most nights.  
Inception this seems to be the "old new thing" right now.  Really, it is part of RUP.  Inception is the first phase of software development where you try to broadly grasp the problem and approximate how much effort will be required.  So, there is an up-front focus on requirements gathering and design.  Um, that sounds kinda waterfall-ish to me (but don't tell an Inception Guru that).  I worked at a place that spent millions to bring in a bunch of Incepticons (Inception Consultants) to teach us how to do Inception right.  They were gone within a year, and so was Inception.  We went back to scrum/agile.  We would an entire sprint "incepting" only to realize by the end of a release that the product looked nothing like what we incepted.  To save face in the wake of wasting millions, management decided that we should use "just-in-time inception" instead, which is a mini-inception at the beginning of each sprint. Um, I call that "basic planning".   Complete failure if you follow the religious dogma.
DevOps a lot like kanban but with an emphasis on developers working with ops people to deliver more supportable products.  I'm actually going somewhere with this post...it's going to be the first of many posts on DevOps and how to move towards DevOps.    This is a game-changer.  

So, which one is the best?  

Any of these processes can work if practiced properly.  I have no affinity to any of them (except DevOps which I'm starting to really like).  They each have their good and bad points.  I can work and succeed in any of them.  Or fail in any of them.  

I would NEVER make any of these statements though:

  • "Our project succeeded because we finally realized the truths in <insert software development method here> and practiced it.  "
  • "We failed to meet our objectives because we didn't follow <insert software development method here> the way we should have.  "

There are common denominators that each process has.  Reliance on structure, process, and measurement for example.  But this isn't radical.  

HPCs (Highly Paid Consultants)

If you are looking for a new method, maybe because your software team is not as productive as you feel it should be, you should be aware that you'll find lots of HPCs who will want to sell you their expertise in a given method.  These shills will ALWAYS exhibit these characteristics:  

  • "If you adopt this approach then all of your development problems will go away. " 
  • "We are consultants that would love to come in and train your people on The Way.  We don't come cheap but you can't afford not to learn from our experience."
  • "Every other process is a fraud and will never work.  We have lists of how our process is better than whatever process you currently use.  Use this one to ensure success."
  • The "Who We Are" slide of their deck is always a mugshot of each of the two HPCs giving the presentation.  The first HPC is in the process of a deep belly laugh.  This connotes how they will make your failing projects a joy to work on.  The second HPC mugshot is a contemplative pose..."we have wisdom."  
  • One of the slides will tell you how they've been training The Way for the past x years with success.  They won't list specific companies or software, but they'll tout their successes with both small and large firms.  They don't know how to code in any specific language or domain, but they can manage the process.  They'll have lots of photos of their teams huddled around kanban boards collaborating over a bunch of color-coded Post-It notes, trying to determine what they are going to fix next...world hunger or peace?  They'll conveniently "forget" to mention how this collaboration is going to work with offshoring.  
  • They'll always provide you with lots of swag.  It may be "planning poker" cards with their logo on it, a board game they devised to use in their training classes, or bags with catchy logos like "Stop Starting...Start Finishing".  

All of this is a complete waste of time and money.  Just my two cents.  A HPC cannot possibly know your business domain, your software, or your people.  To assume that what worked previously for the HPC will work again for you is very Cargo Cult-ish.  

So, what does work?  

I don't like to complain without offering alternatives.  This is what I feel are the most important concepts that any method should espouse:

  • Never shackle people.  If your framework commands, "Thy shalt have daily standups where we discuss 'pulling cards from the left' (Kanban)", but the team likes to do Scrum-style standups, then allow people some latitude.  Let the team alter the framework to be assistive, not restrictive.  
  • No multi-tasking.  I call multi-tasking "faulty-tasking", because it DOES NOT WORK.  Human beings are not computers, they cannot multi-task.  Humans, instead, can context switch.  A developer can work on one and only one piece of code at a time.  Allow your IT people to complete a task, do not make them context switch.  
  • Don't rely too much on metrics.  Burn down charts and kanban cards and points are nice, but the focus should be on customer satisfaction.  
  • Don't spend millions on HPCs or special "Kanban TVs" for each team room.  It annoys people when large sums of money are spent on this nonsense yet raises were non-existent the last 2 years.  
  • Ensure everyone is allowed to participate.  Too often I've seen cases where a new method was instituted for the onshore workers, but not for the offshore guys...they aren't important after all.  
  • Practice MBWA.  Management By Wandering Around was practiced by HP back in the 1970's.  It's simple...managers get out of their offices and visit employees randomly, staying quiet and observing.  Good managers will quickly determine lots of little things that are impeding process and can remove those barriers.  
  • Consider adopting a new framework every other year on a trial basis.  Have your people learn the new method and bring it in-house.  Then do retrospectives often to see what is working and what should be discarded.  It's always a good idea to try new things.  I never thought software development management theory could improve but then I started to learn DevOps and realized that it in fact does contain a lot of useful concepts that are obvious, but under-employed by most organizations.  

Over the course of my career I've seen the companies I've worked for bring in lots of management frameworks and paradigms to try to solve what is wrong with their IT departments.  All of these "tools" have a bit of truth and value, but I hold that you can't solve a people problem with a slick framework.  The software development process is broken because fundamentally it is a people problem, not a process problem.  There are patterns and anti-patterns that I list above that I've seen work universally.  Don't be dogmatic about any process and don't be part of a Cargo Cult.  

You have just read The Cargo Cults of Software Development Management on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.