DaveWentzel.com            All Things Data

Management

PMP Certified!

Last week I sat for, and passed, the PMP certification.  In this post I'll discuss problems with vendor certifications and why an industry certification like the Project Management Institute's Project Management Professional certification is valuable.  Even a non-manager, propellor-head should consider the PMP certification to advance his career and improve his brand.  

Entrepreneurial Programmers

The definition of 'entrepreneur' is misunderstood.  An entrepreneur is so much more than a businessman.  Entrepreneurial programmers are those technologists that exhibit similar traits.  The best programmers are entrepreneurial programmers (EPs).  You want these folks on your team.  And you want to be one of them.  In this post I'll describe what it means to be an EP.  Do you qualify?  

On Points

Software estimation "points" were never meant to be applied universally in scrum and agile.  They were meant to help dysfunctional teams meet "goals."  Points solve the wrong problems, and solve them in a way that fosters distrust with management and customers.  There are better alternatives.  

The #NoEstimates Movement is Nuts

Most software developers hate giving estimates.  Who doesn't?  But now there is a movement to ditch estimation entirely (#NoEstimates) based on a bunch of faulty logic.  They've gone too far.  Estimates serve a legitimate purpose.  Those behind this movement are an embarrasment to those of us who want to see our clients and employers succeed.  

On Resumes

I have debated for months whether I should actually publish this post.  

I've finally decided to do it.  

--dave

Understand accounting and you'll understand the trends in software development and IT

(Overheard at the water cooler many years ago)..."Why are we moving to kanban and agile?  I don't understand it, our teams always deliver."

(Overheard at a recent SQL Saturday I attended)..."One of our ISVs (Independent Software Vendors) is now offering subscription-based licensing in addition to their standard release-based licensing.  The subscriptions are almost twice as much as release-based licensing, yet my stupid company just switched to subscription.  There goes my raise."

(Overhead by a DBA in the hallway)..."I can't believe we are moving all of our data to the cloud.  Security is going to suck and by my calculations it's going to actually cost us more money.  I don't understand the rationale for this.  I guess we'll all be laid off soon."  

There are trends in our industry that baffle IT folks.  The above are just a few examples.  There are (somewhat) hidden reasons for these trends.

Accounting "gimmicks".  

If you are involved in software development or IT it behooves you to understand this stuff. YOUR CAREER AND FUTURE IS AT STAKE.  

Let's dive right in.  

Quick Accounting Primer on Expenses

Companies spend money on stuff.  These are expenses.  Expenses are always classified as either Capital Expenses or Operational Expenses.  A table is a good way to visually represent the differences.  

  Capital Expense Operational Expense
aka capex opex
definition

the cost of developing or providing non-consumable goods that support the business.

the ongoing cost of operating a business.  
"value" the expense has a usable life of more than a year, hence it has longer-term value the expense is consumed within a year and then has zero value
example a 3D printer the toner material for the 3D printer
another example a new delivery truck toilet paper  (it has no value once consumed)
one last example building a new warehouse that the company owns leasing a new warehouse from another company that owns the land and erected the building for you
accounting treatment it is added to an asset account and the company's cash flow statement as an investment.   shown as a current expense, recorded immediately and subtracts from income, reducing net profit and thus taxes
affect on profits and taxes is deducted from earnings/reduces profits over its usable life.  this is called depreciation deducted from earnings and will reduce profits and taxes in the year it is paid/incurred
extremely simple example a truck is purchased for 100K and is added as a fixed asset with a useful 10 year life.  10K may be deducted to offset profit/taxes in each year for the next 10 years (VERY simplified example).  The annual 10K allotment is handled similarly to opex for that year.   Toilet paper purchases are expensed and deduct from profits/taxes in the year the tp is used.  

One quick note...R&D Expenses

The above table shouldn't be difficult to understand.  If it is, just trust me, or do your own research.  

Now let's start to get a bit tricky.  GAAP (Generally Accepted Accounting Principles) has a classification for expenses called "research and development".  These expenses are opex.  This is sometimes counter-intuitive for people.  If I'm a pharma company my R&D could lead to a breakthrough drug that nets me millions in profits over decades.  Shouldn't these be capex?  Is this an "investment"?  

Not generally.  The rationale is that at any point an R&D project might be abandoned and there will be no "asset" to the pharma company.  

If you work in software development then you may consider yourself R&D.  Right?  

Not always.  

But let's not get ahead of ourselves.  

Which is better...capex or opex?

First of all, there are a lot of gray areas where a given expense might be classified as capex or opex depending on just how you are willing to bend the GAAP rules and justify your position.  

In situations where an expense could be either capex or opex some companies will prefer one or the other based on how it will benefit them the most. 

A good auditor/investor/accountant can learn a lot about a company's management and goals by looking at how it handles decisions around whether an expense is on-the-books as capex and opex.  Frankly, many expenses, especially IT, like developing and maintaining software, could be either capex or opex depending on your world view.  Here are some generalities that I believe that others may disagree with:

Entity Capex Opex Reasoning
pre-IPO companies x   Pre-IPO companies prefer large expenses to be capex because the expense can be split amongst many years which has the appearance of inflating current year's profits at the expense of future profits.  Pre-IPO companies want to look as profitable as possible to get investors excited.  
Investors   x See above.  Investors would rather see higher opex because nothing is "hidden" by using depreciation methods that hide expenses in later years.  
Companies interested in minimizing taxes   x Costs are accounted for sooner.  This also has a "smoothing" affect on profits.  Forecasting is easier and you won't see as many huge "one time charges" to profits for some big capex item.  (Note that investors don't want to lose their profits to taxes, which is why investors like companies that don't try to hide big capex expenses).  
software development expenses (R&D) at ISVs (Independent Software Vendors)   x Many will disagree with me on this.  I'll discuss this more below.  
IT department expenses (including software development) for non ISVs (banks, pharma companies, finance, etc) x   Here is the sticking point and what I feel is the answer to all of those questions at the beginning of this post.  Companies want to move to Opex nowadays.

Some companies may want to defer as much expense as possible to look profitable (those pre-IPOs).  Those companies will want to capitalize as much as they can.  Otherwise, generally, companies nowadays prefer opex to capex.  

Even within a company there are conflicting interests.  Some folks want opex, some want capex.  A good CFO/CEO will favor opex because that is what their investors want to see.  But within those companies the CIO/CTO may feel differently.  Many companies view IT costs as out-of-control.  How better to make the budget look smaller than to shift expenses to capex?  So now you have the CIO/CTO working at cross-purposes with the goals of the company.  

Isn't this stuff complicated?  

The Rules for Various IT Expenses

Here are some "rules" for various IT expenditures.  Look at the list carefully and you'll see that the trends in our industry today is to move away from capex and towards opex.  This has been the general trend in business since at least Enron and the Dot Com Bubble.  

Expenditure Capex Opex Reasoning
software licenses x   They have a usable life of more than one year.  
software subscriptions   x You are only "renting" the software.  Fail to pay the rent and you have no asset.  
purchased laptop x   Has a usable life of more than one year.  
leased laptop   x No asset after the lease expires.  
"cloud"   x "renting"...no asset...do you see the trend?
data centers x   Huge upfront costs.  This is an asset.  
software licenses for software deployed in the cloud   x Yup, you can buy a license for an IDE and deploy it on AWS and move it all to opex.  I'll say it again.  If you purchase software that would otherwise run in your data center, yet deploy it on a VM in the cloud, magically the capex becomes opex.  

 

The Rules for Software Development Expenses

There are actually FASB and IFRS rules that govern how to expense software development.   They are very complex.  This post is a simplification of the issues.  Feel free to use google to confirm.  You may find a lot of information that conflicts what I have here.  I suggest you actually read the rules and you may find your company is doing things incorrectly, at its detriment.  But I'm not an accountant nor do I play one on TV.  

First your company's primary business activity determines whether you should be using capex or opex for software development/IT costs.   

  ISV non-ISV
Core Business The company's core business is developing software to sell to others.   The company's core business is not software development but it undertakes software development to increase efficiencies in its core competencies.  
Example DaveSQL LLC creates tools that are sold to other companies to aid them in SQL Server DBA tasks.  Dave BioPharma, Inc creates new drugs.  It buys DaveSQL's products to help manage its servers.  
IT expenditures should be... always opex.  People disagree with this...they are wrong.  Go read the rules.  R&D is always opex.  If the ISV cancels the development effort at any time there is no asset.   at times this is capex, at other times, opex.  More in the next section.  

For non-ISVs...when is software development capex vs opex?

Remember, all software development costs (well, most I guess) should be opex for an ISV.  This table is solely for IT shops at traditional companies.  The "phase" of the software development project at a non-ISV determines how the expense is handled.  

Expenditure Capex Opex Reasoning
Functional design/"Evaluation Phase"   x If the project is not feasible and is scrapped there is no asset, so it is R&D, which is opex in the traditional sense.  
Development Phase including detailed technical specifications x   The outcome is an asset.  Even if the software is useless or obsolete by the end of this phase and is scrapped, it is still capex.  There is still an asset.  That asset may be worthless and can't be sold, but it is still an asset.  
Post-implementation   x This one should be obvious.  This is production support.  

 

 

 

 

 

Expenditure

Capex Opex Reasoning
software licenses x    
software subscriptions   x You are only "renting"
"cloud"   x You are "renting" and therefore there is no asset after the lease expires.  
data centers x   Huge upfront costs.  
software licenses for software deployed in the cloud   x Yup, you can buy a license for an IDE and deploy it on AWS and move it all to opex.  

The Rules for Software Development Expenses

Expenditure Capex Opex Reasoning
software licenses x    
software subscriptions   x You are only "renting"
"cloud"   x You are "renting" and therefore there is no asset after the lease expires.  
data centers x   Huge upfront costs.  
software licenses for software deployed in the cloud   x Yup, you can buy a license for an IDE and deploy it on AWS and move it all to opex.  

The Rules for Software Development Expenses

If you haven't noticed thus far in this post, there is a tendency for most companies to prefer opex over capex.  This is not an iron-clad rule, but that is the trend in the business world today.  So, if we were accountants/CFOs/analysts/investors we would want to figure out ways to get more opex and less capex from our software development efforts.  This helps us pay less taxes.  

First thing you should note is that the last table is very waterfall-ish in its "phases".  Design to development to ops.  But what if we were agile and used cross-functional teams?  Could we make some of that capex into opex?  Yep.  And there's the trick.  

Waterfall generates assets too quickly under accounting rules.  It has detailed design documents after all...and those are assets.  So there's another reason why agilists tout "Working Sofware over comprehensive documentation".  I'll bet you didn't know that.  Agile, if practiced properly and understood by your Finance Guys, will have less capex.  

Agile is the best way I've ever seen to shift almost all non-ISV software development costs to opex.  Just get some working software out the door and then bug fix it.  That probably seems like an oversimplication to a technologist, but not-so-much to an accountant.  

Bending the Rules Even More

You can justify anything you try hard enough.  For instance, you can front-load opex using waterfall if you lump that comprehensive documentation as part of your "evaluation phase" documentation.  Using that trick we could re-classify just about anything.  

Please note that pre-IPO companies can also bend the rules in the reverse direction to generate more capex to make their current year profits higher.  Like I said at the beginning of this post, this is all "accounting gimmicks".  

The Ultimate Rule Bender...the Cloud

Quick thought experiment...your customer comes to you and says, "Your software doesn't work because it doesn't do X properly."  You decide that you agree and proceed to rectify it.  Is this work capex or opex?  Here is the rule...upgrades and enhancements to non-ISV software is capex...maintenance and bug fixes are opex.  So, is the work you are about to undertake capex or opex?  That depends.  Your customer would probably label the "issue" a bug (hence opex), but your company may disagree and deem it a "requirements change", hence an enhancement, hence capex.  

But wait, we don't want capex...we want opex, so do we have to admit our software is buggy to get an opex classification?

Nope.  

Enter the cloud.  

All cloud application development, even enhancements and upgrades, is opex because the software is rented.  Nice.  Now you can get an opex expenditure and never admit that your software was buggy.  

More on the Cloud and Software Subscriptions

With traditional release-based licensing an ISV would only make money when the next release was available.  This had an unpredictable effect on profits.  If you missed a release date you may not make any money.  Subscription-based licensing fixes that by "smoothing" out the profits.  Recently Adobe moved their software packages to a subscription-only model.  When they first released their earnings under this model their profits were down radically based on where most of their customers were in the release cycle.  They basically picked an inopportune time to change their model.  

The buyer of software loves subscriptions for the same reason.  "Smoothed" expenses and no capex.  

Open Source Software and "Services"

I'm convinced that the opex/capex debate is one of the key reasons for the rise in the open source software (OSS) movement.  Most OSS vendors offer a version of their software for free and then try to make money by offering services.  To the user of OSS this is very appealing.  There is no upfront cost for the software (capex) and the customization services are opex.  

Not all OSS uses this model, but it is prevalent.  

Think of every blogger that offers free software to do performance analysis for SQL Servers.  Altruism aside, they do this to get you to use their tools hoping that you will attend their seminars to learn more.  Or purchases their consulting services.  It's really a great model.  

 

A History Lesson and Concluding Thoughts

Back in the Dot Com Days every company preferred capex to defer the hit to profits.  And times were good for IT guys who didn't look quite so expensive because their salaries were more-or-less capitalized.  Then the dot com bubble burst, the economy tanked, Enron blew up the world, and Sarbox came along.  Now most companies want to be as transparent as possible with their accounting.  And that means opex and less capex "one-time" charges to earnings.  

Every trend in our industry in the last 15 years is geared toward the move to opex.  

  • Why is there a push to be more agile and to use Kanban?  To get us to opex faster.  
  • Why are developers asked to do support work...a la "DevOps"?  To get more people under the support budget (opex).  
  • Why is every company tearing down their 10 year old data centers and moving to the cloud?  (Do I have to say it?)  
  • Why are ISVs moving to subscription-based software models?  So that their customers can opex the "rent".  (This also "smooths" the ISV's monthly recurring revenues too).  
  • Why is your company trying to move your on-premise SQL Servers to SQL Azure (or whatever it is called now)?  I think you got it.  

It behooves all technologists to understand the basics of accounting and economics.  Many of the trends in our industry can be traced to how those trends will ultimately affect profits.  You should be designing your software accordingly.  I have no clue what the future holds in our industry, but I sleep better knowing that I understand the economics of the decisions being made.  


You have just read "Understand accounting and you'll understand the trends in software development and IT" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

DevOps

There is a lot being written about DevOps, most of it is not accurate.  The biggest inaccuracy is that DevOps is a simple merging of your development group with your operations group...ie,"dev" and "ops".  Nothing could be further from the truth.  I am a huge proponent of DevOps, and if it is properly understood and implemented it can turn around a faltering IT organization.  I have seen this and experienced it first hand at both large shops (120 developers/50 admins) and small ones (13 devs/3 ops guys).  

DevOps is the melding of Development and Operations

Let's just get this one out of the way first.  I see everywhere on the internet that DevOps is the merging of your developers and ops guys.  This is a dangerous over-simplicatioin of DevOps.  IMHO.  Yes it is true that DevOps takes the best-of-breed approach of both worlds and makes them more compatible, but we aren't just redefining job roles.  

I can think of few things worse than having my development staff also handle operations activities.  That's a recipe for disaster.  This misconception probably started with with small startups (or smaller groups in larger organizations) that didn't have the money or time to hire dedicated ops guys.  The easiest solution is to have your developers eat their own dog food.  But this isn't really DevOps.  It's NoOps.  It works great for NetFlix but might not work great for you.  At small shops NoOps is a reality and a necessity when you don't have the headcount for anything more than getting the product shipped to a happy customer so you can generate revenue.  Nothing wrong with that.  

But at some point the jack-of-all-trades model won't scale anymore.  Your product will become too big (in terms of code volume or customer volume) for everyone to wear every hat.  It is a core premise of DevOps that their is a degree of cross-training and understanding of every role, but that does NOT mean every engineer should be doing every role.  

Further, some people are better suited to be either a developer or a ops guy.  This is basic division of labor.  I don't want my smartest developer patching a system, just like I don't want my CEO taking out the garbage.  It's not that developers are better than ops, or the CEO is more important than the janitor, it's just the reality that we want our people working on what they are best suited for.  

Are devs and ops guys diametrically opposed?

I've worked at large ISVs where the ops guys spend all of their time keeping things running smoothly and the dev guys spend all of their time making changes for the next release.  Change is the enemy of the former and the friend of the latter.  Sales guys need whiz bang features and product management is tasked with having the devs deliver it.  But after every release the ops guys are scrambling because the systems are no longer stable.  Performance probably sucks again and web containers need hourly recycling.  While product management is declaring victory with the release the ops guys are getting no sleep dealing with the new issues.  Who can't forgive an ops guy for wanting fewer releases?  But isn't that anti-agile?  

At these large shops I've seen entire management teams that bicker over whether the next release should even be released, and when it should be released, and what it will do to system stability.  At the last minute risky features are removed which causes more risk because this new configuration was never tested.  All of this friction ensures that no real work gets done.  New features aren't being developed and operational tasks aren't being completed.  That is the core of DevOps...stop the bickering and let's ensure the teams are working together toward the same goal -- profitability.  

Lastly, there is far more to DevOps than just devs and ops guys.  QA is a part of it.  Change review boards need to be integrated into DevOps.  Product management, scrum masters, sales...everyone.  

So, What is DevOps?

For me, in a nutshell, it has each group try to think and behave like the other group and instill best practices in each other.  That begs the question, "what are the best practices?"

DevOps embraces tools, methods, and technologies, that can meld your operations and development staff so they work together better.  Some examples:

Example Behavior How DevOps Helps
Your developers use Visual Studio for stored procedure development but your DBAs use management studio.   Get your Ops people using VS.  
Your developers have a build/deploy process for their code but your DBAs don't follow SCM for their processes.   Your DBAs should have the same build/deploy process.  Visual Studio makes it simple to keep operational scripts under the same source control as everything else, or you could use a third party build/deploy tool like MD3.  Everyone follows the same processes.  
Your developers use Git, your ops people use nothing, and your QA folks use a separate Git repository, or SVN.  The Change Review Board uses a file share with neither versioning nor permissioning.   Everyone uses the same repository and follow the same policies.  
Your developers add indexes based on their dev findings.  Your DBAs drop them and add new ones with included columns or whatever.   DBAs need to educate developers as to why their indexing choices are not working and developers need to LISTEN.  
Develops code against systems that aren't properly patched, have varying versions of the application stack installed, use ancient 32 bit systems that were prod 10 years ago.   As part of a build deployment consider deploying on a new VM with the approved stack.  This ensures that if a developer needs to upgrade WebSphere that it is also upgraded in the VM build.  Nowadays this is easy to script, especially if you use something like Aptitude.  This is called "infrastructure-as-code" or "software-defined environments."  If your ops guys tell you their architecture is too complex to do this then you need to educate them that more things are moving to the cloud because of this mentality.  It's not the 1970s anymore.  
Your developers need to change a piece of infrastructure and there is nowhere to version control their script that an Ops person can use.  For instance, you use SQL Replication but there are no scripts that a developer can alter to change an article and re-snapshot.  Or the developer doesn't even know that replication is installed, therefore a table change they make breaks production.   Everything should be scripted and available to your developers.  Good developers can help the Ops people do this.  Good Ops people use the scripts for everything.  No settings are changed via SSMS GUI.  Everything is scripted.
Your Ops people ask your developers to automate a process and they code it in C# and your Ops people don't know C#.  (Hint:  your dev guys should've used PoSH and your Ops guys better learn it if they don't).   Everyone should use a scripting language like PoSh for these tasks.  PoSh is simple for Ops people to learn to do a modicum of modifications, and it is very similar to C# so it should not be a huge learning curve for the dev people.   
Non-prod deployments are totally automated with TeamCity or Jenkins, but when we move to prod then it is a manual process.   Everyone uses the same processes.  More on this below.  
Your developers use VersionOne for defect management.  Your Ops guys use Jira.   Use the same tools for bug tracking.  
Your developers pair with your BAs on new features.  Since you are Agile you don't document the nitty gritty details of each business process.  Your QA folks struggle to test it because they know nothing of what is not documented.   You could immediately state that all requirements be documented, or you could simple expand your pair to include a QA person that can then understand the decisions and possibly document them as a QA artifact.  

If this seems common-sensical then good for you.  However, if you are a developer who couldn't care less about how your code is operationalized, then maybe you should learn more about DevOps.  But, the above just scratches the surface of what DevOps is.  In the next few blog posts I'll cover the far more important concept of "work" and how to manage it properly in a DevOps environment. 

Even simpler, DevOps is an extension of agile where the ops guys get to participate in agile development processes too.  

 

How Does DevOps Affect Existing Processes

Whenever a new software management method arrives on the scene it freaks people out.  We all know certain parts of our software organization is broken but we don't want wholesale change either.  Here is a list of items that are important to large ISVs and the impact of DevOps upon them:

  • Change Management: if your people think your cm process is broken, then it probably is.  If it is viewed as just a "process" where people know they are going to be harangued because they weren't consulted prior to the change, then the cm process really shows no value.  Collaboration can fix this.  So can automation.  If you automate the change process then people have more confidence.  
  • Release Management:  at large companies there is usually total automation for non-prod envs.  Click a button in Jenkins and your build is deployed to your env.  When ops guys move the change to prod the process is thrown out the window and there is manual intervention again.  The rationale is that there needs to be backups and swapping of hardware, etc.  The fact is if there are special "prod-only" processes then those should be baked into the automation process.  Get the Ops guys to collaborate with the devs to make this a reality.  

 

Summary

This post was my summary on the basics of DevOps.  I clearly feel DevOps is not just another software management method that we can jam down the throats of our people.  It does contain a lot of new and controversial ideas, like having your Dev and Ops teams work together.  This is tough to do in highly-siloed organizations.  But in my experience DevOps works VERY well.  In the next post I'll cover some additional tenants of DevOps that you don't read about much that can also help you on your DevOps journey.


You have just read "DevOps" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

The Cargo Cults of Software Development Management

Managing the software development process is broken.  Just my opinion.  I have yet to put my finger on exactly what needs to be fixed, and guess what?  No one else has either.  If they did the software dev process wouldn't be broken.  Yet corporations and management constant search for the elusive fix, like Francisoco de Orellana searching for the Lost City of Gold.  Fred Brooks' classic, The Mythical Man Month, comes the closest to flat-out telling us, "There is no El Dorado."  

This is a lead-in post to a DevOps series I'm going to do. DevOps is the latest software development method out there.  It is one that I truly believe can deliver on its promises of fixing some of the ills of our industry.  

The Cargo Cult

Sometimes people observe a successful outcome and believe that if they replicate the circumstances that they too can reproduce the same outcome.  When success isn't forthcoming it is because the circumstances had absolutely nothing to do with the outcome.  This is a fallacy called, post hoc ergo propter hoc...after this, therefore because of this.  Like any fallacy, understanding when you are susceptible to it is half the battle.  And in my opinion, software architects primary responsibility is understanding when their organizations are falling for fallacious concepts.  

So why is it a Cargo Cult?  World War II brought western wealth to indigenous peoples of once-isolated, tiny Pacific islands.  These people had little exposure to the West and led more primitive lives.  In many cases these islands didn't see the bloodshed of war, instead they were used as supply depots.  Some jungle would be cut down and a landing strip would be made with landing lights to guide the supply planes.  The cargo was often food that wasn't always easy to get on these islands.  After the war these islands were abandoned by the war powers and suddenly the indigenous people didn't see food and cargo any more.  There are documented cases where these people would cut down more trees and build more landing strips and then light bonfires and wave discarded signal flags, yet no planes would land.  This baffled the people who assumed that these circumstances were responsible for their food.  

And it is this cargo cult behavior that I see today when adopting new software development methods.  Management says (or hears) that some process (agile maybe) was involved in some wildly successful software project.  Management believes that therefore if they adopt the process that they too will experience success.  

Other Cargo Cult Examples

Hilarious!

 

Various Software Development Methods

I've worked in, and think I understand, *at least* the following development methods:  

Method Dime Tour Does it work?
Waterfall actually I'm not sure about this one.  I've never had any company actually claim to be waterfall.  I think that is because "waterfall" has a negative connotation.  But some of us know when we've worked in a waterfall environment.  And it isn't all bad.  I've worked in waterfall environments that were wildly profitable and fun.  And I've seen waterfall projects be death marches too.  
Scrum/Agile I lump these together.  They are separate and unique but in every environment I've worked in the practitioners claim they practice both.  Works ok but practitioners sometimes adhere to the tenants like it was a religion.  It isn't, don't drink the Kool-Aid
XP another variant of agile Works for me.  
ICONIX I started out on this.  I guess this is roughly waterfall. Worked for me.  
Rational Unified Process ok, maybe I didn't understand this one when I was forced to use it.  Kinda waterfall-ish I guess.  We shipped software with it and were profitable, and that's the metric I use for a working method.  
Kanban a just-in-time method.  It attempts to focus developers on what is really important and on getting that done.  Runs the software dev process like a manager would from a catwalk on a manufacturing floor.   Works well when people understand more than the simple platitudes like "stop starting and start finishing" and really understand how "work works"
Microsoft Operations Framework I actually worked at a place that took MOF and made it into a software engineering process.  M$ even came in and did a lecture series on how to make it work.   Seemed ok...we made money...I got home at a decent hour most nights.  
Inception this seems to be the "old new thing" right now.  Really, it is part of RUP.  Inception is the first phase of software development where you try to broadly grasp the problem and approximate how much effort will be required.  So, there is an up-front focus on requirements gathering and design.  Um, that sounds kinda waterfall-ish to me (but don't tell an Inception Guru that).  I worked at a place that spent millions to bring in a bunch of Incepticons (Inception Consultants) to teach us how to do Inception right.  They were gone within a year, and so was Inception.  We went back to scrum/agile.  We would an entire sprint "incepting" only to realize by the end of a release that the product looked nothing like what we incepted.  To save face in the wake of wasting millions, management decided that we should use "just-in-time inception" instead, which is a mini-inception at the beginning of each sprint. Um, I call that "basic planning".   Complete failure if you follow the religious dogma.
DevOps a lot like kanban but with an emphasis on developers working with ops people to deliver more supportable products.  I'm actually going somewhere with this post...it's going to be the first of many posts on DevOps and how to move towards DevOps.    This is a game-changer.  

So, which one is the best?  

Any of these processes can work if practiced properly.  I have no affinity to any of them (except DevOps which I'm starting to really like).  They each have their good and bad points.  I can work and succeed in any of them.  Or fail in any of them.  

I would NEVER make any of these statements though:

  • "Our project succeeded because we finally realized the truths in <insert software development method here> and practiced it.  "
  • "We failed to meet our objectives because we didn't follow <insert software development method here> the way we should have.  "

There are common denominators that each process has.  Reliance on structure, process, and measurement for example.  But this isn't radical.  

HPCs (Highly Paid Consultants)

If you are looking for a new method, maybe because your software team is not as productive as you feel it should be, you should be aware that you'll find lots of HPCs who will want to sell you their expertise in a given method.  These shills will ALWAYS exhibit these characteristics:  

  • "If you adopt this approach then all of your development problems will go away. " 
  • "We are consultants that would love to come in and train your people on The Way.  We don't come cheap but you can't afford not to learn from our experience."
  • "Every other process is a fraud and will never work.  We have lists of how our process is better than whatever process you currently use.  Use this one to ensure success."
  • The "Who We Are" slide of their deck is always a mugshot of each of the two HPCs giving the presentation.  The first HPC is in the process of a deep belly laugh.  This connotes how they will make your failing projects a joy to work on.  The second HPC mugshot is a contemplative pose..."we have wisdom."  
  • One of the slides will tell you how they've been training The Way for the past x years with success.  They won't list specific companies or software, but they'll tout their successes with both small and large firms.  They don't know how to code in any specific language or domain, but they can manage the process.  They'll have lots of photos of their teams huddled around kanban boards collaborating over a bunch of color-coded Post-It notes, trying to determine what they are going to fix next...world hunger or peace?  They'll conveniently "forget" to mention how this collaboration is going to work with offshoring.  
  • They'll always provide you with lots of swag.  It may be "planning poker" cards with their logo on it, a board game they devised to use in their training classes, or bags with catchy logos like "Stop Starting...Start Finishing".  

All of this is a complete waste of time and money.  Just my two cents.  A HPC cannot possibly know your business domain, your software, or your people.  To assume that what worked previously for the HPC will work again for you is very Cargo Cult-ish.  

So, what does work?  

I don't like to complain without offering alternatives.  This is what I feel are the most important concepts that any method should espouse:

  • Never shackle people.  If your framework commands, "Thy shalt have daily standups where we discuss 'pulling cards from the left' (Kanban)", but the team likes to do Scrum-style standups, then allow people some latitude.  Let the team alter the framework to be assistive, not restrictive.  
  • No multi-tasking.  I call multi-tasking "faulty-tasking", because it DOES NOT WORK.  Human beings are not computers, they cannot multi-task.  Humans, instead, can context switch.  A developer can work on one and only one piece of code at a time.  Allow your IT people to complete a task, do not make them context switch.  
  • Don't rely too much on metrics.  Burn down charts and kanban cards and points are nice, but the focus should be on customer satisfaction.  
  • Don't spend millions on HPCs or special "Kanban TVs" for each team room.  It annoys people when large sums of money are spent on this nonsense yet raises were non-existent the last 2 years.  
  • Ensure everyone is allowed to participate.  Too often I've seen cases where a new method was instituted for the onshore workers, but not for the offshore guys...they aren't important after all.  
  • Practice MBWA.  Management By Wandering Around was practiced by HP back in the 1970's.  It's simple...managers get out of their offices and visit employees randomly, staying quiet and observing.  Good managers will quickly determine lots of little things that are impeding process and can remove those barriers.  
  • Consider adopting a new framework every other year on a trial basis.  Have your people learn the new method and bring it in-house.  Then do retrospectives often to see what is working and what should be discarded.  It's always a good idea to try new things.  I never thought software development management theory could improve but then I started to learn DevOps and realized that it in fact does contain a lot of useful concepts that are obvious, but under-employed by most organizations.  

Over the course of my career I've seen the companies I've worked for bring in lots of management frameworks and paradigms to try to solve what is wrong with their IT departments.  All of these "tools" have a bit of truth and value, but I hold that you can't solve a people problem with a slick framework.  The software development process is broken because fundamentally it is a people problem, not a process problem.  There are patterns and anti-patterns that I list above that I've seen work universally.  Don't be dogmatic about any process and don't be part of a Cargo Cult.  


You have just read nodetitle on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

The Sunk Cost Trap

"We want you to float on Project Automation and get back to us in a week with your thoughts.  We are a little concerned that this project has cost us $14 million and 14 months.  It is scheduled for code freeze in 4 months and we have yet to see a working proof-of-concept."  The worry was audible in the VP's voice.  
 
And this is how it starts.  I both relish and fear these moments.  On one hand I love coming in and saving a failing project, on the other hand, you never win and friends doing it.  You see, there was a customer perception that our enterprise software required too much manual intervention and Project Automation (PA) was meant to relieve some of that burden.  It was a contractual demand from our largest customers.  We either succeeded or we went out of business.  
 
I quickly began spending my days in the PA team room, at first absorbing what I was hearing, then asking questions, then offering suggestions.  When you get to that last stage the reticence is noticeable from your co-workers.  Within two weeks it was clear to me that most of the team viewed this as a death march project.  The team knew key aspects needed to change if it was to be a success but everyone was hesitant to make those changes knowing that there was only 4 months left.  Left alone, inertia guaranteed this project would fail.  
 
Within three weeks I had a working list of features that, as implemented, would NOT meet customer requirements regardless of time or money spent at this point.  The approach was wrong.  I also worked out a better approach that would meet the requirements, albeit scaled back for Version 1, and could be coded-up and unit tested within the remaining 3 months.  Granted, this would require some serious hunkering down and probably some OT, but we would show success.  
 
It took another week until I could convince the entire team that the only way they could show success was to ignore the sunk costs and change direction.  Within that week I had a somewhat-working proof-of-concept that I could demo and the team was won over.  It took another week to convince management to ignore the sunk cost and change direction.  Management was not happy about time and money being flushed away, but it was the only way to show success.  
 
We delivered Project Automation about a month late and with scaled-back requirements.  In the enterprise software development space, that's an unqualified success!  And we worked almost no overtime (I strongly believe in work/life balance).  By the time the release was GA'd we already had a service pack to address the missing requirements and even added a few new features that we realized were lacking after usability testing.  Customers were thrilled.  
"We want you to float on 'Project Automation' and get back to us in a week with your thoughts.  We are a little concerned that this project has cost us $14 million, 140 man-years, and 14 months of time.  It is scheduled for code freeze in 4 months and we have yet to see a working proof-of-concept."  If 'worry' has a sound it was audible in the VP's voice that day.  
 
And this is how it starts.  I both relish and fear these moments.  The moment that I am asked to work on an impossible project.  On one hand I love coming in late and saving a failing project, on the other hand, I never win friends doing it.  And I've failed at one of these opportunities.  That's not braggadocio.  There were times I went days without sleep to get to success.  It's just my nature, and I'm neither the only one nor the exception to the rule.  
 
Project Automation (PA) was critical to the future of our software.  You see, there was a customer perception that our enterprise software required too much manual intervention and PA was meant to relieve some of that burden.  It was a contractual demand from our largest customers.  We either succeeded at PA or we went out-of-business.  The details of what PA did are irrelevant to this story.  Just understand that it was critical.  
 
I wasn't the first person asked to help out with PA.  Like I said, there are lots of people far better than me.  For months management tried to get some control.  They tried the obvious things...shorter sprints, a move from scrum to kanban, enforced pair programming, more unit testing, less unit testing.  Nothing worked.  Those obvious things, while often helpful, will not turn around a failing project.  I know that, management sometimes does not.  
 
I quickly began spending my days in the PA team room, at first only listening.  Soon I began asking questions, then later offering suggestions.  When you get to that last stage the reticence is noticeable from your co-workers.  Within two weeks it was clear to me that most of the team viewed this as a death march project.  The team knew key components needed to change if it was to be a success but everyone was hesitant to make those changes knowing that there were only 4 months left.  Left alone, inertia guaranteed this project would fail.  
 
Within three weeks I had a working list of features that, as implemented, would NOT meet customer requirements regardless of time or money.  The approach was wrong.  I worked out a better approach that would meet the requirements, albeit scaled back, and could be coded-up and unit tested within the remaining 3 months.  Granted, this would require some serious hunkering down, but we would show success.  
 
It took another week until I could convince the entire team that the only way they could show success was to ignore the sunk costs and change direction.  Within that week I had a somewhat-working proof-of-concept that I demo'd and the team was won over.  It took another week to convince management to ignore the sunk cost and change direction.  Management was not happy about time and money being flushed away, but it was the only way to show success.  More out of desperation for their failing business than faith in my ideas, they acquiesced.  The PA team would change direction and do it my way.  
 
We delivered Project Automation about a month late and with scaled-back requirements.  In the enterprise software development space, that's an unqualified success!  And we worked almost no overtime (I strongly believe in work/life balance...and it was NHL playoff time too).  By the time the release was GA'd we already had a service pack to address the missing requirements and even added a few new features that we realized were lacking after usability testing.  Customers were thrilled.  We remain(ed) in business.  

Why did this project succeed?  It wasn't because I'm some kind of architect-superstar.  We know that's not true.  It was a success solely because I could convince others to ignore the sunk costs, especially so late in the development cycle, and switch focus to something else that would work instead.   

The Sunk Cost Trap

I feel I'm constantly explaining the sunk cost trap to project and product managers.  Recognizing a sunk cost trap when you see it in your software projects can save your company.  

A sunk cost in accounting parlance is a cost that has already been incurred and can't be recovered.  A sunk cost is not necessarily bad if it is a cost that can yield a positive return in the future.  The sunk cost trap arises when people cannot see that an investment will not ever show a return, yet they continue to waste resources pursuing it.  

There are lots of pithy aphorisms that deal with the sunk cost trap that you might've heard:  

  • "When you first find yourself in a hole, stop digging."
  • "Your first loss is your best loss."  
  • "Don't throw good money after bad."  
  • "Let bygones be bygones."  

Each of these aphorisms is conveying the message, more or less, not to fall for the sunk cost trap.  Yet people fall for this fallacy all the time.  We have the mistaken belief...it is really very irrational...that we've already committed a lot of resources to a given endeavor and we are past the point of no return.  We must continue with the chosen path and see it through.  The sunk cost trap, like all fallacies, can cost you and your company a lot of money when not identified.  

It is important as a software architect to not be a victim of fallacies.  This may even be the most important part of a software architect's job description.  

In your everyday life you probably fall for the sunk cost trap more than you realize.  Have you ever finished watching a two-hour movie that you were not enjoying after the first half hour?  You already had thirty minutes invested in it right?  It might get better so you keep watching.  Or you've finished a terrible dinner at a posh restaurant because you aren't the type to complain and walk out without paying.  If you're going to pay anyway, why not finish the meal?  Or you hold onto a bad stock in your portfolio because it is down too much to sell now, even though you'd never buy more of it today.  

Try to recognize fallacious activity. If you suddenly realize that a bad decision was made and it is preventing you from attaining your goals, and a better alternative exists, then you need to ensure that you change.  The truly good architects will do this...the mediocre architects will focus on the sunk costs and will try, come hell or high water, to make a subpar solution work at any cost.  

Prevention and Cure

Here are some tricks that I use to prevent sunk cost traps in my projects and overcome common objections when they do arise:   

  • "Fail early instead of fail late".  I preach this.  Never embark on a 6 month (or longer) project.  Instead, pare down the scope until you can have a proof-of-concept (POC) deliverable in a few sprints.  In your sprint retrospectives you should be evaluating whether or not the POC is working and if there are any alternatives that may be better.  POCs are really just "science projects" but many people are not willing to deviate after a few sprints due to the sunk cost trap.  I just finished a very lengthy blog series on various POCs I did using NoSQL solutions.  As soon as we found an issue that caused a given product to be sub-optimal for our needs we dropped that POC immediately.  We did not continue to throw good money at it just because we already wrote a few hundred lines of code.  I dropped a couple NoSQL POCs after just a few hours because of fundamental flaws I saw in the products given our requirements.  
  • Understand lost opportunity costs.  By continuing to invest in something that is sure to fail you lose TWICE.  You are flushing money down the toilet on the existing solution and you can't properly invest in an alternative that is more promising (that is the "lost opportunity cost").  Management fails to understand opportunity costs constantly.  For instance, senior, strategy-minded architects are forced to perform support tasks because there is a shortage of support personnel.  The opportunity cost is the senior architect cannot think about how to relieve the technical debt that causes inflated support costs in the first place.  
  • "This was a learning cost, albeit an expensive one".  At a minimum, make your mistakes educational so they won't happen again.  

For years after Project Automation I still had managers griping about the $14 million "learning cost" we spent before we changed direction.  This is entirely the wrong thing to focus on.  It isn't that we wasted $14 million, it is that we are still in business and are profitable.  I can assure you that we never had another fiasco like PA again.  Of course we made sure we had better project governance, but most importantly we recognized when the sunk cost trap was causing us to not do the right things.  

Metrics That Count (and it ain't "points")

I've lived through my fair share of software productivity and management frameworks.  Kanban, agile, scrum, XP, waterfall...there's probably more that I'm trying to subconsciously suppress as I write this.  Each of these is concerned in some respect with metrics.  How do we evaluate how much work is getting done and who is doing it?  How do we use metrics to improve "cycle times"?  How do we improve "burn down"?  How can we use these metrics at performance review time?  Well, IMHO, none of these "metrics" really matter.  What matters is shipped software, delighted customers, rising stock prices, and stress-free employees who get to go home at the end of the day and spend quality time with their spouses, significant others, and/or kids.  Nothing else matters.  

Management thinks it needs hard numbers and metrics to determine if the program is meeting its software development goals.  It also needs numbers to determine which developers are meeting expectations and which are unsatisfactory.  One problem is that software development is not assembly line work.  In a Toyota factory, management has various mechanisms to determine efficiency of individuals.  Number of defects, speed of the line, number of work stations mastered, the ability to train new employees, etc etc.  

Art vs Science

Software development is *not* assembly line work, no matter what new language or Big Data system the cool kids are all using.  Software development is more "art" than "science".  And management of "art", with its inherent lack of metrics, is so much harder to do than managing something with defined metrics and "numbers"...something like science or math.  

Think I'm exagerrating?  Do you think Pope Julius II evaluated Michelangelo based on the number of cherubs he painted on the ceiling of the Sistine Chapel everyday?  It's true that they argued over the scope of the work and budget, but the Pope never tried to evaluate The Master based on some concocted metric. 

There is so much in software development that simply cannot be measured up-front.  We generally call these things the "non-functional requirements."  Some shops call them "-ilities".  Performance is generally considered a non-functional requirement.  We all try very hard to evaluate the performance of our software before it ships, often using tools such as LoadRunner.  But more often than not we find that we have not met the necessary performance metrics once the software is in the customer's hands.  So, how do you measure a performance metric early?  You really can't.  So, do we ding the team or individual for failing this metric? 

The only metric that matters

... in software development is working, released features that a customer wants.  If the feature has not shipped then you get zero credit.  There is no A for Effort.  Even if you are 80% feature-complete, you get no credit.  If you shipped it but the customer doesn't like it, you get no credit either.  I hear developers complain that it isn't their fault that the product failed...the requirements from the analysts were wrong and the developers merely implemented the requirements as given.  I appreciate that argument, and I feel your pain, but the only metric that counts is a happy customer.  When your company goes bankrupt because your product failed because of bad requirements, I'm not sure your mortgage company is going to care.  

Other Metrics

There are lots of metrics management uses to evaluate us.  Here are a few and my rebuttal as to why they don't work for project evaluation:

  • Tickets closed:  I've worked at shops where each branch of code needed its own ticket for check-in purposes.  And we always had 4 supported versions/branches open at any time.  So a given bug may need 4 tickets.  That's called "juking the stats."

  • Lines of code written:  so now we are incentivizing people to write more code instead of elegant, short-and-sweet, supportable code.  More lines of code = more bugs.  

There are two ways to design a system: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies

  • Story Points:  A quick google search for "what is a story point?" yielded this article which pretty much concludes that you shouldn't use story points for metrics.  Oops.  
  • Velocity:  this supposedly shows management the "rate of progress" in our development.  In a perfect world our velocity will improve when we develop new tools that help us work smarter and faster, such as automation.  But many times velocity is merely going up because developers are incentivized to make the velocity improve and they do this the simplest way possible...cut corners.  
  • Code Test Coverage:  there are lots of tools that will analyze how many lines of code you have that have no unit tests.  I covered this in my blog post Paradox of Unit Testing?.  This leads to people juking the stats again...writing a bunch of tests to make the code coverage analysis tool happy.  
  • Unit Tests Written:  see nodetitle again.  I have worked with people who have refused to add new features to their code because there were too many unit tests that would need to be rewritten.  

The last two are the WORST offenders.  Most developers realize that lines of code, points, and tickets closed are ridiculous metrics, but many otherwise thoughtful developers fall for static code analysis and unit tests.  I've seen whole teams spend entire sprints writing unit tests for code that was 5 years old with no reported bugs because there was no test coverage.  It sucks the life out of the product and the team.  

I once tried to remedy this situation, and I hate to admit this, by merely adding empty test bodies to get the metrics acceptable.  And I've seen lots of people merely comment out broken tests to avoid getting weenied for that.  

Why do we rely on metrics?

Numbers suggest control.  Management likes control.  But this is totally illusory.  I've worked on teams where every sprint had some task called something like "refactor the Widget interface settings" that was assigned 15 points.  If the team had a bad sprint they merely claimed these points to make the numbers like good.  No work was ever really getting done and management had no idea.  That same team, after a 12 month release cycle, had ZERO features to contribute to the product's release.  Management was not happy.  But every sprint showed progress and burndown.  

Heisenberg and Perverse Incentives

When something is measured too much then the measurement itself will skew the system under measurement.  Loosely, this is known as the Heisenberg Uncertainty Principle.  I've worked on teams where there was an over-reliance on points as the metric to determine productivity.  People knew they were being measured and they geared their activities to those things that could generate the most points.  This usually meant pulling simple cards with small points that were low-risk.  The more important, higher point but longer duration "architecture" cards were never worked on.  They were too risky, you either got all of the points, or none of them.  

Summary

I'm sorry about this long rant on software development metrics.  Every team is unique and will determine how best to structure itself for optimal efficiency.  So many of these metrics are an effort to shoe-horn the team into a structure that management is comfortable with, even if that is forsaking the goals of the program.  Let the team figure out what metrics it needs.  Management's only metric of concern should be shipped features.  Nothing else matters.  When management evaluates its development teams on that metric I believe you will see teams that are super-productive because they are no longer juking the stats and wasting time on anything that is not leading them to more shipped features.  

But nothing is going to change.  It is too risky for management to not have a solid metric in which to evaluate us and our products.  Very depressing.   


You have just read "Metrics That Count" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.  

Pages

Subscribe to RSS - Management