DevOps
There is a lot being written about DevOps, most of it is not accurate. The biggest inaccuracy is that DevOps is a simple merging of your development group with your operations group...ie,"dev" and "ops". Nothing could be further from the truth. I am a huge proponent of DevOps, and if it is properly understood and implemented it can turn around a faltering IT organization. I have seen this and experienced it first hand at both large shops (120 developers/50 admins) and small ones (13 devs/3 ops guys).
DevOps is the melding of Development and Operations
Let's just get this one out of the way first. I see everywhere on the internet that DevOps is the merging of your developers and ops guys. This is a dangerous over-simplicatioin of DevOps. IMHO. Yes it is true that DevOps takes the best-of-breed approach of both worlds and makes them more compatible, but we aren't just redefining job roles.
I can think of few things worse than having my development staff also handle operations activities. That's a recipe for disaster. This misconception probably started with with small startups (or smaller groups in larger organizations) that didn't have the money or time to hire dedicated ops guys. The easiest solution is to have your developers eat their own dog food. But this isn't really DevOps. It's NoOps. It works great for NetFlix but might not work great for you. At small shops NoOps is a reality and a necessity when you don't have the headcount for anything more than getting the product shipped to a happy customer so you can generate revenue. Nothing wrong with that.
But at some point the jack-of-all-trades model won't scale anymore. Your product will become too big (in terms of code volume or customer volume) for everyone to wear every hat. It is a core premise of DevOps that their is a degree of cross-training and understanding of every role, but that does NOT mean every engineer should be doing every role.
Further, some people are better suited to be either a developer or a ops guy. This is basic division of labor. I don't want my smartest developer patching a system, just like I don't want my CEO taking out the garbage. It's not that developers are better than ops, or the CEO is more important than the janitor, it's just the reality that we want our people working on what they are best suited for.
Are devs and ops guys diametrically opposed?
I've worked at large ISVs where the ops guys spend all of their time keeping things running smoothly and the dev guys spend all of their time making changes for the next release. Change is the enemy of the former and the friend of the latter. Sales guys need whiz bang features and product management is tasked with having the devs deliver it. But after every release the ops guys are scrambling because the systems are no longer stable. Performance probably sucks again and web containers need hourly recycling. While product management is declaring victory with the release the ops guys are getting no sleep dealing with the new issues. Who can't forgive an ops guy for wanting fewer releases? But isn't that anti-agile?
At these large shops I've seen entire management teams that bicker over whether the next release should even be released, and when it should be released, and what it will do to system stability. At the last minute risky features are removed which causes more risk because this new configuration was never tested. All of this friction ensures that no real work gets done. New features aren't being developed and operational tasks aren't being completed. That is the core of DevOps...stop the bickering and let's ensure the teams are working together toward the same goal -- profitability.
Lastly, there is far more to DevOps than just devs and ops guys. QA is a part of it. Change review boards need to be integrated into DevOps. Product management, scrum masters, sales...everyone.
So, What is DevOps?
For me, in a nutshell, it has each group try to think and behave like the other group and instill best practices in each other. That begs the question, "what are the best practices?"
DevOps embraces tools, methods, and technologies, that can meld your operations and development staff so they work together better. Some examples:
| Example Behavior | How DevOps Helps |
|---|---|
| Your developers use Visual Studio for stored procedure development but your DBAs use management studio. | Get your Ops people using VS. |
| Your developers have a build/deploy process for their code but your DBAs don't follow SCM for their processes. | Your DBAs should have the same build/deploy process. Visual Studio makes it simple to keep operational scripts under the same source control as everything else, or you could use a third party build/deploy tool like MD3. Everyone follows the same processes. |
| Your developers use Git, your ops people use nothing, and your QA folks use a separate Git repository, or SVN. The Change Review Board uses a file share with neither versioning nor permissioning. | Everyone uses the same repository and follow the same policies. |
| Your developers add indexes based on their dev findings. Your DBAs drop them and add new ones with included columns or whatever. | DBAs need to educate developers as to why their indexing choices are not working and developers need to LISTEN. |
| Develops code against systems that aren't properly patched, have varying versions of the application stack installed, use ancient 32 bit systems that were prod 10 years ago. | As part of a build deployment consider deploying on a new VM with the approved stack. This ensures that if a developer needs to upgrade WebSphere that it is also upgraded in the VM build. Nowadays this is easy to script, especially if you use something like Aptitude. This is called "infrastructure-as-code" or "software-defined environments." If your ops guys tell you their architecture is too complex to do this then you need to educate them that more things are moving to the cloud because of this mentality. It's not the 1970s anymore. |
| Your developers need to change a piece of infrastructure and there is nowhere to version control their script that an Ops person can use. For instance, you use SQL Replication but there are no scripts that a developer can alter to change an article and re-snapshot. Or the developer doesn't even know that replication is installed, therefore a table change they make breaks production. | Everything should be scripted and available to your developers. Good developers can help the Ops people do this. Good Ops people use the scripts for everything. No settings are changed via SSMS GUI. Everything is scripted. |
| Your Ops people ask your developers to automate a process and they code it in C# and your Ops people don't know C#. (Hint: your dev guys should've used PoSH and your Ops guys better learn it if they don't). | Everyone should use a scripting language like PoSh for these tasks. PoSh is simple for Ops people to learn to do a modicum of modifications, and it is very similar to C# so it should not be a huge learning curve for the dev people. |
| Non-prod deployments are totally automated with TeamCity or Jenkins, but when we move to prod then it is a manual process. | Everyone uses the same processes. More on this below. |
| Your developers use VersionOne for defect management. Your Ops guys use Jira. | Use the same tools for bug tracking. |
| Your developers pair with your BAs on new features. Since you are Agile you don't document the nitty gritty details of each business process. Your QA folks struggle to test it because they know nothing of what is not documented. | You could immediately state that all requirements be documented, or you could simple expand your pair to include a QA person that can then understand the decisions and possibly document them as a QA artifact. |
If this seems common-sensical then good for you. However, if you are a developer who couldn't care less about how your code is operationalized, then maybe you should learn more about DevOps. But, the above just scratches the surface of what DevOps is. In the next few blog posts I'll cover the far more important concept of "work" and how to manage it properly in a DevOps environment.
Even simpler, DevOps is an extension of agile where the ops guys get to participate in agile development processes too.
How Does DevOps Affect Existing Processes
Whenever a new software management method arrives on the scene it freaks people out. We all know certain parts of our software organization is broken but we don't want wholesale change either. Here is a list of items that are important to large ISVs and the impact of DevOps upon them:
- Change Management: if your people think your cm process is broken, then it probably is. If it is viewed as just a "process" where people know they are going to be harangued because they weren't consulted prior to the change, then the cm process really shows no value. Collaboration can fix this. So can automation. If you automate the change process then people have more confidence.
- Release Management: at large companies there is usually total automation for non-prod envs. Click a button in Jenkins and your build is deployed to your env. When ops guys move the change to prod the process is thrown out the window and there is manual intervention again. The rationale is that there needs to be backups and swapping of hardware, etc. The fact is if there are special "prod-only" processes then those should be baked into the automation process. Get the Ops guys to collaborate with the devs to make this a reality.
Summary
This post was my summary on the basics of DevOps. I clearly feel DevOps is not just another software management method that we can jam down the throats of our people. It does contain a lot of new and controversial ideas, like having your Dev and Ops teams work together. This is tough to do in highly-siloed organizations. But in my experience DevOps works VERY well. In the next post I'll cover some additional tenants of DevOps that you don't read about much that can also help you on your DevOps journey.
You have just read "[[DevOps]]" on davewentzel.com. If you found this useful please feel free to subscribe to the RSS feed.
Dave Wentzel CONTENT
other management devops