Software Implementation Decision Calculus

Here is the decision matrix I use when evaluating software purchasing, implementation, and deployment decisions

As a Microsoft Technology Center technical evangelist, and former CTO, I get asked every day:

What are your opinions on software package x?
We need a solution for ETL. Should we purchase a commercial software package or deploy some open-source project?
Should we use a first-party (native) Azure service, or the underlying open source code the Azure service is built on?

Basically, folks are asking me for my unbiased opinions on software purchasing and deployment decisions based on my experiences. I’m going to assume you’ve already determined the most critical decision points:

What are the requirements of your project? You certainly wouldn’t purchase an ETL tool if your requirement was a self-service data lake query tool.
What are the existing skill sets within your organization? If you are looking for a self-service analytics tool and your analysts have no knowledge of SQL, then you might want to start with a graphical querying tool.
What is your timeline, budget, etc? If you have zero available budget for software you certainly wouldn’t want to purchase a COTS solution.

Disclosures: I am a Microsoft employee. I tend to recommend first-party native Azure services for most data and analytics needs. But not because I’m paid to. Our services are really pretty good. I pride myself on always giving my unvarnished opinions and it has gotten me into trouble at times. I’m a technologist that prefers writing a few lines of python vs using a no-code/low-code tool because I value long-term flexibility over perceived short-term velocity gains when using these tools. I prefer writing a little SQL over having a point-and-click query writer.

So, let’s assume you know what type of software you need and you are evaluating what is available on the market. For every package and vendor, here are the key decision points I believe you should consider:

Does the software have an open-source or commercial license?
Who are the package’s reference customers?
Community: Does the software have a presence on StackOverflow and what is the state of the user community? Is there an open user forum?
Does the software run as a PaaS, SaaS, or containerized offering? Or will I need to deploy the solution on VMs and deal with backups, patching, security, etc?
What does Gartner and the other research advisories think?
What are the development paradigms? Is it customizable? Is there an API? Can I shell out to run a custom script? What is the learning curve?
Do a rapid prototype. Was your team comfortable using the package?

Let’s dive deeper into each of these questions.

Licensing Models/Open Source

Is the software open source (OSS)? If it isn’t, I strongly look elsewhere. Although I rarely look at the source code, sometimes you have to if you come across a bug and the vendor’s on-call support is weak. Grokking source code scares a lot of technologists. It should not. Many times I have coded a simple solution that throws errors and the software does not appear to be working as the documentation suggests. Looking at the source code is a quick way to confirm my assumptions.

Be careful! Not all OSS licenses are created equally. If you are an ISV it is critical you understand the OSS license of all software you deploy. Some OSS licenses will require YOU to also open source ALL of your IP as part of their terms. I’ve written about these concerns.

Some organizations would rather not deal with “self-supporting” OSS and would rather have a software vendor with commercial licenses that can be called at 2am when emergency issues arise (think Microsoft SQL Server). If this is your organization’s culture then certainly stick with commercially-licensed software. There’s even a hybrid model that is the best of both worlds you may want to consider: some companies will provide commercial support for OSS offerings. An example: Microsoft provides commercially-supported and cloud-hosted versions of many OSS offerings…including PostgreSQL. PostgreSQL is a well-known and well-liked OSS database, but commercial support offerings are difficult to find.

Reference Customers

ISVs will always provide a list of reference customers on their website, usually a little banner of their biggest customers. Here’s one I just scraped off an ISV’s website (I won’t disclose who):

What value am I getting from this data?

If these are reference customers then I want to speak to a user at one of these organizations, hopefully in my industry, and have a frank conversation about the software I’m about to purchase. In all of my years in IT I have never once, yet, had an ISV that would allow me to speak directly to a reference customer.

I wonder why.

I don’t want to read a testimonial or see a quote. I want to speak to an actual user and ask questions about their satisfaction, implementation hurdles, on-call support nightmares, etc.

While I’m surfing a vendor’s website I also like to look for spelling mistakes and similar issues that make me think the vendor does not value quality and user experience.

Stackoverflow

I always research software I’m not familiar with on stackoverflow. SO is the one website that most technologists can’t live without. If a software package has a weak presence on SO then I won’t deploy that tool. Period.

I like to spend at least an hour reading the top entries on SO for the tool. What are the quality of the questions? What are the recurring questions? Are users generally satisfied or are there a lot of comments that indicate the software is functionally incomplete?

Thanks to stackoverflow I haven’t written an original line of code in at least 5 years. I copy/paste all of my code from stackoverflow. My guess is most developers do too!

Community Interest

Does the software package have a thriving user community? Does the vendor host user groups, sponsor meetups, or speak at industry conferences? Years ago you could gauge community interest by looking at the forums on the vendor’s website. I’m less interested in these forums since most discourse is occurring on stackoverflow these days. However, if I see a forum is “closed”, requires a subscription or paid support, or is heavily moderated…it raises my suspicions. Why should I pay a fee to have access to a forum where the users are primarily solving problems I have with the software?

One benefit of forums is participants tend to be more honest about their true feelings about the software and especially the support (or lack of it) team. This discourse is often eye-opening. Not all vendors will allow this much openness. Be wary of paywalled support forums.

PaaS, SaaS, containerization

In today’s world I don’t want to deploy software that requires my team to manage infrastructure. Minimally I expect the software to run in a container that I can customize and scale on Kubernetes. I don’t want to deal with backups and VM patching. I always prefer first and third party Azure offerings for these reasons. I want to focus on solving business problems, not operations problems.

Research Advisories’ Opinions

It’s always worthwhile to see what Gartner, Forrester, and the other industry analysts think of a software vendor and where it ranks versus its peers. Minimally, you will get a list of alternative software packages and vendors to research.

To be honest, I don’t put much faith in industry analyst opinions. In most cases the analysts have never actually used the software and are generating their opinions based on interview data and feedback from large corporate customers. Many of these big customers are, in fact, defending their purchasing decisions instead of objectively evaluating the software under review. And then of course there are the uncomfortable relationships between vendors and analysts. It’s not always clear who is funding a research report.

Development Concerns

Customizability: When I hit a limitation with the tool, how easy is it to “shell out” and build something custom? Can I do that with python or C# or am I locked in to the vendor’s bespoke scripting language which I now must learn?
APIs and scriptability: Can the software be operated via automation? If this software is going to participate in your DevOps pipelines it needs to be scriptable.
Training: Frankly, if I need training to use the software, I’m immediately concerned. I don’t need my team taking even more training. And, honestly, most developers don’t want to take more “tool-based” training. Tool-based knowledge is not aspirational for most developers. Most developers don’t want to learn another language, they want to learn another domain. Domain-based knowledge is what most technologists really want. Example: Most .NET developers don’t want to learn python, they want to learn what differentiates python from .NET, why it is the lingua franca of data science, and how they can use domain-based data science training to become better .NET developers.
What is more important: Fit for purpose vs fit for use? It’s the former. If I have a team of python developers, I don’t want to force a low-code/no-code solution onto them. It will feel constraining…like a toy, not a productivity tool.

I see this a lot…salesmen present the wrong tools to development teams. Don’t do this. Understand your audience.

Most developers just want to know if the tool is going to add more friction to their day.

Do a Rapid Prototype/Bake-Off

Most software vendors want to send you to free training. Training is great, but is not a substitute for hands-on experience. Demand a rapid prototype. Use the business problem you want to solve with this new software as the use case to do a 3 day rapid prototype. You won’t solve the business problem in 3 days, but you want your team to walk away with a better idea of “a day in the life” of using this software. Is the software cumbersome? Does it handle your team’s development concerns? What are the rough edges? A few pointers on rapid prototypes:

Don’t waste time installing or setting up the software. This is a “once and done” activity and is not something that is done daily.
Let the vendor know this is a rapid prototype and they are being evaluated on how well the software fits the needs of your team. This isn’t training. You are expecting results.
Tell the vendor you intend on running similar bake-offs with their competitors. The vendor likely knows the rough edges of their competition.
Conduct a retrospective after the RP. Is this a tool your team wants to use? Did it solve the problem it claims to solve?

How the Microsoft Technology Center can help

The Microsoft Technology Center helps customers make technology decisions. The MTC mandate is to be the Trusted Advisor for our customers. We do that by showing how data can add business value. The tech is easy, what’s hard is understanding the processes that work. We never make technology recommendations until we understand your business problem and the culture and capabilities of your team. (I’ve been told this is refreshing).

I recommend to every customer that they run a small-scale Rapid Prototype before making any technology decisions. Training is great, but there is no replacement for hands-on experience. At the MTC we advocate 3-day rapid prototypes. We know we can’t solve your business problem in a few days, but we want you to walk away with a better understanding of our best practices and processes, as well as how our software will fit with your team.

MTC architects are seasoned technology veterans, consultants, and former executives. We’ve used our tools for years, we’ve worked with our product teams to make them better, and we can evaluate which tools will work best for your unique team. We also know the tool is less important than understanding the underlying domain-knowledge, patterns, and processes that lead to successfully solving complex business problems.

Does that sound like a partner you can trust? Does “3 days” for a rapid prototype sound like a good investment of your time? Before you start your next technology project contact me on LI and let me show you a different approach to solving problems.

Are you convinced your data or cloud project will be a success?

Most companies aren’t. I have lots of experience with these projects. I speak at conferences, host hackathon events, and am a prolific open source contributor. I love helping companies with Data problems. If that sounds like someone you can trust, contact me.

Thanks for reading. If you found this interesting please subscribe to my blog.

Dave Wentzel 2021-08-31 CONTENT
Digital Transformation Architecture