DaveWentzel.com            All Things Data


On Points

Software estimation "points" were never meant to be applied universally in scrum and agile.  They were meant to help dysfunctional teams meet "goals."  Points solve the wrong problems, and solve them in a way that fosters distrust with management and customers.  There are better alternatives.  

The #NoEstimates Movement is Nuts

Most software developers hate giving estimates.  Who doesn't?  But now there is a movement to ditch estimation entirely (#NoEstimates) based on a bunch of faulty logic.  They've gone too far.  Estimates serve a legitimate purpose.  Those behind this movement are an embarrasment to those of us who want to see our clients and employers succeed.  

Benevolent Dictators

I'm currently working on a team of data architects.  We usually agree quickly on a given design for most problems, but occassionally we do not.  When there is disagreement, the question becomes whose design wins?  How should we determine the "winning" design?  If the debate drags on too long the team's velocity comes to a crawl.  

In other situations there is a designated "lead architect" whose design approach may not match what most other team members feel is correct.  But again, on-going debate causes velocity to suffer, tensions to flare, etc.  How do you solve these issues?  

How to Sell Agile to a Waterfall-ist or How Even I Learned How to Embrace Some Agile Concepts

"Insanity is doing the same thing over and over again and expecting different results."

--Albert Einstein

Some people are so change-averse they won't try something new even if the current "something" is recognized as a total failure by everyone.  This is my current situation regarding agile vs waterfall software development methodologies.  Here's the refrain:  "We've been developing the same software for 30 years using this same method before they even had a fancy term for it like 'waterfall' and we're not going to change now."  That's a powerful statement and anyone who wasn't pig-headed like me would've run for the hills as soon as it was uttered.  

Here are the problems from my vantage point:

MS VS Scrum 1.0

This is another in my series of posts on TFS Tips for the Data Professional.  Last week MS released Visual Studio Scrum 1.0.  This is a series of Work Item Types, Reports, and a SharePoint portal site that enable your team to utilize TFS in a scrum/agile environment by using scrum/agile terminology and processes (sprint,burndown, velocity, etc) instead of having to map your scrum concepts onto the default TFS schema (or extending it manually).  Here are some installation notes.  TFS already has the MSF Agile template which was always close to scrum, now we have something that is even closer.  The Sprint and Release Burndown reports ease what can be a PITA if you are not your team's full time scrum master.  What is especially nice is that you can migrate your existing TFS projects to Scrum 1.0 quite easily with the TFS Integration Platform.  Finally, if you've read a lot about scrum/agile but have never tried it, or have tried it with limited success, or just need a refresher, MS has an excellent Scrum Guidance Document that helps you to understand scrum concepts and map them directly to Scrum 1.0.  

Doing Database CIT via TFS

Continuous Integration Testing (CIT) is an agile concept where you are constantly testing your code checkins to ensure they build, compile, and have successful unit tests.  Notification of failures for smaller shops is usually just an email to an SCM who then tracks down what broke using the version control system and notifies that person.  In larger shops a special CIT machine will sit in the scrum room with a big red or green light indicating if the builds are passing CIT.  When it goes red someone usually steps up to determine cause. 

In most shops this CIT concept is a no-brainer for the application code (tools like Apache Ant and CruiseControl really help with the automation and scheduling) , but is rarely done for the database scripts.  One reason is that most applications have a database build process that doesn't lend itself to CIT-style processing easily (for instance, requiring db-level object diffs to determine what needs to be applied).  My database build process is more of an evolutionary process, so by definition it does lend itself to CIT and change quite nicely.  

This post is the next in a series of posts showing useful TFS tips for the data professional.  I'll cover how I created a very simple script that handles database CIT using TFS as the source control repository.  You could also plug in a different VCS quite easily. It uses vbscript and batch files but PowerShell or a compiled application would work just as well, the concept is the important thing.   

I have the actual files available here for download.  Here is the process:

A controlling batch file (called RunCIT.bat) is scheduled using AT.exe or similar scheduler, this handles the automation component.  Let's look at some of the interesting points of the batch file...

Lines 1 - 14...the batch file has a series of parameters that tell it which server to run the db scripts against, user/pwd, location of the logs, and the TFS connection string.  

It then builds a loop (line 19) and checks  "branches.txt" (a csv file) for a list of branches that we want to run CIT against.  In our environment we have at least 5 active code/release branches at any time so I want to run CIT against all branches.  Our SCM can change this file whenever a new branch is created or deprecated.

As a side note, if you've ever tried to do looping in a batch file, it's a real PITA.  If you've ever tried to expand your DOS variables within that loop you know it's almost impossible.  I found a nifty workaround to make this happen using "SETLOCAL ENABLEDELAYEDEXPANSION" and alternative variable naming schemes.  See Lines 3 and 27 for examples. 

Within the loop we then run a series of tf.exe commands to Get Latest for the source code for that branch.  This means that your build machine needs to have Visual Studio loaded on it (unless someone knows how to get the tf.exe commands to run without it).  

The scripts for the given branch are then executed, first against a "net new" database and then a "ported" (upgraded) database, each is set up via manipulate_db.v-b-s.  I've found that naming a vbscript file as anything other than .vbs and then calling it with cscript/wscript is an easy way to bypass UAC on newer Windows OSs

Why run CIT against a "new" and "upgraded" database?

Unlike C# code, where I simply need to test if I compile and my unit tests succeed, a db build is a little more tricky because I need to be concerned about data history and upgrading data.  If I'm changing a UNIQUE INDEX from ColA,ColB to just ColA I can't just CREATE INDEX WITH DROP_EXISTING since it's possible an existing db will have violations.  I need to test that.  If I'm adding ColC to a table I need to remember to add that to the CREATE TABLE statement (for new databases) and an appropriate ALTER for upgraded dbs.  In each case the "new" and "upgraded" database take different codepaths through the sql scripts. 

So your upgraded or "ported" db should probably be a customer db with some substantial data.  You can even run against multiple upgraded dbs, perhaps one from each version of your software that you may have to upgrade. 

But the key is Line 73, GenerateDiscrepancyReport.vbs.  This compares netnew vs ported dbs and reports on missing (or schematically diferent) tables, views, columns, indexes discrepancies (different cluster keys, cols), and it will even report any data changes if you have defined system or master data tables that you ship with your db.  This is handled via 2 stored procs (here and here). 

Email is sent on success and failure with the logs attached. 

Happy Testing!!

Agile and Scrum tools for TFS

I just stumbled across some nifty tools for those of us trying to practice agile/scrum in a Visual Studio world.  

Subscribe to RSS - agile