10,000 Foot Developer

I was working with an “expert”. He was working on a major refactor that was going to break several hundred of our tests.

Now, probably the best way to do this with ANY repo would be either to branch and make the change, and then do the fixes and merge when it was all good — or just not check in at all. Just stay away from everyone else’s working code!!!!!

Well guess what — the expert made a big boo boo. HE WAS SUPPOSED TO BRANCH DAMMIT BUT DI’NT!!! He did the code change and then did his incremental check in, and proceeded to push ALL the test breakages to the main trunk!!! Well, we didn’t have a Jenkins job running on our trunk to check it (we used git svn, and the job ran on svn/Jenkins while we had our skunk works).

Seeing where this is going? Right. Now the team all checked out and rebased to the breakage. This cost us several hours as a team to resolve. The expert was quite sorry, but also blamed it on no build check. I’d have more sympathy if that particular person wasn’t the one who was the repo nazi for our team. Tsk tsk.

I also ran into an interesting situation.  This error had messed up a lot of my local changes.  Although I’m in the habit of stashing before I check out, I don’t build between checkout and applying the stash  — and then I usually stash pop in the prescribed manner which generally blows away any hope of reapplying the stash again.   Fortunately I had an OK stash I could use, and rolled back before these errored commits came in. Then I branched – which in effect made my branch the trunk while I waited for the fix.  Applied my stash into my branch and continued until the guy fixed what he had.   I could have branched his stuff, checked it in as a remote branch, and rolled the master back but I decided not to out of respect for him to fix his error.  There were some messy merges along the way with this, very time consuming.

There are some very important things to learn working with these distributed repos.

First, a build server is a good buffer for monstrous breakage. But . . . in this case we were all pulling quickly as we worked (me and another developer swapping code changes) so we weren’t waiting for the builds to check out. Do you wait for a good build before checking out? What ensures you didn’t get something in between the build and the result? Nothing . . . . even on a locked down TFS system some years ago we still ran into this problem.

Second, no matter what greater deity of code you may be, you are fallible. Check yourself over and over, more than a monkey looking for lice in its fur.

And third — distributed repos haven’t solved this problem at all.

A new workflow could be deduced from this:  build,stash, build,pull/rebase, build, stash pop,build.  hahaha  🙂   Also, we could all start getting that nervous Ctrl-S tick I have back from the 1990’s, but manifest in git stash over and over.  I do increment commits which require builds — but this behavior also raises the question: should we *even* increment commit if our local build doesn’t work? It could create an accident such as this.

And what if your build takes a LONG time?  Such as the TFS worksite — upwards of 25 minutes with integration tests!  Sweet Fancy Moses . . . .

I suppose one way to fix this would be to pull the revision from the last greatest build. Maybe something can be configured in Git or Hg to do that? Dunno.