Agile Fatale – Human Nature

“In fact, I am surprised how little improvement there has been in human evolution. Oh, there has been technical advancement, but, how little man himself has changed.” – Khan to Captain Kirk in Star Trek Episode “Space Seed”.

I was discussing a recent a survey from Version One with a colleague, and we noticed an interesting statistic — who champions Agile at their companies.  To “champion” means to advocate; the advocate would introduce Agile, push for it, bring in tools etc.  The number that came up was astounding – 63% of  management is the Agile champion at companies according to this survey.

agilev1essay

Why would I be surprised that it was management, and not the developers, who are the champions?  Honestly, these days, I am not.  Agile is now seen as almost exclusively a management technique and not an engineering technique by the industry.

Agile is not developers methodology anymore, its managements — on the social scale.

In the old days, at the start, this wasn’t true.  In fact at a shadow level on projects this still isn’t true — there are still scopes of “agile” developers use, naturally, to create software that is still a bit out of purveyance of management because there is no way to track these activities, yet.

Agile encompasses a lot of things — and management is part of it.  But management as developers intended it was meant to be self management to get their own work done.  We used to use XPlanner, for instance — and the PMs didn’t care.  Now middle management controls and even enters what is to be in the modern XPlanners — the scope has been changed.  Less useful for developers, more useful for middle management and up.  Still we, as developers, need a way to manage our work.  So when you walk by you’ll see instances of Sublime Text or Text Mate all running, in their documents the creative process of  managing their work.

In the early days of “agile” — mid to late 90’s for me — we all thought there would be a democratization of software building.  Less management, more developer’s taking on roles.  It happened for a while, but Agile morphed back to almost-waterfall when the value to our tracking became apparent to management.   It’s just reality.

Reality – Agile Ignores Human Nature

Evaluations

The reality is, someone is responsible for the work that gets done.  The reality is, people are evaluated on an individual basis and not as a group.  People get hired of fired based on these judgements  not teams. Agile can’t circumvent this reality.  Agile proponents made the mistake of ignoring human nature.  Management needs indicators to judge people and products on — and that’s what’s there.

Estimates

The impact of PEOPLE is why the estimate feedback loops we used to create in the tools like Rally are all but gone.  Someone would make a swag estimate for a story, then the team would get together and make a task estimate, then real time would be tracked against those estimates and next time supposedly the process of estimation and building would get better.  But this never happened properly.  Fear would rule — because you could get fired for missing a target if HONEST.  At a lot of places I would see over estimation used as a weapon by developers to get more pay, or protect themselves.  Its all due to lack of trust, and that is due to human nature.

No one, still, believes estimates are at bet a 50% +/- activity.  That’s a sad reality.  There’s no way around it, because technology is a creative process and you can’t accurately always estimate it.

Natural Ways

If you’ll notice, a lot of Agile is almost waterfall-like now.  I’ve even seen charts from internal accounting departments putting cost centers on phases of “agile” and “waterfall” — equivocating the two.  There we are again with scope of management.  And you know — a release schedule at a management level IS waterfall, because its natural to them.

Still though we sit in meetings with management where they play the old Microsoft Project game of  microtasking.

Fear of not meeting a deadline drives this.  Again — it is folly.  Yet we still play with these rituals.

The Next Agile

So Agile as we talk about it has two scopes in software: development and management.  Right now we are on a development downtick and a management uptick.

Management is following up on what is already there because they are concerned with scale and capital — and neither of these matter until there is a product or an idea of a product first.  That’s OK.  Even though I have not really seen any improvement in management the whole time.   Management should really concern itself with culture — dealing with making human nature a strength in their organization.  I have not seen any progress in this though, sadly.  There are some outstanding individuals but, I know, I know — what do I expect.  A process doesn’t make people better, and training doesn’t make them ethical.  Etc.

Developers have contributed the most to the process: inventing better tools like continuous integration, automated rollouts, test based designs or test suites at all levels.  In my opinion, that’s what has enabled Agile to even move forward — not better management, better tools.

So that’s my focus.  I still have to fight the human nature battles — project scope changes, contracts, personalities etc.  But the tools, the TOOLS, are the next great step all the time.  There is no key to changing human nature,  just tools.

 

IE8 Not Loading BackboneJS/LessCSS Application

Kinda funny — we have a new application and none of us were testing it in IE8!  Sound familiar? 🙂

The application wouldn’t load at all, not at all, in IE8. Debugging was tedious with the IE8 debug tool.

Our specs are pretty up to date but IE8 is still on the table for our customers.  The headless testing isn’t set up with any IE instances to cover this either. The UI piece is Backbone, Require, Underscore, JQuery, and Less.  All tried javascript technologies.

Fix the Tests
First, I noticed none of the QUnits ran in IE8.  So tracing it back, I noticed there was a library someone had put into the applications Require configuration, but not in the test config — and that did the trick.  For whatever reason it worked in the other browsers.

Fix the Application
Next, the application wasn’t loading at all in IE8 at runtime.  I had to do three more things to get it working:

1.  Move a console.log patch to load with the application — IE8 can’t do console.log unless its debugging and developers were putting them all over in the code. This technique is well documented on StackOverflow and other sites. Looks like this:

//mock_log_console.js
if (typeof ( console ) !== 'undefined' && console != null) {

} else {
    console = {
       log:function () {
           //alert("log"); //if you want, I usually do not
       },
       warn:function () {
       },
       error:function () {
    }
  }
}

Make sure this script is called before everything else in your index.html or whatever:

<script type="text/javascript" src="mock_log_console.js"></script>

2. Change the doctype. We had

<!DOCTYPE html>

but it was causing IE8 not to load the page, so I put in this, that seemed to solve it:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

3. IE8 was hating it when we set the type as follows:


<script type="application/javascript"></script>

So “type=’application/javascript’ “had to come out — nothing set now at all.

Closing
Debugging and fixing consisted of:

  1. Making sure the right libraries got loaded.
  2. Checking the syntax compliancy of the code to IE8 specs — for instnace, console.log(), DOCTYPE and acceptable script properties.
  3. A lot of elbow grease.

On other environments, Less is already compiled — but continues to be a problem loading slow and needed refreshes for IE8 when resolving at runtime.  We decided there was nothing we could do to solve this, except point at optimized CSS files at the right time.  There are ways of compiling everything but what you are woking on too.

Again, elbow grease.  Who ever said this shit gets any easier?

Simple Script to Run QUnit in All Browsers Locally

Before you check in you should always run a full build.  If you have  javascript — you should run all your tests locally first before the build.  This is because most automated test systems like JSCover or PhantomJS only use Firefox or WebKit libraries. And most shops do not have access to the full test-in-all-browsers system like BrowserStackAPI with Bunyip yet.  Someday we’ll be doing that, but until then, run one of these (on Windows — Mac/Linux are easy enough).  Just make sure to have a web server running, serving up the QUnit code.  In my case I use HFS Http Server pointing at the source directory (tests and src).


REM this will start up 4 parallel browser instances and run the QUnit tests.

start "Firefox" "<full path to>\firefox.exe" http://localhost:88/test/js/qunit-require-tests.html?noglobals=true

start "IE" "<full path to>\Internet Explorer\iexplore.exe" http://localhost:88/test/js/qunit-require-tests.html?noglobals=true

start "Safari" "<full path to>\safari.exe" http://localhost:88/test/js/qunit-require-tests.html?noglobals=true

start "Chrome" "<full path to>\chrome.exe" http://localhost:88/test/js/qunit-require-tests.html?noglobals=true

REM end code

How about a Grand Piano Application that let’s a Novice Play Like Mozart?

Route the Warp Drives to the Food Replicators!

Seriously.

I was reading an article somewhere about how “we don’t need data analysts anymore, we just need tools to do what the analysts do.  I’m a developer for a living, and at times have to do data analysis and let’s see we want a tool, maybe a nice little button that will let Shirley Temple:

  • Do cubes   . . . what what’s a cube
  • Do factor analysis
  • How about some laymen second derivatives?
  • Square Root Transformations for Polticial Science majors
  • Show me your item nonresponse while your at it
  • What, the fuck, ever

Again, seriously.  Do we have to go back to that again?  The rational Rose/WebSphere Designer  “management and BA’s just put in your objects and an application comes out” bullshit again?

The industry fails, over and over again, to understand progress.  Whenever a tool is invented, that is NOT the end.  It still takes knowledge and hard work to understand how to use something.  And, once that is done, something newer, better, and more complex will be build on top of it.

Progress is advancement towards simplifying complexity; but that does not mean complexity or the brains to understand it ever go away.

It struck me as I looked to all these new San Francisco dot-com ideas being put up with tools like NodeJS and the new UI frameworks that these little apps are nothing more than Palm Pilot executables moved to the web in a different technology.  A todo list in HTML 5?  GEE!  Anything new about that list?  Nope.

If we developers are going to be using words about ourselves like “craftsmen” then maybe its high time we look at what that word *really* means:  hard work and innovation.

Service busses are boring and dry and don’t write themselves.  MVC frameworks have been evolving over years and years and don’t invent themselves.  And for every advancement, there’s yet another level of complexity to be uncovered and understood.  Businesses that think “well, this is good enough” aren’t around anymore.  Evolve, or go extinct.

Reroute Your Production Static JavaScript To Debug DeMinified

We deploy our javascript code minified and obfuscated to most of our environments.  Only at the localhost level does anyone really see the full form of the code.  Sometimes this creates difficult debug scenarios; environments vary from localhost, to dev, to test, to production.  Fortunately there is a very easy was to reroute the javascript requests on a site to the source code on your local machine using the proxy tool Fiddler.

I’ve tried a few other tools like Wireshark (which is better for complete packet monitoring on a network card), Charles (a great commercial product),  and Burp (which works well).  But this method is as about as simple as can be and almost every developer runs Fiddler now days.

There are a couple of prerequisites:

  1. You have the source code.  If not, you can still de-minify the code with a Firebug un-minify plugin but unfortunately the variables will remain obfuscated.
  2. Have FireFox with Firebug installed so that you can set breakpoints, etc.

Ok here’s what you do:

  1. Download and install Fiddler (from fiddler.com).
  2. Start up Fiddler.
  3. Hit your site in Firefox.  At this point, when you pull your javascript files across they will be minified/obfuscated.
  4. Click on the AutoResponder tab on Fiddler.  Make sure auto responses is enabled.
  5. No you can debug, either:
  6. A single file by adding a rule, putting in a URL (I choose a realtive one like /resources/js/myfile.js) then adding the local file path to the rule (like c:\js\myfile.js).
  7. Many files at once in a directory.  Go to your local directory and drag it directly into the Autoresponder rule window in Fiddler.  It will enter all the files!

Now you can make local changes, debug, whatever you need.  To disable the routing simply shut down Fiddler or uncheck the enable autoresponse checkbox.

Maven 3 Multithreading

Maven 3 multithreading works very well – as long as your plugins support it!  It’s still in an experimental mode but we are using it on our project to cut our build times in half.  And its much easier to implement than Ant’s parallel thread functionality.    On another project with Ant I multithreaded our build  – would take out almost 25% with parallelizing ant builds.  Maven claims a 50% reduction and I believe it.

So . . .good news – it works great.

Bad news – not all plugins are set up for multithreading yet (from Maven), like sonar, so it can’t be used it in a full build process yet.  You could probably write a two-step process though; one multithreaded and one not.

Running

mvn -T 4 clean install –>ran our build with 4 total threads, cuts build time almost in half!

I have an 8-core machine, so if I run it with one thread per core:

mvn –T 1C clean install –> goes even faster, 8 threads on my machine and saved almost 75% of the time needed to build!

Here’s Maven’s documentation:

https://cwiki.apache.org/MAVEN/parallel-builds-in-maven-3.html

Apache HTTP File Compression

Here are the settings I use on an Apache HTTP Server to compress files automatically when served.  Most browsers have the tools internal to decompress the payloads once they get them.

Configuration

To do automatic file compression for test type files from the Apache HTTP server, in httpd.conf:
1. Uncomment this entry; the file is included by default in the Apache installation:

LoadModule deflate_module modules/mod_deflate.so

2. Add this configuration to the bottom of the file:

#Compression
#compress these types
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/json application/javascript application/xml

#filter out browsers that can't handle compression
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

#turn this logging on to test, but off for normal deployment
#DeflateFilterNote Input input_info
#DeflateFilterNote Output output_info
#DeflateFilterNote Ratio ratio_info
#LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
#CustomLog logs/deflate.log deflate

3. Restart the Apache HTTP server.

This will compress the static content files, bypassing images.  The browser will automatically decompress the files for the user.

Testing

To test your configuration, uncomment the logging section (restart) and hit the application a few times, you will see entries in logs/deflate.log. Make sure to comment the logging again so the file does not grow (restart).

Don’t Build the Maven Jar

I’m using the maven-assembly-plugin to build a zip for some static content in a maven module. But when I build the whole project I also end up getting a jar file for that module. To avoid this use one of two tricks –  either specify your build as a type “pom” or  tell the jar plugin to do nothing at all.

1.  POM type packaging

<groupId>"group ID"</groupId>
<artifactId>"artifact ID"</artifactId>
<version>1.0</version>
<packaging>pom</packaging>

2.  “Nulling” the jar plugin

<plugin>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>default-jar</id>
<phase>none</phase>
<configuration>
<finalName>nothing</finalName>
<classifier>nothing</classifier>
</configuration>
</execution>
</executions>
</plugin>

What Ports are Open in Windows?

I was installing a few different versions of Tomcat — to support legacy and new applications — on an existing server.  I needed to know what ports were available.

The windows command for doing this is:

netstat -a -no

You’ll get a list like this back:

IntelliJ and Mercurial Won’t Play Right?

Click to Enlarge

 

Well let’s see — you have Hg installed on your machine via TortoiseHg.  You are using IntelliJ.  You’ll notice the locations and options in each UI you use – command line, Tortoise right-click, tortoise UI, or IntelliJ menus depending on where you are — have almost zero parody.

I noticed that one of my IntelliJ Hg projects wasn’t recording changes in the IntelliJ UI.  Found out that IntelliJ needs a little setting after you open your Hg-based project to get everything running on all cylinders:

Go to File->Settings->VersionControl and enter your project directory and select Mercurial for your repository.  Do this for each project.

Then your IntelliJ should play nicely.