Better Software Development

Martin Fowlers Pattern talk at #jaoo made me make this 12 patterns of development song.

Have fun

Michael

On the twelfth day of Development,
my true dev sent to me
Twelve bridges bridging,
Eleven factories making,
Ten observers observing,
Nine builders building,
Eight visitors a-visiting,
Seven composites composing,
Six iterarators iterating,
Five golden states,
Four calling proxies,
Three nice adaptors,
Two commands commanding,
And a singleton in a pair tree!

Guy Steeles Talk on Fortress reminded me on the possibility of using Unicode characters in strings and identifiers in most current programming languages.

That also opens the way for using them in DSLs. Either in internal or external ones. If the special symbols have already a defined meaning in the domain (perhaps as part of a standard notation) this should make DSLs even more readable for the people with the domain knowledge.

All that came to my mind when talking with Dan North about the great new JBehave 2 BDD Framework for Java. When parsing and converting scenario step text tokens, the usage of unicode symbols could improve conciseness and readability.

From Sam Aaron I learned that a ton of unicode characters is just an (ALT-)Keystroke away on a Mac keyboard. He also showed me that in Ruby Unicode is no problem for identifiers and string content.

Today was the last day of this great JAOO Conference. Rickard, Emil and Irene had to leave. I'll stay for two more day. Tomorrow some tutorials and squill hacking with Jevgeni .

I had some really great discussions over the last couple of days. The conference atmosphere is the best place to breed new ideas.

As a final gift to myself I had my MacBook Air signed by some of the speakers. I wanted to do that already when attending the QCon London in March, but back then it was to late when the idea crossed my mind.



I am happy that the following (in no particular order) people where so kind to sign my daily development tool :

Dan North, Big Dave Thomas, Dick Gabriel, Guy Steele, Martin Fowler, Ola Bini, Neal Ford, Kirk Pepperdine, Erik Meijers, Linda Rising, Jeff Sutherland, Kresten Krab Thorup, Kevlin Henney, Frank Buschmann and Bill Venners.

Becoming a PC was a very unpleasant experience for it but its still working and hasn't shown a blue screen since.

I now just need some very resistant coating for that.

Looking forward to the next conference (QCon in March) and to the lots of interesting follow up projects in the mean time.

I thought about extending Lego Time Tracking to a kind of Agile story and task management. There you take the time estimated for the task and put that many bricks of the color assigned to the task away to a central location where it is clearly visible (area of stacked bricks). Everyone contributing to the task takes away the bricks he needs for tracking his work time done on this task. The count of bricks still available at the information radar visualizes the remaining estimated time needed for the task. If the bricks are used up and the task not done its obvious that the team overpromised on this task (also noticeable much earlier).

This is something XylonSextuswrote in the comments to the lifehacker post. I just recently discovered the comment.

I quickly came to the conclusion that the existing eight colors available (plus the ones on sale at the online lego shop) are by far not enough to cover the tasks for a whole iteration. When pondering it I came across the printing possibilities my friend Mattcher has available at his company.

So we did a quick prototype on printing onto lego bricks and here are the results:




I'm currently thinking about further uses of this idea besides puzzles, learning materials, additional time tracking/project management colors...

Regarding the additional colors: As you can print a color on both sides you can reuse the bricks for different task (one of the two colors).

Performance Problems
After struggling for some time with the performance problems with recent IDEA versions on my clients workstations, I've finally found a solution. After upgrading from version five the performance of IDEA began to degrade.
I had many hangs and a general sluggish feel.

The machine is a single core 3,4GHz, 4 GB RAM, Maxtor 7200 SATA drive Windows XP station that shouldn't be that slow.
The projects I'm mostly working on contain about 7000 classes. Unfortunately some of these are quite large. But these posed no problems for former versions of IDEA.

Unsuccessful Attempts
Of course I'm always shutting down the Anti-Virus program before starting development.

I started creating a 512 RAM disk for the build process output files (.class, .jar etc).
But this didn't improve anything for the IDE (just the build process).

There were several articles talking about IDEA optimizations. Mostly settings and JVM startup params.
10 Tips by Hamlet d'Arcy
IDEA Performance Tuning by J2EE blogger
IDEA VM options from the JetBrains blog

I played with all the JVM startup parameters available but this didn't help either. Its currently running with
-server -ea -Xmx768m.

Following the tip of the JetBrains blog I also used DiskKeeper to optimize the MFT segment of the disk.

But none of the above had any significant impact.

IO-bound IDE
When looking at the IO Read/Write information of the task manager it is very clear that the IDE is mostly IO-bound.
Then I used sysinternal's filemon to look which files were accessed at which times by IDEA (especially during the 3-30 second hangs I experienced regularly). These were mostly the caches, ideas own jar files and my application's class files.

What I don't understand is why the "operating system" is not able to cache that file information in memory as quite a lot of the available 4GB is still free.

Decision for SSD
When listening to the google talk of IDEA tech lead Dmitry Jemerov the mentioning of having 10000 rpm drives helps a lot with IDE performance was an obvious hint in the right direction. But I neither want nor need an 10k rpm drive in my workstation.
As since February I'm the proud owner of a MacBook Air with an 64gb solid state drive the solution was at hand.

Buying the SSD
I looked through serveral reviews of smaller SATA-SSD drives and decided to go for on of the fastest available. The 0 millisecond access time together with read/write rates of 120gb/80 gb respectively made it a clear favorite.

So I ordered a 16GB 2,5" MTRON SSD Mobi 3000 from the german reseller for about 170 EUR.

Installation
After plugging it into the machine I copied the IDEA installation directories to the new drive, changed the cache location in the idea.properties file and copied my project files there as well. Next things were the JDKs and some additional libraries (potential candidates would also be the maven repository and some local libraries).


After that IDEA worked like a charm, no more hangs, no hiccups just flowing from my fingertips and not hindering me at all. That's great feeling this again.

Finally I've found it: The mysterious "Anykey" is living under the strange name of "QuickFix" in your IDE of choice.

Pressing Alt+Enter (Cmd+Enter) in Intellij IDEA or Ctrl+1 in Eclipse enables you to program on a "per example" basis. You can do

introduction of variables, fields, classes,
renaming
changing method signatures
completing structure
surrounding code with live template
importing dependencies
and much more ...



and much more just by pressing the "AnyKey" at any offending piece of code. Especially the ton (600+) of intentions of IDEA fixes almost each possible syntax problem using a single key.

Kent Beck used the AnyKey (QuickFix) when describing the Refactoring By Example.

You can find further keystroke reference cards in the help menus of your IDE.

DZone made up a nice collection of refcardz:
IntelliJ IDEA DZone Refcard
NetBeans 6.1 Refcard

I've had troubles with time tracking my worktime for all the years. I always found this to be a tedious burden and inconvenience. So one morning in my blue hour (reading in a cafe before work) I spent the time pondering the alternatives.

I started listing software and realworld solutions to timetracking that are possible and tried to contemplate if they would (or had) worked out for me.

software:
spreadsheets
plain text files
browser based timetracking
Outlook/iCal
popup applications/widgets asking for the current task

corporeal:
sticky notes
paper
tally sheets
notebook
diary / filofax

But none of these seemed to work sufficiently for me. But the arrangement of events of a day in iCal struck me as beeing a stack of time boxes atop each other. Having played recently with my daughters legos (duplo), I quickly jumped on the thought of using them to build these stacks in the real world.


So I bought an box of 600 legos from amazon for around 19 Euros. Opening the package showed me that fortunately there was a variety of colors (white, black, brown, red, orange, yellow, green, chartreuse, blue) and lots of different sizes (lengths) and two widths (one row and two rows of studs).


As a child I hated these one-rowers as they were not useful in building stuff. But here and now they seemed a perfect fit. Small enough and in the right sizes. I chose a time partitioning of a quarter of an hour. So I can use the lengths 1,2,3,4 to build 15,30,45 and 60 minutes worth of time in a row representing an hour.

Stacking these hourly rows on top of each other builds up the whole day. I use the different colors for the projects I'm involved in (8 are just enough), putting them on the stack whenever I want and have time to do so (but mostly quite instantly).


I made up a single width column as ruler for the work hours (from at around 10 am up to 6 pm). So I can easily see whats missing and at what time I did something. For the days of the workweek I chose the rainbow color scheme (red, orange, yellow, green, blue - Monday to Friday) for the longer base row that I stack my hours on. So I can gather a whole week of time tracking until I have to enter them in some time sheet (software). I put the columns of a whole week on top of a green building plate to fix them.


You can easily see how much work you did for any given project as you recognize the colored areas rather than time ranges (8:45-11:15). Having the relative time shares as part of this setup helps as well.

You can even plan your work by prebuilding your days on temporary bases with the planned amount of time for each activity (or putting at least the estimated amount of bricks aside).

The benefits are obvious:
it works (for about 4 months now)
I have something to play with while pondering stuff
it looks great
it's incredibly fast with no overhead
planning is possible


The single disadvantage:
coworkers coming to your place and disassembling your time tracks


I'm still looking for a great name for the whole thing. Here's what I have collected by now:
Names:
BrickLog
Daily Column
TimeStack
TrackStack
The timeful way of building

I appreciate any suggestion for a great name.

One last thing I have been thinking about is getting these daily columns recorded automatically. So using your webcam or phone camera, you just hold the "day" in front of it. After taking the picture it is processed. The extracted information can be used in any possible way.

Update: I started writing a small java application that shall exactly do this. It's almost finished, so stay tuned. Writing the data to iCal should pose no problem as well.

Early preview image of the scan application:


Update:
Post on Lifehacker
InfoQ follow up article.

I added a poll for the name of "Lego Time Tracking to the blog" Feel free to vote or to suggest new names.

I've been reading Neal's blog for a while. So I've been looking forward to the book. (I even accidentally ordered it twice - one was the pre-buy at amazon, which I forgot about).

I spend the last two days reading the book and found it quite helpful. There are a lot of concrete tips and examples for immediate use and daily improvement of your mechanic skills. Many of the experiences regarding the effective use of the tools at hand that he describes are well known to me. I can't really understand how developers are not keen to improve their productivity.
Neal's book is a good addition to the PragProgs masterpiece. It concentrates more on the mechanics and on some principles of productive software development. So the triad of values-principles-patterns got a son named mechanics.

What I missed in the book was:

• a comprehensive list of the notes at the end.


• Christopher Alexanders appearance as one of the philosophers.


• the notion of cheat sheets/refcards


• references to Martin Odersky's Scala the scalable language


• references to Kent Becks "Implementation Patterns" (especially in the SLAP section)

As being a developer myself I was a bit disappointed by the quality of the examples (the solutions not the starting points) and a bit by the correctness of the text (typos). I spotted several errors, some bad designs and some uninformed choices even on the first read of the book. I'll post them to the errata page.

Neals suggestion of an online repository at productiveprogrammer.com of productive programmers tools, tips and mechanics is a great idea. I'd really like to join this effort.

Sitting a mere 200 km from home, 7 hours late by now and hopefully there at dawn, I'll try to recapture the events of this day.
After the last day of the exciting QCon week ended on Friday, I used the time on Saturday morning to buy some family gifts in Covent Garden. Taking the Gatwick Express from Victoria, I arrived about one and half hour before my flight was scheduled to leave. Checking in at the check-in computer was no problem at all. The second step in the procedure was to proceed to the Fast Bag Drop. I looked around but could only see hundreds of people queuing at "Bag Drops" on both sides of the terminal hall. A staff of 6 was handling the large amounts of luggage the people wanted to drop quickly. So the obvious approach was to put all passengers of all departing flights together in a waiting queue and handle them with the same priority. Great idea. After waiting too long, there was a staff member yelling for passengers to Pisa. My flight was scheduled before the one to Pisa, it had about three quarters of a hour to liftoff. I asked her for assistance, she told me that the boarding for the BA flights is done a 3/4 hour before liftoff. Although she tried to phone her way through - no chance. So I had waited all the time in vain and missed my flight. I promised myself never to take any luggage except hand luggage on the next one-week trip.
I was directed to the ticket sales counter for assistance. As this was the only direct flight to Dresden for the day, I had to be rerouted. So she typed all the possible combinations she could think of into her terminal - is there no software for calculating routes? I suggested routing to Berlin and to travel home from there by train. As they don't have internet access at the ticket counter, I was sent to the shopping area to look up the train connections myself. Unfortunately, the last train to Dresden leaves Berlin (capital city!) at 9 pm, thank you Deutsche Bahn.
The alternative routes were: getting from Gatwick to Heathrow and flying there directly to Berlin, arriving at 10:15 pm, or flying from Gatwick via Zurich to Berlin and arriving at 9:45 pm. As she didn't want me to be queued again I was directed to a "special care" express counter. For the first time ever, my bagpack was not manageable by the luggage transport system (because of the straps). So I had to declare it as oversized luggage, which took only another fifteen minutes as the poor woman before me had some problems with the guy at the luggage counter.
Next was security. The usual taking cosmetics bottles from women stuff was even topped this time. After the regular security checks we're all used to, there is an additional x-ray scan of your shoes at Gatwick. So all passengers took off their shoes to have them x-rayed. I don't know what they expect to find in there except terrible smells. You could have taken everything from the shoes when crossing the distance between the two security checkpoints.
Just before boarding I called my wife who was desperately expecting my return as being ill and pregnant herself, she had to take care of our ill little daughter for the entire conference week. She told me that the public transport staff in Berlin was on strike. Fine.
The flight to Zurich had only a one-hour delay, so nothing special. The only funny thing was the woman next to me doing some brain exercise on her Nintendo DS and cursing all the time. Getting the boarding card in Zurich was also no problem. When walking to the Gate of the Lufthansa flight, a friendly gentleman redirected all people to another exit. "It's shorter and nicer that way." I was quite suspicious. The solution to that weird behaviour was that Swiss airports do security checks even on transit passengers. But no shoe scanning this time. And no one took my MacBook Air to be a "device", as Michael Nygard had experienced in the US.
So arriving in Berlin, I took the fortunately still operating bus shuttle from Tegel Airport and managed to get via the cold and closed-down main station to the east-station where I'm now sitting at McDonalds(R) "McCafe(R)" waiting for my train to depart in about 3,5 hours time.

Update: The train from Berlin was half an hour late, this was some official rescheduling but neither the information screens nor the ticket machine knew of this. Another problem was the train having been in the station over night with heating switched off and doors open, so it was cooled down to a very low temperature and the heating could not catch up. This caused me to add another layer of clothing to be able to sleep within the train. When I arrived in Dresden it rained, no surprise there.

Lessons learned:
arrive 3 hours before departure at the airport
travel light with hand luggage only
don't expect airlines being able to understand queuing theory and staff capacity
clean your shoes for security scan
don't trust Swiss when they speak of shortcuts
forget about returning home on time when you're needed most

Its very funny how many guys are interested in your laptop if its a MacBook Air. I actually spotted only one other at the SpringSource Booth. At Monday I got awarded the Canadian maple leaf tag by Kirk Pepperdine, who forced me during the next days on several occassions to show off :)

The benefit was that a lot of people got the chance to actually hold in in their hands and have a closer look at the slick thing. And yes I'm a proud big child showing around my latest toy. I expect the situation to change until JAOO 2008, we'll see.

As the Haskell session with Lennart Augustsson was unfortunately canceled, Mathias and I decided to attend the Architectural Implications of RESTful design talk by Peter Rodgers of 1060research. It was very insightful.

The basic principles of the talk were:
1) all resources are named by an URI
2) resources are immutable and copied
3) you can construct arbitrary URI which present a computation and use other URIs as parameters
(e.g. active:imageOperation+operation@fllcc:/doc/rotate45.xml+image@http://imageurl)

With these precodition Peter showed a kind of functional programming approach. You just write (or have tools write) your programm (function, expression) as a cascade of URIs.

It was a real eye-opener, just like in a functional programming languages your functions don't have sideeffects and therefore an URI presenting an expression is always referrring to the same value (as long as the resources used are really immutable, one may propagate changes). So it is very easy to cache them based on usage count and expensiveness of computation. Even with a very limited dynamic cache this kind of program runs faster that the traditional imperative one which is recomputed over and over.

Another advantage is that you address cross cutting concerns by partitioning the address space of the URIs and putting zones around them which contain the aspects (AOP, e.g. security, transactions).

So with a very simple approach and a powerful runtime used for routing, caching and executing or retrieving the resources pointed to by the URIs - NetKernel it's another tool in our polyglot language toolbelt.

So I attended another agile session on the mashup of agile methodologies. Unfortunately it turned out to be just a basic introduction of the different agile practices. The real mashup of practices regardless of the borders of the methodologies I expected was not presented by Rachel. I.e. something like Kents XP graphs depicting which practices (and principles) depend on each other and which have positive impact when used together, etc.

In my opinion the Agile Coach should be responsible for knowing all the agile practices, introducing the relevant ones to the team, helping them using and evaluating and improving their use of the practices and enable them as a self-empowered team to decide what works best for them. The coach also bears the responsbility of keeping the use of practices consistent i.e. having the team use an increasing part of the "pattern language".

This year's qcon I decided to attend no technical tutorials but more development process oriented ones - i.e. Agile.

On Monday I was delighted to attend Lindas tutorial. It had a very personal touch, was very interactive. She presented the patterns for introducing new ideas into organizations using a play with us participants as actors. Actually this was the story of herself introducing Design Patterns back in 1996. During the discussion many of the subtleties of the patterns (i.e context and forces) were addressed. Having us contributing by playing and asking lots of questions helped a lot to really absorb the essence of the patterns. It was a lot of fun.

The introduction of the pattern concept (reference to Christopher Alexander and his great book which i recently finished rereading was also very engaging).

One of the most important points she made during the tutorial was, that these patterns for changing the attitude of people can be used and misused to introduce ideas regardless of their moral value. So they are very powerful.

And the tutorial helped to solve real problems. In the First Aid Clinic part we used the stuff just learned to give three of us advice how to approach difficult situations they currently have at work.

Most of the patterns of change are deeply rooted in human behaviour. So all the knowledge that was gathered by brain scientists back all the patterns rediscovered. After the tutorial we stayed for a while to further discuss some of these things.

Linda is a great speaker with deep understanding and lots of wisdom to share. Thanks a lot for the tutorial.

Books recommended at the tutorial:
Fearless Change
A Timeless Way of Building
Predictably Irrational: The Hidden Forces That Shape Our Decisions
The Tipping Point
Gang of Four - Design Patterns
Patterns of Software Architecture (1,4,5)

While reviewing Martin Fowlers upcoming DSL book I started thinking (again) about closures in Java and the current overly verbose syntax for Single Abstract Method Interfaces. And then there was the blog entry of Anders Norås about Deferred Execution and Kicker Methods, both relaying to the DSL stuff I just read.

So starting with a SAM closure.

new Closure() {

    public Integer eval() {

       return input.length();

    }

}

closure.eval();

So I thought about the Java Instance Initializer Block, which is executed before the constructor is called.

class A {

{

   initializer code;

}

... constructor and class body ...

}

So, combining the Anonymous inner class with the initializer Block gave me.

new Closure() {{

     result=input.length();

}}

closure.result

the problem is that the code there is evaluated when defining the class and creating the anonymous instance, so there is no deferring possible.

Using a SuperClass for the closure which basically does this.

class Closure {

    public R eval() {

        return newInstance().result;

    }

    private Closure newInstance() {

        try {

            final Constructor syntheticConstructor = getClass().getDeclaredConstructors()[0];

            return (Closure) syntheticConstructor.newInstance(createParams());

        } catch (Exception e) {

            throw new RuntimeException("Error creating new instance of closure", e);

        }

    private Object[] createParams() throws IllegalAccessException {

        final Field[] fields = getClass().getDeclaredFields();

        final int fieldCount = fields.length;

        final Object[] params = new Object[fieldCount];

        params[0] = fields[fieldCount - 1].get(this); // enclosing Instance comes last

        for (int i = fieldCount - 2; i >= 0; i--) {

            params[i + 1] = fields[i].get(this);

        }

        return params;

    }

    }

}

I didn't think so but it is possible to create additonal new Instances of Anonymous Inner Classes, you just have to watch for the synthetic constructor and the reference to the enclosing class. If you don't use constructor parameters to pass into the closure, you'll have to support all the referenced final outer variables, so passing parameter is certainly more useful.

so the result is:

new Closure() {{

     result=input.length();

}}

assertEquals(10,closure.eval());

to reduce noise one can skip the generics and cast the types accordingly.

non final input data can be done be creating a constructor in the closure which takes vararg arguments of declared generic types and
keeps them in protected final fields for reuse in the expression block;
The remaining problem is that one has to use non primitive Objects as integrated final local variables.

I put the complete code on my website.

Last week I finally got the book. Implementation Patterns from Kent Beck. I've been waiting for it for quite a while.

The first thing I noticed was that the book is actually Java Best Practice Pattterns, although it doesn't mention in in its title.
The book is a part of the condensed knowledge of an experienced developer how to write communicating, simple and flexible code. Its level is a bit high profile for new developers. Although I enjoyed to reflect my own work, I really liked many of the topics taken much further and discussed in greater scope and more precisely.

I would have wanted more structure in the flow of the book. Perhaps smaller chapters focused to some subtopic of the current chapters.

What I missed most in the book were references to the other important works in this context, developing software even at the language level is never isolated from the other forces and possibilities one has.

What I missed most were:

discussion of package concept, its usefulness for communication, structuring, documenting and visibility

a consistent example carrying through the book, either from the JUnit library or an extended version of the Performance Measurement appendix

Generics: used in many examples but not explained and no implementation patterns for them

Enum: very shortly discussed in the framework section but imho belong to the class and interface chapter, very important as type-safe alternative to constants, the constant object would be a
important accompanying concept

Annotations although you use them in examples and have surely used them a lot in JUnit

the immutable java.lang.*; objects which are the counterparts of the primitive types

going with that Autoboxing, its many advantages and problems together with varargs

explicit discussion of Packages as structuring element

more finger-pointing on potential concurrency problems, e.g. when sharing state, safe copy and the like

regarding to communication: principle of least surprise - consistent usage of patterns throughout an application and commonly accepted usage of language features (e.g. java naming schemes etc).

law of demeter

the danger of having overridable methods called in constructors and the necessity of

an explicit Chapter on Properties, e.g. published, encapsulated attributes, the pro and cons, problems with published fields, with a discussion of the Java-Beans concept

within the "Creation" Chapter a discussion on dependencies and dependency handling, publish them don't publish them, the creational patterns shown

discussion of Martin Fowlers POJO notion

taking the language created by a framework or library further to the point where it covers its (small) domain -> further reading Martins DSL book

how to handle cross cutting concerns ? use a technology for this or provide the hooks yourself

discussion of Inversion of Control, using callbacks to transfer the flow of control to a framework having my own code executed in a certain context/environment (especially resource handling etc)

a more positive discussion of the final keyword (esp. for fields, locals, parameters) , it communicates a lot, helps to prevent a lot of errors (compiler catches them) and assures immutability
declaring classes and methods final is a very different side of the coin

A problem with the Appendix exampel is that it ignores some advice of the book, contains bugs and false assumptions. The statement that initializing collecitons with a size doesn not matter is wrong. Kent measured the wrong things to prove that.

I read the book twice since, writing a lot of notes in it (errata and commentary). I asked Kent to discuss the stuff and fortunately he agreed. So I hope to contribute to the evolution of the book.

I hope I don't offend Kent with my long commentaries and errata lists. I like reviewing books. you get so much to think about and to take the auhtors thoughts further along the road.

Kent has set up a mailing list to discuss the content of the book and the importance of style. If one takes the metaphor furter, the values are the style, that is defined by the elements of style (i.e. princpiles) which get then implemented by the practices.