When I joined Redpoint I noticed a disproportionate number of musicians in the firm. So much in fact that I frequently heard people ask, "When will we see the Redpoint band play?" Today while reading the book Rise Up Singing in preparation to play some holiday tunes with the family, I ran across a quote from Pete Seeger: "Musicians can teach the politicians; it's fun to swap the lead. Musicians can teach the planners – economists, engineers, lawyers: Plan for improvisation." From my experience in the software industry, this quote rings true. Over the years, writing computer software has sought to become formulaic, much like a construction engineering project. This simply isn't appropriate; Software is abstract rather than concrete. In graduate school I was taught that there is no "correct" solution to difficult problems.
I have never been on a project that went exactly as planned. I have yet to meet anyone in the industry who has. While learning the trade of project management, I was expected to add contingency to all project estimates. I completely understand the need for projects to have predictable goals and specific end point as well as a way to track project performance and anticipate if the project will be done on time. Agile allows improvisation in a formal way that's traceable and repeatable. This seems like exactly what the industry is asking for.
To use a music analogy: Traditional software development is like classical music while agile is similar to heavily improvisational music such as jazz music. Both type attract highly skilled individuals that spend years practicing their skill in order to even enter the lowest level of the profession. In classical music the best forms are predictable, while jazz music is fluid and evolves in real-time. Since we acknowledge projects will change and evolve as we build them, we must plan for improvisation rather than predictably.