Problems with software development

Yet another thought on what is wrong with software development. The inherent difficulties in software project management have been thrashed over many a times. Also, developement methodologies have been vilified incessantly in technology literature. I rather hold a contrarian view.

First, we should celebrate software achievements. In a mere half century, this discipline has achieved true great heights and is the foundation of all hi-tech. Think about this, we have satellites and spaceships which can correct themselves as they travel through the space. NASA Mars toys had fixed hardware structure but software-wise they were truly a piece of beauty. Once the landers were on Mars, depending on the conditions and surroundings, command center at earth uploaded the newer version of software and obtained optimal performance from the hardware. This becomes more important for ships like pioneer which would be reaching their destination in years down the road. By upgrading the software, we could at least make sure that the new strides in technology could be put in use, even long after the launch.

Anyway I digress, the issue at hand is the software development problems. Now that we understand that the scene actually is not so bad, lets see what plagues us.

1. Team structure: More than any other engineering discipline, I find, software requires the power of experience. Its an infinitely malleable science, where one needs to understand at what point one should stop. Yes, the most important aspect is to discover the optimal ground where a solution is rightly engineered. Watchful eyes, looking for the under and specially the over-engineered design, are a pre-requisite for any team. Yet, the fast pace of evolution means that the experience goes obsolete pretty soon too. This is the real challenge. More than anything I feel we need a balanced team structure in terms of experience and raw energy. Have right amount of youth who is uninhibited in terms of exploring new ideas and the weathered minds to restrain the zealots. Still, in my experience most of the teams are ignorantly composed. Either its the strong belief in youth or amalgamation of like minded industry veterans. Few places where this was managed judiciously, I found the most productive teams.

2. Overloading: The abstract nature of software is its one of the most appealing characteristics and the deadliest too. What seems clever in trying to get a lot out of little, by overloading existing modules in terms of definition and usages, is most of the time a bobby trap. Having seen lots of well intentioned overloading going topsy-turvy, its about time we should put "design guidelines" on strong "no-reuse" of existing resources in new set-ups. Once recent one which I observed in one of my esteemed PhD, 10 yr veteran colleague was in overloading a date field. In existing setup it denotes, the end of a relationship but suddenly he has put a new twist on it, and made it work somehow. His rational is, creating something more straightforward would have been "overkill".

3. Of course the usual suspects, which have been covered pretty well by others:
- Ever changing or poorly defined requirements.
- Dumb or premature optimization.
- Bloated or complex design.
- It aint broken!

BTW I have stolen my new mantra from Google's bag.
"Creating simplicity is a complex task."

"Simply" yours:).

Improving product development

"The competitor has come up with the killer feature", "The customer wants this feature", "Apocalypse, if the launch does not happen by March 31st". Ah, our revered product managers, coming up with same old lines all the time. Most of these product managers are not worth the shining business cards they carry. Lets see:

1. Reactive: Show me one product manager who came up with true innovation. OK, that was harsh, but the point stands. Most of the time requirements document is a copy of competitors' features or rehashing the customer need. Granted these are important avenues to set the requirements but frankly this is no creativity. The genuine talent is in foreseeing the customer need and simplifying the features. Nevertheless, even on the reactive part of their job, most product manager's perform poorly. They either misunderstand or miscommunicate the customer requirements, causing long and never ending product cycles.

2. Poor planning: Product management is an art, I agree, but its an art with numbers. Manager has lots of tools and techniques to plan a project but they easily botch this process. How? Amazingly the product manager's are apt at make stupid engineering decisions. Yes, lots of them do have a background in engineering but after being away from hands on technology for 1o years, many of them are in the bastion of obsolescence. Answer, reinvent yourself product managers.

What is the remedy?
Get rid of product management, nope, the group has a role to play but lets redefine its role. Innovation generally flows out of R&D or marketing departments. We should introduce more marketing principles into product management. A not-so-radical idea would be to rotate product managers into marketing departments. The other dimension is engineering. Engineering should be given a bigger stake at the beginning of product design cycle. Technologists have a complete picture of intricacies and boundaries of the system. The integration and oft-missed problems like internationalization could be taken into account at the right time in planning. In short it is time to bring marketing and engineering near to each other, where product management has failed in past.