Enterprise IT Project Insanity

A study published in the Harvard Business Review  has again shown that many IT projects continue to come in late and over budget. In addition it shows that there is a higher than expected number of large scale failures. These failures are massively over budget (200% in this study) and over deadline (70% overruns) and it cites examples where this has even contributed to the collapse of the company or at best reduced its profit forecasts leaving it at the mercy of the City. It’s interesting reading with the study showing that in total 27% of the 1471 projects overran in some way.

Now you could argue that these are mostly major IT transformation projects and hence the risks associated with them are bound to be significant (although this highlights another fact in that if the project was seen as transformational then perhaps there is an absence of a culture of continuous improvement at those organisations). Regardless though this is still damming evidence of the ability for this industry to implement IT projects. A comparison with other industries has been made on numerous occasions and IT tends to come out worst for achieving project success (although it’s not alone with many large defence sector projects for example suffering too).  There are many reasons for IT projects to fail and I’m not going to cover them all here but instead ask the question why are we still repeating the same mistakes in many organisations. It has been 36 years since Frederick P Brooks wrote the seminal book “The Mythical Man Month” yet many of its key themes within his essays remain problems today. Walk around many traditional enterprise IT departments today and you need to avoid the Tar Pit on your way to joining the  Death March on the “Tower Of Babel” project. For those of you not familiar with the book, the Wikipedia article on it summarizes it well. Some concepts will seem obvious and basic pitfalls but don’t forget this was written in a different age. The key points made in the book being the importance of progress tracking, tooling, communication and the iconic Mythical Man Month (whereby assigning more programmers to a project running behind schedule will make it even later, because of the time required for the new programmers to learn about the project, as well as the increased communication overhead).

So what has changed in 36 years?  In some ways a lot has changed: The emergence of Lean/Agile practices and Motivation 3.0? (check out Daniel Pink’s work on intrinsic motivation) have dragged the thought leaders in the industry in the right direction (usually forced via a groundswell movement) but crucially the impact of these innovations vary by company and corporate cultures. It seems in many traditional enterprises little has changed. We continue to march on for projects destined to fail with old world approaches, planning in the ‘mythical man months’ based on ‘lies’ (sorry ‘estimates’). We still see people being thrown at a problem to resolve it despite this being a known anti-pattern.  Many of the new approaches aimed at addressing these problems (e.g. lean/agile etc.) thrive in the smarter/learner enterprises and technology companies but have have struggled to get traction in the traditional red brick enterprises despite the impressive results of agile/iterative approaches. Even those enterprises that have adopted them have struggled to instil the philosophy behind Agile and instead just dogmatically implement an specific Agile methodology. The shift is happening but what’s stopping faster evolution in these enterprises?

Many of the issues could be classed as cultural, organizational or management issues and so can we can blame the senior management? Well that is way too simplistic and all stakeholders in an IT project have a part of play in its success or failure.  Perhaps more likely is that the management tools, processes, procedures and attitudes that manage a modern enterprise don’t naturally fit to managing software development. In some ways this is not surprising as these enterprises are busy managing their core competencies (finance, manufacturing, construction, logistics etc.) with software development being done internally without the focus it perhaps needs. Take the world of banking as an example. A bank might spend millions on its IT and perhaps even consider some of it’s IT systems to be a competitive advantage, however it is a bank, not a software house and any activity within the organisation must fit the internal processes whether that fit is a natural one or not. In my post about the “Future of the IT Department” I covered how these IT departments have become large and unwieldy beasts. Fitting software development into a non-IT enterprise can be difficult and rigid. How many times have you had to bend software development reporting to fit into a model for which it doesn’t naturally fit? We have all had to dumb down technical analysis of issues/risks/progress into bite size chunks of simplistic bullet points that fit neatly into a PowerPoint slide but convey little in the way of accuracy. No doubt you get dragged into needless conference calls as a result of someone misunderstanding the technical situation. That’s not to say that those we report to are not intelligent (often far from it) but that they often lack the required skills to manage technical details. So what about the IT experts who progress up to management, they understand the problem right? Well yes they did, but it doesn’t take long to adjust to the culture of the organisation as it is a necessity for them to do so. They cannot do their job without following the processes that run the rest of the business and they need information/milestones/gateways to track progress. The industry is getting wise to this down at the development team level however as new tools emerge through the agile space that track the progress and velocity of a development team in easy to consume visual ways (e.g TFS v.next). Whilst these are not intended to be consumed at exec level I expect that more accurate reporting at the local level will result in better reporting flowing up the organisational structure. The pace of change in the industry is to blame also here for the managers of today were the IT guys of yesterday when different architecture landscapes were around and Mainframes were king.Where there is an inability for management and IT professionals to communicate and share a culture the gulf is  filled with consultants and IT sales reps which can compound the problem. The use of Offshoring and outsourcing can also make the problem worse as the barriers to communication are now formal 3rd party engagements. We need to find a common ground to fill this cultural divide in an effort to help speed up the evolution of change.

It’s not unusual for these large enterprises to use Waterfall methodologies and tightly control the IT department resources so that IT can be managed neatly within the organisations financial and resource planning models. This will only get more engrained over the next few years as the economic climate dictates tighter financial controls and risk monitoring. This of course does little to aid the progress of IT evolution unless risks are taken to counteract the cost squeeze with more agility. The study mentioned above also highlighted that those projects that do succeed are often using Agile approaches and focusing on customer value, which makes sense. IT projects are one-off bespoke creations and yet are managed as though they were identical widgets produced on a production line. Enterprises are organised into a post war hierarchical command and control structure that is structured to actually avoid the communication between teams yet we still apply that model to the IT organisation in many enterprises. If we look at the likes of Google and Facebook, their developers are given more autonomy (and responsibility) for the software they build and its implementation, test, adoption and its evolution. They are encouraged to deploy often and innovate with trust placed upon them based on the fact that Software development is a skill and one developer is not as replaceable with another as is often assumed (explained brilliantly in John Miano’s post here on The Myth of the Interchangeable Programmer). In comparison the enterprise developer more often than not sits in India creating code based on detailed specifications (based on wrong assumptions at a point in time) separated by offshore coordinators, managers, on-shore coordinator’s, and a few thousand miles of undersea cable from the business they serve - stifling their ability to add value. When quality of output into test drops the next project is assigned more testing time and resource to catch defects, but at the expense of the design/development time. this leads to rushed design/build and again lower quality, so again more testing is required. And then in response there is a whole bureaucratic process of change management policies to rightly protect service. I’m not suggesting that change management processes and IT policies are not important (or indeed vital) but there is a balance between innovation and risk aversion that must be reached for the benefit of the company and for enterprise IT to evolve.

As the downturn in the economy bites cost controls will be tightened and IT departments trimmed, more often than not replaced by off shore labour, but this is an opportunity to rethink the approach and evolve the IT organisation into something leaner backed by more autonomy and lean processes. Agile/Iterative approaches have been proven to reduce costs and improve quality in comparison to traditional methods. Enterprises need to fully embrace Agile and encourage innovation within smaller business aligned teams of IT, relax some of the processes and abandon the often rigid enforcement of “Waterfall For All” and instead enable teams to choose the methodology that best fits.

IT within enterprises is evolving slowly but perhaps a revolution is required to speed things up, are you ready to revolt?