Agile – RichHewlett.com

The Growth Of Business IT

In my popular post on “The Future of the IT Department” I covered how IT is changing rapidly in enterprises and touched on how business aligned IT teams are going to become more relevant. Some of these agile ‘business focused development and delivery teams’ will be official IT sponsored initiatives whilst others will be somewhat rogue business division sponsored teams working without the IT department as a response to the expensive, often poor quality service provided by the IT division.

The rapid pace of marketplace innovation and the lack of flexibility of many IT organisations within enterprises, when fuelled with the consumerization of IT and the growth of cloud computing is leading to a boom in DIY business application development. Gartner predicts that…

“Rank-and-file employees will build at least a quarter of all new business applications by 2014, up from less than 5% in 2007.” [Gartner]

For many years now there has always been the power in the business to harness Excel macros and VBA to enhance end-user productivity, but now this is being enhanced by new friendly end-user tools such as easy mobile app development, the ability to host new websites in the cloud in a few clicks and a whole SaaS model to replace your IT in house infrastructure over night .

business-it-support The business benefits of this boom are clear to see. The ability of end-users and Business IT teams to manipulate the data and process flows to meet the shifting demands of the market are attractive. Customer demands can in theory be more easily met by those closet to the customer building applications quickly and with their day to day use clearly in mind. As the market changes the user can adjust their homebrew application to fit the market, or throw it away and start a new one. Instead of a business analysis working closely with the developer to create an application she can reduce the communication overhead by just building it herself. Even if the application is only to be used as a POC this is a very efficient process to find out what works and what doesn’t. In this article on BusinessWeek the CEO of NetApp explains the benefits seen by encouraging employees to build their own tools, such as cost savings and customer satisfaction. It’s not all peachy though. There are obvious pitfalls to this approach. the IT organisation may be slow and expensive but they often have genuine reasons for being that way. Interoperability, support, security, regulatory concerns, supplier contracts and economies of scale are all topics the IT organisation has to consider and so too does the business if its going to promote this DIY application approach.

Business run IT teams can do very productive work and react quickly to change, but from my experience the problem comes when they have to rely on the IT department to support the implement their change and that’s where tension can arise. Teams outside the IT structure can find it hard to understand the constraints of the IT department. I find developers in business sponsored teams have a real desire to be productive for customers but lack some of the rigor that is prevalent in IT based teams (particularly around maintainability and change control). The IT department can seem to be a blocker to the teams agility when it is unable to adhere to the timescales expected by the business teams. I think some effort needs to be made on both sides to understand the constraints the other teams are under and work together. Critically I feel the IT department needs to realise that this trend will continue and the IT org is at risk of becoming irrelevant (other than to keep the lights on and maintain legacy systems). Perhaps this is the natural evolution of the consumerisation of technology but I do think that IT organisations can have a very relevant role to play in this shift. By sponsoring agile business centric development teams to support the business better the IT organisation of the future can have a very relevant role and IT professionals are ideally positioned to populate these teams and support the growth in DIY applications whilst adding some beneficial structure.

It’s All About Culture (Enterprise IT Beware)

This interesting post by PEG recently highlights an organisations culture as being in reality the only differentiating factor that they have. In his view assets, IP, cost competitiveness, brand and even people can be copied or acquired by your competition but it is your company culture that will lead to success/failure. I agree with his assertion on the importance of culture, and would add that whilst it has always been the case that culture is critical the rapidly changing new world is bringing with it new challenges to competitiveness resulting in culture becoming even more important for competitive advantage. I must clarify here that I see a huge gulf between the ‘official’ culture of an organisation that is documented and presented by senior management and the real culture that is living and breathing on the shop floor (and they rarely match).

This excellent highscalability.com post by Todd Hoff recently highlights the way that the rules framing IT (and business) are changing and how start-ups are becoming a beacon for investigating this new world. When we look at this new world the key elements are flexibility, adaptability and innovation. These traits thrive in start-ups where the ‘culture’ encourages them. Many of these new industry shakers lack the assets, brand and IP to use but excel in using their ‘culture’ to outmanoeuvre bigger rivals and drive innovation in their industry.

Some enterprises get this and are trying hard to foster a more innovative and customer focused culture. The emergence of agile development practices can help to focus the team on the true business value of features and aid flexibility in the use of resources. SAP has recently designed its new office environment to fully promote Agile development practices (see this post), and whilst this on its own can’t change corporate culture, it can remove some of the blockers to a more agile culture emerging. Of course there are many other enterprises that still rely on military inspired hierarchical structures and attempt to enforce a desired culture. This InfoQ article by Craig Smith summarizes recent articles covering how mainstream management are missing the benefits of an agile approach within their organisations.

"…The management world remains generally in denial about the discoveries of Agile. You can scan the pages of Harvard Business Review and find scarcely even an oblique reference to the solution that Agile offers to one of the fundamental management problems of our times.”

The benefits of Agile practices are well documented and so if this fundamental approach is still not connecting with mainstream managers then how long will it be before they grasp the bigger paradigm shift that is occurring underneath them. This shift is being forged by start-ups and enabled via cloud computing.

Let’s look into what is happening in the start-up space; Todd Hoff’s post again highlights the use of small dedicated autonomous teams with the power (and responsibility) to make rapid changes to production systems. these teams being responsible for the entirety of their systems from design to build, test, deployment and even monitoring. How does this work? Well it can only work effectively via a shared culture of innovation, ownership and excellence. Facebook/Google staff have stated that their biggest motivator is the recognition of their good work by their peers (see here and here). This is ‘Motivation 3.0’ in action with the intrinsic rewards of the job at hand being the main motivation to succeed. Compare this with the traditional and still prevalent command and control approach used in a lot of enterprises with tightly controlled processes, working habits and resources. Splitting the responsibility for each stage of the development lifecycle between different teams (usually separated by reporting lines, internal funding requests, remote locations etc.) and then expecting a coherent solution to prevail is not going to work in this new world.

We are now starting to see the emergence of the cloud as a major force in driving the democratising of computing power and enabling the emergence of empowered teams. Todd’s post covers in detail how the cloud is making the once impossible possible and I recommend you take the time to read it. Only time will tell if the cloud’s impact on Enterprise IT will act as a catalyst to driving the emergence of more agile competitive corporate cultures in enterprises in the future.

A Trial Kanban Project – What Worked. What Didn’t.

Recently I introduced Kanban to an inflight development project with the objective of trialling this technique with the team to raise productivity by overcoming various communication and process issues. In the first part of this 3 part series I presented an introduction to Kanban. This second part will cover the results obtained by introducing it to an in-flight project and part 3 will give some fuel to the “Physical versus Virtual Kanban boards” debate.

Project Background

This was a service release aiming to investigate and resolve a fixed set of production defects on a newly implemented enterprise scale system. The team consisted of 6 people onshore including Project Manager, Work Packet Manager, Technical Architect, System Analyst and 12 offshore developers (with limited experience of the system). This was a Waterfall project with the project plan based on the agreed prioritized defect list, a fixed resource and time-boxed approach. The number and complex nature of the defect investigations together with dependency management (with other teams) made the tracking of items within the project difficult and opaque which exposed the lack of a structured workflow within the team.

Approach Taken

Mid project I suggested the team adopt a Kanban approach and after explaining the concept to the team quickly got their buy-in. A quick virtual task-board was introduced (in Microsoft Excel initially) with columns defined based on the current informal process with added enforced quality assurance steps. Each work item (defect or change request) was added to the board and tracked, with updates applied daily by the team, as a very minimum, during a morning stand-up.

The task-board immediately brought a visual dimension to the project status making managing work in progress (WIP)easier and it also helped to bring focus to the daily stand-ups. After this initial success, to overcome the limitations of the virtual board and to get wider team involvement, the project moved to a physical board with very positive results.

The new board evolved from the previous version based on feedback and was simpler to use. Its physical nature and the fact that it was always on display for the team to consume made a huge difference and resulted in full team commitment to the approach. The physical board became the focal point for the daily stand-ups and for ad-hoc progress updates. The off-shore’s teams updates were applied regularly by the onshore coordinator. Team members used the Kanban queues to action work and move them to the next queue, monitoring/resolving blockages.

What worked well

Implementing Kanban was particularly easy compared to other methods and approaches. As the method logically represents a team to-do list it was easy to explain to people and therefore was not seen as a new complicated process which no doubt aided adoption with the team. It was immediately apparent that this method would compliment and not conflict with existing methodologies and processes. The fact that the board initially should mirror you current processes combined with the fact that it is a method not a methodology meant that it easily slotted in. Again this helped with team adoption and it received a very positive reception from the project team.

The team liked that Kanban immediately brought a visual dimension to project status. They could quickly see items in their queues and where the bottlenecks (if any) were. One quick glance at the board showed the current status of all items. This enabled the team to quickly provide progress updates and set realistic expectations with stakeholders outside the project team. Effort was saved by avoiding wasted deployments into test environments by being able to see at a glance what was going to be ready to deploy in the next build and then set early expectations with the QA team on whether they wanted to take the build or not.

As the flow of work was visualized and monitored it became easier to enforce the current process stages more explicitly. An item would have to pass through the “code review” stage before it could get to the “Ready to Deploy” stage – not only that but the process was completely transparent and clear to everyone involved. This aided communication too which facilitated the current process. As the process was transparent, and the team were able to use the stages as a common ‘vocabulary’, amendments to the process were easily discussed and small incremental process improvements were made, increasing the efficiency of the team. Kanban encourages this constant improvement and the flexibility of the method ensures that it is not a barrier to change.

From a project management perspective the board provided an excellent starting point for meeting updates and status tracking. With all WIP now visible the control of that work, and the resources involved became easier. It was this visibly that actually triggered a useful resource allocation investigation resulting is resources being discovered to be under-utilised in one area and enabled reallocation.

On this project the move from a virtual software task-board (albeit a very simple one) to a physical board made a huge difference and it made the whole method very tangible and “real”. Not just another piece of software to update but something different, something big and bold and in the room!

What didn’t work so well

With any new method a team has to find its feet and test the boundaries of what is productive and what isn’t. The rules to our process and our task board were not as clearly communicated to the whole team as at first thought resulting in some lack of full team ownership and involvement in the first few days. This was evidenced by some of the team acting as purely ‘read-only’ users initially until they were encouraged to fully participate and engage with the method. Despite being a simple concept, without an explanation of the columns their meaning was open to a users interpretation. These were not problems with the Kanban method but merely communication problems that were quickly rectified.

It was discovered that the Kanban board could become misused. There was an urge to overcomplicate the board by adding more detail to the columns and the cards (such as expected delivery date and resource names). As our first board was virtual we found that additional information could easily be added without much control. This was stamped out early on after discussions within the team led to the information that was only required for reporting and tracking reasons being added or supplied from elsewhere as required. The board was simplified again and then a subsequent move over to a physical board helped eradicate this problem.

We found that it was possible for the Kanban board to be used to just display the current status of work as opposed to being used as intended to drive work allocation. The board was being updated daily to reflect the days progress but the board was not driving the workflow and the team was not using the queues to allocate all their work. This meant that it was still adding value as an information radiator but not enabling its full potential. A similar problem was that some WIP items were not added to the board at all but managed separately. This was compounded if stand-ups were driven from an alternative list (defect tracker tool, meeting notes) instead of the Kanban board. All team members had to be reminded that all WIP had to be on the board no matter what the work was was, as it all had to be made visible. Luckily these issues were overcome by reminding people to use the board as the “one truth” and ensuring that all work was progressed via the board, which in a short time became embedded in the team practices.

An area where Kanban (and indeed most Agile methods) suffers is the ‘Remote Team’ scenario. The common assumption that all the teams are co-located is a common failing but one that has to be overcome using practical solutions where possible. Ideally teams need to be co-located for maximum efficiency but the use of telepresence, chat tools and conference calls can help. On this project we managed with an on-shore coordinator acting as a proxy to the offshore team and the Kanban board, combined with usual modern communication methods. Use of a good virtual Kanban board would potentially resolve some of these issues. To be clear Kanban did not introduce any additional impediments to remote teams than didn’t already exist within the team and its methodology.

Summary

This was just an informal trial of Kanban on a inflight project and so in the future I would like to extend the trial to other projects but with some adjustments. At project kick off I would not assume that the whole team had an understanding of the columns and the workflow process, and would ensure full buy-in. More monitoring of ‘cycle time’ would be advised to gain more insight into the current process. I would also like to trial some of the professional virtual task-board software on the market, together with possible integration with Team Foundation Server Work Items.

In summary the Kanban method worked very well on this project and with its simple/flexible approach I’m sure that it can be useful on a huge variety of projects. Critically the feedback from this project team was extremely positive, so much so that team members actually went on to implement Kanban on their other projects afterwards. It was seen by the team (and stakeholders) as a successful work tracking and communication technique, whilst also providing the opportunity to continuously improve. I highly recommend considering trialling Kanban on your future projects, and if you can’t manage to go that far then at least start to think how you can visualize your work in progress.

An Introduction to Kanban

Recently I introduced Kanban to an inflight development project with the objective of trialling this technique with a hope to raising productivity by overcoming various communication and process issues. In this three part series I’ll cover my real world experiences and the results felt by the project/team. This first part offers an introduction to Kanban, part 2 will cover the results obtained by introducing it to an in-flight project and part 3 will give some fuel to the “Physical versus Virtual Kanban boards” debate.

Kanban has been used in manufacturing for many years and is rooted in the Kaizen (continuous improvement) philosophy documented in the Toyota Production System. It has been used in the manufacturing industry for many years. It’s focus in the field of Software Development came with the book by David Anderson. Kanban in the Software Development sense is …

“… a method for developing products with an emphasis on just-in-time delivery while not overloading the developers. It emphasizes that developers pull work from a queue, and the process, from definition of a task to its delivery to the customer, is displayed for participants to see”. – Wikipedia

Kanban is not a methodology but an enabler to the goal of Continuous Improvement: The Kanban Method encourages small continuous, incremental and evolutionary workflow changes that stick over time. When teams have a shared understanding about the flow of work they are more likely to be able to build a shared comprehension of a problem and then suggest improvements that can be agreed by team consensus.

Visualization of work has become very popular in agile development practices. XP, for example, has “informative workspaces” that display project progress at a glance and the use of “Information Radiators” is gaining ground (i.e. graphs or charts on the wall showing graphically the work in progress within the team or project. This work is too often hidden and so by publicizing the flow of work we can start to understand the real processes in place and how work flows to completion. Once you have that understanding (and can easily share it) you can start to make improvements.

Kanban development revolves around a visual task-board used for managing work in progress: A progress board but with rules! The columns on the board represent the different states or steps in the workflow and the cards/post-it-notes represent a unit of work (the feature/user story/task etc.). We then move the cards along the board through each process stage (i.e. column) to completion. A look at the board at any one time shows which items of work are currently within each stage of the process.

In order to limit the concurrent work in progress within the team we add rules to the Kanban board and set WIP limits on each column, e.g. only 6 items can be in QA at any one time as that is the max capacity of that team. No more work can be assigned to the QA queue whilst it is full. This limit enforces that new work is only “pulled” into the QA queue when there is available capacity. This immediately reveals the bottlenecks in your process so that they may be addressed. Once we spot a bottleneck we stop adding more work to the bottlenecked queue and collaborate within the team to eliminate the blockage (e.g. move resources off build tasks and onto QA tasks). It’s important to note here that we are not artificially introducing a limit here, as we have created the board initially based on the current workflow and therefore the limit exists in reality (e.g. only a single team member can perform part of the process) but it may not have been currently visible to the team.

Don’t underestimate the power of a visual progress board! It’s useful for all project stakeholders to see the latest project status, including senior managers. It might even replace that dreaded status report. The flow of work through each state in the workflow can be monitored, measured and reported (if desired). By actively managing the flow the continuous, incremental and evolutionary changes to the system can be evaluated to have positive or negative effects on the system. Ultimately we want to figure out how to keep development and throughput running smoothly by tracking down bottlenecks and then adjusting our workflow or our resources to eliminate them. We may be able to relieve repeated bottlenecks by changing the number and types of people in each role and cross training our team. One thing that can be monitored is the ‘cadence’ or rhythm of the pace of change. How long does it take for an item to be completed once its added to the board (i.e. its travel time from left to right)? This cycle time can be calculated as:

Cycle Time = Number of Things in Process / Average Completion Rate

A critical benefit of Kanban is that it is very simple to implement, use and improve. Knocking up a task board on your current process is straight forward and team members soon pick up the approach which really aids team engagement. In addition as it is not a methodology in its self it can be used within your existing methodology and actually provides a more gradual evolutionary path from Waterfall to Agile. It is also less dependent on cross functional teams than Scrum. Using Kanban to manage tasks within the Design, Build and Test phases of a Waterfall project can prove very successful regardless of the wider methodologies in use. In addition, the ability for it to handle irregular, ad-hoc work makes it also especially good for managing defect resolutions within a build team or for a Operations/Service team handling incidents.

A 2010 case study with the BBC Worldwide team using Lean methods including Kanban recorded a 37% increase in lead time improvement, a 47% increase in delivery consistency and a 24% drop in defects reported. This was not all due to Kanban, but Kanban appears to have been a key ingredient together with smaller batch sizes, stand-ups and an empowered team.

I think that Kanban is a very powerful approach that can fit within your existing methodology well (including Waterfall) to drive benefit and importantly Continuous Improvement. For more information on Kanban check out these links:

http://www.kanban101.com/
http://en.wikipedia.org/wiki/Kanban_(development)
http://www.agileproductdesign.com/blog/2009/kanban_over_simplified.html
http://www.crisp.se/kanban
http://www.infoq.com/articles/agile-kanban-boards
http://www.infoq.com/articles/hiranabe-lean-agile-kanban
http://www.kanbanblog.com
http://en.wikipedia.org/wiki/Lean_IT

Enterprise IT Project Insanity

A study published in the Harvard Business Review has again shown that many IT projects continue to come in late and over budget. In addition it shows that there is a higher than expected number of large scale failures. These failures are massively over budget (200% in this study) and over deadline (70% overruns) and it cites examples where this has even contributed to the collapse of the company or at best reduced its profit forecasts leaving it at the mercy of the City. It’s interesting reading with the study showing that in total 27% of the 1471 projects overran in some way.

Now you could argue that these are mostly major IT transformation projects and hence the risks associated with them are bound to be significant (although this highlights another fact in that if the project was seen as transformational then perhaps there is an absence of a culture of continuous improvement at those organisations). Regardless though this is still damming evidence of the ability for this industry to implement IT projects. A comparison with other industries has been made on numerous occasions and IT tends to come out worst for achieving project success (although it’s not alone with many large defence sector projects for example suffering too). There are many reasons for IT projects to fail and I’m not going to cover them all here but instead ask the question why are we still repeating the same mistakes in many organisations. It has been 36 years since Frederick P Brooks wrote the seminal book "The Mythical Man Month" yet many of its key themes within his essays remain problems today. Walk around many traditional enterprise IT departments today and you need to avoid the Tar Pit on your way to joining the Death March on the "Tower Of Babel" project. For those of you not familiar with the book, the Wikipedia article on it summarizes it well. Some concepts will seem obvious and basic pitfalls but don’t forget this was written in a different age. The key points made in the book being the importance of progress tracking, tooling, communication and the iconic Mythical Man Month (whereby assigning more programmers to a project running behind schedule will make it even later, because of the time required for the new programmers to learn about the project, as well as the increased communication overhead).

So what has changed in 36 years? In some ways a lot has changed: The emergence of Lean/Agile practices and Motivation 3.0? (check out Daniel Pink’s work on intrinsic motivation) have dragged the thought leaders in the industry in the right direction (usually forced via a groundswell movement) but crucially the impact of these innovations vary by company and corporate cultures. It seems in many traditional enterprises little has changed. We continue to march on for projects destined to fail with old world approaches, planning in the ‘mythical man months’ based on ‘lies’ (sorry ‘estimates’). We still see people being thrown at a problem to resolve it despite this being a known anti-pattern. Many of the new approaches aimed at addressing these problems (e.g. lean/agile etc.) thrive in the smarter/learner enterprises and technology companies but have have struggled to get traction in the traditional red brick enterprises despite the impressive results of agile/iterative approaches. Even those enterprises that have adopted them have struggled to instil the philosophy behind Agile and instead just dogmatically implement an specific Agile methodology. The shift is happening but what’s stopping faster evolution in these enterprises?

Many of the issues could be classed as cultural, organizational or management issues and so can we can blame the senior management? Well that is way too simplistic and all stakeholders in an IT project have a part of play in its success or failure. Perhaps more likely is that the management tools, processes, procedures and attitudes that manage a modern enterprise don’t naturally fit to managing software development. In some ways this is not surprising as these enterprises are busy managing their core competencies (finance, manufacturing, construction, logistics etc.) with software development being done internally without the focus it perhaps needs. Take the world of banking as an example. A bank might spend millions on its IT and perhaps even consider some of it’s IT systems to be a competitive advantage, however it is a bank, not a software house and any activity within the organisation must fit the internal processes whether that fit is a natural one or not. In my post about the "Future of the IT Department" I covered how these IT departments have become large and unwieldy beasts. Fitting software development into a non-IT enterprise can be difficult and rigid. How many times have you had to bend software development reporting to fit into a model for which it doesn’t naturally fit? We have all had to dumb down technical analysis of issues/risks/progress into bite size chunks of simplistic bullet points that fit neatly into a PowerPoint slide but convey little in the way of accuracy. No doubt you get dragged into needless conference calls as a result of someone misunderstanding the technical situation. That’s not to say that those we report to are not intelligent (often far from it) but that they often lack the required skills to manage technical details. So what about the IT experts who progress up to management, they understand the problem right? Well yes they did, but it doesn’t take long to adjust to the culture of the organisation as it is a necessity for them to do so. They cannot do their job without following the processes that run the rest of the business and they need information/milestones/gateways to track progress. The industry is getting wise to this down at the development team level however as new tools emerge through the agile space that track the progress and velocity of a development team in easy to consume visual ways (e.g TFS v.next). Whilst these are not intended to be consumed at exec level I expect that more accurate reporting at the local level will result in better reporting flowing up the organisational structure. The pace of change in the industry is to blame also here for the managers of today were the IT guys of yesterday when different architecture landscapes were around and Mainframes were king.Where there is an inability for management and IT professionals to communicate and share a culture the gulf is filled with consultants and IT sales reps which can compound the problem. The use of Offshoring and outsourcing can also make the problem worse as the barriers to communication are now formal 3rd party engagements. We need to find a common ground to fill this cultural divide in an effort to help speed up the evolution of change.

It’s not unusual for these large enterprises to use Waterfall methodologies and tightly control the IT department resources so that IT can be managed neatly within the organisations financial and resource planning models. This will only get more engrained over the next few years as the economic climate dictates tighter financial controls and risk monitoring. This of course does little to aid the progress of IT evolution unless risks are taken to counteract the cost squeeze with more agility. The study mentioned above also highlighted that those projects that do succeed are often using Agile approaches and focusing on customer value, which makes sense. IT projects are one-off bespoke creations and yet are managed as though they were identical widgets produced on a production line. Enterprises are organised into a post war hierarchical command and control structure that is structured to actually avoid the communication between teams yet we still apply that model to the IT organisation in many enterprises. If we look at the likes of Google and Facebook, their developers are given more autonomy (and responsibility) for the software they build and its implementation, test, adoption and its evolution. They are encouraged to deploy often and innovate with trust placed upon them based on the fact that Software development is a skill and one developer is not as replaceable with another as is often assumed (explained brilliantly in John Miano’s post here on The Myth of the Interchangeable Programmer). In comparison the enterprise developer more often than not sits in India creating code based on detailed specifications (based on wrong assumptions at a point in time) separated by offshore coordinators, managers, on-shore coordinator’s, and a few thousand miles of undersea cable from the business they serve – stifling their ability to add value. When quality of output into test drops the next project is assigned more testing time and resource to catch defects, but at the expense of the design/development time. this leads to rushed design/build and again lower quality, so again more testing is required. And then in response there is a whole bureaucratic process of change management policies to rightly protect service. I’m not suggesting that change management processes and IT policies are not important (or indeed vital) but there is a balance between innovation and risk aversion that must be reached for the benefit of the company and for enterprise IT to evolve.

As the downturn in the economy bites cost controls will be tightened and IT departments trimmed, more often than not replaced by off shore labour, but this is an opportunity to rethink the approach and evolve the IT organisation into something leaner backed by more autonomy and lean processes. Agile/Iterative approaches have been proven to reduce costs and improve quality in comparison to traditional methods. Enterprises need to fully embrace Agile and encourage innovation within smaller business aligned teams of IT, relax some of the processes and abandon the often rigid enforcement of "Waterfall For All" and instead enable teams to choose the methodology that best fits.

IT within enterprises is evolving slowly but perhaps a revolution is required to speed things up, are you ready to revolt?

Embedding Pro-Active Tasks In Your Dev Team

We have made huge advances over recent years in the tools available to the development team, including the more proactive and investigative tools (profiling tools, code analysis, performance analysis, debugging etc). However demanding project timelines mean that we have increasingly less time to investigate, trial and use these tools. Compounding the problem is that unfortunately the first thing to get abandoned on a tight project are the proactive development tasks that lead to a better quality project but that don’t necessarily help get ‘something’ out the door. Obviously we need to try to get these approaches embedded into the development lifecycle despite their upfront costs. At first glance this seems a difficult challenge but then consider automated unit testing. This proactive task of developing tests alongside your code was a hard pill for many to swallow initially (and still is in some organisations) but as an industry we embedded the believe that the effort was worthwhile and the result was better quality, more tested and rigorous code. The same approach needs to be considered for other proactive tasks. Here’s a simple guide for getting something proactive adopted by your dev team:

Firstly the task need to be qualified: what is the task, what benefit does it provide, when would it be best used and what are the costs associated with not doing it?
Evangelize to the wider team. Hold demo’s of the approach to build awareness. Try to focus on one approach first and build a buzz around it so that it fixes in people’s daily psyche so it seems odd not to take the approach. Don’t forget to include the wider development stakeholders (Business Analysts, Architects, Project Managers) too as they may be impacted for better or worse. Use the concept of ‘Technical Debt‘ to help justify the long term impact of decisions affecting system quality.
Automate, automate, automate! How can you make it easier and quicker to get the initiative embedded in your development process? Can it be incorporated into your automated Continuous Integration solution for example?
The effort involved with the approach will need to be quantified so that they can be factored into development task estimates for project planning early enough to enable projects to be planned with these initiatives included. it is much harder for to find the time (and project manager commitment) for unplanned tasks.
Pilot the approach on small projects to be able to refine the approach and prove the benefits.
Include the approach in your ‘done lists’.
Vocalise and Visualise. Document the results/benefits of the approach and shout loudly when it avoids a production incident or missed deadline.

I would say that the most important step is the second one – Evangelising to your colleagues. it’s hard to stop a groundswell of enthusiasm for a new approach and you’ll progress much faster with peer support.

Is Your Gym Like Your Dev Team?

I recently managed to drag myself out of the office and into the gym, but unfortunately my mind was still on the office and my observations of what makes a dev team tick. In between sets I observed my fellow gym-goers and I witnessed similarities with my experiences of IT development teams. Parallels between your developer teams and the local gym can made both in terms of personas and in the approaches to gym training:

Exercise Machines vs. Free Weights:

When the Nautilus training machines (variants of which now fill every Gym) appeared in the early 1970’s they facilitated a revolution in exercise training and professional gyms. In contrast to free-weights (barbells, dumbbells etc) exercise machines provide a convenient and safe way of training. They don’t require a watcher and force the user through the ‘correct’ range of motion to avoid injury, also enabling forced reps etc. Like software frameworks these machine are built by expert engineers using solid (but also opinionated) ideas. They both enable new starters to get started easily and safely but they also share the trade offs. Machines/frameworks can lack fluidity and shelter the user from needing to understand the underlying principles at work. If the machine is out of service the gym user may not appreciate that they can achieve the same results via other methods. Remove the abstraction that the framework provides to the software developer and it may expose their lack of underlying skills (e.g. an ASP.NET Web Forms developer not appreciating HTTP). Of course the ‘correct’ way of doing something is always debateable and may not suit your needs for every project. Interestingly some pure "Bodybuilders" refuse to use machines for snobby reasons even when it would prove useful, whilst the majority of other gym users only use machines. The same can be said for developers and frameworks. An experienced all-rounder will happily use machines/frameworks where they are useful for productivity but will also utilise free-weights/alternative methods to achieve specific requirements.

Metrics:

Ask any professional athlete or Bodybuilder what calories they consume or the weights/reps/sets in their last gym session and they’ll tell you in detail. This is because they know the value of recording metrics and how to use them to track progress. The same principles can be applied to software development teams. What’s your current burn-down rate? What’s the average code churn figure for a nightly build? How many hours effort really went into building that MVC view compared to the estimate? A productive team that is continuously improving will be using these metrics to drive progress.

Agility:

Whilst solid athletes measure and plan they are also agile in their training – because they have to be. They have to adapt to changing training environments and to the subtle messages from their bodies to avoid injury and maintain productivity by focusing on the end goal. You wouldn’t expect them to stick rigidly to a plan defined months before despite changing circumstances (e.g: injuries, soreness), things change and so the journey towards the goal must be managed with flexibility.

Gym Buddies:

The benefits of having a gym buddy are clearly documented in the fitness world and for obvious reasons (shared motivation towards goals etc) and these benefits are so often overlooked in the development team. Pair Programming is a step in the right direction and is one technique that springs to mind but it is also just as important to foster a shared vision within the team and promote discussions and peer learning. A performing team is usually greater than the sum of its parts because people’s performance feeds off the ideas and motivation of their peers.

The Miracle Widget:

For those who don’t want the sweat and pain there’s always the miracle widget that will yield amassing results with little effort. Whether it’s a new machine, wonder drug, electronic shock training, sofa gyms, or SOA, Cloud Computing and BPM they need to be viewed with some apprehension. That’s not to say they aren’t the next big thing, but more that they are not silver bullets and they are used best within a cohesive thought out strategy.

Doping:

Taking steroids can rapidly improve an athletes performance but that improvement comes at a cost of unwanted side effects. The end goal may be rapidly becoming achieved but at the cost of internal physical or mental damage. This is form of extreme technical debt, taking a short cut here and there may be acceptable to ship the product but reliance on that short cut can build making it harder to reverse that debt.

Below I’ve noted some general stereotypical personas from the Gym and how they mirror development team personas. Do you recognise these roles in your gym/dev team? Warning: These are fun generalisations so don’t get upset or take it too seriously!

The Bodybuilder:

This guy has one goal in mind, to get ‘big’. All his exercises are anaerobic aimed at building muscle and developing his physique. He doesn’t do aerobic training as it detracts energy from his primary goal. He shows a strong ‘engineer like’ expertise of one discipline and he probably has excellent in-depth knowledge of that area and is very focused on learning more about it. He can be slightly intimidating to approach but generally happy to share his knowledge and experience and enjoys being able to show off his skills. This persona fits well with many traditional experienced software developers, who are experts in their chosen areas of discipline and increasingly seek to learn more about that technology area, often ignoring the benefits of others. They are dedicated and seen as experts in their field but outside their field they struggle and sometimes the imbalance with other disciplines has a negative effect.

The Endless Runner :

Similar to ‘the Bodybuilder’ above but this time in a different discipline. These guys want to run faster/longer and focus on aerobic exercises and building endurance. Again a solid, expert software engineer but this guy is not in it for the showy technology but more for building the plumbing infrastructure required to keep systems operation.

The All-Rounder :

He is not the biggest or fastest guy in the gym but he is the typical all rounder. He probably has experience of working in the various disciplines above (maybe mastering both) but prefers breath over depth. The All Rounder is able speak everyone’s language and can compete admirably with anyone else but has to submit to the overwhelming expertise of the guys listed above. Often this guy is a bridge between the different disciplines and chats in the corner with both. He gains the benefits that the variation and breadth of knowledge provides but is often at risk of not keeping up with the pace of change in either. His nearest IT persona is the architect due to his all round skills and his comfort liaising with all the required disciplines. He is happy to share his experiences when asked or when he sees someone really struggling, but is often less opinionated about one approach or another as he see’s all sides of the technology argument.

One Routine Guy:

A consistent gym attendee but does the same routine for years. We all know developers like this. They lack true ambition for the vocation and hence don’t build up a true understanding of the changing world around them. They are happy to use what they know and they feel works but the lack of willingness to learn new things puts them at risk of hitting a progress wall and finding themselves obsolete, eventually quitting.

Bored Stiff Guy :

They have decided to go the gym but have no real desire to do the workout. He runs through the motions, moving from task to task with little effort or intensity. We have all no doubt worked with developers who are going through the motions without any passion for the art of software development. Similar to ‘One Routine Guy’ they know what they need to know and lack any enthusiasm to learn new skills etc. I often refer to these as ‘Part Time Programmers’ as they see the job as 9 to 5 and the thought of picking up a new skill without company sponsored training course is alien to them.

The Biceps Only Guy:

Only focuses his energy on what can be shown off. He is just playing with the flashy stuff but without building a strong foundation to balance it with. For this guy ‘Gloss is Boss’. Some developers are happy playing with new technologies and building hundreds of "Hello World" apps but yet actually rarely innovate for the team as they fail to see the bigger picture.

The Poor Form Guy:

Is energetic and enthusiastic about training with heavy weight but inadvertently uses dangerously bad form in his exercises. Sometimes developers/architects can become so absorbed in delivering big solutions that they fail to assess their actions. They design complicated solutions using patterns they often don’t understand regardless of the project risks and the potential long term problems around stability/maintainability etc. Like ‘Poor Form Guy’ this is often a case of poor teaching or poor controls. These guys needs a coach or community to check their form.

The Personal Training Guru:

An expert in his field that takes in many disciples and guides people in their goals. Sort of a more senior experienced all rounder that is now dedicated to helping others. He has respect from his community and his advice is well respected. These are the guru’s in the tech world, experienced consultants/authors (e.g. Martin Fowler).

The Impatient For Results Guy:

He wants results and fast! He’s usually the customer for the ‘Miracle Wonder Widget’ (see below). Happy to take the easy option and cut corners on quality if he can. No doubt we’ve all worked with some bad development/project managers like this.

The Newbie:

He’s new to the gym and very intimidated. He’s still finding his feet with the machines and social etiquette. Just like new developers these are the live blood of the community as they bring enthusiasm and new ideas, but they need to be guided. They need assistance to get up the steep learning curve and be shown the right way to behave. If we make it hard for them to add value quickly then we risk them giving up and going elsewhere, or at least becoming a Bored Stiff Guy.

The Non-Conformist:

This guy is in the corner of the Gym doing his own thing. He’s probably using the equipment in a unique way, or using a less known training technique. He is innovative and might capable of producing amazing results using non standard approaches. He can be found in your development team too bashing out productivity tools and reviewing the latest open-source offerings. Regardless of his personal success he provides a fresh approach and generates new ways of working. He needs keeping in check though to ensure that his solutions are viable longer term.

The Non-Committed Local Gym Supervisor:

Whilst many gym supervisors are like the Personal Trainers some can be over focused on numbers (subscriptions, machine usage rules) more than real results. Once the new recruit is brought in they get given the user guide and then are left to it, with poor form being ignored as long as basic safety rules are adhered to. There can be a lack of evangelism of techniques, ideas etc., or of facilitating the creation of a real community in dev teams too which can lead them to fail. A lot of the success/failure of teams can result from the performance of the development manager / technical lead and their willingness to support the team to keep them productive.

Summary:

Of course the conclusion is that I should have been working out instead of ‘people watching’ but the fact remains that there are parallels that can be drawn between our work communities and many other walks of life. This opens up the ability for us to view situations from different perspectives which can then help us to improve our understanding.

The Future Of The IT Department

Recently I have been witness to rapid, often painful, change within my own internal IT division over the last few years and observed the on-going developments in the industry. It is clear that IT departments changed dramatically in a short amount of time and the pace is not relenting. This has led me to try to picture what IT will look like within large institutions in the future. It is becoming more and more apparent that the structure of our internal IT organisations are very often based on the traditional legacy models that served enterprises well in the past. Big IT investments and centralised systems are best managed and maintained by an rigid organisational structure. The IT department and the business units are today usually far more disconnected than many CIOs would care to admit. IT used to be something that was done by the IT department based on fairly static business processes. However we’re now in a different world, where IT is seen increasing as just a commodity and business processes need to be able to react quickly to changing economic conditions. No longer is the IT department responsible for big monolithic systems (e.g. payroll etc.) but IT is now embedded in every business process so in some sense every department is an IT department. Surely if the IT organisation doesn’t aid the business then it will be eventually pushed aside and replaced.

The Journey From Past to Present

This excellent post by PEG covers this subject well. PEG paints the picture of the traditional IT organisation as it was in many enterprises and then slices it up to represent the current model once outsourcing/off-shoring has been considered. The left hand diagram showing the more traditional split, and the right showing the emerging norm:

Diagrams from PEG: The IT department we have today is not the IT department we’ll need tomorrow

It surprises me how many people consider their jobs as not being under threat from outsourcing as they’re role is above the bottom tier on this sort of diagram, but as you can see it is inevitable that the line between permanent staff and outsource partner staff will continue to rise to the point represented in the triangle on the right, with a good cross section of IT roles being fulfilled by partner organisations. This represents where many large enterprises are at present whereby some “doing” roles are maintained in-house but the management and planning layers are also supplemented by outsource/offshore partners. The bulge in the middle represents the extra permanent resources required to cover the additional overhead of managing partner resources. Taking a bank to be the textbook example of a large enterprise with a significant scale IT organisation then this research into European banks activities provides some insight into the strategy driving these changes. Unsurprisingly cost reduction is key, but its not the only factor…

“Survey participants cited cost reduction as the primary reason to outsource IT functions, followed by cost variability (for example, the flexibility to respond to peak demand without ramping up internal resources) and access to know-how or skilled personnel. The main benefits of outsourcing were access to know-how or skilled personnel and a guaranteed level of service. (The cost benefits associated with outsourcing often fell short of expectations.) The biggest disadvantages of outsourcing were high switching costs and limited control over critical elements of the IT environment. On the whole, however, the survey shows that banks have embraced outsourcing. Only 3 percent of the banks surveyed were planning to decrease their outsourcing activities. The case for offshoring was slightly different. Although banks used offshoring primarily for the same reason they used outsourcing—to reduce costs—the main benefit of offshoring was less stringent foreign labour laws. The biggest disadvantages of offshoring were opposition among domestic personnel, large overhead, and loss of control.”

Both partner strategy models are therefore seen as suffering from elements of losing control of assets or deliverables and somewhat adding to management overheads, but providing some agility by providing a mechanism to ramp up or down resources as required.

PEG extends his model to show that in the future there will be an increased reliance on SaaS and automation tools and therefore a chunk of the IT organisation structure will be replaced by these as well as outsourcing/offshoring roles.

Diagram from PEG: The IT department we have today is not the IT department we’ll need tomorrow

Within the current model, management layers have often become too complex and unwieldy. With the IT organisation being a business entity itself within the enterprise and with 65% of IT spend just being used to maintain current service, business functions and IT often clash over priorities and the allocation of funding. In many instances resulting in the business going outside of the IT Org to secure services or growing their own ‘black ops’ internal capability just to get things done. This again challenges the traditional IT organisational model where IT keeps a tight control.

Changing Objectives

Tighter financial conditions, increasingly competitive environments and a desire to maximise returns is leading to a model of pay per use and more utilising of partners and outsourcing models. Technology advances are making this transition possible (e.g. Cloud Computing, SaaS). Future IT departments will increasingly utilise these external services resulting in them adopting a very different structure. Whilst the traditional IT organisation has been geared to building and maintaining large complex systems and is staffed with technical people, the rapidly emerging model is one where IT skills are outsourced to numerous vendors and IT staff become the negotiators and orchestrators of these relationships and contracts. Instead of managing systems changes internally the IT organisation is increasingly just the middleman between the business and the outsource/offshore partners. The role becomes one of managing projects more than technically implementing them. Reports can be found of in-house IT departments cutting 90% of headcount with a rapid shift to offshore/outsourcing with the remaining staff focusing on the planning and relationship management tasks. This Boston Consulting Group paper suggests there is an essential move from “doer” to “orchestrator”, with the IT Organisation “doing fewer of the traditional ‘run the business’ activities” instead leaving them to external providers and doing more coordinating of (one or many) providers activities to meet the design. This “network of external providers and integrators” needs monitoring and tuning and the structure of the IT Organisation will need to centre around these activities.

A quote from Reinventing The IT Organisation by Antoine Gourevitch, Stuart Scantlebury & Wolfgang Thiel…

“Unless CIOs take swift action, the IT organisation will be at risk of being reduced to a thin layer between the business and the specialist outsourcing firms.”

The outcome will presumably be either a slim organisation staffed with Change Managers and Project Managers responsible for liaising with the partners to satisfy business requirements, or alternatively these changes could prove the catalyst required to move to true business driven IT, where IT skills are integrated with the business units to enable them to react rapidly to changing business needs. Larry Dignan in his post welcomes the idea of breaking up the traditional IT organisation, seeing it as an anachronism. He classes CIOs as often “out of their league”, “process jockeys” who would “rather be scouting new technologies” than innovating. I would agree that this appears to be the case in many large organisations where IT, some would argue, has frustratingly become detached from the goal of driving business value through technology, losing itself in bureaucratic processes. These organisations can seem a long way from delivering core bottom line business value. PEG discusses the detachment of Enterprise Architecture and the business, together with a description of little ‘a’ and big ‘A’ architects, here and its well worth a read. Even where IT organisations do deliver real value its often to timescales that seem painfully long to the business customer but painfully short to the IT guy wrapped up in bureaucratic red tape. Perhaps this isn’t ITs fault as such but more the arcane structure of the IT organisation as we have come to accept.

One way suggested for IT organisations to remain relevant and address future challenges is for the business and IT to move closer together than ever. This has been talked about for many years but with the demise of the monolithic IT organisation the next few years could see this model mature. Perhaps decentralised pockets of business IT shops closely aligned to the business units will be the norm, introducing new challenges around how to control these pockets.

This shift towards IT/business integration could be very rewarding for an enterprise as in reality modern business processes are often tightly intertwined with the LOB applications in use and so anything that can be done to ensure that those LOB applications support the business processes instead of restricting the pace of business change will be welcomed. Dreischmeier & Thiel suggest new ways of working may be required as IT organisations are forced to adjust their operating model to become faster, more agile and to embrace rapid-development approaches. The business can’t afford to be held back by a slow and unwieldy IT organisation.

One concept I particularly like is the concept of “introducing Product or Solution Managers” to address the “lack of end to end ownership within IT Orgs”. The person would “own the IT product/solution across all technical layers”. This role should improve TCO and aid business & IT priority alignment. Dreischmeier & Thiel also see the CIO as a key player in ensuring that the IT organisation is “Proactively Engaging in Business Transformation Activities” and that even the IT organisation is very well positioned to be a key player in this transformation as it is aware of the end to end business processes (in theory). They suggest:

“Creating, together with the business, a new-business-model team that seeks out and addresses the changes in economics of the relevant industry as it changes through increased competition and environmental forces”.

The growth of agile development practices have a a part to play here too. Having innovative IT teams that ‘fail fast and often’ and use lean agile techniques to maximise business value could replace traditional models. Smaller, focused development teams under the direct control of the business units using Agile practices and being supported by a central infrastructure function (probably outsourced) could prove a very effective way of actually building what the business really need. The evolution of Cloud Computing technologies provides real opportunities to make these teams very capable. A business unit based developer could ‘mashup’ cloud services together with core on-premise web services to produce a powerful line of business application that is then deployed to PaaS cloud based infrastructure. Forester Analyst Alex Cullan sells the benefits of this model with the term “Empowered BT (Business Technology)” where IT’s role is to empower the business to utilise the technology that they need in order to remain competitive. The traditional arguments against this approach such as the expected system proliferation and business technology decisions being driven by hype, are dismissed as actually not as bad as we in IT would believe. He argues successfully that some proliferation is acceptable if it empowers the business, but there would have to be trust in business leaders to choose the right path for this to work. Is that trust there at this moment in time? Well not according to this MIT & Boston Consulting Group survey where it shows that current CIOs believe that business leaders are not positioned to lead IT enabled business transformation. Only 33% of CIOs consider their company’s senior execs effective at driving business value with IT, and 40% consider them effective at prioritizing IT investments. However perhaps this reflects the differences in the current differing priorities of the of traditional IT Organisations and the business units, with IT enforcing its traditional maintenance role (“keeping the lights on”) and role of application development/innovation more than a real distrust. The paper does however highlight the benefits that can be achieved when the IT organisation avoids the simple “middle man” role and takes the lead role of driving business change (such as lower maintenance costs, faster realisation of business benefits from new systems, and higher employee satisfaction). Perhaps the future of the IT organisation is that of a business in its own right, an internal consulting firm offering assistance in business process design, innovation and development management.

Proctor and Gamble run their IT Organisation as a business within the enterprise running alongside other business services (e.g accounting etc.). Their services are branded and marketed to the enterprise and billed on a usage basis with business units empowered to choose to consume these services or go elsewhere. The emphasis is on running this as a viable competitive internal business that is in tune with its customers (in this case the internal business units) needs. They have Brand managers responsible for “the innovation, pricing and commercialization of the services” that ensuring that the total end to end offerings can match that of 3rd party offerings. Underpinning this though is a collection of external partner relationships that still need to be managed and so in essence this is still heading towards becoming an integrator, orchestrating these partner services into a clear cohesive branded, and hopefully relevant, service. The key here though is the added value provided by this internal IT business service that crucially understands the business and offers competitive services that are completely relevant to the business. This is supported by the BCG research that found where IT Organisations really drove business change they often delivered their IT services as shared services and placed more emphasis on relevant prices and alternative service levels. They tended to centralise IT with lower levels of recorded “shadow” IT being instigated by the business, which could perhaps suggest that these business units felt they were getting sufficient value from their shared IT services, even though it was under central control.

Future Skills

All these changes have massive implications on the skills required within the IT organisation of the future. In the current model maintaining a relevant skilled workforce can be tricky with many key staff feeling demotivated by the outsourcing/offshoring partner model and the subsequent removal of technical roles from their organisation. The loss of junior IT roles to partner resources destroys any future progression opportunities and shows that this model is unsustainable moving forward. Engaging technical people will be increasingly difficult in the current model but perhaps a move to more business aligned IT can help skilled staff remain technical if they wish and also benefit the business through enhanced IT innovation and passion for their roles, instead of forcing good techies to oversee offshore/outsource relationships.

It seems essential now that IT staff of the near future will be expected to have an enhanced level of business acumen and market knowledge to fulfil their roles. Will this come at the expense of excellent technical skills? Maybe! Perhaps the technical skills will be embedded within the offshore/outsource partners and the relevant ‘technical’ skills required in the IT Organisation will be those around technical process design and system analysis. Knowledge of the business will perhaps be more important than any technical skill (for the majority of roles) and therefore it makes more sense to recruit IT staff from within the business units themselves. This is evident in a number of studies with CIOs, such as this BCG study…

“In general, CIOs told us that Internal IT staff roles are shifting away from application development and towards process analysis and engineering, business relationship management, project management and architecture design and implementation.”

Within the previously mentioned Proctor & Gamble organisation the same theme emerges as the skills reflect the role of IT within the organisation:

“..traditional IT is just 30% of what we do. If traditional IT is all a person masters, he or she will never be a leader here. The rest is about business knowledge. Those who embrace that approach will certainly increase their value…”

This view was supported by the previously mentioned study into European Banking, but it also went further, pointing out that technical skills were being neglected …

“…many banks appear to be underestimating the value of technical tools and skills, which are critical to developing high-impact applications, maintaining an efficient infrastructure, and managing outsourcing partners.”

So where does this leave you and I? Well, I expect the relevant number of deeply technical IT professionals will decline in Western countries but this decline will be dwarfed by the increase in semi-professional developers, working in the business but using end-user computing tools to develop systems that are meant to be rapid, easy and throw away. Where more complex solutions are sought then outsource partners will happily fill that gap. Escaping the large enterprises and fleeing to the small and medium enterprises will not be sustainable longer term either as the partner model will win there too eventually. It is entirely possible that the partner model will lose some of its lustre (it’s already happening in places) and there may be some swing back to in-house technical teams. If that happens then the IT community needs to be ready to promote a new ‘agile’ alternative that understands and drives true business benefits.

This evolution of the IT organisation is natural in such an immature industry as this but one thing is definite the future is different and we need to adapt. Whichever direction the future takes for you spend some effort in the meantime trying to understand your business customers needs better and keep innovating for them!