DevTeam – RichHewlett.com

Moving Sonar Rules closer to the Developer with ESLint

Shift code quality analysis to the left by moving your static analysis from the CI/CD pipeline to the developers IDE where possible. In this post I cover what we did, how to set up Sonar Lint and how we ultimately moved the Sonar rules into ESLint instead.

Problem

As a development team working on a JavaScript application we sometimes had issues where a difference between the rules enforced in our CI/CD pipeline, using SonarQube, and local rules, using ESLint, led to discrepancies in standards and ultimately, at times, broken builds. If you are enforcing rules in your project and in the CI/CD pipeline with SonarQube then indeally you need them to mostly match but this is not as easy as it sounds. Any failure in the pipeline are more time-consuming to resolve than if they happened locally before the developer pushed their code to the repository.

Solution

Import the Sonar rules into ESLint and force ESlint in both the IDE and the CI/CD pipeline.

In order to strike a balance of quality assurance and flexibility in the implementation of rules we introduced an approach that combines ESLint and Sonar rules with the emphasis on shift-left with rule enforcement done in the IDE as code is written and then re-enforced later in the CI/CD pipeline.

Firstly, Sonar Source (developers of SonarQube) provide a plugin for several IDEs (including VSCode) called SonarLint that helps address the issue of running Sonar rules in an IDE. Once installed it will analyse your code against default Sonar rules and report issues. What if you’re not running the default set of rules on your Sonar server, well no worries as the plugin can be set to connect to your server and download the right quality profile and rules.

To do this install the SonarLint extension into your IDE (many are supported, e.g. VS Code, Visual Studio etc) and then set the extension properties as per the instructions for your particular IDE. For VS Code it goes like this:

To link a server set the “sonarlint.connectedMode.connections.sonarqube” setting which has to be a USER setting (oddly). Then in workspace settings for the project you can configure the projectKey for your project. Workspace setting files are created in the .vscode folder in a settings.json file whcih can be added to source control so this only needs to be setup once per project. Once done, press F1 > type sonar > select “SonarLint: Update all project bindings to SonarQube” which will refresh the plugins cache and force it to download the rules from your Sonar server.

Now whilst SonarLint is a useful tool it is not as powerful as ESLint for linting in the IDE (in my opinion). For example ignoring a rule (for a genuine reason) at file or line level is not possible in a satisfactory way (you can only ignore from a line/file from ALL sonar rules). Also Eslint provides more power and flexibility especially where you have a centrally managed sonar server with shared rule profiles and quality gates that are not easy to change (which may be a good thing for your organisation).

So instead of, or even in additon to, SonarLint checking you can actually import the Sonar JavaScript scanner rules into ESLint. To do this install the npm package: eslint-plugin-sonar then configure your ESLint config to use the Sonar js/recommended) JS rules.

npm install eslint-plugin-sonar
Add it to your eslint config file as an extends
extends: [
“plugin:sonarjs/recommended”
]

Now ESLint will report quality errors that would previously only been highlighted in Sonar during a CI/CD pipeline build. This immediate feedback is more useful to the dev team and reduces the time wastage associated with broken builds. For us this ensured that apart from a few rules the majority are now in ESLint where developers can see them and resolve them, preventing the need for a CI/CD pipeline to highlight the problem (and broken builds).

To enforce the ESLint rules at build time we run the ESlint analysis during the CI/CD Pipeline by calling ESlint as a build step.

eslint --config eslintrc.json src

This means no ESLint errors will be let through. We also have a Sonar Quality Gate check configured in the Pipeline but as the majority of Sonar rules are now in ESLint we should only get a failure where a rule is breached that is only in the server Sonar Profile.

As an additional step we can also import all the ESlint issues found in to Sonar so that we can see them in the Sonar dashboard. You can export the ESlint rules as JSON for Sonar to import (during the build). To do this run this command in the build (ideally create a new npm script for it) assuming your src folder contains the source:

eslint --config .eslintrc.json --output-file ./eslint-report.json --format json src

Next set this Sonar property in your sonar-project.properties file or via command line argument (where eslint-report.json is the output report produced above).

sonar.eslint.reportPaths=eslint-report.json

Any issues from the ESLint report will then appear in Sonar issues marked with an EsLint badge. It appears warnings are added as majors and errors as Criticals and unfortunately I’ve not yet found a way to change this.

As a side note this command is also useful with eslint to output a HTML report of any errors which is great for reviewing or sharing:

eslint --config .eslintrc.json --output-file ./eslint-report.html --format html src

Summary

In summary, quality is being maintained as the same rules are enforced but now developers only need to ensure that ESlint is happy before committing their changes to source control to ensure a smooth server build. To make this easier you can add a new npm run script that runs all pre-commit checks that is triggered automatically (e.g. git hooks) or manually by the developer.

Enterprise IT Project Insanity

A study published in the Harvard Business Review has again shown that many IT projects continue to come in late and over budget. In addition it shows that there is a higher than expected number of large scale failures. These failures are massively over budget (200% in this study) and over deadline (70% overruns) and it cites examples where this has even contributed to the collapse of the company or at best reduced its profit forecasts leaving it at the mercy of the City. It’s interesting reading with the study showing that in total 27% of the 1471 projects overran in some way.

Now you could argue that these are mostly major IT transformation projects and hence the risks associated with them are bound to be significant (although this highlights another fact in that if the project was seen as transformational then perhaps there is an absence of a culture of continuous improvement at those organisations). Regardless though this is still damming evidence of the ability for this industry to implement IT projects. A comparison with other industries has been made on numerous occasions and IT tends to come out worst for achieving project success (although it’s not alone with many large defence sector projects for example suffering too). There are many reasons for IT projects to fail and I’m not going to cover them all here but instead ask the question why are we still repeating the same mistakes in many organisations. It has been 36 years since Frederick P Brooks wrote the seminal book "The Mythical Man Month" yet many of its key themes within his essays remain problems today. Walk around many traditional enterprise IT departments today and you need to avoid the Tar Pit on your way to joining the Death March on the "Tower Of Babel" project. For those of you not familiar with the book, the Wikipedia article on it summarizes it well. Some concepts will seem obvious and basic pitfalls but don’t forget this was written in a different age. The key points made in the book being the importance of progress tracking, tooling, communication and the iconic Mythical Man Month (whereby assigning more programmers to a project running behind schedule will make it even later, because of the time required for the new programmers to learn about the project, as well as the increased communication overhead).

So what has changed in 36 years? In some ways a lot has changed: The emergence of Lean/Agile practices and Motivation 3.0? (check out Daniel Pink’s work on intrinsic motivation) have dragged the thought leaders in the industry in the right direction (usually forced via a groundswell movement) but crucially the impact of these innovations vary by company and corporate cultures. It seems in many traditional enterprises little has changed. We continue to march on for projects destined to fail with old world approaches, planning in the ‘mythical man months’ based on ‘lies’ (sorry ‘estimates’). We still see people being thrown at a problem to resolve it despite this being a known anti-pattern. Many of the new approaches aimed at addressing these problems (e.g. lean/agile etc.) thrive in the smarter/learner enterprises and technology companies but have have struggled to get traction in the traditional red brick enterprises despite the impressive results of agile/iterative approaches. Even those enterprises that have adopted them have struggled to instil the philosophy behind Agile and instead just dogmatically implement an specific Agile methodology. The shift is happening but what’s stopping faster evolution in these enterprises?

Many of the issues could be classed as cultural, organizational or management issues and so can we can blame the senior management? Well that is way too simplistic and all stakeholders in an IT project have a part of play in its success or failure. Perhaps more likely is that the management tools, processes, procedures and attitudes that manage a modern enterprise don’t naturally fit to managing software development. In some ways this is not surprising as these enterprises are busy managing their core competencies (finance, manufacturing, construction, logistics etc.) with software development being done internally without the focus it perhaps needs. Take the world of banking as an example. A bank might spend millions on its IT and perhaps even consider some of it’s IT systems to be a competitive advantage, however it is a bank, not a software house and any activity within the organisation must fit the internal processes whether that fit is a natural one or not. In my post about the "Future of the IT Department" I covered how these IT departments have become large and unwieldy beasts. Fitting software development into a non-IT enterprise can be difficult and rigid. How many times have you had to bend software development reporting to fit into a model for which it doesn’t naturally fit? We have all had to dumb down technical analysis of issues/risks/progress into bite size chunks of simplistic bullet points that fit neatly into a PowerPoint slide but convey little in the way of accuracy. No doubt you get dragged into needless conference calls as a result of someone misunderstanding the technical situation. That’s not to say that those we report to are not intelligent (often far from it) but that they often lack the required skills to manage technical details. So what about the IT experts who progress up to management, they understand the problem right? Well yes they did, but it doesn’t take long to adjust to the culture of the organisation as it is a necessity for them to do so. They cannot do their job without following the processes that run the rest of the business and they need information/milestones/gateways to track progress. The industry is getting wise to this down at the development team level however as new tools emerge through the agile space that track the progress and velocity of a development team in easy to consume visual ways (e.g TFS v.next). Whilst these are not intended to be consumed at exec level I expect that more accurate reporting at the local level will result in better reporting flowing up the organisational structure. The pace of change in the industry is to blame also here for the managers of today were the IT guys of yesterday when different architecture landscapes were around and Mainframes were king.Where there is an inability for management and IT professionals to communicate and share a culture the gulf is filled with consultants and IT sales reps which can compound the problem. The use of Offshoring and outsourcing can also make the problem worse as the barriers to communication are now formal 3rd party engagements. We need to find a common ground to fill this cultural divide in an effort to help speed up the evolution of change.

It’s not unusual for these large enterprises to use Waterfall methodologies and tightly control the IT department resources so that IT can be managed neatly within the organisations financial and resource planning models. This will only get more engrained over the next few years as the economic climate dictates tighter financial controls and risk monitoring. This of course does little to aid the progress of IT evolution unless risks are taken to counteract the cost squeeze with more agility. The study mentioned above also highlighted that those projects that do succeed are often using Agile approaches and focusing on customer value, which makes sense. IT projects are one-off bespoke creations and yet are managed as though they were identical widgets produced on a production line. Enterprises are organised into a post war hierarchical command and control structure that is structured to actually avoid the communication between teams yet we still apply that model to the IT organisation in many enterprises. If we look at the likes of Google and Facebook, their developers are given more autonomy (and responsibility) for the software they build and its implementation, test, adoption and its evolution. They are encouraged to deploy often and innovate with trust placed upon them based on the fact that Software development is a skill and one developer is not as replaceable with another as is often assumed (explained brilliantly in John Miano’s post here on The Myth of the Interchangeable Programmer). In comparison the enterprise developer more often than not sits in India creating code based on detailed specifications (based on wrong assumptions at a point in time) separated by offshore coordinators, managers, on-shore coordinator’s, and a few thousand miles of undersea cable from the business they serve – stifling their ability to add value. When quality of output into test drops the next project is assigned more testing time and resource to catch defects, but at the expense of the design/development time. this leads to rushed design/build and again lower quality, so again more testing is required. And then in response there is a whole bureaucratic process of change management policies to rightly protect service. I’m not suggesting that change management processes and IT policies are not important (or indeed vital) but there is a balance between innovation and risk aversion that must be reached for the benefit of the company and for enterprise IT to evolve.

As the downturn in the economy bites cost controls will be tightened and IT departments trimmed, more often than not replaced by off shore labour, but this is an opportunity to rethink the approach and evolve the IT organisation into something leaner backed by more autonomy and lean processes. Agile/Iterative approaches have been proven to reduce costs and improve quality in comparison to traditional methods. Enterprises need to fully embrace Agile and encourage innovation within smaller business aligned teams of IT, relax some of the processes and abandon the often rigid enforcement of "Waterfall For All" and instead enable teams to choose the methodology that best fits.

IT within enterprises is evolving slowly but perhaps a revolution is required to speed things up, are you ready to revolt?

Embedding Pro-Active Tasks In Your Dev Team

We have made huge advances over recent years in the tools available to the development team, including the more proactive and investigative tools (profiling tools, code analysis, performance analysis, debugging etc). However demanding project timelines mean that we have increasingly less time to investigate, trial and use these tools. Compounding the problem is that unfortunately the first thing to get abandoned on a tight project are the proactive development tasks that lead to a better quality project but that don’t necessarily help get ‘something’ out the door. Obviously we need to try to get these approaches embedded into the development lifecycle despite their upfront costs. At first glance this seems a difficult challenge but then consider automated unit testing. This proactive task of developing tests alongside your code was a hard pill for many to swallow initially (and still is in some organisations) but as an industry we embedded the believe that the effort was worthwhile and the result was better quality, more tested and rigorous code. The same approach needs to be considered for other proactive tasks. Here’s a simple guide for getting something proactive adopted by your dev team:

Firstly the task need to be qualified: what is the task, what benefit does it provide, when would it be best used and what are the costs associated with not doing it?
Evangelize to the wider team. Hold demo’s of the approach to build awareness. Try to focus on one approach first and build a buzz around it so that it fixes in people’s daily psyche so it seems odd not to take the approach. Don’t forget to include the wider development stakeholders (Business Analysts, Architects, Project Managers) too as they may be impacted for better or worse. Use the concept of ‘Technical Debt‘ to help justify the long term impact of decisions affecting system quality.
Automate, automate, automate! How can you make it easier and quicker to get the initiative embedded in your development process? Can it be incorporated into your automated Continuous Integration solution for example?
The effort involved with the approach will need to be quantified so that they can be factored into development task estimates for project planning early enough to enable projects to be planned with these initiatives included. it is much harder for to find the time (and project manager commitment) for unplanned tasks.
Pilot the approach on small projects to be able to refine the approach and prove the benefits.
Include the approach in your ‘done lists’.
Vocalise and Visualise. Document the results/benefits of the approach and shout loudly when it avoids a production incident or missed deadline.

I would say that the most important step is the second one – Evangelising to your colleagues. it’s hard to stop a groundswell of enthusiasm for a new approach and you’ll progress much faster with peer support.

Is Your Gym Like Your Dev Team?

I recently managed to drag myself out of the office and into the gym, but unfortunately my mind was still on the office and my observations of what makes a dev team tick. In between sets I observed my fellow gym-goers and I witnessed similarities with my experiences of IT development teams. Parallels between your developer teams and the local gym can made both in terms of personas and in the approaches to gym training:

Exercise Machines vs. Free Weights:

When the Nautilus training machines (variants of which now fill every Gym) appeared in the early 1970’s they facilitated a revolution in exercise training and professional gyms. In contrast to free-weights (barbells, dumbbells etc) exercise machines provide a convenient and safe way of training. They don’t require a watcher and force the user through the ‘correct’ range of motion to avoid injury, also enabling forced reps etc. Like software frameworks these machine are built by expert engineers using solid (but also opinionated) ideas. They both enable new starters to get started easily and safely but they also share the trade offs. Machines/frameworks can lack fluidity and shelter the user from needing to understand the underlying principles at work. If the machine is out of service the gym user may not appreciate that they can achieve the same results via other methods. Remove the abstraction that the framework provides to the software developer and it may expose their lack of underlying skills (e.g. an ASP.NET Web Forms developer not appreciating HTTP). Of course the ‘correct’ way of doing something is always debateable and may not suit your needs for every project. Interestingly some pure "Bodybuilders" refuse to use machines for snobby reasons even when it would prove useful, whilst the majority of other gym users only use machines. The same can be said for developers and frameworks. An experienced all-rounder will happily use machines/frameworks where they are useful for productivity but will also utilise free-weights/alternative methods to achieve specific requirements.

Metrics:

Ask any professional athlete or Bodybuilder what calories they consume or the weights/reps/sets in their last gym session and they’ll tell you in detail. This is because they know the value of recording metrics and how to use them to track progress. The same principles can be applied to software development teams. What’s your current burn-down rate? What’s the average code churn figure for a nightly build? How many hours effort really went into building that MVC view compared to the estimate? A productive team that is continuously improving will be using these metrics to drive progress.

Agility:

Whilst solid athletes measure and plan they are also agile in their training – because they have to be. They have to adapt to changing training environments and to the subtle messages from their bodies to avoid injury and maintain productivity by focusing on the end goal. You wouldn’t expect them to stick rigidly to a plan defined months before despite changing circumstances (e.g: injuries, soreness), things change and so the journey towards the goal must be managed with flexibility.

Gym Buddies:

The benefits of having a gym buddy are clearly documented in the fitness world and for obvious reasons (shared motivation towards goals etc) and these benefits are so often overlooked in the development team. Pair Programming is a step in the right direction and is one technique that springs to mind but it is also just as important to foster a shared vision within the team and promote discussions and peer learning. A performing team is usually greater than the sum of its parts because people’s performance feeds off the ideas and motivation of their peers.

The Miracle Widget:

For those who don’t want the sweat and pain there’s always the miracle widget that will yield amassing results with little effort. Whether it’s a new machine, wonder drug, electronic shock training, sofa gyms, or SOA, Cloud Computing and BPM they need to be viewed with some apprehension. That’s not to say they aren’t the next big thing, but more that they are not silver bullets and they are used best within a cohesive thought out strategy.

Doping:

Taking steroids can rapidly improve an athletes performance but that improvement comes at a cost of unwanted side effects. The end goal may be rapidly becoming achieved but at the cost of internal physical or mental damage. This is form of extreme technical debt, taking a short cut here and there may be acceptable to ship the product but reliance on that short cut can build making it harder to reverse that debt.

Below I’ve noted some general stereotypical personas from the Gym and how they mirror development team personas. Do you recognise these roles in your gym/dev team? Warning: These are fun generalisations so don’t get upset or take it too seriously!

The Bodybuilder:

This guy has one goal in mind, to get ‘big’. All his exercises are anaerobic aimed at building muscle and developing his physique. He doesn’t do aerobic training as it detracts energy from his primary goal. He shows a strong ‘engineer like’ expertise of one discipline and he probably has excellent in-depth knowledge of that area and is very focused on learning more about it. He can be slightly intimidating to approach but generally happy to share his knowledge and experience and enjoys being able to show off his skills. This persona fits well with many traditional experienced software developers, who are experts in their chosen areas of discipline and increasingly seek to learn more about that technology area, often ignoring the benefits of others. They are dedicated and seen as experts in their field but outside their field they struggle and sometimes the imbalance with other disciplines has a negative effect.

The Endless Runner :

Similar to ‘the Bodybuilder’ above but this time in a different discipline. These guys want to run faster/longer and focus on aerobic exercises and building endurance. Again a solid, expert software engineer but this guy is not in it for the showy technology but more for building the plumbing infrastructure required to keep systems operation.

The All-Rounder :

He is not the biggest or fastest guy in the gym but he is the typical all rounder. He probably has experience of working in the various disciplines above (maybe mastering both) but prefers breath over depth. The All Rounder is able speak everyone’s language and can compete admirably with anyone else but has to submit to the overwhelming expertise of the guys listed above. Often this guy is a bridge between the different disciplines and chats in the corner with both. He gains the benefits that the variation and breadth of knowledge provides but is often at risk of not keeping up with the pace of change in either. His nearest IT persona is the architect due to his all round skills and his comfort liaising with all the required disciplines. He is happy to share his experiences when asked or when he sees someone really struggling, but is often less opinionated about one approach or another as he see’s all sides of the technology argument.

One Routine Guy:

A consistent gym attendee but does the same routine for years. We all know developers like this. They lack true ambition for the vocation and hence don’t build up a true understanding of the changing world around them. They are happy to use what they know and they feel works but the lack of willingness to learn new things puts them at risk of hitting a progress wall and finding themselves obsolete, eventually quitting.

Bored Stiff Guy :

They have decided to go the gym but have no real desire to do the workout. He runs through the motions, moving from task to task with little effort or intensity. We have all no doubt worked with developers who are going through the motions without any passion for the art of software development. Similar to ‘One Routine Guy’ they know what they need to know and lack any enthusiasm to learn new skills etc. I often refer to these as ‘Part Time Programmers’ as they see the job as 9 to 5 and the thought of picking up a new skill without company sponsored training course is alien to them.

The Biceps Only Guy:

Only focuses his energy on what can be shown off. He is just playing with the flashy stuff but without building a strong foundation to balance it with. For this guy ‘Gloss is Boss’. Some developers are happy playing with new technologies and building hundreds of "Hello World" apps but yet actually rarely innovate for the team as they fail to see the bigger picture.

The Poor Form Guy:

Is energetic and enthusiastic about training with heavy weight but inadvertently uses dangerously bad form in his exercises. Sometimes developers/architects can become so absorbed in delivering big solutions that they fail to assess their actions. They design complicated solutions using patterns they often don’t understand regardless of the project risks and the potential long term problems around stability/maintainability etc. Like ‘Poor Form Guy’ this is often a case of poor teaching or poor controls. These guys needs a coach or community to check their form.

The Personal Training Guru:

An expert in his field that takes in many disciples and guides people in their goals. Sort of a more senior experienced all rounder that is now dedicated to helping others. He has respect from his community and his advice is well respected. These are the guru’s in the tech world, experienced consultants/authors (e.g. Martin Fowler).

The Impatient For Results Guy:

He wants results and fast! He’s usually the customer for the ‘Miracle Wonder Widget’ (see below). Happy to take the easy option and cut corners on quality if he can. No doubt we’ve all worked with some bad development/project managers like this.

The Newbie:

He’s new to the gym and very intimidated. He’s still finding his feet with the machines and social etiquette. Just like new developers these are the live blood of the community as they bring enthusiasm and new ideas, but they need to be guided. They need assistance to get up the steep learning curve and be shown the right way to behave. If we make it hard for them to add value quickly then we risk them giving up and going elsewhere, or at least becoming a Bored Stiff Guy.

The Non-Conformist:

This guy is in the corner of the Gym doing his own thing. He’s probably using the equipment in a unique way, or using a less known training technique. He is innovative and might capable of producing amazing results using non standard approaches. He can be found in your development team too bashing out productivity tools and reviewing the latest open-source offerings. Regardless of his personal success he provides a fresh approach and generates new ways of working. He needs keeping in check though to ensure that his solutions are viable longer term.

The Non-Committed Local Gym Supervisor:

Whilst many gym supervisors are like the Personal Trainers some can be over focused on numbers (subscriptions, machine usage rules) more than real results. Once the new recruit is brought in they get given the user guide and then are left to it, with poor form being ignored as long as basic safety rules are adhered to. There can be a lack of evangelism of techniques, ideas etc., or of facilitating the creation of a real community in dev teams too which can lead them to fail. A lot of the success/failure of teams can result from the performance of the development manager / technical lead and their willingness to support the team to keep them productive.

Summary:

Of course the conclusion is that I should have been working out instead of ‘people watching’ but the fact remains that there are parallels that can be drawn between our work communities and many other walks of life. This opens up the ability for us to view situations from different perspectives which can then help us to improve our understanding.