Team Development – RichHewlett.com

Moving Sonar Rules closer to the Developer with ESLint

Shift code quality analysis to the left by moving your static analysis from the CI/CD pipeline to the developers IDE where possible. In this post I cover what we did, how to set up Sonar Lint and how we ultimately moved the Sonar rules into ESLint instead.

Problem

As a development team working on a JavaScript application we sometimes had issues where a difference between the rules enforced in our CI/CD pipeline, using SonarQube, and local rules, using ESLint, led to discrepancies in standards and ultimately, at times, broken builds. If you are enforcing rules in your project and in the CI/CD pipeline with SonarQube then indeally you need them to mostly match but this is not as easy as it sounds. Any failure in the pipeline are more time-consuming to resolve than if they happened locally before the developer pushed their code to the repository.

Solution

Import the Sonar rules into ESLint and force ESlint in both the IDE and the CI/CD pipeline.

In order to strike a balance of quality assurance and flexibility in the implementation of rules we introduced an approach that combines ESLint and Sonar rules with the emphasis on shift-left with rule enforcement done in the IDE as code is written and then re-enforced later in the CI/CD pipeline.

Firstly, Sonar Source (developers of SonarQube) provide a plugin for several IDEs (including VSCode) called SonarLint that helps address the issue of running Sonar rules in an IDE. Once installed it will analyse your code against default Sonar rules and report issues. What if you’re not running the default set of rules on your Sonar server, well no worries as the plugin can be set to connect to your server and download the right quality profile and rules.

To do this install the SonarLint extension into your IDE (many are supported, e.g. VS Code, Visual Studio etc) and then set the extension properties as per the instructions for your particular IDE. For VS Code it goes like this:

To link a server set the “sonarlint.connectedMode.connections.sonarqube” setting which has to be a USER setting (oddly). Then in workspace settings for the project you can configure the projectKey for your project. Workspace setting files are created in the .vscode folder in a settings.json file whcih can be added to source control so this only needs to be setup once per project. Once done, press F1 > type sonar > select “SonarLint: Update all project bindings to SonarQube” which will refresh the plugins cache and force it to download the rules from your Sonar server.

Now whilst SonarLint is a useful tool it is not as powerful as ESLint for linting in the IDE (in my opinion). For example ignoring a rule (for a genuine reason) at file or line level is not possible in a satisfactory way (you can only ignore from a line/file from ALL sonar rules). Also Eslint provides more power and flexibility especially where you have a centrally managed sonar server with shared rule profiles and quality gates that are not easy to change (which may be a good thing for your organisation).

So instead of, or even in additon to, SonarLint checking you can actually import the Sonar JavaScript scanner rules into ESLint. To do this install the npm package: eslint-plugin-sonar then configure your ESLint config to use the Sonar js/recommended) JS rules.

npm install eslint-plugin-sonar
Add it to your eslint config file as an extends
extends: [
“plugin:sonarjs/recommended”
]

Now ESLint will report quality errors that would previously only been highlighted in Sonar during a CI/CD pipeline build. This immediate feedback is more useful to the dev team and reduces the time wastage associated with broken builds. For us this ensured that apart from a few rules the majority are now in ESLint where developers can see them and resolve them, preventing the need for a CI/CD pipeline to highlight the problem (and broken builds).

To enforce the ESLint rules at build time we run the ESlint analysis during the CI/CD Pipeline by calling ESlint as a build step.

eslint --config eslintrc.json src

This means no ESLint errors will be let through. We also have a Sonar Quality Gate check configured in the Pipeline but as the majority of Sonar rules are now in ESLint we should only get a failure where a rule is breached that is only in the server Sonar Profile.

As an additional step we can also import all the ESlint issues found in to Sonar so that we can see them in the Sonar dashboard. You can export the ESlint rules as JSON for Sonar to import (during the build). To do this run this command in the build (ideally create a new npm script for it) assuming your src folder contains the source:

eslint --config .eslintrc.json --output-file ./eslint-report.json --format json src

Next set this Sonar property in your sonar-project.properties file or via command line argument (where eslint-report.json is the output report produced above).

sonar.eslint.reportPaths=eslint-report.json

Any issues from the ESLint report will then appear in Sonar issues marked with an EsLint badge. It appears warnings are added as majors and errors as Criticals and unfortunately I’ve not yet found a way to change this.

As a side note this command is also useful with eslint to output a HTML report of any errors which is great for reviewing or sharing:

eslint --config .eslintrc.json --output-file ./eslint-report.html --format html src

Summary

In summary, quality is being maintained as the same rules are enforced but now developers only need to ensure that ESlint is happy before committing their changes to source control to ensure a smooth server build. To make this easier you can add a new npm run script that runs all pre-commit checks that is triggered automatically (e.g. git hooks) or manually by the developer.

Physical vs Virtual Kanban Boards

In previous posts I covered an introduction to Kanban and a review of a trial Kanban project. In this, my third in a series of Kanban posts, I’ll cover physical versus software Kanban boards. For some this is an almost religious debate, and many would not question their decision for one other another. Whilst the answer for which is best will, of course, always be ‘it depends’, there are some fundamental differences that exist. I generally find that there is an expectation within IT that you virtualise everything, because well that’s what we do. Whilst this is often true I would add that a physical board should not be underestimated as out-dated and indeed should be considered equal for its significant merits in many areas.

Due to recent growth of Kanban there are now many software Kanban tools providing varying levels of functionality but all offering at the very least a virtual multi-user Kanban task-board. For a comprehensive list of these tools check out this AgileScout post. If you don’t like these tools then could always create your own in MS Access, Excel or even Word if you like. These might be enough to get you started for a Kanban proof of concept project but will of course lack the features built into specialist software such as MI, history tracking, cadence monitoring and reporting. Similarly physical boards can start out as being lines drawn on a whiteboard or string pinned out on a pin-board, then littered with post-it-notes. Either way the barrier to entry is low, although the Software As A Service (SaaS) model used by the many web based Kanban board providers will mean there will be a long term cost associated with utilizing a 3rd party tool. An alternative to a SaaS model would be to use on-premise Kanban software. For teams using Team Foundation Server you can easily build on your existing infrastructure by adding a Kanban viewer for your TFS work items. Telerik Work Item Manager for TFS has a taskboard view that can be customised and there are other Kanban based offerings that integrate with TFS (e.g. http://visualwip.codeplex.com). Microsoft can obviously see the power of this approach considering it’s now built into Team Foundation Server 11 which will have a web based taskboard view via the Web Access application, complete with workflow rules and drag & drop updates. Check out the screenshot below.

The first real difference between a physical and virtual board is the sheer presence that a physical board gives. On my project documented in part 2 of this Kanban series the move from a virtual software taskboard (albeit a very simple one) to a physical board made the whole method much more tangible and “real” to the team. It moved from being another software tracker to update to something different, something big and bold and in the room! You’re not going to miss a physical board on your way to the coffee machine forcing them to become a talking point in the office. Project stakeholders pay attention to it as it stands out when they’re in the team’s work area. The fact that we have hundreds of existing software based processes makes the introduction of a physical one slightly alien to the team and therefore grabs attention. Size is important here too, as teams can gather around the board for stand-ups ensuring a shared vision. Teams struggle to gather around a PC screen or print out. This ‘presence’ feature can be replicated for a virtual Kanban board by utilizing a large LCD screen and I would recommend that if you can do it.

Of course the presence of a physical board is only good if your team is co-located. Remote teams are a challenge with Kanban as they are with any method or methodology and therefore a virtual board may be the only viable solution. In my recent project I utilised the on-site coordinator to act as a remote team proxy for the board which can work but is by no means ideal. A virtual tool allows everyone in the team to contribute regardless of their location.

A key aspect of the Kanban method is its simplicity and flexibility. Any board whether physical or virtual must be easily accessed, read, updated and changed. Continuous improvements to your processes are key and changing the board to reflect those changes should not offer any resistance. Complicated physical boards can be difficult to change as can some of the virtual tools. There’s no clear winner here but I would keep this in mind when reviewing Kanban software tools or when building a complex physical board.

Virtual boards provide the ability to easily produce MI and reporting information from your workflow, something just not possible with a physical board without manual work which could slow down the process. The ability for a team to track underlying trends and have up to date cadence figures is a huge benefit for an experienced Kanban team. I would suggest it may have less benefit for a team new to the process however as the Kanban method suggests that the board alone should be able to provide the data you need, but of course its not prescriptive and is open to adaption as a team sees fit.

One hurdle that some organisations struggle with is data security and privacy. Whether you class the data on your board as sensitive or not may influence your decision on the type of board to use. On one hand Virtual boards provide the ability to restrict access to verified users therefore maintain data security but on the other hand, many Kanban tools are externally hosted and therefore you are trusting your data to a third party. It’s not just security of course but also reliability of the vendor, e.g I’d hate to be mid project when my Kanban board disappears because the vendor has ceased trading. A physical board has its own challenges. Anyone in your office can see the board with no restriction on user roles. Without an automated change history it is also open to malicious changes from disgruntled colleagues. Backing up a virtual board is second nature but its harder to backup/restore a physical board, however taking regular photos is the best way I have found to date.

When I consider what would be my ideal board it would have to be easy, flexible, big and as integrated as possible with our existing systems. My ideal would be a virtual board with all the benefits that brings (especially remote teams) but it would have to have real presence in the office such as a large LCD on the wall constantly displaying the work in progress. In addition the screen would be a touch screen or alternatively put a terminal under display that was always logged in and used solely for the purpose of team members to update the board, thus removing the barrier for people to update the board. Of course the team could log on from their machines and update board but I see real benefit in walking up to the board to make changes, especially if required during a stand-up. This replicates the ability for you to physically move that post it note on the physical board. This setup should provide the majority of benefits of both approaches.

In summary, the choice is a personal one for your team and your circumstances, but for teams new to Kanban I wouldn’t underestimate the importance of having something bold that stands out and demands attention from the team and this is usually easier to implement with a physical board. Once the process is embedded and the board becomes a key part of the teams day then this would become less important and the power of virtual boards can be maximized.

For an excellent list of Kanban tools check out: http://agilescout.com/best-kanban-tools/.

A Trial Kanban Project – What Worked. What Didn’t.

Recently I introduced Kanban to an inflight development project with the objective of trialling this technique with the team to raise productivity by overcoming various communication and process issues. In the first part of this 3 part series I presented an introduction to Kanban. This second part will cover the results obtained by introducing it to an in-flight project and part 3 will give some fuel to the “Physical versus Virtual Kanban boards” debate.

Project Background

This was a service release aiming to investigate and resolve a fixed set of production defects on a newly implemented enterprise scale system. The team consisted of 6 people onshore including Project Manager, Work Packet Manager, Technical Architect, System Analyst and 12 offshore developers (with limited experience of the system). This was a Waterfall project with the project plan based on the agreed prioritized defect list, a fixed resource and time-boxed approach. The number and complex nature of the defect investigations together with dependency management (with other teams) made the tracking of items within the project difficult and opaque which exposed the lack of a structured workflow within the team.

Approach Taken

Mid project I suggested the team adopt a Kanban approach and after explaining the concept to the team quickly got their buy-in. A quick virtual task-board was introduced (in Microsoft Excel initially) with columns defined based on the current informal process with added enforced quality assurance steps. Each work item (defect or change request) was added to the board and tracked, with updates applied daily by the team, as a very minimum, during a morning stand-up.

The task-board immediately brought a visual dimension to the project status making managing work in progress (WIP)easier and it also helped to bring focus to the daily stand-ups. After this initial success, to overcome the limitations of the virtual board and to get wider team involvement, the project moved to a physical board with very positive results.

The new board evolved from the previous version based on feedback and was simpler to use. Its physical nature and the fact that it was always on display for the team to consume made a huge difference and resulted in full team commitment to the approach. The physical board became the focal point for the daily stand-ups and for ad-hoc progress updates. The off-shore’s teams updates were applied regularly by the onshore coordinator. Team members used the Kanban queues to action work and move them to the next queue, monitoring/resolving blockages.

What worked well

Implementing Kanban was particularly easy compared to other methods and approaches. As the method logically represents a team to-do list it was easy to explain to people and therefore was not seen as a new complicated process which no doubt aided adoption with the team. It was immediately apparent that this method would compliment and not conflict with existing methodologies and processes. The fact that the board initially should mirror you current processes combined with the fact that it is a method not a methodology meant that it easily slotted in. Again this helped with team adoption and it received a very positive reception from the project team.

The team liked that Kanban immediately brought a visual dimension to project status. They could quickly see items in their queues and where the bottlenecks (if any) were. One quick glance at the board showed the current status of all items. This enabled the team to quickly provide progress updates and set realistic expectations with stakeholders outside the project team. Effort was saved by avoiding wasted deployments into test environments by being able to see at a glance what was going to be ready to deploy in the next build and then set early expectations with the QA team on whether they wanted to take the build or not.

As the flow of work was visualized and monitored it became easier to enforce the current process stages more explicitly. An item would have to pass through the “code review” stage before it could get to the “Ready to Deploy” stage – not only that but the process was completely transparent and clear to everyone involved. This aided communication too which facilitated the current process. As the process was transparent, and the team were able to use the stages as a common ‘vocabulary’, amendments to the process were easily discussed and small incremental process improvements were made, increasing the efficiency of the team. Kanban encourages this constant improvement and the flexibility of the method ensures that it is not a barrier to change.

From a project management perspective the board provided an excellent starting point for meeting updates and status tracking. With all WIP now visible the control of that work, and the resources involved became easier. It was this visibly that actually triggered a useful resource allocation investigation resulting is resources being discovered to be under-utilised in one area and enabled reallocation.

On this project the move from a virtual software task-board (albeit a very simple one) to a physical board made a huge difference and it made the whole method very tangible and “real”. Not just another piece of software to update but something different, something big and bold and in the room!

What didn’t work so well

With any new method a team has to find its feet and test the boundaries of what is productive and what isn’t. The rules to our process and our task board were not as clearly communicated to the whole team as at first thought resulting in some lack of full team ownership and involvement in the first few days. This was evidenced by some of the team acting as purely ‘read-only’ users initially until they were encouraged to fully participate and engage with the method. Despite being a simple concept, without an explanation of the columns their meaning was open to a users interpretation. These were not problems with the Kanban method but merely communication problems that were quickly rectified.

It was discovered that the Kanban board could become misused. There was an urge to overcomplicate the board by adding more detail to the columns and the cards (such as expected delivery date and resource names). As our first board was virtual we found that additional information could easily be added without much control. This was stamped out early on after discussions within the team led to the information that was only required for reporting and tracking reasons being added or supplied from elsewhere as required. The board was simplified again and then a subsequent move over to a physical board helped eradicate this problem.

We found that it was possible for the Kanban board to be used to just display the current status of work as opposed to being used as intended to drive work allocation. The board was being updated daily to reflect the days progress but the board was not driving the workflow and the team was not using the queues to allocate all their work. This meant that it was still adding value as an information radiator but not enabling its full potential. A similar problem was that some WIP items were not added to the board at all but managed separately. This was compounded if stand-ups were driven from an alternative list (defect tracker tool, meeting notes) instead of the Kanban board. All team members had to be reminded that all WIP had to be on the board no matter what the work was was, as it all had to be made visible. Luckily these issues were overcome by reminding people to use the board as the “one truth” and ensuring that all work was progressed via the board, which in a short time became embedded in the team practices.

An area where Kanban (and indeed most Agile methods) suffers is the ‘Remote Team’ scenario. The common assumption that all the teams are co-located is a common failing but one that has to be overcome using practical solutions where possible. Ideally teams need to be co-located for maximum efficiency but the use of telepresence, chat tools and conference calls can help. On this project we managed with an on-shore coordinator acting as a proxy to the offshore team and the Kanban board, combined with usual modern communication methods. Use of a good virtual Kanban board would potentially resolve some of these issues. To be clear Kanban did not introduce any additional impediments to remote teams than didn’t already exist within the team and its methodology.

Summary

This was just an informal trial of Kanban on a inflight project and so in the future I would like to extend the trial to other projects but with some adjustments. At project kick off I would not assume that the whole team had an understanding of the columns and the workflow process, and would ensure full buy-in. More monitoring of ‘cycle time’ would be advised to gain more insight into the current process. I would also like to trial some of the professional virtual task-board software on the market, together with possible integration with Team Foundation Server Work Items.

In summary the Kanban method worked very well on this project and with its simple/flexible approach I’m sure that it can be useful on a huge variety of projects. Critically the feedback from this project team was extremely positive, so much so that team members actually went on to implement Kanban on their other projects afterwards. It was seen by the team (and stakeholders) as a successful work tracking and communication technique, whilst also providing the opportunity to continuously improve. I highly recommend considering trialling Kanban on your future projects, and if you can’t manage to go that far then at least start to think how you can visualize your work in progress.

An Introduction to Kanban

Recently I introduced Kanban to an inflight development project with the objective of trialling this technique with a hope to raising productivity by overcoming various communication and process issues. In this three part series I’ll cover my real world experiences and the results felt by the project/team. This first part offers an introduction to Kanban, part 2 will cover the results obtained by introducing it to an in-flight project and part 3 will give some fuel to the “Physical versus Virtual Kanban boards” debate.

Kanban has been used in manufacturing for many years and is rooted in the Kaizen (continuous improvement) philosophy documented in the Toyota Production System. It has been used in the manufacturing industry for many years. It’s focus in the field of Software Development came with the book by David Anderson. Kanban in the Software Development sense is …

“… a method for developing products with an emphasis on just-in-time delivery while not overloading the developers. It emphasizes that developers pull work from a queue, and the process, from definition of a task to its delivery to the customer, is displayed for participants to see”. – Wikipedia

Kanban is not a methodology but an enabler to the goal of Continuous Improvement: The Kanban Method encourages small continuous, incremental and evolutionary workflow changes that stick over time. When teams have a shared understanding about the flow of work they are more likely to be able to build a shared comprehension of a problem and then suggest improvements that can be agreed by team consensus.

Visualization of work has become very popular in agile development practices. XP, for example, has “informative workspaces” that display project progress at a glance and the use of “Information Radiators” is gaining ground (i.e. graphs or charts on the wall showing graphically the work in progress within the team or project. This work is too often hidden and so by publicizing the flow of work we can start to understand the real processes in place and how work flows to completion. Once you have that understanding (and can easily share it) you can start to make improvements.

Kanban development revolves around a visual task-board used for managing work in progress: A progress board but with rules! The columns on the board represent the different states or steps in the workflow and the cards/post-it-notes represent a unit of work (the feature/user story/task etc.). We then move the cards along the board through each process stage (i.e. column) to completion. A look at the board at any one time shows which items of work are currently within each stage of the process.

In order to limit the concurrent work in progress within the team we add rules to the Kanban board and set WIP limits on each column, e.g. only 6 items can be in QA at any one time as that is the max capacity of that team. No more work can be assigned to the QA queue whilst it is full. This limit enforces that new work is only “pulled” into the QA queue when there is available capacity. This immediately reveals the bottlenecks in your process so that they may be addressed. Once we spot a bottleneck we stop adding more work to the bottlenecked queue and collaborate within the team to eliminate the blockage (e.g. move resources off build tasks and onto QA tasks). It’s important to note here that we are not artificially introducing a limit here, as we have created the board initially based on the current workflow and therefore the limit exists in reality (e.g. only a single team member can perform part of the process) but it may not have been currently visible to the team.

Don’t underestimate the power of a visual progress board! It’s useful for all project stakeholders to see the latest project status, including senior managers. It might even replace that dreaded status report. The flow of work through each state in the workflow can be monitored, measured and reported (if desired). By actively managing the flow the continuous, incremental and evolutionary changes to the system can be evaluated to have positive or negative effects on the system. Ultimately we want to figure out how to keep development and throughput running smoothly by tracking down bottlenecks and then adjusting our workflow or our resources to eliminate them. We may be able to relieve repeated bottlenecks by changing the number and types of people in each role and cross training our team. One thing that can be monitored is the ‘cadence’ or rhythm of the pace of change. How long does it take for an item to be completed once its added to the board (i.e. its travel time from left to right)? This cycle time can be calculated as:

Cycle Time = Number of Things in Process / Average Completion Rate

A critical benefit of Kanban is that it is very simple to implement, use and improve. Knocking up a task board on your current process is straight forward and team members soon pick up the approach which really aids team engagement. In addition as it is not a methodology in its self it can be used within your existing methodology and actually provides a more gradual evolutionary path from Waterfall to Agile. It is also less dependent on cross functional teams than Scrum. Using Kanban to manage tasks within the Design, Build and Test phases of a Waterfall project can prove very successful regardless of the wider methodologies in use. In addition, the ability for it to handle irregular, ad-hoc work makes it also especially good for managing defect resolutions within a build team or for a Operations/Service team handling incidents.

A 2010 case study with the BBC Worldwide team using Lean methods including Kanban recorded a 37% increase in lead time improvement, a 47% increase in delivery consistency and a 24% drop in defects reported. This was not all due to Kanban, but Kanban appears to have been a key ingredient together with smaller batch sizes, stand-ups and an empowered team.

I think that Kanban is a very powerful approach that can fit within your existing methodology well (including Waterfall) to drive benefit and importantly Continuous Improvement. For more information on Kanban check out these links:

http://www.kanban101.com/
http://en.wikipedia.org/wiki/Kanban_(development)
http://www.agileproductdesign.com/blog/2009/kanban_over_simplified.html
http://www.crisp.se/kanban
http://www.infoq.com/articles/agile-kanban-boards
http://www.infoq.com/articles/hiranabe-lean-agile-kanban
http://www.kanbanblog.com
http://en.wikipedia.org/wiki/Lean_IT