Moving Sonar Rules closer to the Developer with ESLint

Shift code quality analysis to the left by moving your static analysis from the CI/CD pipeline to the developers IDE where possible. In this post I cover what we did, how to set up Sonar Lint and how we ultimately moved the Sonar rules into ESLint instead.

Problem

As a development team working on a JavaScript application we sometimes had issues where a difference between the rules enforced in our CI/CD pipeline, using SonarQube, and local rules, using ESLint, led to discrepancies in standards and ultimately, at times, broken builds. If you are enforcing rules in your project and in the CI/CD pipeline with SonarQube then indeally you need them to mostly match but this is not as easy as it sounds. Any failure in the pipeline are more time-consuming to resolve than if they happened locally before the developer pushed their code to the repository.

Photo by NESA by Makers on Unsplash

Solution

Import the Sonar rules into ESLint and force ESlint in both the IDE and the CI/CD pipeline.

In order to strike a balance of quality assurance and flexibility in the implementation of rules we introduced an approach that combines ESLint and Sonar rules with the emphasis on shift-left with rule enforcement done in the IDE as code is written and then re-enforced later in the CI/CD pipeline.

Firstly, Sonar Source (developers of SonarQube) provide a plugin for several IDEs (including VSCode) called SonarLint that helps address the issue of running Sonar rules in an IDE. Once installed it will analyse your code against default Sonar rules and report issues. What if you’re not running the default set of rules on your Sonar server, well no worries as the plugin can be set to connect to your server and download the right quality profile and rules.

To do this install the SonarLint extension into your IDE (many are supported, e.g. VS Code, Visual Studio etc) and then set the extension properties as per the instructions for your particular IDE. For VS Code it goes like this:

To link a server set the “sonarlint.connectedMode.connections.sonarqube” setting which has to be a USER setting (oddly). Then in workspace settings for the project you can configure the projectKey for your project. Workspace setting files are created in the .vscode folder in a settings.json file whcih can be added to source control so this only needs to be setup once per project. Once done, press F1 > type sonar > select “SonarLint: Update all project bindings to SonarQube” which will refresh the plugins cache and force it to download the rules from your Sonar server.

Now whilst SonarLint is a useful tool it is not as powerful as ESLint for linting in the IDE (in my opinion). For example ignoring a rule (for a genuine reason) at file or line level is not possible in a satisfactory way (you can only ignore from a line/file from ALL sonar rules). Also Eslint provides more power and flexibility especially where you have a centrally managed sonar server with shared rule profiles and quality gates that are not easy to change (which may be a good thing for your organisation).

So instead of, or even in additon to, SonarLint checking you can actually import the Sonar JavaScript scanner rules into ESLint. To do this install the npm package: eslint-plugin-sonar then configure your ESLint config to use the Sonar js/recommended) JS rules.

  1. npm install eslint-plugin-sonar
  2. Add it to your eslint config file as an extends
    extends: [
    “plugin:sonarjs/recommended”
    ]

Now ESLint will report quality errors that would previously only been highlighted in Sonar during a CI/CD pipeline build. This immediate feedback is more useful to the dev team and reduces the time wastage associated with broken builds. For us this ensured that apart from a few rules the majority are now in ESLint where developers can see them and resolve them, preventing the need for a CI/CD pipeline to highlight the problem (and broken builds).

To enforce the ESLint rules at build time we run the ESlint analysis during the CI/CD Pipeline by calling ESlint as a build step.

eslint --config eslintrc.json src 

This means no ESLint errors will be let through. We also have a Sonar Quality Gate check configured in the Pipeline but as the majority of Sonar rules are now in ESLint we should only get a failure where a rule is breached that is only in the server Sonar Profile.

Photo by Nicole Wolf on Unsplash

As an additional step we can also import all the ESlint issues found in to Sonar so that we can see them in the Sonar dashboard. You can export the ESlint rules as JSON for Sonar to import (during the build). To do this run this command in the build (ideally create a new npm script for it) assuming your src folder contains the source:

eslint --config .eslintrc.json --output-file ./eslint-report.json --format json src

Next set this Sonar property in your sonar-project.properties file or via command line argument (where eslint-report.json is the output report produced above).

sonar.eslint.reportPaths=eslint-report.json

Any issues from the ESLint report will then appear in Sonar issues marked with an EsLint badge. It appears warnings are added as majors and errors as Criticals and unfortunately I’ve not yet found a way to change this.

As a side note this command is also useful with eslint to output a HTML report of any errors which is great for reviewing or sharing:

eslint --config .eslintrc.json --output-file ./eslint-report.html --format html src

Summary

In summary, quality is being maintained as the same rules are enforced but now developers only need to ensure that ESlint is happy before committing their changes to source control to ensure a smooth server build. To make this easier you can add a new npm run script that runs all pre-commit checks that is triggered automatically (e.g. git hooks) or manually by the developer.

Some Recommended VS Code Extensions

One of the things that makes Visual Studio Code (VSCode) such a great editor is the many extensions that have been built for it. Extensions in VSCode are explained here. As a reference for myself when building new machines,and anyone else who might find this useful, below is a list of my most used extensions:

  • Bracket pair colorizer – Colours your brackets and braces for easy identification. I avoid many missing bracket errors with this one!
  • XML Tools – XML Formatting, XQuery, and XPath Tools.
  • ESLint – Extension to integrate ESLint into the IDE.
  • SonarLint – Sonar Rules in VSCode to check your code quality as you go.
  • Prettier – Integrate Prettier into your IDE
  • GitLens – Extend the Git capabilities of VSCode with this tool.
  • PowerShell – Develop PowerShell scripts inside VSCode.
  • Docker – Develop Dockers scripts inside VSCode. Adds syntax highlighting, commands, hover tips, and linting for Dockerfile and docker-compose files.
  • REST Client – Allows you to send HTTP requests and view the response directly in VSCode.
  • JS Refactor – Automated refactoring tools to smooth your JavaScript development workflow.

Check out more on the VSCode marketplace.

What are you using?

Visual Studio 2019 Offline Installer

Microsoft have now released Visual Studio 2019 and like VS2017 there is no offline installer provided by Microsoft, but you can generate one by using the web installer setup program to write all the packages to disk.

To create the offline installer just download the usual web installer exe from the Microsoft download site and then call it from the command line passing in the layout flag and a folder path like this:

vs_community --layout  "C:\Setup\VS2019Offline"

In the example above I’m dowloading the Community verssion, but if its the Enterrpise editions installer then the setup file you downloaded will be called vs_enterprise.

The packages will all be downloaded and a local setup exe installer created.

If you want to limit to English then pass –lang en-US flag too.

vs_community --layout  "C:\Setup\VS2019Offline" --lang en-US

You can also limit what workloads to download if you know their names by listing them after a –add flag.

Enjoy your offline installs.

New WP Code Snippet Editor Online Tool

You can now get the benefits of my Live Writer plugin in your browser without using Live Writer.

Use WordPress…..? Post code snippets….? Well now you can customise the look and feel of the snippets whilst previewing them in a new online tool at https://WPCodePreview.com.

Ten years ago I blogged here about how to create a plugin for the popular (at that time) Microsoft Windows Live Writer blog editor and made the plugin available for download. Over time I added some new features and it has proved very popular, but times change and Windows Live Writer was dumped by Microsoft, and then resurrected as an open source project – now called Open Live Writer (and the plug was updated). Over that time more people are using other editors and platforms to edit their content (and so am I) so I have now replicated the main features of the plugin into an online web application. No more need for Live Writer unless you still like using it, in which case carry on.

Like the Live Writer plugin before it, the site provides a simple way to edit a code snippet and get it looking just how you want – line numbers, line highlighting, language syntax to use etc. All features provided by the WordPress “code” short-code functionality documented here . Once you have the snippet looking how you want, then copy it and paste it into your blog post editor of choice.

For more infnromation checkout the site at https://WPCodePreview.com or the User Guide.

Cmder – A Better Windows Console

Whilst Linux treats console users as first rate citizens and provides many useful and powerful terminal emulators Windows has always lagged behind. This is evermore noticeable now that many developer and IT Ops workloads are done via the terminal. Modern web development and DevOps tooling requires at least some interaction with the terminal, and with the world moving to git for source control developers everywhere are having to embrace consoles.
Whilst Microsoft have traditionally neglected the Windows console they have started to add new features and improvements. For a background on the Windows Console and its architecture check out this blog series. Windows 10 has the best Windows console to date, but there are better out there from 3rd parties and I’ve really got into Cmder.
Cmder is a smart per-configured bundle of the ConEmu emulator software with some extras thrown in. To quote directly from their website:
 

Cmder is a software package created out of pure frustration over the absence of nice console emulators on Windows. It is based on amazing software, and spiced up with the Monokai color scheme and a custom prompt layout, looking sexy from the start.

It can be run portable on a USB Stick if you wish and it has full Git and Bash support. You can emulate the Windows Command Prompt or PowerShell, Bash, Windows SubSystem for Linux (WSL), even the VS Developer Command Prompt among others. All in a slick feature rich emulator.

cmder

It has hundreds of settings that can be tweaked to get everything just the way you like it and it also has the awesome Quake mode so it can slide down from the top of your display.
Cmder2
Support for Cmd, PowerShell, Bash and many more is included out the box, but if you are a Visual Studio user and want to emulate the Developer Command Prompt for VS2017 (reommended) then check out the simple instructions in this guide by Ricardo Serradas on Medium.
I’ve been using it for months and its been stable, performant and has also caught the eye of collegues due to those good looks which make it a pleasure to work in compared to the plain Windows console. Give it a try.

Some SonarQube Upgrade Issues & Fixes

I recesq-ci-72xntly upgraded a SonarQube server installation from v5.6.2 to v6, and unfortunately hit a few issues along the way which I thought I’d share here in case others experiences the same issues. All were resolved in the end and if you are yet to be running SonarQube to analyse your software assets please don’t be put off my these small issues. SonarQube is an outstanding tool to have in your Quality Control armoury and it is really incredibly easy to set and run. In fact you can download it and run it  straightaway in under two minutes without installing anything (check out this link Get Started in Two Minutes to learn how).

Anyway the first problem I hit with the upgrade was an error message in the log when the service was trying to connect to the database (in my instance an MS SQL Server):

Unsupported JDBC driver provider: jtds 

Apparently support for jtds was changed to a bundled version by  SonarQube at some point and so it needs to be removed from the connection string:

Original connection string:
sonar.jdbc.url=jdbc:jtds:sqlserver://ServerName;instance=sonar;databaseName=sonar

New connection string:
sonar.jdbc.url=jdbc:sqlserver://ServerName;instance=sonar;databaseName=sonar

This change made I still could not connect to the database but this time due to different error, which was:

Can not connect to database. Please check connectivity and settings. The TCP/IP connection to the host ServerName1, port 1433 has failed. Error: “Connection refused: connect. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.”

After verifying the database was indeed up, running, not blocked by a firewall and indeedsqlserverconfigmgrexample open on the specified port I found that I had to turn off dynamic ports on my Sonar DB server. To do this open the SQL Server Configuration Manager application, under SQL Server Network Configuration – Protocols for Sonar, right click TCIP/IP and choose properties. Under IP Addresses ensure that TCP Port is 1433 for all entries (including IPAll) AND ensure that TCP Dynamic Ports is blank. My TCP Dynamic Ports value was “0” which actually enables dynamic ports! After this change DB connectivity was successful.

At this point the auto-upgrade step failed and after integrating the logs I found this problem:

Cannot resolve the collation conflict between “Latin1_General_CI_AS” and “Latin1_General_CS_AS” in the equal to operation.

After some googling I hit this very useful Stack Overflow post where the problem is explained. I choose to manually update the database collation (option 3). After running the suggested query I was able to work out the indexes that needed to be dropped and recreated to enable the collation to be updated.

After this I deleted the data out of the SonarQube temp folder (ensuring that the Sonar Service had been stopped) and restarted the service and this triggered the upgrade process again which this time completed successfully.

Using PowerShell for your VS Code Integrated Terminal

Microsoft’s superb Visual Studio Code editor has an integrated terminal which is accessed via the ‘View’menu or via the Ctrl+’ shortcut keys. On Windows by default the terminal used is the Windows Command Prompt (cmd.exe) terminal, however you can easily configure VS Code to use a different terminal such as Windows PowerShell.

Open the User Settings config file (the ‘settings.json’ file accessed via File > Preferences > User Settings) and modify the setting for which terminal to run on Windows:

The default setting is:

 “terminal.integrated.shell.windows”: “C:\\WINDOWS\\system32\\cmd.exe”,

To use the PowerShell terminal instead add this to your settings.json user settings file:

“terminal.integrated.shell.windows”: “C:\\Windows\\sysnative\\WindowsPowerShell\\v1.0\\Powershell.exe”,

Now PowerShell will be used instead of cmd.exe. Currently only one terminal can be configured in VS Code and so you can’t have both PowerShell and cmd.exe so you’ll have to choose your favourite for now. You can however access mutliple instances of the terminal via the drop down on the terminal window.

vscodeposhterminal2

Finally whilst on the subject of VS Code and PowerShell I recommend installing Microsoft’s PowerShell Extension which lets you code and debug PowerShell scripts directly within VS Code (and benefit from its features, e.g. git integration etc).

NPM config for web access via a proxy

If you are using NPM for to install your JavaScript modules and you are sitting behind a corporate proxy server  with a strict firewall then you will likely be having problems. If NPM cannot find its way out to the web you will likely be getting a timeout error like the one below:

npm-logo

npm ERR! argv “C:\\node.exe” “C:\\nodejs\\node_modules\\npm\\bin\npm-cli.js” “install” “package1”
npm ERR! node v4.2.1
npm ERR! npm  v2.14.7
npm ERR! code ETIMEDOUT
npm ERR! errno ETIMEDOUT
npm ERR! syscall connect
npm ERR! network connect ETIMEDOUT 185.31.18.162:443
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network ‘proxy’ config is set properly.  See: ‘npm help config’

To resolve this problem you need to tell NPM the address of your web proxy, including the username/password to authenticate, so that it can route outgoing HTTP requests via that proxy. NPM stores its configuration in a config file and can be edited via the console/terminal using “NPM Config” command. Use this command to set  set both the HTTP and HTTPS values replacing the username/password and proxy address with your custom values:

npm config set proxy http://username:password@yourproxy.yourcompany.com:8080/
npm config set https-proxy http://username:password@yourproxy.yourcompany.com:8080/

To view the current proxy settings, or to check that your change worked, you can run “npm config get” (as opposed to “npm config set”) to read the settings.

“npm config get proxy”
“npm config get https-proxy”

Alternatively running only “npm config get” will show ALL NPM config settings.

Should you want to remove the npm setting you can do it like this:

“npm config rm proxy””
”npm config rm https-proxy”

For more information checkout the NPM documentation here:https://docs.npmjs.com/misc/config

Backing Up Your Blog Content Using HTTrack

I’m pretty strict on making sure I have my data backed up in numerous places and my blog content is no different. I would hate to lose all these years of babbling. In this post I cover how I back up this blog, and this will apply to any blog engine or indeed any website.

This blog is hosted on WordPress.com and I trust the guys at ‘Automatic’ to keep my data safe, but accidents do happen. Ideally I want an up to date backup of this blog together with any images used. Personally I?m not too concerned about having it in a WordPress format but rather actually prefer having the raw content that I could use to recreate the blog elsewhere.

The HTTrack Tool

The tool I use is HTTrack (http://www.httrack.com) which is a web site copying tool. It essentially re-creates a working copy of the site on the local disk (which you can navigate in a browser too). The tool has many features and includes command line interface which makes it easy to run via a scheduled task. The various command line switches are documented here http://www.httrack.com/html/fcguide.html , but I use this simple command below:

C:\HTTrack\httrack.exe http://richhewlett.com -O c:\TargetFolderPathHere

For interest the –O switch tells HTTrack to output the site to disk and hence produce a copy.

You can create a Windows Scheduled Task that periodically runs this command line and you have an automated backup. I personally go a bit further and wrap this command into a Windows PowerShell script. This script creates new folder each time with the current date and implements error handling which writes to the system eventlog.

This script is an example only and comes with no guarantees that it will work for you without modification:


#===============================================
# Backup blog to disk using HTTRACK
# (Created by Rich Hewlett, see blog at RichHewlett.com)
#==============================================
clear-host
write-output "---------------------Script Start--------------------"
write-output " HTTrack Site Backup Script"
write-output "-------------------------------------------------------"

# set file paths
$timestamp = Get-Date -format yyyy_MMM_dd_HHmmss
$TargetFolderPath="F:\MyBlogBackUp\$timestamp"
$HTTrackPath="C:\HTTrack\httrack.exe"

write-output "Backup target path is $TargetFolderPath"
write-output "HTTrack is at $HTTrackPath"

# set error action preference so errors don't stop and the trycatch
# kicks in to handle gracefully
$erroractionpreference = "Continue"

try
{
    write-output "Creating output folder $TargetFolderPath ..."
    New-Item $TargetFolderPath -type directory

    write-output "Download data ..."
    invoke-expression "$HTTrackPath http://MyBlog.com -O $TargetFolderPath"
    write-output "Done with downloading."    

    write-eventlog -LogName "Network" -Source "HTTrack" -EventId 1 -Message "Downloaded blog for backup"

}
catch
{
    # error occurred so lets report it
    write-output "ERROR OCCURRED DURING SCRIPT " $error

    # write an event to the event log
    write-output "Writing FAIL to EventLog"
    write-eventlog -LogName "Network" -Source "HTTrack" -EventId 1 -Message "Download blog for backup FAILED during execution. $error" -EntryType Error
}

write-output "------------------------------------Script end------------------------------------"

I run this job monthly via a Windows scheduled task.

UPDATE: A working Powershell script can be found on my GitHub site.

Using WordPress Export Feature

If you run a WordPress blog you can also do an export via the Dashboard which will export all site content (including comments) which is useful. In addition to the above raw HTML backup above I also use the export tool periodically manually (and therefore infrequently). For more information on this feature check out http://en.support.wordpress.com/export/

Critical Preview Fix For Live Writer Code Plugin

WLW2011TemplatePreview

Just a quick note to announce a new version of my Windows Live Writer plug-in, for Source Code Syntax Highlighting in WordPress.com posts, has been released.

CaptainKernel posted a comment on this blog to say he was having issues with the preview feature not formatting the code correctly. After some investigation this has been caused by a change on the WordPress.com.

Anyway a new fixed version, v.1.4.2, is now available to download here which resolves this issue.