Building a Python Flask Web UI For Raspberry Pi Sure Elec LCD

In an earlier post I outlined how I setup a Sure Electronics LCD screen with my Raspberry Pi 3 using a Python driver. Whilst updating the LCD via command line is immensely useful I decided to build a UI to control the LCD send messages too it. By using a browser based UI I could update the LCD screen from anywhere. Essentially this was a chance to play with a Python web framework and write some code!

BlogFlaskUISureElecLCD2 

I’ve passed the UI’s URL round my family’s devices at home and they now send me messages whilst I’m in my study working/playing.

The end result can be found in my GitHub repo.

As my driver was in Python and I’m enjoying coding in Python at the moment I decided to use a Python Web framework to serve the HTML/JavaScript UI and host RESTful services on the server side to accept LCD commands. After some reading I went with Flask which  seemed perfect for my needs. I could have gone with Django but Flask seemed for appropriate for my needs. For a good comparison see this CodeMentor.io post. For a great tutorial on Flask checkout this series by Miguel Grinberg and this great post by Scotch.io.  

Building the server side web framework was easy and logical in Flask and I was able to get something setup in one file which served my needs. However after reading some Flask best practices I spread my solution out into a more appropriate structure. Flask will seem familiar to web developers with experience of ASP.net MVC, Web API, Node/Express etc. You define routes to handle incoming requests. The key aspects my solution are outlined below. I am running the Flask server directly on my Raspberry Pi and using it to serve the pages and host the services for commanding the LCD screen.

To install Flask (on a Pi) first install Python Pip (a popular Python Package Manager) via “apt-get install python-pip” or “apt-get install python3-pip” (for a Python v3 specific Pip) and then install Flask via “pip install flask”.

Flask comes with a small lightweight development server which runs your app in Dev mode and also auto restarts after code changes. I found this fast and robust enough for my needs. 

Lets check out the main parts of the code:

run.py:  This is the entry point for the app. When run it calls run in the app file and here I have optionally passed IP/Port I want the app to run on which enables the app to be exposed to the internal network so I can connect from other machines on the network. 

from app import app 
if __name__ == '__main__':
    app.run(host="192.100.100.100", port=5000)

app/__inti_.py & config.py: This is the app initialisation code and where it points to the config.py file where config settings can be set for the app.

app/views.py: This is the heart of the app. After importing the relevant python components and instantiating the Smartie LCD driver (from previous post), the routes for the app are defined.

@app.route("/")
def show_homepage():
    return "Home Page!"

@app.route("/lcd")
def show_lcdpage():
    name="Jeff"
    return render_template("lcd.html", name=name)

The route for root will just return the text “Home Page” whereas the route for /lcd will call render_template to return a templated HTML page (lcd.html) and passes any relevant data (e.g. “Jeff” which is irrelevant in this example”). Templates will be covered shortly below.

@app.route("/lcd/clear", methods=["POST","GET"])
def display_clear():
    smartie1.clear_screen()
    return "Success"

@app.route("/lcd/displaymessage", methods=["POST"])
def display_message():
    if not request.json:
        abort(400)
    smartie1.clear_screen()
    smartie1.backlight_on()
    smartie1.write_lines_scroll(request.json['Lines'])
    return "Success" 

Any POST or GET on http://SERVERADDRESS:PORT/lcd/clear will result in the smartie drivers clear screen method being called. A POST to “/lcd/displaymessage” will be validated to ensure that the request contains JSON data and then the data will be passed to the driver for display.

/app/templates/lcd.html: This is the main HTML page that enables the user to type the messages to display.

BlogFlaskUI2

The CSS and JavaScript used by this page is found in the static folder and referenced in the usual way………….

<!-- CSS for our app -->         
 <link rel="stylesheet" href="/static/lcd.css"/>

<!-- JS for our app --> 
<script type="text/javascript" src="/static/lcd.js" charset="utf-8"></script>

So we need to ensure that the flask server returns these static files, but we don’t want to have to define  a specific app.route for each one so instead we use this one in our views.py :

@app.route('/<path:filename>')  
def send_file(filename):  
      return send_from_directory('/static', filename)

This basically states that any requests for a file path are sourced from the /static folder directly. So we can just place any files in the static folder that we want to be served directly (the CSS and JavaScript files in our case).

/app/static/lcd.js:  From this JavaScript code we can consume the services hosted by Flask for our application. It’s using the XMLHttpRequest object to make AJAX requests to the Flask server. The SendCommand function takes callback methods which will be called on success or error.

function SendCommand(url, httpVerb, data, successCallback, errorCallback){
                
  var dataToSend;
  if(data!=null){
      var dataToSend = JSON.stringify(data);          
  }
   var request = new XMLHttpRequest();
  request.open(httpVerb, url, true);
  request.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');
   request.onload = function() {
      if(this.status >= 200 && this.status < 400){
          // success here
          var returnedData; 
          if (this.response != null){
              successCallback(returnedData, this.status);
          }                                                                        
      }
      else{
          //error returned from server
          errorCallback("Error response returned from server", this.status);
      }
  }
   request.onerror = function() {
          errorCallback("Error contacting server", this.status);
  }

  if (dataToSend != null){
      request.send(dataToSend);
  }
  else{
      request.send();
  }
}

That’s mostly it. Run the app by running the run.py module (e.g in the Python IDLE or terminal) and direct your browser to http://SERVERADDRESS:5000/lcd.

The code for my Python driver and this web app is available on GitHub here https://github.com/RichHewlett/smartie and https://github.com/RichHewlett/LCD-Smartie-Web.

Using a Sure Electronics LCD with the Raspberry Pi using Python

Using a Sure Electronics LCD with the Raspberry Pi using Python

After receiving a new Raspberry Pi 3 at Christmas I quickly set off looking for uses for this wonderful machine, and quickly found myself hooking it up to an LCD display. This post covers the Python driver I used and modified, as well as some other useful resources.

RPi3

Firstly as an owner of the original Raspberry Pi model B I was pleasantly surprised how capable the Raspberry Pi version 3 is. The performance is much improved and the new built in wireless is a superb addition which makes this version of the Pi more  usable and reduces the friction caused by having too many USB devices hanging off it like a mutant spider. The Pi v3 is powerful enough to be used as a basic workstation and I found myself working on it directly (within Raspbian OS) making the coding workflow more enjoyable as a result than the previous models. Also the as VNC Server is now built into the Raspbian OS it makes setting up the Pi as headless very easy. In fact once you have installed Raspbian and setup VNC then you can run it with just the power supply connected which makes it use little space and you can locate it anywhere.

SureElecLCD1Once my Pi3 was up and running I decided to connect an old LCD matrix screen to it. This LCD is Sure Electronics 20×4 screen which I bought years ago to attach to my HomeServer but never got around to it. The LCD has a USB interface and was used on Windows via the LCD Smartie application. A quick google showed that LCDProc is an equivalent tool for using LCD screens with Linux, and I found some useful tutorials (e.g. here & here). You can install LCDProc via “apt-get install lcdproc”.

I only had limited success with LCDProc but soon realised what I really needed was a programmatic way of controlling the LCD as opposed to a general display tool, so after some more googling I found a few variations of python drivers were being used to control similar LCDs, and this “Smartie” driver by Steve Davis worked best for me, which in turn was inspired by this driver. These drivers use the “pyserial” Python module, easily installable via Python PIP package manager:

“apt-get install python-pip” or “apt-get install python3-pip” (for python v3 pip)
followed by “pip install pyserial==3.0”.

RaspberryPiWhilst the smartie driver initially worked I found that it failed to work after a reboot. After some serious head scratching I realised that it would always work after running LCDProc, meaning the initalisation code for LCDProc setup up the LCD successfully for the Python driver to communicate.  After digging into the Sure Electronics manual and digging into the source code for the SureElec driver for LCDProc (here & here) I was able to find the initisiation command codes for the LCD and add this to the smartie python driver. This enabled the Python driver to be used without relying at all on LCDproc or similar software.

 I have also added some new functionality to the driver which includes flashing, wrapping text and scrolling multiple lines etc. Also included is a demo function that runs through the various functions provided by the driver to show what’s possible and to help testing after making any modications. My updated driver can be found on my GitHub repo (https://github.com/RichHewlett/smartie). With this driver i was able to control my LCD backlight, display text and get a temperature reading. The fact that it was writen in Python was a bonus as I find Python a great langage for coding fun projects.

SureElecLCD2

If you have a Sure Electronics LCD you may be able to take this and use it, or modify it as required, although there are many variations of these devices. Once you have connected your LCD and rebooted the machine just check that the LCD is on the right USB port in Linux (e.g. /dev/ttyUSB0) and that the user has permissions on that port. If you have a Sure 16×4 screen then this will probably work by modifying the SCREEN_WIDTH constant to 16. If you have a different LCD Screen then you may need to modify the initalisation codes and command strings that get sent to the LCD by each method. A good source of refercence is to check out the LCDProc source code for their bundled drivers as they support many screens.

In the next post I’ll cover my next step of building Python Web App using Flask to control the LCD screen from a browser.

Archiving Adobe Lightroom Back Ups with PowerShell

Archiving Adobe Lightroom Back Ups with PowerShell

LightroomLogoIf you are an Adobe Lightroom user it is critical to have regular backups of your photo library catalogue. Luckily this is a simple task thanks to the fact that Lightroom has features built in to regularly taka a backup for you (which in effect means making a copy of your current catalogue file into a new location in the location you have specified in the user preferences of the application.

For information on how to configure the backup settings in Lightroom check out this Adobe link: https://helpx.adobe.com/lightroom/help/back-catalog.html

Lightroom unfortunately does nothing to clear out old backups and prior to Lightroom version 6 these backups were not even compressed, which together can mean the space required to store backups grows very quickly. It was always frustrating as the catalogue files can be compressed by a huge margin (80-90% in cases). Luckily newer versions of Lightroom now compress the backups into zip files which makes their size much less of an issue.

Anyway for those familiar with PowerShell I have a script that I use which after each backup to remove old backups, compress the new backups and move the backup to a new location (to a separate drive to guard against drive failure).

powershellLogo1The script is called LR_Zip_Tuck as it zips the backups and tucks them away. There are two versions of the script. V1 is for Lightroom versions before V6/CC as it includes the additional compression step which is no required since Lightroom V6. This still wo9rks with Lightroom V6 but is slower , and so V2 of the Script is recommended.

The script first waits until the Lightroom application is no longer running before proceeding. This means that you can run this script on exit of Lightroom as it is still backing up (if you have it set to backup on exit) and it will wait until Lightroom has finished (I run it from a desktop shortcut when I still in Lightroom or it is backing up on exit).

## check if Lightroom is running, and if so just wait for it to close
$target = "lightroom"

$process = Get-Process | Where-Object {$_.ProcessName -eq $target}

if($process)
{
	Write-Output "Waiting for Lightroom to exit..."
	$process.WaitForExit()
	start-sleep -s 2
}

It then loops each folder in the backup location looking for catalogue backups that Lightroom has created since the last time the script was run. It then copies it to the off drive backup location and then deletes local the file.

## loop each subfolder in backup location and process
foreach ($path in (Get-ChildItem -Path $LocalBkUpFolder -Directory))
{
	## find zip file in this folder and rename
	$path | Get-ChildItem | where {$_.extension -eq ".zip"} | Select-Object -first 1 | % { $_.FullName} | Rename-Item -NewName {$path.Name + ".zip"}

	## move file to parent folder (as dont need subfolders now)
	$SourceFilePath = $path.FullName + "\" + $path.Name + ".zip"
	Move-Item $SourceFilePath -Destination $LocalBkUpFolder

	## copy zip to remote share location
	Write-Output "Tucking backup away on remote share"
	Copy-Item $NewFileName -Destination $RemoteBkUpFolder

	## delete folder
	Remove-Item -Recurse -Force $path.FullName
}

It then does some house keeping ensuring that only the configured number of old backups exist in the local and remote locations (ensuring that the oldest are deleted first). This prevents the backups building up over time.

## cleanup zip files (local)
Remove-MostFiles $LocalBkUpFolder *.zip 8

## cleanup zip files (remote)
Remove-MostFiles $RemoteBkUpFolder *.zip 20

That’s about it. The scripts are available on my GitHub repo here (as LR_ZipTuck_V1.ps1 and LR_ZipTuck_V2.ps1).

Some SonarQube Upgrade Issues & Fixes

I recesq-ci-72xntly upgraded a SonarQube server installation from v5.6.2 to v6, and unfortunately hit a few issues along the way which I thought I’d share here in case others experiences the same issues. All were resolved in the end and if you are yet to be running SonarQube to analyse your software assets please don’t be put off my these small issues. SonarQube is an outstanding tool to have in your Quality Control armoury and it is really incredibly easy to set and run. In fact you can download it and run it  straightaway in under two minutes without installing anything (check out this link Get Started in Two Minutes to learn how).

Anyway the first problem I hit with the upgrade was an error message in the log when the service was trying to connect to the database (in my instance an MS SQL Server):

Unsupported JDBC driver provider: jtds 

Apparently support for jtds was changed to a bundled version by  SonarQube at some point and so it needs to be removed from the connection string:

Original connection string:
sonar.jdbc.url=jdbc:jtds:sqlserver://ServerName;instance=sonar;databaseName=sonar

New connection string:
sonar.jdbc.url=jdbc:sqlserver://ServerName;instance=sonar;databaseName=sonar

This change made I still could not connect to the database but this time due to different error, which was:

Can not connect to database. Please check connectivity and settings. The TCP/IP connection to the host ServerName1, port 1433 has failed. Error: “Connection refused: connect. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.”

After verifying the database was indeed up, running, not blocked by a firewall and indeedsqlserverconfigmgrexample open on the specified port I found that I had to turn off dynamic ports on my Sonar DB server. To do this open the SQL Server Configuration Manager application, under SQL Server Network Configuration – Protocols for Sonar, right click TCIP/IP and choose properties. Under IP Addresses ensure that TCP Port is 1433 for all entries (including IPAll) AND ensure that TCP Dynamic Ports is blank. My TCP Dynamic Ports value was “0” which actually enables dynamic ports! After this change DB connectivity was successful.

At this point the auto-upgrade step failed and after integrating the logs I found this problem:

Cannot resolve the collation conflict between “Latin1_General_CI_AS” and “Latin1_General_CS_AS” in the equal to operation.

After some googling I hit this very useful Stack Overflow post where the problem is explained. I choose to manually update the database collation (option 3). After running the suggested query I was able to work out the indexes that needed to be dropped and recreated to enable the collation to be updated.

After this I deleted the data out of the SonarQube temp folder (ensuring that the Sonar Service had been stopped) and restarted the service and this triggered the upgrade process again which this time completed successfully.

Using PowerShell for your VS Code Integrated Terminal

Using PowerShell for your VS Code Integrated Terminal

Microsoft’s superb Visual Studio Code editor has an integrated terminal which is accessed via the ‘View’menu or via the Ctrl+’ shortcut keys. On Windows by default the terminal used is the Windows Command Prompt (cmd.exe) terminal, however you can easily configure VS Code to use a different terminal such as Windows PowerShell.

Open the User Settings config file (the ‘settings.json’ file accessed via File > Preferences > User Settings) and modify the setting for which terminal to run on Windows:

The default setting is:

 “terminal.integrated.shell.windows”: “C:\\WINDOWS\\system32\\cmd.exe”,

To use the PowerShell terminal instead add this to your settings.json user settings file:

“terminal.integrated.shell.windows”: “C:\\Windows\\sysnative\\WindowsPowerShell\\v1.0\\Powershell.exe”,

Now PowerShell will be used instead of cmd.exe. Currently only one terminal can be configured in VS Code and so you can’t have both PowerShell and cmd.exe so you’ll have to choose your favourite for now. You can however access mutliple instances of the terminal via the drop down on the terminal window.

vscodeposhterminal2

Finally whilst on the subject of VS Code and PowerShell I recommend installing Microsoft’s PowerShell Extension which lets you code and debug PowerShell scripts directly within VS Code (and benefit from its features, e.g. git integration etc).

Interactive file reading with Powershell

Interactive file reading with Powershell

Sometimes you want to see the contents of text file whilst it is still being updated, a common example is where you are outputting to a log file and need to see the output interactively without having to keep opening the file to check for progress, or to see if a job has complete.

Luckily there is the very useful Get-Content Powershell Command.

This can take a “-wait” parameter that will reread the file every second or so and check for updates, displaying it in the console (until you end the command with Ctrl&C as usual).

So for example the command below will read the file constantly until you tell it to stop:

Get-Content C:\Logs\Log.txt -wait

In addition there is the “-Tail” parameter which is the Powershell equivalent to the Unix tail command. This will read the last few lines of the file only and not the whole file. This can be used on its own like this:

Get-Content C:\Logs\Log.txt -tail 5

…which displays the last 5 lines of the file. Or you can combine -wait and -tail together to constantly read the last (specified) number of lines:

Get-Content C:\Logs\Log.txt -wait -tail 5

For more information, see Get-Content Powershell Command on MSDN.

Useful React.JS Learning Resources

Useful React.JS Learning Resources

Below are some links that you might find useful for learning React.js and Flux, Facebook’s successful JavaScript UI framework.  There are a lot of resources out there but here are some of the best that I have collected for members of my team.

Introductions and overviews of React.js:

Flux:

Tutorials for Flux & React:

Books: 

Prefer the old school approach of reading a book then instead check out this: React.js Essentials by Artemij Fedosejev

Break on Exceptions in Visual Studio 2015

Break on Exceptions in Visual Studio 2015

Looking for the option to break on exceptions during debugging in Microsoft Visual Studio 2015? Well Microsoft dumped the old exceptions dialog and replaced it with the new Exception Settings Window. To see it to show that window via the menu: Debug > Windows > Exception Settings.

vsexceptionsettingsmenu

Use the Exception Settings window to choose the types of exceptions on which you wish to break. Right click for the context menu option to turn on/off the option to break or continue when the exception is handled (see below). To break on all exceptions you’ll want to ensure this is set to off (not ticked).

vs2015exceptionssettingsdiag2

For more information check out these MSDN links:

https://blogs.msdn.microsoft.com/visualstudioalm/2015/02/23/the-new-exception-settings-window-in-visual-studio-2015/

https://blogs.msdn.microsoft.com/visualstudioalm/2015/01/07/understanding-exceptions-while-debugging-with-visual-studio/

Disable Start Menu Web Search in Windows 10

Disable Start Menu Web Search in Windows 10

If like me you like the Windows 10 “start” menu to only provide applications and Windows settings in the search results and not web search results you need to configure it using these steps.

Using the Start Menu find “Cortana & Search Settings” , then click the settings icon (the cog),  turn Cortana off, and then turn off “Searh Online and Include Web Results”.

Allow PowerShell Execution

Allow PowerShell Execution

By default PowerShell’s execution policy is very restrictive which is a good thing for security. If you are editing or running scripts on your machine you may want to relax it slightly. As I often forget to do this on new machines I’m making a note of the command in this post:

Open PowerShell prompt as Administrator, and then run:

set-executionpolicy remotesigned

Remotesigned means local scripts can be run but downloaded ones must be signed. You can remove all restriction via:

set-executionpolicy Unrestricted

To view the current setting on your machine use get* instead of set*:

get-executionpolicy

For more information see technet here:
https://technet.microsoft.com/en-us/library/ee176961.aspx