Calculate a file hash without 3rd party tools on Windows & Linux.

Calculate a file hash without 3rd party tools on Windows & Linux.

If you need to generate a hash of a file (e.g. MD5, SHA256 etc) then there are numerous 3rd party tools that you can download but if you are restricted to only built in tools or don’t need to do this often enough to install something then there are built in OS tools for Windows and Linux that can be used.

Windows:

For Windows there is “certUtil” which can be used from the command prompt console with  the “-hashfile” option to generate a hash for a supplied file:

CertUtil [Options] -hashfile filePath [HashAlgorithm]

The [HashAlgorithm] options are MD2, MD4, MD5, SHA1 (default), SHA256, SHA384 and SHA512.

For example to get an MD5 hash of a file use:

CertUtil -hashfile C:\ExampleFile1.txt MD5

More documentation for CertUtil can be seen here.

For those with access to PowerShell v4  and above (Windows 8.1 & Win Server 2012 R2) you can use the built in commandlet called get-filehash like this:

Get-FileHash C:\ExampleFile1.txt  -Algorithm MD5 | Format-List

The algorithms supported are SHA1, SHA256 (default), SHA384, SHA512, MACTripleDES, MD5 & RIPEMD160.

For Powershell versions prior to V4 there are numerous scripts available on the web that will work out the hash for you using various methods.

Linux:

For Linux use the correct  hashalgorithmSUM command in the terminal for the algorithm you are looking for, i.e. for an MD5 hash use md5sum or for SHA512 hash use sha512sum.

For example:

md5sum /home/rich/Documents/ExampleFile1.txt 
sha1sum /home/rich/Documents/ExampleFile1.txt
sha512sum /home/rich/Documents/ExampleFile1.txt

 

 

Advertisements

Archiving Adobe Lightroom Back Ups with PowerShell

Archiving Adobe Lightroom Back Ups with PowerShell

LightroomLogoIf you are an Adobe Lightroom user it is critical to have regular backups of your photo library catalogue. Luckily this is a simple task thanks to the fact that Lightroom has features built in to regularly taka a backup for you (which in effect means making a copy of your current catalogue file into a new location in the location you have specified in the user preferences of the application.

For information on how to configure the backup settings in Lightroom check out this Adobe link: https://helpx.adobe.com/lightroom/help/back-catalog.html

Lightroom unfortunately does nothing to clear out old backups and prior to Lightroom version 6 these backups were not even compressed, which together can mean the space required to store backups grows very quickly. It was always frustrating as the catalogue files can be compressed by a huge margin (80-90% in cases). Luckily newer versions of Lightroom now compress the backups into zip files which makes their size much less of an issue.

Anyway for those familiar with PowerShell I have a script that I use which after each backup to remove old backups, compress the new backups and move the backup to a new location (to a separate drive to guard against drive failure).

powershellLogo1The script is called LR_Zip_Tuck as it zips the backups and tucks them away. There are two versions of the script. V1 is for Lightroom versions before V6/CC as it includes the additional compression step which is no required since Lightroom V6. This still wo9rks with Lightroom V6 but is slower , and so V2 of the Script is recommended.

The script first waits until the Lightroom application is no longer running before proceeding. This means that you can run this script on exit of Lightroom as it is still backing up (if you have it set to backup on exit) and it will wait until Lightroom has finished (I run it from a desktop shortcut when I still in Lightroom or it is backing up on exit).

## check if Lightroom is running, and if so just wait for it to close
$target = "lightroom"

$process = Get-Process | Where-Object {$_.ProcessName -eq $target}

if($process)
{
	Write-Output "Waiting for Lightroom to exit..."
	$process.WaitForExit()
	start-sleep -s 2
}

It then loops each folder in the backup location looking for catalogue backups that Lightroom has created since the last time the script was run. It then copies it to the off drive backup location and then deletes local the file.

## loop each subfolder in backup location and process
foreach ($path in (Get-ChildItem -Path $LocalBkUpFolder -Directory))
{
	## find zip file in this folder and rename
	$path | Get-ChildItem | where {$_.extension -eq ".zip"} | Select-Object -first 1 | % { $_.FullName} | Rename-Item -NewName {$path.Name + ".zip"}

	## move file to parent folder (as dont need subfolders now)
	$SourceFilePath = $path.FullName + "\" + $path.Name + ".zip"
	Move-Item $SourceFilePath -Destination $LocalBkUpFolder

	## copy zip to remote share location
	Write-Output "Tucking backup away on remote share"
	Copy-Item $NewFileName -Destination $RemoteBkUpFolder

	## delete folder
	Remove-Item -Recurse -Force $path.FullName
}

It then does some house keeping ensuring that only the configured number of old backups exist in the local and remote locations (ensuring that the oldest are deleted first). This prevents the backups building up over time.

## cleanup zip files (local)
Remove-MostFiles $LocalBkUpFolder *.zip 8

## cleanup zip files (remote)
Remove-MostFiles $RemoteBkUpFolder *.zip 20

That’s about it. The scripts are available on my GitHub repo here (as LR_ZipTuck_V1.ps1 and LR_ZipTuck_V2.ps1).

Using PowerShell for your VS Code Integrated Terminal

Using PowerShell for your VS Code Integrated Terminal

Microsoft’s superb Visual Studio Code editor has an integrated terminal which is accessed via the ‘View’menu or via the Ctrl+’ shortcut keys. On Windows by default the terminal used is the Windows Command Prompt (cmd.exe) terminal, however you can easily configure VS Code to use a different terminal such as Windows PowerShell.

Open the User Settings config file (the ‘settings.json’ file accessed via File > Preferences > User Settings) and modify the setting for which terminal to run on Windows:

The default setting is:

 “terminal.integrated.shell.windows”: “C:\\WINDOWS\\system32\\cmd.exe”,

To use the PowerShell terminal instead add this to your settings.json user settings file:

“terminal.integrated.shell.windows”: “C:\\Windows\\sysnative\\WindowsPowerShell\\v1.0\\Powershell.exe”,

Now PowerShell will be used instead of cmd.exe. Currently only one terminal can be configured in VS Code and so you can’t have both PowerShell and cmd.exe so you’ll have to choose your favourite for now. You can however access mutliple instances of the terminal via the drop down on the terminal window.

vscodeposhterminal2

Finally whilst on the subject of VS Code and PowerShell I recommend installing Microsoft’s PowerShell Extension which lets you code and debug PowerShell scripts directly within VS Code (and benefit from its features, e.g. git integration etc).

Interactive file reading with Powershell

Interactive file reading with Powershell

Sometimes you want to see the contents of text file whilst it is still being updated, a common example is where you are outputting to a log file and need to see the output interactively without having to keep opening the file to check for progress, or to see if a job has complete.

Luckily there is the very useful Get-Content Powershell Command.

This can take a “-wait” parameter that will reread the file every second or so and check for updates, displaying it in the console (until you end the command with Ctrl&C as usual).

So for example the command below will read the file constantly until you tell it to stop:

Get-Content C:\Logs\Log.txt -wait

In addition there is the “-Tail” parameter which is the Powershell equivalent to the Unix tail command. This will read the last few lines of the file only and not the whole file. This can be used on its own like this:

Get-Content C:\Logs\Log.txt -tail 5

…which displays the last 5 lines of the file. Or you can combine -wait and -tail together to constantly read the last (specified) number of lines:

Get-Content C:\Logs\Log.txt -wait -tail 5

For more information, see Get-Content Powershell Command on MSDN.

Allow PowerShell Execution

Allow PowerShell Execution

By default PowerShell’s execution policy is very restrictive which is a good thing for security. If you are editing or running scripts on your machine you may want to relax it slightly. As I often forget to do this on new machines I’m making a note of the command in this post:

Open PowerShell prompt as Administrator, and then run:

set-executionpolicy remotesigned

Remotesigned means local scripts can be run but downloaded ones must be signed. You can remove all restriction via:

set-executionpolicy Unrestricted

To view the current setting on your machine use get* instead of set*:

get-executionpolicy

For more information see technet here:
https://technet.microsoft.com/en-us/library/ee176961.aspx

 

Backing Up Your Blog Content Using HTTrack

Backing Up Your Blog Content Using HTTrack

I’m pretty strict on making sure I have my data backed up in numerous places and my blog content is no different. I would hate to lose all these years of babbling. In this post I cover how I back up this blog, and this will apply to any blog engine or indeed any website.

This blog is hosted on WordPress.com and I trust the guys at ‘Automatic’ to keep my data safe, but accidents do happen. Ideally I want an up to date backup of this blog together with any images used. Personally I?m not too concerned about having it in a WordPress format but rather actually prefer having the raw content that I could use to recreate the blog elsewhere.

The HTTrack Tool

The tool I use is HTTrack (http://www.httrack.com) which is a web site copying tool. It essentially re-creates a working copy of the site on the local disk (which you can navigate in a browser too). The tool has many features and includes command line interface which makes it easy to run via a scheduled task. The various command line switches are documented here http://www.httrack.com/html/fcguide.html , but I use this simple command below:

C:\HTTrack\httrack.exe http://richhewlett.com -O c:\TargetFolderPathHere

For interest the –O switch tells HTTrack to output the site to disk and hence produce a copy.

You can create a Windows Scheduled Task that periodically runs this command line and you have an automated backup. I personally go a bit further and wrap this command into a Windows PowerShell script. This script creates new folder each time with the current date and implements error handling which writes to the system eventlog.

This script is an example only and comes with no guarantees that it will work for you without modification:

#===============================================
# Backup blog to disk using HTTRACK
# (Created by Rich Hewlett, see blog at RichHewlett.com)
#==============================================
clear-host
write-output "---------------------Script Start--------------------"
write-output " HTTrack Site Backup Script"
write-output "-------------------------------------------------------"

# set file paths
$timestamp = Get-Date -format yyyy_MMM_dd_HHmmss
$TargetFolderPath="F:\MyBlogBackUp\$timestamp"
$HTTrackPath="C:\HTTrack\httrack.exe"

write-output "Backup target path is $TargetFolderPath"
write-output "HTTrack is at $HTTrackPath"

# set error action preference so errors don't stop and the trycatch
# kicks in to handle gracefully
$erroractionpreference = "Continue"

try
{
    write-output "Creating output folder $TargetFolderPath ..."
    New-Item $TargetFolderPath -type directory

    write-output "Download data ..."
    invoke-expression "$HTTrackPath http://MyBlog.com -O $TargetFolderPath"
    write-output "Done with downloading."    

    write-eventlog -LogName "Network" -Source "HTTrack" -EventId 1 -Message "Downloaded blog for backup"

}
catch
{
    # error occurred so lets report it
    write-output "ERROR OCCURRED DURING SCRIPT " $error

    # write an event to the event log
    write-output "Writing FAIL to EventLog"
    write-eventlog -LogName "Network" -Source "HTTrack" -EventId 1 -Message "Download blog for backup FAILED during execution. $error" -EntryType Error
}

write-output "------------------------------------Script end------------------------------------"

I run this job monthly via a Windows scheduled task.

UPDATE: A working Powershell script can be found on my GitHub site.

Using WordPress Export Feature

If you run a WordPress blog you can also do an export via the Dashboard which will export all site content (including comments) which is useful. In addition to the above raw HTML backup above I also use the export tool periodically manually (and therefore infrequently). For more information on this feature check out http://en.support.wordpress.com/export/

How to Backup To USB Drive Only If It’s Connected

How to Backup To USB Drive Only If It’s Connected

A key part of most personal data backup strategies involves backing up data to an external USB drive but I don’t want to leave it constantly connected. In this post I cover how to backup to an external drive using a scheduled automated process but only if the external drive is connected at the time.

I don’t believe in leaving external backup USB drive always connected to my system (PC or Server) to avoid the data being corrupted or deleted. Also if the backup drive is for off-site storage then its not possible to be always connected anyway. Using the steps below I can perform a full data backup by simply physically connecting the drive (connecting a USB drive or docking the SATA drive into a USB dock for example), and then leaving it overnight. In the morning I can safely physically disconnect it and store it without even having to log onto the machine.

The basic flow:

1) Assuming that the machine is always on, which my server is, a scheduled task runs every night and executes a backup script (regardless of the backup drive being connected or not). If your machine is not always on then vary this by setting the scheduled task at a time when the machine is usually on.
2) The script checks for the existence of a specific folder on the connected drive (e.g: U:\backup\). If the drive isn’t connected then the folder path won’t exist and the script just exits happily. However, if the drive has been connected then the folder path will exist and the backup script will continue and copy over the data.
3) After a successful backup the script safely disconnects the USB drive. This step is technically optional as Windows supports pulling a USB drive out without doing a soft eject but its highly recommended to tell Windows first to avoid data corruption.
4) Optionally you can also output a backup log somewhere to enable you to check the logs periodically without having to reconnect the USB drive to verify the job worked.

More detailed steps:

Firstly connect the external USB drive and make a note of the drive letter it uses. Changing the drive letter to something memorable might help (B for backup, U for USB, O for Offsite etc). We’ll use U for this example. Next we need to write a DOS command script, a simple program or Powershell script to perform the backup of the data using the backup tool of your choice. I use Robocopy to copy the files via a Powershell script and below is a simplified version of my script.

#==========================================================================================================
# Checks for presence of offsite backup USB drive, and backs up relevant data to drive if present,
# exits gracefully if not present. Run script everynight and it only backs up data when offsite
# USB external drive is turned on.
#==========================================================================================================
clear-host

# set file paths and log file names
$timestamp = Get-Date -format yyyyMMdd_HHmmss
$LogBasePath="D:\Logs\OffSiteUSBBackup"
$LogFile="$LogBasePath\USBBkUp_$timestamp.txt"
$USBDriveLetter="U"
$USBDriveBackupPath="U:\Backup"

# set error action preference so errors don't stop and the trycatch kicks in to handle gracefully
$erroractionpreference = "Continue"

try
{
	# Check USB drive is on by verfiying the path
	if(Test-Path $USBDriveBackupPath)
	{
		# now copy the data folders to backup drive
		invoke-expression "Robocopy C:\Docs $USBDriveBackupPath\Docs /MIR /LOG:$LogFile /NP"
		invoke-expression "Robocopy C:\Stuff $USBDriveBackupPath\Stuff /MIR /LOG+:$LogFile /NP"

		# Copy the log file too
		invoke-expression "Robocopy $LogBasePath $USBDriveBackupPath\Logs /MIR /NP"

		# Sleep for 60 to ensure all transactions complete, then disconnect USB drive
		Start-Sleep -Seconds 60
		Invoke-Expression "c:\DevCon\USB_Disk_Eject /removeletter $USBDriveLetter"
	}
}
catch
{
	# Catch the error, log it somewhere, but make sure you still eject the drive using below
	Start-Sleep -Seconds 60
	Invoke-Expression "c:\DevCon\USB_Disk_Eject /removeletter $USBDriveLetter"
}

It is key to include in the script a check for the existence of a specific folder on the drive letter belonging to the external drive (U in our example). Only if its present do we continue with the backup.

I make sure that Robocopy logs the output to a file and that file is on the server and copied to the USB drive (as a record of the last backup date etc). I also report the running of the PowerShell script to the Eventlog for reporting purposes but this is outside the scope of this post.

It’s safer to tell Windows that you’re gonna pull the drive out and so I call  USB_Disk_Eject  from within my script, passing in the drive letter. I then wait 30 seconds to ensure the drive has had sufficient time to disconnect before I exit the script. There are a few tools available for ejecting USB drives such as Microsoft’s Device Console (DevCon.exe) but I use USB Disk Ejector (https://github.com/bgbennyboy/USB-Disk-Ejector).

Now set up a Scheduled Task in Windows to run the script every night at a set time.  As the script is scheduled to run every night all I have to do if I want to perform a back-up is connect my backup drive and leave it until the morning. The script will run overnight, find the drive, backup and disconnect. In the morning I can just physically disconnect the drive safely without having to log onto the machine. Periodically I’ll check the backup logs and the backup drive to make sure all is well and to check remaining drive space etc.

UPDATE: A working Powershell script can be found on my GitHub site.

Do you like this approach? Got a better idea? Let me know via the comments.

Using SlickRun For Fast PowerShell Commands

I’m a big fan of SlickRun and if you’ve not used it I recommend you download it and give it a try. I find it an invaluable tool not only for launching apps but also web sites, collections of applications and directories etc. It’s the simplicity of SlickRun that makes it so powerful. Sure there are newer, and certainly prettier launchers out there but SlickRun is still the best in my opinion. Whilst it’s usefulness is slightly diminished by the Windows 7 start menu search it still has its place for providing shortcuts of any name you like, and for running multiple apps off one command. Anyone reading this blog will no doubt have spotted that I also am a fan of Windows PowerShell, so I’ve now combined the two. The objective being to run simple PowerShell commands within SlickRun and have the results appear quickly in a console (with no coding).

PowerShell can be run with start up parameters (see the documentation here). The one’s we need to use are:
-NoExit to not exit the console window after running the command (so you can see the results)
-command to executes the specified command as though they were typed at the Windows PowerShell command prompt.

The SlickRun setup for the MagicWord (I’ve used ‘POSH’ in this example) is:
MagicWord: POSH
Filename: powershell
Parameters: powershell -NoExit -command "$I$"

The filename can be just "powershell" or if you feel better you can put in the full path to the PowerShell exe. Passing the command text as "$I$" just passed in what you typed in SlickRun after the magic word.

We can then just use the POSH keyword followed by the command we wish to run. For example this keyed into SlickRun…

Posh1

launches the PowerShell console and the results…

Posh2

Of course this approach can be used from other application launchers, directly from the command line or the Start-Run dialog but then of course you will have more to type as you will need to key the start-up parameters each time.

Android Remote Desktop Client

2XClient_LogonI find that I am increasingly relying on the computing power of my Android smartphone (a HTC Desire) and finding novel ways of using it to make my IT life easier. Sometimes I just want to connect to my PC that is in another room, or more often for me it’s my headless Windows Home Server, and so I scouted for a Remote Desktop client that I could run on my phone. The key requirement was for it to use the Windows native Remote Desktop protocol and therefore not require any software to be installed on my PC or Server, which ruled out a lot of the VNC based Apps. Luckily 2x.com have released an excellent FREE App that ticks all the boxes.

2XClient for Android can be found here or on Android Market here. It is dead easy to set up the target machines and there are several display optimisation options. The key thing though is that it’s actually very easy to navigate the target machines desktop via a custom keyboard and a nifty mouse icon that can be dragged around with a left and right mouse button attached (left image below).  In these images I’m logging onto my Windows Home Server (a Windows 2003 based OS) but I also use it with my Windows 7 PC too. One thing to note for Windows 7 though is that I needed to set my Remote Desktop settings (via My Computer > System Properties > Remote Settings)  to “Allow connections from computers running any version of Remote Desktop” as opposed to the default setting of enforcing Network Level Authentication.

 2XClient_Mouse  2XClient_Keyboard  2XClient_StartMenu

It is surprisingly easy to do simple tasks on the target machine, especially after a bit of practice. Here I am using PowerShell and checking my Home Server Console.

2XClient_WHS12XClient_POSH

A very powerful tool to have on your phone and ideal for those quick techy tasks when you can’t be bothered to get off the sofa.

Windows PowerShell Console Fun

Windows PowerShell Console Fun

1

Whilst watching Die Hard 4 the other day I noticed the funky transparent console windows that were being used to battle out cyber warfare, and being a traditional geek I immediately liked the idea of doing the same for my PowerShell console. Sure I know these guys are using Linux and transparent consoles have been around for years but still I fancied some of that slickness on my Windows.

POSh_PssGlassIt didn’t take long to find PSGlass on CodePlex (http://powershellglass.codeplex.com/) which is a neat little exe that runs in your system tray and hunts out any PowerShell console windows, and any it finds it converts to transparent using the Windows Aero effects. The peek into the source code shows that it’s checking for a window with a process name of “powershell” or “cmd” and then uses the DwmEnableBlurBehindWindow API to make it transparent.  The effect is shown on the left. It’s simple and effective and could be extended easily to do more. Not content with this I wanted to achieve the same result from within PowerShell itself so set about using the API in a script. Luckily for me Oisin Grehan had already written a script to achieve the same result and posted it on PoshCode.org, check it out at http://poshcode.org/2052. For this script the DwmExtendFrameIntoClientArea API is used to create a sheet of glass effect with no borders and the effect is much more striking (right screen shot). The fact that POSh_Glassyou don’t have to have an application running in the background is of course much better and as its a script you can add it to your profile to always take effect on Powershell start-up. I have found it useful to create a function in my profile to toggle the glass effect on/off depending on my mood and what actions I am trying to perform.

As the console was now transparent I quickly found myself wanting it to stay on the top of other windows and so set about looking for a script for that too. Again the excellent PoshCode.org came to the rescue with this script from Shay Levy at http://poshcode.org/1837. Again I downloaded the script and added a function to my profile to be able to toggle it on/off at will from within the console itself. In order to access the Set-TopMost() function that sits within the TopMost.ps1 script I used dot sourcing (described here by Jesse Hamrick) and my functions are shown below:

function OnTop
{
	. TopMost.ps1
	Set-TopMost (get-process -id $pid).mainwindowhandle
}

function NoOnTop
{
	. TopMost.ps1
	Set-TopMost (get-process -id $pid).mainwindowhandle -disable
}

PoshMatrixFor total geek heaven why not go the extra mile and put a Matrix style screensaver within the console itself. If you haven’t seen it I would recommend checking it out at http://www.nivot.org…. The video on the site shows how it works but it essentially runs a Matrix code screensaver inside the console (not the whole desktop).  Now that’s one clever PowerShell script!

2017 UPDATE:

As PoshCode.org site has been down a while I’ve added alternative links to be able to find the above mentioned scripts from the awesome web.archive.org instead of PoshCode.org: