Automating Cisco IOS updates with Unimus

Intro

We believe that keeping the firmware / software of networking devices up-to-date is one of the most important security benefits in any network. Sadly, this is much easier said than done in any network at scale. Previously, we brought you a guide on automating network-wide upgrades of MikroTik's RouterOS (link here).

This time, we tackle automating upgrades of Cisco IOS devices. We will do so in 2 parts, using 2 different approaches - this article is the first part where we will focus on providing a simple and quick solution.

This solution requires 3 main components:

FW image source - a server which will serve as a source / host for FW image(s) used for upgrade
Cisco IOS devices to upgrade, which will be downloading a provided FW image
Unimus' Mass Config Push feature to push commands to your devices and automate the FW upgrades

Here is a conceptual diagram of how our testing topology looks like:

Unimus will be used to perform and automate the IOS upgrades, but also to help us sort and organize IOS upgrade results into output groups, so we don't have to examine every single device output manually. Rather, Unimus will group devices by the outputs we receive to easily identify successful and / or failed upgrades - if any pop up.

Preparing the Image / FW source

You will need a server to serve / host your FW image(s) from. Any server which your Cisco IOS devices can download files from will suffice. For our showcase, we will use a VM running Linux. This way we can host our images easily and can leverage pretty much any transfer method we choose.

FW Transfer methods

When it comes down to transferring files between our FW source and IOS devices, we chose two protocols for this showcase: SCP and HTTP. Usually we see administrators relying on TFTP or FTP, but we seldom see anyone choosing SCP or HTTP. Thus we decided to showcase these, albeit less popular options.

SCP

If you choose SCP, you may need to make some adjustments on your FW image server. As SCP uses the SSH protocol, there is one potential issue you will have to deal with. If you have some older Cisco IOS devices, and especially those which don't offer any sort of addition/exclusion/enforcement of more secure KEX (Key Exchange), Ciphers and Host Key Algorithms; and your source server runs a generally newer SSH server - you may need to enable one or more of the legacy algorithms which your devices are capable of supporting.

We would recommend adding the diffie-hellman-group1-sha1 KEX to your SSH server's config file as a precaution. It should resolve most of these potential issues. If that is not the case though, you will need to check one or more devices manually. Try to download the FW image manually and optionally also turn on terminal monitor and SCP debugging.

HTTP

If you already use / run a web server, then choosing HTTP will save you time. There shouldn't be any extra configuration required on either side - assuming your IOS devices will be able to access your web server.

Preparing Unimus and Mass Config Push preset for Cisco IOS upgrade

Here are the Config Push presets we will be using to pull the FW images and perform the upgrade and reload the devices:

Config Push preset 1 - Upgrade devices

We start with the main Config Push preset, which will do the upgrade - without reloading the devices. We don't want to reload immediately after pulling an image, as that would likely cause problems (due to reloading devices in wrong / random order and causing connection loss). Some devices could reload sooner than others finished their FW transfer, and this could cause pushes to fail for some of our devices.

IOS image transfer using SCP

tclsh
log_user 0
exec "copy scp://SCP_USER:SCP_PASS@FW_SRC_ADDR/UPGRADE_FW_IMAGE flash:"
ios_config "boot system flash:UPGRADE_FW_IMAGE"
tclquit

Where:

SCP_USER - SCP user
SCP_PASS - SCP password
FW_SRC_ADDR - IP or hostname of FW image source device
UPGRADE_FW_IMAGE - file name of a chosen upgrade FW image

Please replace all the example values with your actual ones. Don't forget to check Require "enable" (privileged-exec) mode for this Config Push.

IOS image transfer using HTTP

tclsh
log_user 0
exec "copy http://FW_SRC_ADDR/UPGRADE_FW_IMAGE flash:"
ios_config "boot system flash:UPGRADE_FW_IMAGE"
tclquit

Where:

FW_SRC_ADDR - IP or hostname of FW image source device
UPGRADE_FW_IMAGE - file name of a chosen upgrade FW image

Please replace all the example values with your actual ones. Don't forget to check Require "enable" (privileged-exec) mode for this Config Push.

If you know some of your devices are slow (will take along time to download the IOS image), then we recommend one more step before running the Mass Config Push preset. In my case, I know one of mine devices required as much as 5 minutes to download the IOS image, which would otherwise hit one of Unimus default timeouts. If this is your case as well, we recommend overriding timeouts and giving devices a much longer time to finish. Here's how you can do it via the Advanced settings of a Push Preset:

In my case, I extended it to 5 minutes = 300 seconds = 300,000 milliseconds.

Config Push preset 2 - Reload devices

tclsh
exec "reload in 1"
tclquit

This is a simple preset to reload devices and set it to be executed in 1 minute. Feel free to change it to your needs, there are two formats for reload in command - MMM or HHH:MM, or you can change it to reload at and define a specific date and time for the reload instead.

Let me quickly address why we use the TCL shell to execute a simple reload - in my testing I encountered an inconsistent sequence of prompts when sending the reload command, and I can imagine there can be even more variations between other devices and IOS versions. Use of the TCL shell handles the issue - it just works without needing to handle various IOS inconsistencies.

Quickly setting up a HTTP server

While not recommended for production use, on Linux you can very quickly and easily spin up a HTTP server which will serve files in the current directory by running:

python -m http.server

You can use this to quickly and easily host images for your devices to pull from. The URL to use in your Config Push presets would look something like this:

http://your_machine_ip:8000/image_name.bin

Example of a successful run

And here's how we want to see the upgrade Mass Config Push preset to go:

While we wish everyone had exactly the same results as we did, some of your devices might not finish successfully. It is possible you may encounter errors, such as some devices not finishing the download in time, some devices not being able to download the file at all, or some other small changes in syntax on some devices that are unaccounted for.

If any such issue occurs, Unimus will inform you via its own error assessment of the failed pushes, such as CONNECTION_ERROR, COMMAND_UNSUPPORTED, INTERACTION_ERROR, etc. INTERACTION_ERROR may be a bit more difficult to troubleshoot - as the reasons for this error may vary.

In Part 2 of this series we will improve the upgrade process significantly by using TCL scripting. There will be an added layer of error detection by the upgrade TCL script itself, which should handle and help identify most of the failure scenarios you can run into.

Keep an eye on our blog for the Part 2 of this series coming soon! There is also a topic on our forums you can use to provide feedback or ask questions: https://forum.unimus.net/viewtopic.php?f=11&t=1426