Release Overview - Unimus 2.2.0

With each new release, we also upload a release overview video, so if you prefer a video format, you can find it here:
Youtube - 2.2.0 Release Overview video

For those who prefer readable content, read on!

Device Variables for Config Push

The biggest new feature of this release are the Device Variables. These allow you to inject per-device unique values into generic Config Push presets. This can be used to create Push presets which are generalized and pushed to a large set of devices, but by using variable substitution the data pushed to each device will be tailored to that device.

Basically, Push presets can now behave more like templates, which was previously not as easily possible. More info on variables on our Wiki.

New APIv3

We are releasing the first wave of APIv3 endpoint groups in 2.2.0. In this release, you can now use APIv3 to manage:

• CLI mode change passwords
• Credentials
• Jobs
• Tags
• Zones

Plans are to continue implementing the remaining endpoint groups in point releases after 2.2.0. We are focusing on covering what isn't available in APIv2 first, and after we have API feature parity with the GUI, we will start adding endpoints into APIv3 which are currently covered in v2.

Mass Config Push available over the API

The addition of "Jobs" API endpoints is of significant note, as these allow you to use Config Push over the API.

If you want to build your own custom network automation front-end, Unimus can now be used as its back-end. This means you don't need to deal with all the intricacies of device communication which Unimus already handles, and allows you to leverage our existing 240+ network device drivers from your own applications.

Improvements to API Token management

As a part of API work, we have also substantionally improved API Token management. Tokens can have descriptions and comments now, and new security controls were introduced to specify if tokens should have access to device Credentials and CLI mode change passwords.

Performance improvements

We also put a lot of effort into performance improvements in this release, with a target to support 120.000 devices inside a single Unimus instance. This required significant work across all subsystems in Unimus.

Some highlights of performance differences between 2.1.4 vs. 2.2.0:

• Job initialization time on 120k devices down from 9 minutes to 1.5 minutes
• Average MikroTik RouterOS job duration down from 21 seconds to 9 seconds
• UI component responsiveness massively improved - for example, select all on 120k entities in the UI down from previously 3 minutes to 8 seconds now
• All UI components, screens and tables load data in under 10 seconds (average UI load at ~2 seconds, with 10s being the worst result) with 120k devices in the system
• Full discovery + backup on 120k devices in 2 hours 45 minutes (using 600 concurrent jobs)

We will be publishing a technical blog article with details on performance improvements, as well as another blog article on large-scale Unimus deploy performance tuning in the near future.

Discovery algorithm improvements

During Discovery, Unimus tests and validates which Credentials are available for your devices. Our "Credential Binding" feature serves as a method to prevent Unimus from testing all available credentials on devices, allowing you to set specific credentials that should be used for device communication.

As a part of our work on performance we also optimized device connections during Discovery. If only a single credential is available for a device (either due to Credential Binding, or simply just having one credential in the system), Unimus will now only perform a single connection to the device during Discovery. Previously multiple connection attempts would be performed in this scenario, as the Discovery flow for single and multi-credential discoveries was the same. This has now been optimized.

In case of using SSH, this can result in significant CPU utilization savings during device jobs, as SSH session establishment is computationally expensive. More info on the new behavior on our Wiki.

Security improvements

Another part of Unimus that received heavy focus durign 2.2.0 development was security. We performed an internal security audit of Unimus in advance of full Penetration Testing.

We have found and fixed multiple security-related issues of various severity in this release - please check the full Changelog for more info.

Unimus will undergo a full Penetration Testing cycle during March 2022. We will publish the pentest report publicaly on our Blog - stay tuned.

Other minor new features

In addition to the major features and changes above, this release also brings a bunch of smaller changes and improvements. We added an option to set the UI session timeout, added support for NetXMS v4, RouterOS v7, and many other minor improvements.

Please check the full Changelog below for more information.

Bug fixes and security fixes

As mentioned in a previous section, security and stability were a large time-investment on our end during the development cycle of this release. In addition to security, we have also fixed a slew of bugs, issues and UI inconsistencies of various severity.

All together 33 various bugs, and 20 various security-related issues were fixed. Please check the full Changelog below for full details.

As with each new release, we also added support for a bunch of new networking vendors and devices. In 2.2.0 we are adding support for 12 new device types, from 9 separate networking vendors.

The Changelog for 2.2.0 is quite long, as this is one of our largest releases to date. If you want to see all the changes in this release, please check the full Changelog below:

= Version 2.2.0 =
Features:
  Added option to set UI session timeout (example "-Dserver.servlet.session.timeout=1h")
  Updated NetXMS client library to latest version (4.0.2156)
  Added additional built-in Backup Filters for FortiOS devices
  Added missing search in Config Mode Password binding window (Devices > Edit)
  Unmanaged devices are now displayed with Italic font in "Backups" screen (same as in "Devices")
  Added support for device selection menus on Cisco IOS
  Added support for CLI sections in FortiOS
  Improved Huawei VRP driver compatibility
  Improved detection and grouping of invalid commands in Config Push
  Reordered buttons on the Devices screen into logical groups (better UX)

  New Device Variables feature for Config Push
    - Variables can be defined for devices in the Device screen
    - both single and multi device variables edit are supported
    - Variables can be used in Config Push in the "${variable_name}" format
    - more info: https://wiki.unimus.net/display/UNPUB/Device+Variables

  Added new APIv3:
    - implemented new v3 API, exposing functionality currently missing in APIv2
    - currently available endpoints: "Jobs", "Zones", "Tags", "Credentials", "CliModeChangePasswords"
    - API tokens now have a new "Allow access to credentials" checkbox
    - please check http(s)://your_unimus_address/api/v3/ui for new built-in API docs
    - APIv2 will remain functional for the foreseeable future

  Improvements to API Token management:
    - added "Description" to API tokens
    - API tokens now have a new "Allow access to credentials" checkbox
    - added an "Edit" button for API tokens

  Mass Config Push is now available over APIv3:
    - added an "API Jobs" tab to Config Push if any API jobs exist
    - new retention settings for API Push Job history
    - see above section for details on APIv3

  Performance improvements:
    - general improvements across the application due to DB structure and data access improvements
    - substantial performance improvements in high-concurrency environments due to JDBC datasource change
    - Config Search has been offloaded to the database (as required per DB engine), bringing much better performance
    - optimized job initialization time (10x faster when running jobs on 5.000 devices)
    - a single Unimus instance can now handle 120.000 devices with full discovery + backup on 120k devices in 2 hours 45 minutes
    - UI component responsiveness massively improved (for example, select all on 120k objects in the UI now takes 8 seconds, from 3 minutes previously)
    - with 120.000 devices in Unimus, all screens now load in under 10 seconds max (average screen load at 2 seconds)

  Security improvements:
    - performed an internal security audit of Unimus in advance of full Penetration Testing
    - more info on found and fixed issues in the "Security fixes" section
    - updated user password hashing algorithm to Argon2 (previously Bcrypt2 was used)
    - existing user passwords will be migrated on first successful login
    - Unimus 2.2.0 will undergo a full pentest cycle, results will be published publicly on our Blog

  Optimization of device connection count during Discovery:
    - only open a single CLI session when only a single credential is available for a device
    - applies when credential discovery is not needed due to Credential Binding
    - more info: https://wiki.unimus.net/display/UNPUB/Discovery

  Rewrite of MikroTik RouterOS driver:
    - performance increases, average discovery on ROS down to ~9 seconds (from 21 seconds)
    - added handling for new CLI behaviors introduced in latest ROSv6 versions
    - added support for ROSv7

  Added support for:
    - ArubaOS v6
    - DrayTek VigorSwitch
    - Engage IPTube
    - FiberStore Campus switches
    - Hatteras / Overture Networks
    - Huawei USG
    - JunOS EVO
    - MikroTik RouterOS 7
    - Planet XGS switches
    - other various Planet switches
    - Ubiquiti Dream Machine (UDM)
    - Ubiquiti LTU / LTU-Pro

Fixes:
  Fixed a memory leak if a Core connection connected and disconnected frequently
  Fixed wrong Running Job state could be set on devices during heavy concurrency operations
  Fixed job history records would not be created for devices with extremely long addresses
  Fixed a running Network Scan not being stopped if it's Preset was deleted
  Fixed description missing in Mode Change Password binding (Devices > Edit)
  Fixed running job state could be reverted to a wrong state when Managing / Unmanaging devices while a job was running
  Fixed select all / deselect all and the selection model in general could break in the "Device credentials" table
  Fixed moving devices between Zones could cause the Zone Number to update even if device was not moved due to address conflict
  Fixed changing a user's role to visually break the Backups screen if the affected user was already on it
  Fixed possibility to add Comments to deleted objects if the Comment window was opened while object was deleted
  Fixed actions buttons not working properly in "Backups > Configuration" in specific cases
  Fixed wrong time formatting in "Use management > System access history > Session end" (values were correct in DB)
  Fixed "Other settings > Per-Tag connectors" would not properly show all configured ports for a connector
  Fixed attempting to remove all Users would throw an exception (will now properly remove all users other than yours)
  Fixed the Zones screen not properly refreshing when specific changes were done to Zones by another user
  Fixed select all on tables with extremely large amounts of objects could causing loading for a very long time
  Fixed enabling "Show all passwords" in the "CLI mode change passwords" table could cause bad behavior in the "Device credentials" table
  Fixed search in "Import history jobs" did not work
  Fixed the "port" field being formatted wrongly in the "Notifications > Email" screen
  Fixed changing a user's role to duplicate the Theme selector on the Dashboard if the affected user was already on it
  Fixed Credentials screen did not live-update changes to counters when credentials were Bound / Unbound by another user
  Fixed "Basic import > CSV file import" could throw exceptions to the UI when an invalid CSV file was provided
  Fixed possibility to add Device Access restriction without selecting and account, which resulted in an exception
  Fixed Comment icon column in the Schedules screen was not properly sized
  Fixed rare scenarios where upgrade from 2.0 or 2.1 to latest versions could fail
  Fixed possible invalid input in "Notification settings > Diff before and after lines"
  Fixed multiple rare errors on concurrent operation attempts on already deleted objects during multi-user workflows
  Fixed multiple other minor UI and UX issues and missing live value changes during multi-user workflows
  Fixed discovery failing on some models of Adtran TA
  Fixed discovery failing on JunOS-EVO devices
  Fixed discovery failing to recognize newer Planet switch types
  Fixed Config Push on MikroTik RouterOS could fail on specific commands with long output
  Fixed output formatting in Config Push on some MikroTik RouterOS versions could be broken
  Fixed backup could contain some extra unwanted data on some MikroTik RouterOS versions

Security fixes:
  Completely removed log4j library due to multiple exploits that were identified in this library
  Log out all other user's sessions if a user changes their password (other than the session changing the password)
  Log out all sessions of a user if their password is changed by another Administrator user
  Users logged out due to session timeout are redirected to the Login screen instead of just an overlay on their last screen
  Fixed user could remove Backup Filters applied to Tags the user didn't have access to
  Fixed users could re-run Push presents from output group context menu even if they didn't have access to do this
  Close currently opened "Show password" popups in the Credentials and "Device > Info" screens when a password is set to "High security mode"
  Close currently opened "Show password" popups in the Credentials and "Device > Info" screens when a user's role is changed to READ-ONLY
  Fixed Backups screen would not remove access to already opened device backups if access to a device was lost
  Fixed users without access to the Default Zone could still add devices through "Network Scan"
  Changed APIv2 to no longer expose credential passwords through Device endpoints (there was no way to control this), use APIv3 for credential access

  Fixed multiple instances of "live" access changes not working (screen change / reload was required to apply new access restrictions):
   - for all affected screens affected data will be added / removed immediately after accessibility is changed now
   - fixed Dashboard not listening to live device access changes
   - fixed Zones not listening to live access changes
   - fixed "Mass Config Push > Targets" not listening to live device access changes
   - fixed "Mass Config Push > Output groups" not listening to live device access changes
   - fixed "Other settings > Per-Tag connectors" not listening to live access changes
   - fixed Devices screen not listening to Zone-based device Tag live access changes (Tag propagations to Devices from Zones)
   - fixed "Basic import" not listening to live Zone access changes

Embedded Core version:
  2.2.0

Known issues:
  ISSUE: "Re-discover affected devices when Ports or Connectors change" Advanced Settings option does not work
  WORKAROUND: none
  STATUS: issue scheduled for fixing

  ISSUE: "Stop" in Config Push does not work
  WORKAROUND: none
  STATUS: issue scheduled for fixing

  ISSUE: Some screens in Unimus show time in server's time zone, others in client's (browser's) time zone
  WORKAROUND: none, issue only relevant if client has different time zone than server
  STATUS: we are debating on how to fix this - will likely create a setting to select which TZ should be used