Issues when using backup agents newer than 15.0.26321(C21.01 HF1) on OSes related to Windows 7, when the SHA-2 code signing support patch/update from Microsoft is not installed

Issues when using backup agents newer than 15.0.26321(C21.01 HF1) on OSes related to Windows 7, when the SHA-2 code signing support patch/update from Microsoft is not installed

Affected environments: 

The issue happens on the following operating systems when when the SHA2/SHA256 support patches from Microsoft have not been installed, or have been installed only recently (after encountering problems with backups):

  • Windows Server 2008
  • Windows Server 2008R2
  • Windows Small Business Server 2011
  • Windows 7

Issue type A:

You do an update of the Acronis Agent from an older, pre-C21.02 release (such as, for example, C21.01 HF1/build 15.0.26321) to a more modern agent release (such as, for example, the now-current C21.11/build 28323). After rebooting the machine (either when the Agent installer prompted you that a reboot is necessary, or for any other reason), the machine no longer boots, and loads Recovery Console instead. The Recovery Console displays a boot error about failing verification of the signature of one or more drivers, which are part of the Agent (e.g., and most commonly, snapman.sys ).

Issue type B:

You have managed to troubleshoot and workaround issue type A, but backups are still failing with error messages like:  "The system cannot find the file specified"

Example screenshot from Backup Console:

More info and examples of symptoms:

(Click on the "Details" button of the high-level error shown above)  Example error message details  of a failed backup activity on a Windows Server 2008R2 with Agent version C21.11/build 28323 :

HideExample of full error: 

 

Error
Operating system error: The system cannot find the file specified
MESSAGE Operating system error: The system cannot find the file specified
Additional info:
------------------------
Error code: 20250646
Fields: {"$module":"mms_vsa64_28323","CommandID":"..."}
Message: TOL: Failed to execute the command. Backup plan '....'
------------------------
Error code: 20250646
Fields: {"$module":"agent_protection_addon_vsa64_28323","CommandID":"..."}
Message: TOL: Failed to execute the command. Backup plan '...'
------------------------
Error code: 20119593
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: Failed to execute the command.
------------------------
Error code: 20250646
Fields: {"$module":"mms_vsa64_28323","CommandID":"..."}
Message: TOL: Failed to execute the command. Resolving items to back up
------------------------
Error code: 20250646
Fields: {"$module":"agent_protection_addon_vsa64_28323","CommandID":"..."}
Message: TOL: Failed to execute the command. Resolving items to back up
------------------------
Error code: 20119595
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: Failed to execute an internal command while resolving items selected for backup.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: ASYNC: Unidentified job.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: Job resolves the selection rules specified in the centralized backup plan for managed machine root item backup.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: ASYNC: Unidentified job.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: ASYNC: Lazy barrier action job.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: GXT searches for items by type.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: GXT started going through hierarchy.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: GXT is going through hierarchy.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: ASYNC: Lazy barrier action job.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: ASYNC: Action sequence pending action job.
------------------------
Error code: 16318468
Fields: {"$module":"agent_protection_addon_vsa64_28323"}
Message: GXT is going through hierarchy.
------------------------
Error code: 16318468
Fields: {"$module":"mms_vsa64_28323"}
Message: ASYNC: Action sequence pending action job.
------------------------
Error code: 16318468
Fields: {"$module":"mms_vsa64_28323"}
Message: ASYNC: Full barrier action job.
------------------------
Error code: 9764951
Fields: {"$module":"disk_bundle_vsa64_28323"}
Message: DAAPI has failed to collect information about the machine.
------------------------
Error code: 1048581
Fields: {"$module":"disk_bundle_vsa64_28323"}
Message: Failed to initialize the snapshot manager library.
------------------------
Error code: 9
Fields: {"$module":"disk_bundle_vsa64_28323","code":2}
Message: Unknown status.
------------------------
Error code: 65520
Fields: {"$module":"disk_bundle_vsa64_28323","code":2147942402}
Message: The system cannot find the file specified
------------------------

Example error from a slightly older (and equally affected) Agent build 28156 :

Full machine backup fails with the error:
Windows error: (0x80070002) The system cannot find the file specified

Checking the MMS logs we can find text similar to:

| error 0xf90004: ASYNC: Action sequence pending action job.
| line: 0xd7761d47edeb39a4
| file: d:\81\enterprise\common\async\semaphore.cpp:154
| function: Async::Detail::SemaphoreLogic::SemaphoreJob::Describe
| $module: mms_vsa64_28156
|
| error 0xf90004: ASYNC: Full barrier action job.
| line: 0x368cb7423c1c08db
| file: d:\81\enterprise\common\async\full_barrier.cpp:227
| function: Async::FullBarrier::BarrierJob::Describe
| $module: mms_vsa64_28156
|
| error 0x950057: DAAPI has failed to collect information about the machine.
| line: 0xce47d41ec1026b89
| file: d:\81\enterprise\disk_manager\daapi1\disk_manager_impl.cpp:690
| function: DiskManagement::Da1DiskManagerImpl::GetView
| $module: disk_bundle_vsa64_28156

In SnapAPI logs we can find text similar to:

[20211209-084329-099][SnapAPI][E] Failed to open control device '\\.\Global\SnapApiControl4050': status 2
[20211209-084329-099][SnapAPI][T] Init kernel 0x2
[20211209-084329-099][SnapAPI][T] Initialize status Error: 10 (status 2h) Text: \\.\Global\SnapApiControl4050

Cause

Issue type A:

This issue is completely connected to known issues ABR-330569 and ABR-330765 .

  • [ABR-330569] Agent installation on Windows Server 2008 R2 fails. This is relevant to running the installer of a modern agent version manually on a machine with one of the affected OSes - either as a fresh install (if an Acronis Agent has not been present on the machine before), or as an update-in-place (if an older, pre-C21.02 release of the Agent was already present on the machine, and the new release's installer is now automatically running in updater mode). The installer, in both cases, produces a user-visible error message which says that there is a problem with installing some components due to code signing.
  • [ABR-330765] Updating agent on Win2008R2 requires a restart and sends the machine into recovery mode. This is relevant if the agent gets updated remotely (by triggering an update manually via the Backup Console UI) or if gets configured to auto-update itself (Note: agent auto-update has been available only since C21.02).

If driver signature verification gets disabled by the machine's administrator at its boot screen (Recovery Console), the machine can proceed to boot successfully. However, backups after that may still not work, failing with an error message similar to "The system cannot find the file specified".

Issue type B:

Even when you have managed to troubleshoot and overcome issue type A(the SHA2 patches/KB updates get installed manually or via Windows Update, and when the Agent of an affected version (e.g. C21.11) gets re-installed, and the system is booting fine), backups after that may still not work, failing with an error message similar to "The system cannot find the file specified".

The following workaround can be used in this case, assuming the SHA2/SHA265 patch is already installed via Windows Updates:

  • fully uninstall the agent (using e.g. cleanup_tool)
  • reboot the machine
  • do a clean install of the C21.11/current agent

The clean install should let the disks be "seen" properly by our drivers, and get the backups going.

Background/root cause:

As explained in the Microsoft resources mentioned below, over the last months, stricter driver signing algorithm enforcement started happening on the Windows side, and Microsoft (as well as most software vendors - Acronis included) started using the modern and stronger SHA2/SHA256 cryptographic algorithm for signing code such as drivers and user-space executables. This is all part of the industry-wide initiative to phase out the use of SHA1, which has been deemed no longer sufficiently secure for a few years.

Acronis Agents, starting from C21.02 (February monthly release) for the Cloud variant for the Backup/Cyber Protection product, and corresponding releases published in the same timeframe for the on-premises variant of the product, have switched to using SHA2/SHA256-bit crypto signatures for the various kernel driver components which the Agent needs to operate.

For more information see:

Workaround

In order to tackle issue type A:

  1. To get the affected machine to boot, reboot the machine, and press F8 on system start up and choose the option “disable driver signature enforcement”.
  2. Install the SHA2 support patch from Microsoft manually, or by applying all available updates via Windows Update.
  3. Reinstall the (current release of) the Agent.

In order to tackle issue type A and issue type B (prevent issue type B from occurring at all):

  1. To get the affected machine to boot, reboot the machine, and press F8 on system start up and choose the option “disable driver signature enforcement”.
  2. Install the SHA2 support patch from Microsoft manually, or by applying all available updates via Windows Update.
  3. Do a deep-cleaning full uninstall of the Agent, by using the Cleanup Tool
  4. Reboot the server again.
  5. Do a clean install of the current agent version (e.g. C21.11 at the time of writing)

Make sure to keep your Windows OS sufficiently updated. In particular, apply all stability-critical and security-critical patches from Microsoft before updating your Acronis Agent. Install the SHA2-related patches from Microsoft before updating your Acronis Agent.

Notes about using the cleanup tool in this particular case

In its modern versions, you simply need to start it from a Windows CMD which is explicitly ELEVATED; no need to specify any arguments or flags: the tool just runs and sweeps. After the tool is done (which can take several minutes, and occasionally upwards of 10 minutes), there is now no need to do any of the manual post-checks mentioned in the KB - when the tools finishes the cleanup, its last line of output will let you know you need to reboot.

(!) The cleanup utility does not trigger a reboot on its own, it only informs the user if a reboot is required.

Note: On rare occasions (which have been seen a few times on Windows Server 2008/R2 and 2012/R2), the OS may BSOD once during the first reboot after the cleanup_tool has finished; this is due to how these older Windows versions handle driver removal; subsequent boots would be fine.