NovaBACKUP 20.x Generic S3 Storage Device troubleshooting guide

Note: There is a bug in the initial 20.0 Release version (20.0.1011.1 build) that we are looking to fix that affects certain OS's or combinations of S3 endpoint providers + OS, which even though the https:// prefix is in place in the endpoint URL (top URL field) and all fields are specified properly, that the "Check credentials" function still fails. In that case go ahead and add the Generic S3 device without passing the test, by pressing the OK button, and it should appear in your Device tab list. Then attempt backing up 1 small file to that new Generic S3 device that you just created (even though it fails the "Check credentials" test) and verify if that backup completes successfully, in most cases it should. If it does fail then please verify all of the device fields input and correct them by editing the device, and then perform the telnet tests detailed in the bottom text and view the last screenshot for reference.

Just to be clear this Generic S3 Storage Device article was written specifically for version 20.x, some mention of how it works is applicable to both 19.8 and 20.x, but the troubleshooting steps that are detailed are only meant for and tested in 20.0.1011.1, which is the initial 20.0 Release version. Before we get too far, here are some links to other Generic S3 Storage Devices related KB articles (that apply to both 19.8 and 20.x backup client versions): 

Adding Generic S3 Storage Devices in NovaBACKUP 19.8 and 20.x

How to import backups stored on a Generic S3 Storage Device in NovaBACKUP 19.8 and 20.x

This "NovaBACKUP 20.x Generic S3 Storage Device troubleshooting guide" KB article aims to explain the what to look out for when creating (or editing) a Generic S3 device, as far as troubleshooting it, and performing the "Check credentials" function on a new Generic S3 device or an existing Generic S3 device in NovaBACKUP 20.x. As a general note the Generic S3 backup and restore code was completely revamped in NovaBACKUP v20.0. In that case it is absolutely REQUIRED that you will need to re-create any S3 (Generic or Legacy) device that exist in the Device tab, which would have been created in any version prior to the 20.0 Release (20.0.1011.1), including 19.8.1325.1 (the prior release before 20.0).

The first 20.0 Release build was build number 20.0.1011.1, as a note. In that case if in any prior version of NovaBACKUP backup client you had either a Legacy S3 device, which would be an S3 device created and added in the software prior to version 19.8.1325.1, that device will not work for both backup and restore in 20.0.1011.1, or you had a modern Generic S3 device created in 19.8.1325.1, which in that case would be a Generic S3 device created and added in the software starting in version 19.8.1325.1 (the last release prior to 20.0), backup and restore will not work for you in 20.0 (I saw that as most likely, as there were many changes in the code between 19.8 and 20.0 when it comes to Generic S3 backup and restore, so even "Check credentials" may fail if you added a Generic S3 device in 19.8.1325 and were to upgrade to 20.0.1011.1). The 20.0.1011.1 (the initial 20.0 Release build version) then it will not work in version 20.x. If you did have a Generic S3 device created prior to 20.0.1011.1, like in 19.8 for a Generic S3 device, or a Legacy S3 device created prior to 19.8, that S3 device will likely not work at all for backup/restore/check credentials in 20.0.1011.1, and in that case you will need to create a new device for it in 20.x, after deleting the original device. You can have a maximum of 8 x Generic S3 devices (an error will appear when attempting to add more than that). In addition to the "S3 Settings" tab in the Generic S3 Properties dialog, there is an "[Optional Settings]" tab, which can only be specified when creating a new device (cannot be done later by editing the device), and a "[Security]" tab to enable and specify a Secret encryption key (encryption password), because Encryption for Generic S3 backups is done at the device level and not at the backup job level, and that setting can be specified for a new device as well as you can edit it later to change the key at the device level at any time (a keychain history of the entered key (passwords) are saved in encrypted format locally) but it is always best practice to write down these encryption key (passwords) to keep somewhere safe outside of the backup client computer, as they may be required in order to restore backups, if the key that the backup was completed with at the time of reading that from the Generic S3 device does not match the currently specified key stored in the current Generic S3 device at the time of restoring for instance a very old backup and in that case you had changed the Secret encryption key multiple times after that backup was completed, you may need to have that password written down somewhere to restore a very old backup in that case, in case the keychain history that we store locally in an index file does not match what the backup is expecting (and then apply that old encryption password on the current Generic S3 device, or enter it manually when you are prompted at the time of the restore and it can't find a match in the keychain history).

The Device tab > "Generic S3 Properties" dialog (adding a new device or editing an existing) in the 20.0 client looks like this in the "[S3 Settings]" tab / section, which is where all of your Generic S3 provided server details are specified (all fields are required here to specify), and it contains a "Check credentials" button to verify the connection to the Generic S3 endpoint server can A. connect to and B. pass a credentials check to verify the account can be accessed from the backup client. Note: The "[Optional Settings]" tab and the "[Security]" tab are optional tabs to specify additional settings in, and are not required to add the device and backup and restore from it. The [S3 Settings] tab inside "Generic S3 Properties" dialog, which all fields are required to specify, looks like this when creating a new Generic S3 device, notice the "Check credentials" button to utilize at the bottom of the dialog:

The [Optional Settings] tab in the "Generic S3 Properties" dialog, is basically a way to import your old Device and Backups from that old Device, in that case you specify the Domain + Computer Name in this tab, and this tab is ONLY able to be specified when creating a brand new Generic S3 device, you cannot apply or change the values in [Optional Settings] later after the device is already saved. At the time of creating a new Generic S3 device allows you specify the Domain (keep this field blank if the computer were in WORKGROUP) and the Computer Name (The computer name that matches your original backup client machine that had a Generic S3 device on it prior, and that computer name would need to match the computer name that would have been seen in Windows, so without the domain suffix appended to the end, if you were to type in the "set" command in an Admin Command Prompt in Windows on your original computer where the backup client was installed, then it would be able to report a variable "COMPUTERNAME" and in that case, the value seen there is the computer name we expect here, that was last utilized by your Generic S3 device that you are attempting to restore from now) to basically restore the Generic S3 device on a new or rebuilt replacement backup client computer, like if the backup client computer or hard drive failed where the app was installed and you did not have an Image Backup or otherwise of that backup client computer that contained Generic S3 devices in the client, to be able to restore backups from that old Generic S3 device (the Generic S3 device must have been created in 20.0.1011.1 or newer backup client to support using the Optional Settings tab settings keep in mind), that you had backups stored to already. So this would be utilized in the case that you ever were to need to re-install the backup client from scratch, or re-install your computer OS and all applications from scratch, like due to a hard drive failure or a virus, and you lost your NovaBACKUP application and Program Data (the program's settings and files, including the Device information and Restore index items) and now need to be able to import those prior backups from your old Generic S3 device back into the Restore tab of the newly installed software (from a Generic S3 device that you re-create again as new but then fill in the Optional Settings tab info for the Domain Name and the Computer Name of the original backup client computer that was lost / destroyed for instance). After you add a Generic S3 device that also contains the [Optional Settings] specified correctly, to match your old backup client domain + computername that you are intending to restore backups from, normally this would be on a new or replacement computer to replace the original machine where the backup client had been performing Generic S3 backups to, then in the Restore tab of the software it should show the listing of indexed backups there, that were taken originally by the computer that you specified in the [Optional Settings] tab at the time of creating this new Generic S3 device, which will be seen just after this Generic S3 device is added if you did fill in the [Optional Settings] fields properly, if it does not then either you did not complete any backups to that device that you are trying to now import, or the bucket name was incorrect, or the domain + computer name did not match the server on the Generic S3 provider's end, as it has to be exact (you may be able to utilize the Generic S3 provider's Console website to check the domain + computer name if you browse the bucket where the backups were storing with this old Generic S3 device by NovaBACKUP, to give you a clue there in case you can't recall the exact computer name. This is why it may be important to enable encryption at the Generic S3 device level, as if someone guesses your computer name they may be able to restore backups from that device, but they would also need to know ALL of the settings you had to enter into the [S3 Settings] tab of the software, and in that case since we encrypt that device information to the local files that would not be so easy to figure out. It is still best practice that you enable encryption on any form of Cloud backup like Generic S3 however. Note: Keep in mind the software allows you to add up to 8 x Generic S3 devices, so you don't need to delete your existing / current device if all you are doing here is restoring some backups from another computer. You will be required to know the Secret encryption key (encryption password) however, or have that old key stored in the [Security] tab (that tab is detailed later in this article), if you intend to actually restore data from those backups that you used to Import with this method:

Note: Currently in the first 20.0 Release it is required to utilize all UPPERCASE characters for the fields in this 'Optional Settings' tab, otherwise the Import function will fail, as it won't be able to find that Domain and Computer name if anything but all UPPERCASE characters are utilized for these two fields, if you have already utilized anything but all UPPERCASE characters in these two fields, then the device will need to be re-created, or a second new device created to specify the characters in these two fields in ALL UPPERCASE to allow the Import function to work.

The [Security] tab in the "Generic S3 Properties" dialog, at the time of creating a new Generic S3 device or editing the device after the fact, allows you to enable or disable Encryption on the Generic S3 device, or change the encryption password, so that the backup data is encrypted to Generic S3 when jobs to that device run next, in that case when you go to restore the a backup that does have Encryption enabled on the device, whether normally via the client Restore tab, or after utilizing the Optional Settings to restore the backup index from an old Generic S3 device you had created in 20.0 prior and lost the machine, etc, then it will require you to enter an encryption "password" (detailed below this section):

It is not absolutely necessary to utilize the "Check credentials" button when adding a new Generic S3 device in the software, but it is certainly recommended to do it before adding the device (it is not required to add the device). However, there are some caveats to know that certain Generic S3 providers that WILL NOT be able to pass this test or be able to be utilized to backup. Most providers do work without knowing one additional thing about the URL address that is entered, but it is important to know about this for those providers that DO NOT run a Proxy on port 80/HTTP to forward traffic to port 443/HTTPS which would be necessary (even if you ran self-hosted Generic S3 server application such as min.io, that would require working on HTTPS using a self-signed certificate). All Generic S3 providers operate on port 443/HTTPS, but the dialog allows you to enter the address as just what the provider normally states the address as which would be as "s3.us-east-2.amazonaws.com" for instance for the Amazon AWS provider S3 service, in that case the NovaBACKUP Generic S3 dialog when it comes to the URL field will be able to work if you enter in the endpoint URL at that "s3.us-east-2.amazonaws.com" address, because Amazon AWS runs a Proxy that forwards the traffic from port 80/HTTP to port 443/HTTPS and that is handled on the back end and is not a concern for you. However, not all Generic S3 providers support auto-forwarding the traffic from 80 to 443, and in that case you may have to specify the PROTOCOL as a suffix in the endpoint URL, so in that case you may have to fill in the field as "https://h3f0.la11.idrivee2-14.com" for iDrive E2 Generic S3 provider endpoint URLs (servers), or "https://s3.us-west-001.backblazeb2.com" for Backblaze Generic S3 provider endpoint URLs (servers), in the URL field in the NovaBACKUP Generic S3 Properties dialog. Some providers specifically REQUIRE https:// protocol prefix at the start of the endpoint URL in this field (the example is for the iDrive E2 Generic S3 provider):

Note: All fields are REQUIRED to be specified in the Generic S3 Properties dialog for the "S3 Settings" tab, and by default we fill in the "Device Name" field with "Generic S3 Storage Services.0" as the first Generic S3 device when creating a new device, and the .0 number at the end will be automatically incremented by 1 if you have more than 1 Generic S3 device in place already, however the "Device Name" field name can be changed to what you want. The supported characters for most of the fields for Generic S3 Properties dialog are the following, as S3 in general only supports this character set (unfortunately non-ASCII English language characters cannot be utilized for some fields, such as the Bucket Name, Username, and Backup Directory fields). If you do utilize an invalid character you will get a warning dialog message right away as you attempt to enter those unsupported (invalid) characters into a field that cannot support it: 

Even if you are not able to get the "Check credentials" test to pass when attempting to add a new Generic S3 device to NovaBACKUP 20.0 it does not necessarily mean that continuing after that to add the device, even though the test function did fail, is not going to work for backup; backup to it may still work. There are certain combinations of Operating Systems that this function will not work with, on some Generic S3 providers and we are tracking that issue down currently to fix in a newer 20.0 release. For now if you hit this issue, and even if prefixing the URL with https:// does not allow the "Check credentials" function to work for you when creating a new Generic S3 device, you can still attempt to add the device and backup to it, if it can backup then it works. There is a way to know if you DO NEED to prefix the Generic S3 provider's endpoint URL with https:// in front of the endpoint server address, and that is described below and there is a simple test you can do with the telnet client added to Windows, on a Workstation OS, go to Programs and Features, and add "Telnet client" a feature, or on a Server OS use Server Manager to add a new feature "Telnet client".

If you add a Generic S3 device, even if you cannot pass Check credentials test, then you can edit the device (by double-clicking on it in the Device tab listing) but know that you can only change two fields in that already saved device (even if you have not backed up to it yet), which are the two highlighted fields, those can be edited for instance if you get a new Access Key ID and Secret Access Key to need to re-enter those settings in an existing Generic S3 device by double-clicking on the device in the list and editing it in the "[S3 Settings]" tab, those settings can be changed and saved, and the other settings you can change later by editing an existing Generic S3 device are all of the "[Security]" tab option fields, including disabling or enabling encryption, or just changing your encryption password (each encryption password utilized in the device per backup taken are stored encrypted themselves in a history index that nobody can read, but in that case it is always best practice to write down your encryption password utilized each time you do set one or change one - write it down along with the date of when you made the encryption password change, and keep that information very safe, and not on the backup client computer):

The [S3 Settings] tab if edited, will only allow you to change two settings, the Access Key ID and the Secret Access Key, in case you have to generate a new set of ID/Keys or you get a new account at the same Generic S3 provider, and this is how you would re-enter those details to correct them in the existing device this way using the [S3 Settings] tab after double-clicking on the Generic S3 device in the Device tab list:

The [Optional Settings] tab cannot be edited or specified after the device is already created, it has to be done at the time of creating the Generic S3 device, but in that case you could just create a new device if all you need to do is restore backups from it, as the NovaBACKUP lets you have 8 total Generic S3 devices, so that should not be a problem. The new device you create would only be utilized to restore the older backups anyhow and most likely will just be added temporarily to allow for those restores to take place, and then this extra added Generic S3 device, meant only to restore, could be deleted after that if wanted (notice how the fields in this dialog, when you edit the device after it was already saved, are grayed out meaning you cannot specify those settings when editing a Generic S3 device, just create a new Generic S3 device in that case if needed, as the software can handle up to 8 x Generic S3 devices, and this extra added device could just be used to restore the old backups from in that case):

The [Security] tab, if the device is edited, contains the current masked out Secret encryption key (encryption password), or if encryption was never enabled at the device level, then the field will look blank, and this is where you can enable encryption that is specified at the Generic S3 Storage Services device level, not per backup job, and in that case the setting can be implemented at the time of creating the Generic S3 device as new, or after the fact by editing the device after the device was already created:

Note: The “iDrive E2” and “Backblaze” Generic S3 providers, do not run a proxy at 80/HTTP on their endpoint servers to allow it to auto forward traffic (without a protocol or port specified) to get to 443/HTTPS, so this is why with NovaBACKUP Generic S3 you currently have to prefix the URL for some provider's S3 endpoints like for those two. The current known Generic S3 provider endpoints that need this include iDrive E2 and Backblaze providers. You can see this is true that it will need the protocol prefix in the URL in NBK if you attempt to telnet to port 80 on the known good working servers that do run a proxy on their endpoint servers, using ‘telnet s3.us-east-2.amazonaws.com 80', for Amazon AWS, or 'telnet s3.us-west-1.wasabisys.com 80', for Wasabi, or 'telnet play.min.io 80', for the min.io public server, in that case all of those provider’s endpoint URL's work to connect in NBK, without specifying HTTPS protocol prefix in the URL, which is how user's will typically attempt to enter the URL as for a Generic S3 device, as that is what the provider tells them the address is (the provider likely does not have the protocol prefix or the port specified in the endpoint address in the server details/docs). However, if you try to do that with iDrive E2 or Backblaze, with 'telnet h3f0.la11.idrivee2-14.com 80', for iDrive E2, or 'telnet s3.us-west-001.backblazeb2.com', for Backblaze, then those commands will all fail to connect to 80/HTTP. If the Generic S3 provider does not run a proxy on their server end to support auto forwarding from 80 to 443, this is why Check credentials and backing up to a device that is attempted to be added without an HTTPS prefix can fail, on the providers that you for instance can't telnet to their URL at port 80 on, due to it not ultimately reaching the server at 443/HTTPS. This does mean that for certain Generic S3 providers you will be required to specify the HTTPS protocol in the URL in NovaBACKUP. We could probably make it fall back to attempting HTTPS if no protocol is specified in the URL to then attempt that, or even if HTTP protocol is specified in the URL to then attempt if no connection to HTTPS, to fallback like that if no connection is made to that specified URL (the old Legacy S3 Check credentials code used to attempt at 443/HTTPS if no protocol was specified in the URL), but we are not doing either of those things in the code at this time for Generic S3. Note: If you do make a successful telnet connection you can break out of it using the CTRL+] key combination, which is the official break key for the telnet client.

This is what the ping and telnet test result should look like for a Generic S3 provider that DOES NOT host a proxy running on their endpoint server end at port 80/HTTP which would be able to forward traffic to port 443/HTTPS, and in that case you will BE REQUIRED to add the protocol prefix for https:// in front of the endpoint URL. In the screenshot below, on the left side is the telnet command that did in fact successfully connect to port 443 (which is HTTPS), and it shows on the right side that when we attempt to telnet to port 80 (which is HTTP) the telnet connection failed in that case, for the iDrive E2 provider endpoint address on port 80, this means that that provider's endpoint URL does not have a proxy running on it that would auto-forward traffic to it (since you did not specify a protocol prefix in the URL field that would be the case that it would use port 80/HTTP), some Generic S3 providers DO THIS and others DO NOT. This is where you will have to know to add the https:// protocol as a prefix in the top endpoint URL field (like the second screenshot shows the URL specified as in this example):

Note: There is a bug in the initial 20.0 Release version (20.0.1011.1 build) that we are looking to fix that affects certain OS's or combinations of S3 endpoint providers + OS, which even though the https:// prefix is in place in the endpoint URL (top URL field) and all fields are specified properly, that the "Check credentials" function still fails. In that case go ahead and add the Generic S3 device without passing the test, by pressing the OK button, and it should appear in your Device tab list. Then attempt backing up 1 small file to that new Generic S3 device that you just created (even though it fails the "Check credentials" test) and verify if that backup completes successfully, in most cases it should. If it does fail then please verify all of the device fields input and correct them by editing the device, and then perform the telnet tests detailed in the above text and view the last screenshot for reference.

Note: If you do make a successful telnet connection you can break out of it using the CTRL+] key combination, which is the official break key for the telnet client.