How to use an Amazon S3 Bucket region other than US East (N. Virginia)
NovaBACKUP
Last Updated: Jan 03, 2018 09:34 AM PST
This how-to article is mentioned in our Knowledge Base article Amazon S3 bucket name restrictions and requirements. This is a how-to on using an Amazon S3 bucket that has a region other than US East (N. Virginia) to get that bucket to work with NovaBACKUP software. The primary symptom of an incompatible region utilized for the S3 bucket name when used with NovaBACKUP is that the backup job to Amazon S3 will not be able to connect to Amazon S3 at the start of the backup job and fail to store a backup to S3.
As detailed in the NovaBACKUP Knowledge Base article "Amazon S3 Bucket region and name restrictions and requirements" it mentioned that when creating the Amazon S3 bucket name you will be required to select "US East (N. Virginia)" for the region name. The reason that this specific region has to be utilized for your bucket's region is explained by Amazon in the article here; do a search for the word Important. For those that do not want to read the article you can see the Important text repeated here:
Important
If you use a region other than the US East (N. Virginia) endpoint to create a bucket, you must set the LocationConstraint bucket parameter to the same region. Both the AWS SDK for Java and AWS SDK for .NET use an enumeration for setting location constraints (Region for Java, S3Region for .NET). For more information, see PUT Bucket in the Amazon Simple Storage Service API Reference.
This is the workaround that we have written which you can implement if you do NOT want to use the US East (N. Virginia) region for the bucket region; however this will require installing additional software in Windows, outlined and detailed in the AWS documentation here. It will take about 15 minutes to perform the software installations and then 3 minutes of your time to make the change in NovaBACKUP to your existing Amazon S3 backup job to then test your S3 backup. The required software installation includes the Amazon AWS CLI (Command Line Interface) application to create a new bucket name via AWS CLI with an added "LocationConstraint" bucket parameter. The additional software installations do not have to be done on the machine that NovaBACKUP is installed on (it can be done on a workstation OS). You can follow the instructions that we have written on how to implement the workaround here:
After testing NovaBACKUP and us-west-2 US Oregon region for example this works for us-west-2 (US West Oregon) region in NovaBACKUP after setting the “LocationConstraint” property to us-west-2 (US West Oregon) when creating a new S3 bucket. With the special "LocationConstraint" property set on our S3 bucket name to us-west-2 the backup completes successfully to Amazon S3 where we are using a bucket region of US West Oregon, however without following this workaround it would have failed as by standard NovaBACKUP is only compatible with Amazon S3 that has a region setting of us-east-1 (US N. Virginia). The workaround requires creating a new bucket name in AWS after installing and using the Amazon AWS CLI (Command Line Interface) application (and cannot be done via the AWS Console website) to create the bucket and set the "LocationConstraint" property. The ability to set that "LocationConstraint" property is not something that can be done straight from the AWS Console website, you have to install AWS CLI application and then configure AWS CLI for your S3 credentials and then you can create a new bucket with the “LocationConstraint” parameter added for your specified/preferred region for that new bucket.
Step-by-step instructions for utilizing an Amazon S3 bucket name other than "US East (N. Virginia)" with NovaBACKUP:
1. The AWS Regions and Endpoints web page shows the various regions that Amazon S3 has that can be created. In this instructions example I want to use the S3 region named us-west-2, which is US West (Oregon). It is possible to use NovaBACKUP to perform a backup to this example region however it requires special setup in AWS to be supported and for the backup to actually work and not fail right away. You will be required to view the AWS Regions and Endpoints web page to look for the region that you would like to utilize for your S3 bucket and then take special note that the website will have a last column named "Signature Version(s) Support; this last column contains a value that you will need to know per region. The regions marked "Signature Version(s) Support" with a value of "Versions 4 only" will NOT work with NovaBACKUP, only the regions with a value of "Versions 2 and 4" will work with NovaBACKUP. Do not attempt to use a bucket with a region that has "Versions 4 only" for the value as it cannot work.
The screenshot example below shows the Signature Version(s) supported by Region Name, the Region Names that cannot be used because they support only "Versions 4 only" are marked in red, and the yellow highlighted entries CAN be used because they support "Versions 2 and 4". You can also see the fourth column which is the "Location Constraint" and the data in that column tells you if the "LocationConstraint" property is required to be set on the new bucket that you create. The "LocationConstraint" property can only be set on a new bucket and how to do this is described in Step 8 and S3's Location Constraint requirement is mentioned in the paragraphs above and in Step 8. The only Region Name that does NOT require the "LocationConstraint" property to be set is "US East (N. Virginia)", the column data for that region shows "(none required)" and is circled in blue:
NOTE: You can click on the screenshot above to see the current list of Amazon S3 Regions.
2. Install the AWS Command Line Interface (CLI) on Microsoft Windows, detailed here: https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#install-cli
Download the latest AWS CLI installer package for Windows (64-bit) here:
https://s3.amazonaws.com/aws-cli/AWSCLI64.msi
Install it by double-clicking on the .msi file.
3. Install Python and Pip for Windows, the latest installer version download (3.6.3) is here: https://www.python.org/ftp/python/3.6.3/python-3.6.3-amd64.exe
Install it by double-clicking on the .exe file.
4. Install the AWS CLI packing using pip:
C:\Windows\System32>pip install awscli
5. Now you will need to verify that python, pip, and aws commands are in the environment path and can be executed:
Start a new Admin Command Prompt (do not use an existing Command Prompt session as that won’t have the proper PATH statements added for Python/Pip/AWS commands) and enter these commands in to verify executables are in place:
C:\Windows\System32> python --version
Python 3.6.2
C:\Windows\System32> pip --version
pip 9.0.1 from c:\users\myname\appdata\local\programs\python\python36\lib\site-packages (python 3.6)
C:\Windows\System32> aws --version
aws-cli/1.11.84 Python/3.6.2 Windows/7 botocore/1.5.47
6. Configure AWS CLI to input your S3 access key ID and secret key as well as default region choice, detailed here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
Run this command via Admin Command Prompt to configure the AWS CLI with the access key ID and secret key and the us-west-2 region, note: the 4th prompt will ask for “Default output format [None]: and you can just hit enter at that prompt without configuring anything:
C:\WINDOWS\system32\aws configure
Make sure that the command finishes properly without error, it will return you back to the command prompt if successful without additional messages.
7. Verify that the "aws configure" command that you issued in the prior step was able to be configured properly by repeating the command "aws configure" again from the Admin Command Prompt. It will display a line of configuration data and have you confirm it by pressing the enter key on each of the four configuration items that you performed in the prior step. Just press enter on each line. If you need to change a value in the AWS configuration you have the ability to do that now and in the future.
Make sure that the command completes without error and that it returns you back to the command prompt without additional messages.
8. You will now create a new bucket. The example is our own example so you will be required to change a couple of the properties to match your environment and requirements. Run the example command via Admin Command Prompt, detailed in the second example on this web page: https://docs.aws.amazon.com/cli/latest/reference/s3api/create-bucket.html:
C:\WINDOWS\system32\aws s3api create-bucket --bucket test-nova5 --region us-west-1 --create-bucket-configuration LocationConstraint=us-west-1
In the example above we utilized and created a new bucket name of “test-nova5”, and for the region we are using "us-west-2" (which equates to US West Oregon), and for the "LocationConstraint" variable we are using “us-west-2” (which equates to US West Oregon). This region WILL work in NovaBACKUP but only if you set the “LocationConstraint” variable properly with the above aws s3api create-bucket command properly. All regions will work that are in the AWS S3 Regions and Endpoints list which show in the last column named “Signature Version(s) Support” and that contain a value of “Versions 2 and 4”, NovaBACKUP is not compatible with the regions that display “Versions 4 only” in that column as it can only support Version 2 actually in NovaBACKUP’s AWS/S3 code.
An example of the command to create a new bucket with the AWS CLI tool should output is shown here:
After issuing the command it should allow the creation of the new bucket and then display the "Location" of that bucket on Amazon AWS, and then take you back to the command prompt without any additional messages or errors displayed. You will verify that this new bucket name and selected region are displayed in the AWS Console website in the next step.
NOTE: If you get an error stating "An error occured (InvalidAccessKeyId)" or something to do with an invalid SecretKey then you will need to repeat Step 6 again to perform the "aws configure" command again to configure AWS with the correct and valid AWS Access Key ID and AWS Secret Access Key that matches your account.
9. Using the Amazon AWS Console website - https://s3.console.aws.amazon.com, login to it and then verify that the new S3 bucket was created and that it displays the same Bucket Name and Region properties that you created it with in the prior step. View the list of S3 Bucket names and locate the new bucket that you created in the prior step. If it is not there then something went wrong in the prior steps and just go through it again.
10. Now test out the new S3 bucket by going to NovaBACKUP’s device tab, editing your existing Amazon S3 device, normally named "Amazon Simple Storage Services.0" in the Device list, and replace the text in the Bucket Name field with the new Bucket Name that you just created in Step 8, and then save that change by clicking OK.
11. The final step is to modify your existing Amazon S3 backup job so that it points to the newly modified Amazon S3 device. While in NovaBACKUP click on the Backup tab and then open your S3 backup job, or if your job is scheduled you can also right-click on the S3 backup job on the "Schedule" tab and then click "Modify Job" to modify that backup job. Once the Amazon S3 job is open you will click on the "Backup to..." button (or click Settings -> Destination -> ...) and then re-select the device location for where the backup should be stored to the Amazon S3 device that you modified in the prior step by re-selecting that Amazon S3 device in the device list. Then save the backup job. Now you can test the S3 backup job. It should connect to S3 and complete the file backup job to send data to that new bucket in S3.
Thank you.