Articles in this section

See more

Retention, what is it and how do I use it

Avatar
Jon
  • Updated
Follow

Retention jobs only have one purpose: to remove old backups you no longer wish to keep.

Contents: 

Backup Retention can be accessed through the Schedule tab by selecting the button found at the bottom left of the window: [Backup Retention...]

 

Retention Conceptualization

Depending on your requirements, retention jobs can be performed before or after your backup, and can be configured to look at all of your backups or only one specific job. As such, some basic concepts will differ in the scheduling and number of files that will remain after a retention job is performed.

Example 1: Count 1

  1. Weekly Full File Backup runs in append mode for 4 weeks, creating the following backup files:
    • Weekly Full_dateTime1.nbd
    • Weekly Full_dateTime2.nbd
    • Weekly Full_dateTime3.nbd
    • Weekly Full_dateTime4.nbd
  2. Retention set to run each week before the Weekly Full File Backup set to "Count" 1.
  3. Flow:
    • Backup Week 2
      • Weekly Full_dateTime1.nbd
      • Weekly Full_dateTime2.nbd
    • Retention Week 3
      • Weekly Full_dateTime1.nbd
      • Weekly Full_dateTime2.nbd
    • Backup Week 3
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
    • Retention Week 4
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
    • Backup Week 4
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
  4. With a minimum count set to 1, and the retention job being performed before the backup, you will maintain two backup files after each backup.
    • Changing this to run after the backup will result in two files existing while the backup is being written, with the older file removed when the retention runs, resulting in one file through the week instead of two.

<top>

Example 2: Age 1 Month

NOTE: All partial backups, Differential or Incremental, will be retained since the most recent Full backup

  1. Weekly Full File Backup runs in append mode for 6 weeks, creating the following backup files:
    • Weekly Full_dateTime1.nbd
    • Weekly Full_dateTime2.nbd
    • Weekly Full_dateTime3.nbd
    • Weekly Full_dateTime4.nbd
    • Weekly Full_dateTime5.nbd
    • Weekly Full_dateTime6.nbd
  2. Retention set to run each week before the Weekly Full File Backup set to "Age" 1 month.
  3. Flow:
    • Backup Week 4
      • Weekly Full_dateTime1.nbd
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
    • Retention Week 5
      • Weekly Full_dateTime1.nbd
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
    • Backup Week 5
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
      • Weekly Full_dateTime5.nbd
    • Retention Week 6
      • Weekly Full_dateTime2.nbd
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
      • Weekly Full_dateTime5.nbd
    • Backup Week 6
      • Weekly Full_dateTime3.nbd
      • Weekly Full_dateTime4.nbd
      • Weekly Full_dateTime5.nbd
      • Weekly Full_dateTime6.nbd
  4. With Age set to one month, you will maintain 4 Weekly Full backups, with a minimum of 3 if all jobs run as scheduled (i.e. no manual jobs started)

<top>

Default Retention Settings: Age 1 Month & Count 10

Interpretation

In order for a backup file to be removed when a default retention job runs, the following conditions must be met:

  • Backups must be older than 1 month
  • And there must exist 10 backups

<top>

Don't forget to Schedule Backup Retention jobs

As Backup Retention is performed as separate jobs, you need to make sure to schedule any retention jobs to run after you create them.

Selecting the [Add...] button on the bottom right corner of the Schedule tab will allow you to schedule saved retention jobs that are stored in the Utility folder by default:



<top>

A Note about Incremental Backup Dependencies

Incremental backups are dependent upon the previous Incremental, back to the previous Full backup in order to be “retained” properly. This prevents unintentional loss of backup data.

In different terms: 

  1. Perform a Full file backup (Overwrite or Append mode)
  2. Perform an Incremental file backup (x times)
  3. Retention will not remove the Full or Incremental backups at this point
  4. Perform a Full backup, Retention will be able to remove anything prior to the new Full file backup only
  5. Perform Incremental backups, only those backup prior to the most recent full can be removed, the rest will be dependent

<top>

A Note about Network Shares and Permissions

[Guide] Backup to NAS (with Retention) Walkthrough

Retention jobs do NOT have a way to specify Network Permissions like a Backup Job can. Retention jobs have no way of linking to your Network Device (network storage device) that you may have added to NovaBACKUP in the "Device" tab as a "Network Device". This means that you will have to edit your Retention job script and specify the Network Permissions manually, otherwise when the Retention job runs it will fail to delete appropriate files.

The workaround for this is to adjust your Default Settings to target a Network Device added to the Device tab, then recreate your retention job by selecting [New] in the Backup Retention dialog. Follow this [Guide] Backup to NAS (with Retention) Walkthrough - Configure a Network Device - Part 1 to create a Network Device for NovaBACKUP and adjust default settings.

Retention job cannot delete File Backups on destination network storage device due to network permissions

<top>

A Note about the underscore "_" character in Image Backup names

Retention may not work on your Image Backups if your Image "Backup Name" contains the underscore "_" character. This is due to the way the algorithm identifies backups, stripping the underscore character and any following information to match an Image backup name.

This issue was resolved in version 19.3

Using an underscore will not impact File Backup Retention jobs.

<top>

A Note about Image Backup Retention and "Report Only" mode

There is an issue where if you perform a Retention job in Report Only mode and match Image Backups the internal history used by Retention is cleared for Image Backup files.

We strongly recommend performing Full Image Backups in Overwrite mode and using a manual name rotation strategy, such as "Image A" and "Image B".

 

<top>

"Standard" or Recommended Retention Practices

Schedule Retention jobs to run before your Backup job to ensure you have enough free space for the backup to complete.

Monthly Image - 1 minimum valid backup count, resulting in 2 total

Weekly Full File - 3 minimum valid backup count, resulting in 4 total

Daily Differential File - 4 to 8 minimum valid backup count, resulting in 5 to 9 total

This depends on the space and amount of data in your Daily backups, as well as the number of days you are performing backups on.

<top>

Weekly Full, Daily Differential & Monthly Image Backup Retention Policies

[Guide] Create a weekly full backup, a daily differential backup, and a monthly image backup (With Retention) Step 4. Creating backup retention policies for your scheduled backups

Creating backup retention policies for your scheduled backups

Now that you have that you have all three backup jobs scheduled you are going to make retention policies for each of these backup jobs. Remember, that a retention policy is a way to instruct your backups to delete old backups so you don’t lose space over time. It looks at how many backups you have and if there are more backups than what the retention was set for, it will delete the oldest backups.

So let’s first create a retention policy for our weekly full backup. Click on “Backup Retention” and from there you click on new. Here we are going place the name of the backup we are making the retention policy for and we are going add the words “_retention_policy” so that you know that this is the retention policy and not backup.

Click open, and it will say that this it will reset everything to default. Go ahead and click yes.

From here you are going change the “Backup type to process” as normal backup since this is a file based backup and not an image backup. You are going to change the backup setting to “count” only. This is a way easier system to work with and will make sure there are no confliction or off days on your backup retention policy. Next you are going change the “maximum valid backup count” to what you want you want. Remember that this is a weekly backup so if you change the number from 1 to let’s say 3, then that means that you will have 3 backup files or 3 weeks’ worth of backups. Since we want to have full redundancy we are going to put the maximum valid backup count to 4. This will give us about a month’s worth of backups.

From here click on the advanced tab. You are going to change media type to process to disk because you are going to be backing up to a disc drive. Next, you are going to uncheck the remove index logs options and go to then go to the Process media name field. This is important because this is what tells the retention which backup name to look for so it can place the retention rules on them. I am going to place the exact name I have for my weekly full backup into the field.

Then click on save and the name should already be filled at the bottom because you created a new one when you click on “open” earlier. If the name is not there then place the name of the backup we are making the retention policy for and we are going add the words “retention_policy” so that you know that this is the retention policy and not the backup.

Now if you notice, your Backup retention is not in the schedule tab yet; and that is because you have not added the retention policy to the schedule yet. So you are going to click on add so we can add the retention to the schedule.

From here you are going to click on the small little down arrow at the top right so you can select your backup retention policy that you have created for your weekly full backup. Usually we place them in the Utility folder in the NovaBackup user script folders but if you placed your backup jobs or scripts along with your retention policies scripts in a different folder you can go to them here. Afterwards, click open and from here you can start to create the schedule for the backup retention. You want to make the retention schedule match the backup schedule. The only difference you are going to make is that you are going to place the retentions 5 minutes before the backups. That way, the retention looks to see if it needs to delete any older backups first, then your backups will run 5 minutes later. So for this example you are going to set the date to the 7th like before but instead of 1am you are going to put 12:55am to let the retention go before the weekly full backup. Like before change to run as specific user and check your windows login credentials. Then click ok.

And now you have created your first retention schedule. Now you will do the same process for your daily differential backups and your monthly full image backups.

In the schedule tab and go to backup retention again. Click on "New" and enter the name of the daily differential job name and then add retention_policy and press open.

Like before it will tell you that this will default all the setting; click yes. Now you are going to change backup process to normal and change backup setting to count. From here choose how many files you want to retain before the retention policy deletes the oldest file. Since this is a daily differential backup and you have a weekly full backup once a week, I am going to set this to 6. So after 6 backup files it will delete the oldest differential backup file from your daily differential backups.

Next go to the advanced tab. Change type to process to disk and uncheck remove index logs. Then you are going to place the daily differential media name here so this retention policy knows to look for these particular backups. Double check everything just to make sure, then click on save.

Make sure that your job name has “retention_policy” at the end so you can know the difference between the backup jobs and the retentions.

From here click add and then select the job schedule button so you can add your daily differential retention policy to the schedule. As before you are going to have the retention run 5 minutes before the actual job so it can delete older backups before making new ones. For this daily differential example you will putting the time again to 12:55am so that the retention can run 5 minutes before the daily backups. Like before change to run as specific user and check your windows login credentials. Then click ok

Now it’s time to finish off with creating your monthly image retention policy.

Go back to the schedule tab and go to backup retention. Click on New and place the name of the monthly full image job name and then add retention_policy and press open.

Click ok and reset everything to default values. Next change the Backup type process to "Image backup" since this is for our monthly full image backup and then change the backup setting to count. From here you are going to put 2 as the minimum valid backup count so that you will have 3 months’ worth of full image backups.

From here go to the advanced tab and change the Backup media process to disk and uncheck the remove index logs. (Now I just want to mention that technically you don’t need to place anything in the process media name because this is the only image backup job that you have. If I had more than 1 image job, then you would want to place the media name in this field so the retention can know what backup it needs to look for to place the retention rules on. With that being said, for this example you are still going to place the media name in here in case you decide on making another image backup job in the same destination location.)  After you have double checked everything, then go ahead and click on save.

The last step we are going to do is add that retention policy to the schedule. Click on add to find your retention job that you have just made. Remember that by default your retention policies would be in the utility folder in your Novabackup program data but if you saved them somewhere else, now would be the time to search and locate them. Then change the schedule to monthly and let’s put the retention 5 minutes before the monthly image backup. In this example, that would be on the 13th at 2:55am. Like before change to run as specific user and check your windows login credentials. Then click ok

And that is it. So now you see your Daily differential backup, your weekly full backup, your monthly full image backup, and the retention policies for each. This will ensure you have good redundancy and will ensure that your backups won't clog up space on your drive.

<top>

Cleanup Retention

How to remove entries from the Restore Tab "en masse" for manually removed backup files

Retention jobs will only remove files that exist where the software wrote them, if there is a backup that will match the settings you choose, that you wish to keep, you may want to move or rename the backup file.

  • Click the [Backup Retention] button on the bottom-left corner of the Schedule tab.
  • When the dialog opens, select the [New] button and give the retention job a name, such as "CLEANUP Retention".
  • Set the desired age or count to remove backups older than this value.
  • On the "Advanced" tab, you can specify the specific backup job filename that you wish to match against, or leave it blank to match every backup job filename.
  • Choose the "Remove index/logs even if backup file cannot be deleted" option to allow this retention job to remove the reference to files that no longer exist (i.e. you manually deleted the backup files).
    • Report-only mode will allow you to see what will be removed by this job, allowing you to confirm that files you wish to keep are not removed.
    • If you choose Report-only mode, no references will be removed, no backups will be removed, and you will need to change the setting back after saving the retention job before running it.
  • Select "Save..." then "OK"
  • Select "Backup Retention" again and choose "Run Now"

The only other method to do this is by hand, matching the entries shown on the Restore tab with their output file, removing invalid ones using right-click->delete for each one (reference within NovaBACKUP as well as the output file).

<top>

From the Online Help documentation

Backup Retention Overview

The backup retention feature provides the ability to create jobs to control which backups should be kept. This is useful in cases where too many backups have accumulated, storage space is running too low or you wish to keep future backups within a specified range or condition.

Backup retention jobs also check for dependencies to ensure that no backup gets deleted if another backup depends on it in order to be restored as expected. For example an incremental backup will not be deleted if an older related incremental backup remains.

A backup retention job may be run on demand or scheduled to run automatically. To access this feature, click the schedule menu item, then click the "Backup Retention" button.

Settings contained in the basic and advanced tabs are explained in separate topics, while those common to the main screen are explained here.

Job Name
This field displays the current backup retention job script and location. If you wish to schedule a backup retention job, you will need to specify the script as it would be displayed here.

Run Now Button
Click this button to run the backup retention job using the current settings. This will also save the settings to the job script as shown in the job name field.

New Button
Click this button to open a dialog to specify a name for a new backup retention job. The new job script will have all default settings, which may then be edited and saved to the same name or to a different name.

Open Button
Click this button to open a dialog to select an existing backup retention job. This will load the job settings and set it as the current backup retention job script.

Save... Button
Click this button to open a dialog to save the settings to the current backup retention job script. You can save  to the current job script name or save to a different name if preferred.

Cancel Button
Click this button to close the backup retention screen without saving any changes.

<top>

Basic Tab

Keep backups using this setting:

Age (Default 1 month)
Keep backups for a specified minimum amount of time.

Count (Default 10)
Keep a specified minimum number of backups.

Age and Count (Default)
Keep backups for a specified minimum amount of time after the minimum required count is reached.

 

Backup type to process:

Normal Backup 
The standard backup type which is based on file selection, also known simply as file backup.

Image Backup 
Disk image backup created with the Image Backup tool.

Normal and Image Backups (Default)
Both the normal file and image backups are processed. This is the same process as each type of job were run separately with all settings applied to each type of backup separately.

 

Minimum backup age:
This setting specifies the age when keeping backups based on a minimum age.

Minimum valid backup count: 
This setting specifies the count when keeping backups based on a minimum count.

Maximum warnings allowed for a valid backup: 
If this is checked, a backup will be counted as valid only if it has the set number of warnings or less.

Maximum errors allowed for a valid backup: 
If this is checked, a backup will be counted as valid only if it has the set number of errors or less.

<top>

Advanced Tab

Backup media type to process: 

Disk 
Process only backups that have been saved to disk or network.

Tape
Process only backups that have been saved to tape.

Disk and Tape (Default)
Process backups that have been saved to disk, network or tape.

 

Process only backups containing this media name:
For normal backup types, limit the backup retention process to media containing the name specified. For image backup types, limit the backup retention process to backups with filenames containing the name specified. Just leave this setting blank to process backups of any name.

Delete only incremental and differential backups 
If this is checked, only incremental and differential backups are allowed to be deleted. This will avoid the removal of any full or snapshot backup types.

Remove index/logs even if backup file cannot be deleted
If this is checked, related content gets removed only if the backup was deleted successfully. Related content includes indexes used to list the backup and its log entries as well as actual log files. Note that this setting does not apply to tape backup, because only disk or network backups can be deleted.

Report only mode: just log the results but take no action
If this is checked, only the results of a retention job is logged,  with no changes to the state of any backups. This is similar to a print preview without actually printing.

<top>

Related articles