Search Our Database

The purpose of this article is to show a way to use bash script to automate the backup upload and restore to IPS1 Object storage from a CentOS 7 SSH terminal. Here’s the step in general:

– Create an Object storage bucket in IPserverone’s customer portal
– Prep your CentOS 7 server by installing necessary package
– Generate GPG keys for backup file Encryption
– Make use of a script to automate the Backup process
– Restore data from object storage bucket

Create an Object storage bucket in IPserverone’s customer portal

1. Login to your customer portal account and navigate to the “Object Storage” management page :-
   
2. Click on ‘Create Bucket’ :-
   
3. You may insert your own bucket name, or generate a random name from the system. Once the name has been keyed in, click ‘Create’ button to create the bucket :-
   
4. The new object storage bucket will be created
   
5. Next, you will need an access key to access the new bucket. Navigate to “Access Keys” and click on ‘Create Access Key’ :-
   
6. Enter the following information:-
   1. Enter the Access “Key name”
   2. Enter the “Expiration date” of the key,
   3. “Grant write permission” to this access key for files upload / edit in the object bucket, else if you wish to create a read only key, untick this.
   4. In Access Level, decide on the bucket that this key can access to (The name of the bucket).
   5. Once all details inserted, click “Generate” to generate the new access key.

   

7. You will be given a set of Access key and Secret Key, copy and paste it to somewhere safe and keep it safe.
   

   Prep your CentOS 7 server by installing necessary package

   1. Enable Epel repository for your CentOS 7 box.
   yum install epel-release –y
   2. Install the duplicity package and rng-tools
   yum install duplicity rng-tools
   3. Allowed rng tools to generate random inputs and start the service
   "EXTRAOPTIONS="-r /dev/random""
   service rngd start
   4. Create an ips1 folder in root directory for backup script and gpg key storing purpose
   mkdir /ips1

   Generate GPG keys for backup file Encryption

   1. Generate gpg key, note down the passphrase used to generate the key.
   # gpg --gen-key
   gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.
   
   Please select what kind of key you want:
      (1) RSA and RSA (default)
      (2) DSA and Elgamal
      (3) DSA (sign only)
      (4) RSA (sign only)
   Your selection? 1
   RSA keys may be between 1024 and 4096 bits long.
   What keysize do you want? (2048) 2048
   Requested keysize is 2048 bits
   Please specify how long the key should be valid.
            0 = key does not expire
         <n>  = key expires in n days
         <n>w = key expires in n weeks
         <n>m = key expires in n months
         <n>y = key expires in n years
   Key is valid for? (0) 0
   Key does not expire at all
   Is this correct? (y/N) y
   
   GnuPG needs to construct a user ID to identify your key.
   
   Real name: backups
   Email address: support@ipserverone.com
   Comment: IPS1 Object Storage Backup
   You selected this USER-ID:
       "backups (IPS1 Object Storage Backup) <support@ipserverone.com>"
   
   Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
   You need a Passphrase to protect your secret key.
   
   We need to generate a lot of random bytes. It is a good idea to perform
   some other action (type on the keyboard, move the mouse, utilize the
   disks) during the prime generation; this gives the random number
   generator a better chance to gain enough entropy.
   We need to generate a lot of random bytes. It is a good idea to perform
   some other action (type on the keyboard, move the mouse, utilize the
   disks) during the prime generation; this gives the random number
   generator a better chance to gain enough entropy.
   gpg: key E85FED7C marked as ultimately trusted
   public and secret key created and signed.
   
   gpg: checking the trustdb
   gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
   gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
   pub   2048R/E85FED7C 2021-02-22
         Key fingerprint = B47C 9A36 0838 D0B4 75AA  AFC7 01FF AF74 E85F ED7C
   uid                  backups (IPS1 Object Storage Backup) <support@ipserverone.com>
   sub   2048R/365A5E01 2021-02-22
   2. Use command below to list all created keys.
   # gpg -K --fingerprint
   /root/.gnupg/secring.gpg
   ------------------------
   sec   2048R/E85FED7C 2021-02-22
         Key fingerprint = B47C 9A36 0838 D0B4 75AA  AFC7 01FF AF74 E85F ED7C
   uid                  backups (IPS1 Object Storage Backup) <support@ipserverone.com>
   ssb   2048R/365A5E01 2021-02-22uid                  backups (IPS1 Object Storage backup) <support@ipserverone.com>
   3. Export your gpg keys to a file call ‘backups’ and download it to save keep, if in anyway you had lost the access to the current server, you can still use this backup key to retrieve the uploaded and encrypted backup file in object storage.
   gpg --armor --export backups
   gpg --armor --export-secret-key backups

   Make use of a script to automate the Backup process

   1. Create initial scripts files and log files
   cd /ips1/
   touch obj-backups.sh obj-restore.sh .obj-configrc
   chmod 700 obj-backups.sh obj-restore.sh
   chmod 600 .obj-configrc
   mkdir -p /var/log/duplicity
   touch /var/log/duplicity/logfile{.log,-recent.log}
   2. Add the following contents to .obj-configrc file
   # IPS1 object storage credentials keys
   export AWS_ACCESS_KEY_ID="<ACCESS KEY>"
   export AWS_SECRET_ACCESS_KEY="<SECRET ACCESS KEY>"
   export IPS1_BUCKET="s3://ap-southeast-mys1.oss.ips1cloud.com/<NAME OF YOUR BUCKET>"
   
   # GPG Key information
   export PASSPHRASE="<YOUR GPG KEY PASSPHRASE>"
   export GPG_FINGERPRINT="<YOUR GPG KEY FINGERPRINT>"
   
   # Folder to backup
   export SOURCE="<PATH TO FOLDER TO BACKUP>"
   
   # Will keep backup up to 1 month
   export KEEP_BACKUP_TIME="1M"
   
   # Will make a full backup every 10 days
   export FULL_BACKUP_TIME="10D"
   
   # Log files
   export LOGFILE_RECENT="/var/log/duplicity/logfile-recent.log"
   export LOGFILE="/var/log/duplicity/logfile.log"
   log () {
         date=`date +%Y-%m-%d`
         hour=`date +%H:%M:%S`
         echo "$date $hour $*" >> ${LOGFILE_RECENT}
   }
   export -f log

   *** GPG_FINGERPRINT is “E85FED7C” from the gpg gen-key command above, please insert your respective key fingerprint number.
   The backup policy above is to make a full backup every 10 days and remove all backups older than one month. Kindly adjust according to your needs.

   3. Add the following contents to ‘obj-backups.sh’
   #!/bin/bash    
       source /ips1/.obj-configrc
       currently_backuping=$(ps -ef | grep duplicity  | grep python | wc -l)
   
       if [ $currently_backuping -eq 0 ]; then
               # Clear the recent log file
               cat /dev/null > ${LOGFILE_RECENT}
   
               log ">>> removing old backups"
               duplicity remove-older-than ${KEEP_BACKUP_TIME} ${IPS1_BUCKET} >> ${LOGFILE_RECENT} 2>&1
   
               log ">>> creating and uploading backup to IPS1 object storage"
         duplicity incr --full-if-older-than ${FULL_BACKUP_TIME} \
               --asynchronous-upload \
               --encrypt-key ${GPG_FINGERPRINT} \
               --include ${SOURCE} --exclude '**' \
               --allow-source-mismatch \
               / ${IPS1_BUCKET} >> ${LOGFILE_RECENT} 2>&1
   
               cat ${LOGFILE_RECENT} >> ${LOGFILE}
       fi
   4. Run the script
   ./obj-backups.sh
   5. Check the logs for the backup activity reporting in:
   cat /var/log/duplicity/logfile-recent.log
   6. Add the script running to the server cronjob list so that the backup running can be automated.
   crontab -e
   And insert cronjob line below to run the backup upload at 1am daily.
   0 1 * * * /ips1/obj-backup.sh > /dev/null 2>&1

   Restore a file from Object Storage

   1. Insert the content below to ‘obj-restore.sh’
   #!/bin/bash
       source /root/.obj-configrc
   
       if [ $# -lt 2 ]; then
               echo -e "Usage $0 <time or delta> [file to restore] <restore to>
           Exemple:
           \t$ $0 2018-7-21 recovery/  ## recovers * from closest backup to date
       \t$ $0 0D secret data/  ## recovers most recent file nammed 'secret'";
       exit; fi
   
       if [ $# -eq 2 ]; then
               duplicity --time $1 \
                       ${IPS1_BUCKET} $2
       fi
   
       if [ $# -eq 3 ]; then
               duplicity --time $1 \
                       --file-to-restore $2 \
                       ${IPS1_BUCKET} $3
       fi
   2. To restore all data uploaded, use command below:
   ./obj-restore.sh 0D /tmp/restore/
   You can also recover one specific folder with the following format from a backup 3 days ago with:
   ./obj-restore.sh 3D <folder> /tmp/restore/
   Or recover a specific file from 3 days ago
   ./obj-restore.sh 3D /path/to/thefile /tmp/restore/thefile

   Source: Store Object with Duplicity