Search Our Database

How to perform backup to IPS1 Object Storage bucket from a CentOS 7 Bash terminal

Last updated on |

The purpose of this article is to show a way to use bash script to automate the backup upload and restore to IPS1 Object storage from a CentOS 7 SSH terminal. Here’s the step in general:

– Create an Object storage bucket in IPserverone’s customer portal
– Prep your CentOS 7 server by installing necessary package
– Generate GPG keys for backup file Encryption
– Make use of a script to automate the Backup process
– Restore data from object storage bucket

Create an Object storage bucket in IPserverone’s customer portal

  1. Login to your customer portal account and navigate to the “Object Storage” management page :-
  2. Click on ‘Create Bucket’ :-
  3. You may insert your own bucket name, or generate a random name from the system. Once the name has been keyed in, click ‘Create’ button to create the bucket :-
  4. The new object storage bucket will be created
  5. Next, you will need an access key to access the new bucket. Navigate to “Access Keys” and click on ‘Create Access Key’ :-
  6. Enter the following information:-
    1. Enter the Access “Key name”
    2. Enter the “Expiration date” of the key,
    3. “Grant write permission” to this access key for files upload / edit in the object bucket, else if you wish to create a read only key, untick this.
    4. In Access Level, decide on the bucket that this key can access to (The name of the bucket).
    5. Once all details inserted, click “Generate” to generate the new access key.
  7. You will be given a set of Access key and Secret Key, copy and paste it to somewhere safe and keep it safe.

    Prep your CentOS 7 server by installing necessary package

      1. Enable Epel repository for your CentOS 7 box.
      yum install epel-release –y
      2. Install the duplicity package and rng-tools
      yum install duplicity rng-tools
      3. Allowed rng tools to generate random inputs and start the service
      "EXTRAOPTIONS="-r /dev/random""
      service rngd start
      4. Create an ips1 folder in root directory for backup script and gpg key storing purpose
      mkdir /ips1

    Generate GPG keys for backup file Encryption

      1. Generate gpg key, note down the passphrase used to generate the key.
      # gpg --gen-key
      gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.

      Please select what kind of key you want:
      (1) RSA and RSA (default)
      (2) DSA and Elgamal
      (3) DSA (sign only)
      (4) RSA (sign only)
      Your selection? 1
      RSA keys may be between 1024 and 4096 bits long.
      What keysize do you want? (2048) 2048
      Requested keysize is 2048 bits
      Please specify how long the key should be valid.
      0 = key does not expire
      = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
      Key is valid for? (0) 0
      Key does not expire at all
      Is this correct? (y/N) y

      GnuPG needs to construct a user ID to identify your key.

      Real name: backups
      Email address: support@ipserverone.com
      Comment: IPS1 Object Storage Backup
      You selected this USER-ID:
      "backups (IPS1 Object Storage Backup) "

      Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
      You need a Passphrase to protect your secret key.

      We need to generate a lot of random bytes. It is a good idea to perform
      some other action (type on the keyboard, move the mouse, utilize the
      disks) during the prime generation; this gives the random number
      generator a better chance to gain enough entropy.
      We need to generate a lot of random bytes. It is a good idea to perform
      some other action (type on the keyboard, move the mouse, utilize the
      disks) during the prime generation; this gives the random number
      generator a better chance to gain enough entropy.
      gpg: key E85FED7C marked as ultimately trusted
      public and secret key created and signed.

      gpg: checking the trustdb
      gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
      gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
      pub 2048R/E85FED7C 2021-02-22
      Key fingerprint = B47C 9A36 0838 D0B4 75AA AFC7 01FF AF74 E85F ED7C
      uid backups (IPS1 Object Storage Backup)
      sub 2048R/365A5E01 2021-02-22

      2. Use command below to list all created keys.
      # gpg -K --fingerprint
      /root/.gnupg/secring.gpg
      ------------------------
      sec 2048R/E85FED7C 2021-02-22
      Key fingerprint = B47C 9A36 0838 D0B4 75AA AFC7 01FF AF74 E85F ED7C
      uid backups (IPS1 Object Storage Backup)
      ssb 2048R/365A5E01 2021-02-22uid backups (IPS1 Object Storage backup)
      3. Export your gpg keys to a file call ‘backups’ and download it to save keep, if in anyway you had lost the access to the current server, you can still use this backup key to retrieve the uploaded and encrypted backup file in object storage.
      gpg --armor --export backups
      gpg --armor --export-secret-key backups

    Make use of a script to automate the Backup process

      1. Create initial scripts files and log files
      cd /ips1/
      touch obj-backups.sh obj-restore.sh .obj-configrc
      chmod 700 obj-backups.sh obj-restore.sh
      chmod 600 .obj-configrc
      mkdir -p /var/log/duplicity
      touch /var/log/duplicity/logfile{.log,-recent.log}
      2. Add the following contents to .obj-configrc file
      # IPS1 object storage credentials keys
      export AWS_ACCESS_KEY_ID=""
      export AWS_SECRET_ACCESS_KEY=""
      export IPS1_BUCKET="s3://ap-southeast-mys1.oss.ips1cloud.com/"

      # GPG Key information
      export PASSPHRASE=""
      export GPG_FINGERPRINT=""

      # Folder to backup
      export SOURCE=""

      # Will keep backup up to 1 month
      export KEEP_BACKUP_TIME="1M"

      # Will make a full backup every 10 days
      export FULL_BACKUP_TIME="10D"

      # Log files
      export LOGFILE_RECENT="/var/log/duplicity/logfile-recent.log"
      export LOGFILE="/var/log/duplicity/logfile.log"
      log () {
      date=`date +%Y-%m-%d`
      hour=`date +%H:%M:%S`
      echo "$date $hour $*" >> ${LOGFILE_RECENT}
      }
      export -f log

      *** GPG_FINGERPRINT is “E85FED7C” from the gpg gen-key command above, please insert your respective key fingerprint number.
      The backup policy above is to make a full backup every 10 days and remove all backups older than one month. Kindly adjust according to your needs.

      3. Add the following contents to ‘obj-backups.sh’

      #!/bin/bash
      source /ips1/.obj-configrc
      currently_backuping=$(ps -ef | grep duplicity | grep python | wc -l)

      if [ $currently_backuping -eq 0 ]; then
      # Clear the recent log file
      cat /dev/null > ${LOGFILE_RECENT}

      log ">>> removing old backups"
      duplicity remove-older-than ${KEEP_BACKUP_TIME} ${IPS1_BUCKET} >> ${LOGFILE_RECENT} 2>&1

      log ">>> creating and uploading backup to IPS1 object storage"
      duplicity incr --full-if-older-than ${FULL_BACKUP_TIME} \
      --asynchronous-upload \
      --encrypt-key ${GPG_FINGERPRINT} \
      --include ${SOURCE} --exclude '**' \
      --allow-source-mismatch \
      / ${IPS1_BUCKET} >> ${LOGFILE_RECENT} 2>&1

      cat ${LOGFILE_RECENT} >> ${LOGFILE}
      fi

      4. Run the script
      ./obj-backups.sh
      5. Check the logs for the backup activity reporting in:
      cat /var/log/duplicity/logfile-recent.log
      6. Add the script running to the server cronjob list so that the backup running can be automated.
      crontab -e
      And insert cronjob line below to run the backup upload at 1am daily.
      0 1 * * * /ips1/obj-backup.sh > /dev/null 2>&1

    Restore a file from Object Storage

      1. Insert the content below to ‘obj-restore.sh’
      #!/bin/bash
      source /root/.obj-configrc

      if [ $# -lt 2 ]; then
      echo -e "Usage $0

      if [ $# -eq 2 ]; then
      duplicity --time $1 \
      ${IPS1_BUCKET} $2
      fi

      if [ $# -eq 3 ]; then
      duplicity --time $1 \
      --file-to-restore $2 \
      ${IPS1_BUCKET} $3
      fi

      2. To restore all data uploaded, use command below:
      ./obj-restore.sh 0D /tmp/restore/
      You can also recover one specific folder with the following format from a backup 3 days ago with:
      ./obj-restore.sh 3D /tmp/restore/
      Or recover a specific file from 3 days ago
      ./obj-restore.sh 3D /path/to/thefile /tmp/restore/thefile

    Source: Store Object with Duplicity