Backup and Restore

From OpenNMS
Jump to navigation Jump to search

Backup and Restore OpenNMS

Backing up

Record your OpenNMS Setup

OpenNMS is a complex application build on several technologies: Java, the jetty web server, the PostgreSQL database, RRD... Luckily, your Linux distribution package management should automatically install these components and will take care of version and dependencies.

Still you need to record your setup:

  • OpenNMS Exact Version – OpenNMS repos are always getting obsolete as major versions appear make sure that you know your exact OpenNMS version i.e. 1.8.17-1. OpenNMS old repos can be tricky to find. It is recommended to install this version on another monitoring machine (and every time you upgrade the active OpenNMS) waiting for this rainy day .. :-)
  • Java – type "java -version" on the shell prompt and record the exact Java version
  • DB Version – I myself had many problems restoring OpenNMS to a different PostgreSQL version. This is maybe the most important component. Always use the same exact version when restoring.
  • OS Architecture and Flavor – Had many problem restoring to different flavors, e.g. CentOS —> Ubuntu -- mainly configuration file locations, PostgreSQL different versions and configurations. Make sure to restore to the same operating system and preferably to the same version as was used on the original system.
  • iplike – Did you install custom OS iplike package for PostgreSQL? If so, make sure you install the same package on the new machine

Backing Up the Database

It is always recommended that you use PostgreSQL's "custom" format when performing backups. PostgreSQL is much more flexible in how it deals with restores with the custom format.

To back up your OpenNMS database, run the following:

pg_dump -U opennms -Fc -f /tmp/opennms.pgsql.gz opennms

This will create an "opennms.pgsql.gz" backup file in the /tmp directory.

Backing Up Configuration

All configuration is stored in the $OPENNMS_HOME/etc/ directory. It should be enough to back the entire directory up using tar or zip, like so:

tar -cvzf /tmp/opennms-etc.tar.gz -C $OPENNMS_HOME etc

On Debian and Ubuntu, $OPENNMS_HOME/etc is a symlink, you should back up /etc/opennms instead:

tar -cvzf /tmp/opennms-etc.tar.gz /etc/opennms

Backing Up RRDs

If you have the space, it's probably a good idea to back up your RRD data as well. This is all the historical data on metrics tracked by OpenNMS, such as disk space or network interface traffic history. Copy or tar up the contents of $OPENNMS_HOME/share/ if you wish to back them up as well. Note that your share directory can get VERY large, depending on the number of devices you're managing.

On Debian and Ubuntu, RRD data is in /var/lib/opennms/rrd/.

Backing up any other customizations

If you happen to install any specific Java .jar files not part of the standard OpenNMS packages, not forget to backup these up as well.

Restoring an OpenNMS installation on a new system

Install the OS

  • Open your OpenNMS records and start by installing the OS + all updates (if needed) to match the backed up system

Install Java

  • Install the same Java packages as used on the backed up system

Install OpenNMS

  • Using the wonderful OpenNMS Online Installation Guide, simply install OpenNMS as it was a new clean installation. The MOST important part here is to make sure you install using the package repository containing the same OpenNMS version as was used in the backed-up system.
  • Configure Java and perform other tasks associated with installing OpenNMS.

Restore the OpenNMS configuration and RRD files

  • Stop OpenNMS (if running). Delete the content of the OpenNMS /etc/opennms, RRD folder
  • Using the backup from the old machine, extract from backup the /etc/opennms, RRD folders into the new machine using the same paths as the backed up system
  • Leave OpenNMS stopped

Restore the OpenNMS database

  • Start PostgreSQL (if not running)
  • su as the postgres user and delete the current OpenNMS database

dropdb opennms

  • As the postgres user, create a new clean db owned by the opennms user

createdb -O opennms opennms

  • As postgres user, restore the DB:

pg_restore -U opennms -Fc -d opennms /path/to/db_dump_file

  • Alternately to reduce the amount of time used to perform the restore, add the '-j NUM / --jobs=NUM' option to the command with NUM being the number of parallel jobs used to perform the restore (the number of cores in your server would be a good starting point for the NUM value):

pg_restore -U opennms -Fc -j 24 -d opennms /path/to/db_dump_file

The restore should produce no errors. If there are errors, examine them and fix at this point.

  • As root, restart PostgreSQL

Restore any OpenNMS code customizations

  • If any Java .jar files or other code customizations were in place, restore these

Start OpenNMS

  • As root, start OpenNMS

Test functionality

  • Ensure that all monitored nodes, previous historical data and so on are present
  • see the section on testing OpenNMS in Upgrading

Tips for Backup And Restore

  • If using cron to schedule the db backup, make sure it runs as the postgresql user
  • The configuration files can be backed up using simple tar cron script. Alternatively, you can use rsnapshot. This will also create in-time copies (snapshots) of the data.
  • From time to time, perform test restores to verify your backup sets

Example 1: Sample backup script

  • Tested with OpenNMS 1.10 on Ubuntu 12.04

This is a backup script that performs a weekly backup of OpenNMS configuration files and the database, and copies backup files to a remote SMB share.

The RRD historical data files are not backed up -- if these are desired, uncomment the script section below about files in /var/lib/opennms (this is where these files are on Debian/Ubuntu systems).

Prerequisites

  • Ensure /sbin/mount.cifs exists; install packages as needed

aptitude install cifs-utils

Create the backup script

touch /etc/cron.weekly/opennms-backup
chmod +x /etc/cron.weekly/opennms-backup
vi /etc/cron.weekly/opennms-backup


#!/bin/bash

# define backup destination
LOCALDIR=/mnt/backup
VERSIONINFOFILE=$LOCALDIR/opennms_backup_versions.txt

# backup subroutine
backup_item()
{
    ARCHIVE=$1
    if [ ${ARCHIVE:0:1} == '/' ]; then
       ARCHIVE=${ARCHIVE:1}  # remove leading slash
    fi
    ARCHIVE=${ARCHIVE//\//_}  # convert remaining slashes to underscores

    tar -zcpf $LOCALDIR/$ARCHIVE.tar.gz $1 2>&1 | grep -v 'Removing leading'
}

# ensure the backup directory exists
if ! [ -e $LOCALDIR ] ; then
    mkdir -p $LOCALDIR
fi

# mount the backup target directory using CIFS
mount -t cifs -o username=username,password=secretpassword \\\\smbservername\\OpenNMSBackup /mnt/backup

# back up the OpenNMS PostgreSQL database
su - postgres -c "pg_dump --format=c opennms" | gzip > $LOCALDIR/opennms.postgresql.backup.gz

# save OpenNMS and related program version numbers for future reference
dpkg -l | grep opennms > $VERSIONINFOFILE
dpkg -l | grep postgresql >> $VERSIONINFOFILE
dpkg -l | grep iplike >> $VERSIONINFOFILE
dpkg -l | grep jre >> $VERSIONINFOFILE
java -version 2>> $VERSIONINFOFILE

# back up the OpenNMS etc directory and all subfolders to store the OpenNMS configuration
backup_item /etc/opennms

# back up the OpenNMS var/lib directory and all subfolders to store gathered statistics
#   warning: this can be large (hundreds of MB or several GB)
##backup_item /var/lib/opennms

# unmount the backup target
umount /mnt/backup

Example 2: simple backup script

  • Tested with OpenNMS 1.12 on CentOS 6.5

This is a backup script that performs a daily backup of OpenNMS configuration files, the database and rrd archives and keeps the last 30 backups.

Prerequisites

  • Ensure your backup folder /mnt/backup is mounted

Create the backup script

touch /etc/cron.daily/opennms-backup
chmod +x /etc/cron.daily/opennms-backup
vi /etc/cron.daily/opennms-backup


#!/bin/bash

# define backup destination
LOCALDIR=/mnt/backup

# ensure the backup directory exists
if ! [ -e $LOCALDIR ] ; then
    mkdir -p $LOCALDIR
fi

# Creating timestamp
dsig=`date +%Y%m%d%H%M%S`

# Back up Postgres DB
pg_dump -U opennms -Fc -f $LOCALDIR/$dsig-database.pgsql.gz opennms

# Back up rrd archives
tar cvfz $LOCALDIR/$dsig-rrd.tar.gz /opt/opennms/share/rrd/

# Back up config files
tar cvfz $LOCALDIR/$dsig-config.tar.gz /opt/opennms/etc

# Delete all backups older than 30 days
find $LOCALDIR -name '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]*.gz' -atime +30 -exec echo Deleting: {} \; -delete


Example 3 - detailed backup script

can be run from cron weekly . and additionally keeps sequential backups in a second directory and the most recent separately from it for easy space management

File Makebackup.sh tested on ubuntu 14.04 and opennms 12.9

#!/bin/bash

## BACKUP OPENNMS SCRIPT
## by Paul Cole - omdreams@gmail.com
## intention :  to backup current opennms configuration and data to a directory  and log it
##              expectations - run as sudo - provide  ability to be run as cronjob
##              Best practice - copy to separate dirve or NFS mounted share
##  accepts commandline options  for  $1 = mode of backup   - if not FULL then will accept $2 $3 $4 for additional options
##              FULL or F  = FULL BACKUP with RRD and SQL and ALL ETC
#               ETC or E = ETC directory only
#               RRD or R = RRD data ( historical SNMP statistics data )
#               SQL or S = SQL only  data
#               PARTIAL or P = SQL and ETC only

## Declarations
DATE=$(date +"%Y%m%d.%H%M")
LOGFILE="backuplog.$DATE.log"
BKPPATH="/data/opennms/backups"
BKPOLDPATH="/data/opennms/oldbackups"
BKPLOGPATH="/data/opennms/backups"
BKPMODE="FULL"

## begin
#check if right privs
if [ "$(whoami)" != "root" ]
   then
        echo "Sorry, you are not running with Sufficient Privlidges - try with sudo ."
        exit 1
fi
## check CMNDLINE ARGS
if [ ! -z "$1" ]
   then
        echo "CMD LINE ARGS NOT YET ACCEPTED"
        exit 1
fi
clear
echo "-----------------------------------------------------------"
echo " OpenNMS Backup Script by Paul Cole : omdreams@gmail.com "
echo "-----------------------------------------------------------"
echo "."
#sanity checks
if [ ! -e $BKPPATH ]
   then
        mkdir $BKPPATH
        echo "Backupdir $BKPPATH DOES NOT EXIST - created it"
fi

if [ ! -e $BKPOLDPATH ]
   then
        mkdir $BKPOLDPATH
        echo "Backupdir $BKPOLDPATH DOES NOT EXIST - created it"
fi

if [ ! -e $BKPLOGPATH ]
   then
        mkdir $BKPLOGPATH
        echo "Backupdir $BKPLOGPATH DOES NOT EXIST - created it"
fi

#start logging
LOG="$BKPLOGPATH/$LOGFILE"
echo "Logging to $LOG"
echo "--------------------------------------" | tee $LOG
echo " Moving old backups and initializing "
echo "-------- FileStamp $DATE ----------" | tee -a $LOG
# mv /usr/share/opennms/backup/* /data/opennms/oldbackups
mv $BKPPATH/* $BKPOLDPATH | tee -a $LOG

# Backup the postgres DB opennms only
# echo STOPPING OPENNMS
echo "."
opennms stop | tee -a $LOG
opennms -v status
echo "."
echo "-----------------------------------------------------------" | tee -a $LOG
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): performing vacuum on postgresdb " | tee -a $LOG
        psql -U opennms -c "VACUUM"

## SQL dump - BINARY
#if [ "$BKPMODE" == "FULL" ]; then
        FILE="$BKPPATH/OpenNMS.pgsqlbin.$DATE.gz"
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Backing up Postgres DB as binary dump" | tee -a $LOG
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): pg_dump destination file: $FILE" | tee -a $LOG
        pg_dump -U opennms -Fc -f $FILE >> $LOG
#fi

echo "-----------------------------------------------------------" | tee -a $LOG

## ETC directory
#if [ "$BKPMODE" == "FULL" ]; then
        FILE=$BKPPATH/OpenNMS.etcbkp.$DATE.tar.gz
        SIZE=$( du -chsS /etc/opennms | grep total )
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Backing up OpenNMS ETC config directory"
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Destination file: $FILE" | tee -a $LOG
        echo "          Patience we are compressing $SIZE"  | tee -a $LOG
        tar -acvf $FILE /etc/opennms/ >> $LOG
#fi

echo "-----------------------------------------------------------" | tee -a $LOG

## RRD directory
#if [ "$BKPMODE" == "FULL" ]; then
        FILE=$BKPPATH/OpenNMS.RRD.$DATE.tar.gz
        SIZE=$( du -chsS  /data/opennms/varlib | grep total )
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Backing up OpenNMS Round Robin Historical SNMP statistical data" | tee -a $LOG
        echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Destination file: $FILE" | tee -a $LOG
        echo "          Patience we are compressing $SIZE" | tee -a $LOG
        tar -acvf $FILE  /data/opennms/varlib >> $LOG
#fi

echo "-----------------------------------------------------------" | tee -a $LOG

## REMAINDER OF FILES
FILE=$BKPPATH/OpenNMS.code.$DATE.tar.gz
SIZE=$( du -chsS  /usr/share/opennms | grep total )
echo " $BKPMODE $(date +"%Y%m%d.%H%M"): Backing up remainder of opennms directory Excluding ./etc , ./share ./logs and backups " | tee -a $LOG
echo " $BKPMODE $(date +"%Y%m%d.%H%M"): OpenNMS has consumed $SIZE drive space all inclusive" | tee -a $LOG
tar -acvf $FILE --exclude="$BKPPATH" --exclude="$BKPOLDPATH" --exclude="/usr/share/opennms/share" --exclude="/user/share/opennms/logs"  >> $LOG

echo "-----------------------------------------------------------" | tee -a $LOG

SIZE=$( du -chsS $BKPPATH | grep total)
echo " $BKPMODE $(date +"%Y%m%d.%H%M"): THIS OpenNMS backup consumed $SIZE drive space all inclusive" | tee -a $LOG
SIZE=$( du -chsS $BKPOLDPATH | grep total)
echo " $BKPMODE $(date +"%Y%m%d.%H%M"): historical OpenNMS backups consume $SIZE drive space all inclusive" | tee -a $LOG

echo "--------------------------------------" | tee -a $LOG
echo "-------- COMPLETED  $(date +"%Y%m%d.%H%M") ----------" | tee -a $LOG
echo "--------------------------------------" | tee -a $LOG
opennms start | tee -a $LOG

opennms -v status | tee -a $LOG
echo "."
echo " These Backups are stored in  $BKPPATH   and older backups in  $BKPOLDPATH "
echo " To view detailed log type: less $LOG "
echo "."

ls -rtalh $BKPPATH

Restore script for paul cole's backup script

restores the DB portion only at this time.

#!/bin/bash

LOGFILE="Logs/InstallDB.log"
BackupFile="/opennms/backup/OpenNMS.pgsqlbin.*.gz/"
# restore last backup db
clear
echo -ne "\n ------ RESTORE LAST BACKUP OF DB ------\n"
echo -ne    "---------------------------------------\n"
echo -ne "pg_restore -U opennms -d opennms -c $BackupFile\n"
echo -ne "-------------------------------------------------------------------------\n"
echo -ne " THIS INSTALLS A CLEAN DB from the backup  - you must be SUDO and then run the opennms install -dis after to update teh db\n\n\n"

pg_restore -U opennms -d opennms -c /opennms/backup/OpenNMS.pgsqlbin.*.gz/ | tee $LOGFILE

/opennms/bin/install -dis