Backup and Restore

Back up your SUSE Manager installation regularly, in order to prevent data loss. Because SUSE Manager relies on a database as well as the installed program and configurations, it is important to back up all components of your installation. This chapter contains information on the files you need to back up, and introduces the smdba tool to manage database backups. It also contains information about restoring from your backups in the case of a system failure.

Backup Space Requirements

Regardless of the backup method you use, you must have available at least three times the amount of space your current installation uses. Running out of space can result in backups failing, so check this often.

Backing up SUSE Manager

The most comprehensive method for backing up your SUSE Manager installation is to back up the relevant files and directories. This can save you time in administering your backup, and can be faster to reinstall and re-synchronize in the case of failure. However, this method requires significant disk space and could take a long time to perform the backup.

If you want to only back up the required files and directories, use the following list. To make this process simpler, and more comprehensive, we recommend backing up the entire /etc and /root directories, not just the ones specified here. Some files only exist if you are actually using the related SUSE Manager feature.

  • /etc/cobbler/

  • /etc/dhcp.conf

  • /etc/fstab and any ISO mountpoints you require.

  • /etc/rhn/

  • /etc/salt

  • /etc/sudoers

  • /etc/sysconfig/rhn/

  • /root/.gnupg/

  • /root/.ssh

    This file exists if you are using an SSH tunnel or SSH push. You will also need to have saved a copy of the id-susemanager key.

  • /root/ssl-build/

  • /srv/formula_metadata

  • /srv/pillar

  • /srv/salt

  • /srv/susemanager

  • /srv/tftpboot/

  • /srv/www/cobbler

  • /srv/www/htdocs/pub/

  • /srv/www/os-images

  • /var/cache/rhn

  • /var/cache/salt

  • /var/lib/cobbler/

  • /var/lib/cobbler/templates/ (before version 4.0 it was /var/lib/rhn/kickstarts/)

  • /var/lib/Kiwi

  • /var/lib/rhn/

  • /var/spacewalk/

  • Plus any directories containing custom data such as scripts, Kickstart or AutoYaST profiles, and custom RPMs.

You will also need to back up your database, which you can do with the smdba tool. For more information about the smdba tool, see administration:backup-restore.adoc.
Procedure: Restore from a Manual Backup

  1. Re-install SUSE Manager. For more information about recovering from a backup, see administration:backup-restore.adoc.

  2. Re-synchronize your SUSE Manager repositories with the mgr-sync tool. For more information about the mgr-sync tool, see [syncing.suse.mgr.repositories.scc].

  3. You can choose to re-register your product, or skip the registration and SSL certificate generation sections.

  4. Re-install the /root/ssl-build/rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm package.

  5. Schedule the re-creation of search indexes next time the rhn-search service is started:

    rhn-search cleanindex

    This command produces only debug messages. It does not produce error messages.

  6. Check whether you need to restore /var/spacewalk/packages/. If /var/spacewalk/packages/ was not in your backup, you will need to restore it. If the source repository is available, you can restore /var/spacewalk/packages/ with a complete channel synchronization:

    mgr-sync refresh --refresh-channels

    Check the progress by running tail -f /var/log/rhn/reposync/<CHANNEL_NAME>.log as root.

Administering the Database with smdba

The smdba tool is used for managing a local PostgreSQL database. It allows you to back up and restore your database, and manage backups. It can also be used to check the status of your database, and perform administration tasks, such as restarting.

The smdba tool works with local PostgreSQL databases only, it will not work with remotely accessed databases, or Oracle databases.

The smdba tool requires sudo access, in order to execute system changes. Ensure you have enabled sudo access for the admin user before you begin, by checking the /etc/sudoers file for this line:

admin   ALL=(postgres) /usr/bin/smdba

Check the runtime status of your database with:

smdba db-status

This command will return either online or offline, for example:

Checking database core...       online

Starting and stopping the database can be performed with:

smdba db-start

And:

smdba db-stop

Database Backup with smdba

The smdba tool performs a continuous archiving backup. This backup method combines a log of every change made to the database during the current session, with a series of more traditional backup files. When a crash occurs, the database state is first restored from the most recent backup file on disk, then the log of the current session is replayed exactly, to bring the database back to a current state. A continuous archiving backup with smdba is performed with the database running, so there is no need for downtime.

This method of backing up is stable and generally creates consistent snapshots, however it can take up a lot of storage space. Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.

The smdba tool also manages your archives, keeping only the most recent backup, and the current archive of logs. The log files can only be a maximum file size of 16 MB, so a new log file will be created when the files reach this size. Every time you create a new backup, previous backups will be purged to release disk space. We recommend you use cron to schedule your smdba backups to ensure that your storage is managed effectively, and you always have a backup ready in case of failure.

Performing a Manual Database Backup

The smdba tool can be run directly from the command line. We recommend you run a manual database backup immediately after installation, or if you have made any significant changes to your configuration.

When smdba is run for the first time, or if you have changed the location of the backup, it will need to restart your database before performing the archive. This will result in a small amount of downtime. Regular database backups will not require any downtime.
Procedure: Performing a Manual Database Backup

  1. Allocate permanent storage space for your backup. This example uses a directory located at /var/spacewalk/. This will become a permanent target for your backup, so ensure it will remain accessible by your server at all times.

  2. In your backup location, create a directory for the backup:

    sudo -u postgres mkdir /var/spacewalk/db-backup

    Or, as root:

    install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup

  3. Ensure you have the correct permissions set on the backup location:

    chown postgres:postgres /var/spacewalk/db-backup

  4. To create a backup for the first time, run the smdba backup-hot command with the enable option set. This will create the backup in the specified directory, and, if necessary, restart the database:

    smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup

    This command produces debug messages and finishes sucessfully with the output:

    INFO: Finished

  5. Check that the backup files exist in the /var/spacewalk/db-backup directory, to ensure that your backup has been successful.

Scheduling Automatic Backups

You do not need to shut down your system in order to perform a database backup with smdba. However, because it is a large operation, database performance can slow down while the backup is running. We recommend you schedule regular database backups for a low-traffic period, to minimize disruption.

Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.
Procedure: Scheduling Automatic Backups

  1. Create a directory for the backup, and set the appropriate permissions (as root):

    install -m 700 -o postgres -g postgres /var/spacewalk/db-backup

  2. Open /etc/cron.d/db-backup-mgr, or create it if it does not exist, and add the following line to create the cron job:

    0 2 * * * root /usr/bin/smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup

  3. Check the backup directory regularly to ensure the backups are working as expected.

Restoring from Backup

The smdba tool can be used to restore from backup in the case of failure.

Procedure: Restoring from Backup

  1. Shut down the database:

    smdba db-stop

  2. Start the restore process and wait for it to complete:

    smdba backup-restore start

  3. Restart the database:

    smdba db-start

  4. Check if there are differences between the RPMs and the database.

    spacewalk-data-fsck

Archive Log Settings

Archive logging allows the database management tool smdba to perform hot backups. In SUSE Manager with an embedded database, archive logging is enabled by default.

PostgreSQL maintains a limited number of archive logs. Using the default configuration, approximately 64 files with a size of 16 MiB are stored.

Creating a user and syncing the channels:

  • SLES12-SP2-Pool-x86_64

  • SLES12-SP2-Updates-x86_64

  • SLE-Manager-Tools12-Pool-x86_64-SP2

  • SLE-Manager-Tools12-Updates-x86_64-SP2

PostgreSQL will generate an additional roughly 1 GB of data. So it is important to think about a backup strategy and create a backups in a regular way.

Archive logs are stored at /var/lib/pgsql/data/pg_xlog/ (postgresql).

Retrieving an Overview of Occupied Database Space

Database administrators may use the subcommand space-overview to get a report about occupied table spaces, for example:

smdba space-overview

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH


Tablespace  | Size (Mb) | Avail (Mb) | Use %
------------+-----------+------------+------
postgres    | 7         | 49168      | 0.013
susemanager | 776       | 48399      | 1.602

The smdba command is available for PostgreSQL. For a more detailed report, use the space-tables subcommand. It lists the table and its size, for example:

smdba space-tables

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH


Table                                 | Size
--------------------------------------+-----------
public.all_primary_keys               | 0 bytes
public.all_tab_columns                | 0 bytes
public.allserverkeywordsincereboot    | 0 bytes
public.dblink_pkey_results            | 0 bytes
public.dual                           | 8192 bytes
public.evr_t                          | 0 bytes
public.log                            | 32 kB
...

Moving the Database

It is possible to move the database to another location. For example, move the database if the database storage space is running low. The following procedure will guide you through moving the database to a new location for use by SUSE Manager.

Procedure: Moving the Database

  1. The default storage location for SUSE Manager is /var/lib/pgsql/. If you would like to move it, for example to /storage/postgres/, proceed as follows.

  2. Stop the running database with (as root):

    rcpostgresql stop

    Shut down the running Spacewalk services with:

    spacewalk-service stop

  3. Copy the current working directory structure with cp using the -a, --archive option. For example:

    cp --archive /var/lib/pgsql/ /storage/postgres/

    This command will copy the contents of /var/lib/pgsql/ to /storage/postgres/pgsql/.

    The contents of the /var/lib/pgsql directory needs to remain the same, otherwise the SUSE Manager database may malfunction. You also should ensure that there is enough available disk space.

  4. Mount the new database directory with:

    mount /storage/postgres/pgsql

  5. Make sure ownership is postgres:postgres and not root:root by changing to the new directory and running the following commands:

    cd /storage/postgres/pgsql/
ls -l

    Outputs:

    total 8
drwxr-x---  4 postgres postgres   47 Jun  2 14:35 ./

  6. Add the new database mount location to your servers fstab by editing etc/fstab.

  7. Start the database with:

    rcpostgresql start

  8. Start the Spacewalk services with:

    spacewalk-service start

Recovering from a Crashed Root Partition

This section provides guidance on restoring your server after its root partition has crashed. This section assumes you have setup your server similar to the procedure explained in Installation guide with separate partitions for the database and for channels mounted at /var/lib/pgsql and /var/spacewalk/.

Procedure: Recovering from a Crashed Root Partition

  1. Install SUSE Manager. Do not mount the /var/spacewalk and /var/lib/pgsql partitions. Wait for the installation to complete before going on to the next step.

  2. Shut down the services with spacewalk-service shutdown.

  3. Shut down the database with rcpostgresql stop.

  4. Mount /var/spacewalk and /var/lib/pgsql partitions.

  5. Restore the directories listed in Backing up SUSE Manager.

  6. Start the Spacewalk services with spacewalk-services start.

  7. Start the database with rcpostgresql start.

SUSE Manager should now operate normally without loss of your database or synced channels.

Database Connection Information

The information for connecting to the SUSE Manager database is located in /etc/rhn/rhn.conf:

db_backend = postgresql
db_user = susemanager
db_password = susemanager
db_name = susemanager
db_host = localhost
db_port = 5432
db_ssl_enabled =