welcome: please sign in
location: Hips

Hips backup server

Restoring a backup

  1. Visit ribs. Backups are in /backups/ribs or in /backups/backup (The former is the newer/transitionary place.)

    • Note that ./current contains a full copy of everything that we copy from a system. It is equivalent to daily.0.

    • Read-Only access to the above is provided via CIFS at \\hips.morningside.edu\ribs-zfs and \\hips.morningside.edu\ribs.

  2. Directories of the form daily.x symbolize the age of a backup: x days old.

    • Typically, we do "incremental" backups. This means that what displays in daily.x is only what was changed or deleted on that backup cycle.

  3. Copy a file or directory to its original place on the server that it came from.
    • Note: If you are copying using scp, UNIX permissions are important! For that reason, it may be advisable to use more conservative methods for replacement (for example: email the file for the user to replace, or drop it in their S-Drive for them, etc.)

Fixing samba/winbind connections

  1. Usually a restart of the winbind and samba services is enough.
    •    [root@hips ~]$ svcadm restart samba
         [root@hips ~]$ svcadm restart winbind
  2. If restarting the services does not work, removing and re-adding hips to the domain may be necessary.
    •    [root@hips ~]$ net ads leave -U hasbeenadmin
         hasbeenadmin's password: 
         Deleted account for 'HIPS' in realm 'MUSTANG.MORNINGSIDE.EDU'
         [root@hips ~]$ net ads join -U hasbeenadmin
         hasbeenadmin's password: 
         Using short domain name -- MUSTANG
         Joined 'HIPS' to realm 'MUSTANG.MORNINGSIDE.EDU'
         [root@hips ~]$ svcadm restart samba
         [root@hips ~]$ svcadm restart winbind

Replacing a failed hard drive

Suppose we have a failed drive, c2t6d0:

[root@hips ~]$ zpool status backups
  pool: backups
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: resilver completed after 157h56m with 0 errors on Sun Mar 29 11:04:29 2015
config:

        NAME           STATE     READ WRITE CKSUM
        backups        DEGRADED     0     0     0
          raidz2       ONLINE       0     0     0
            c2t2d0     ONLINE       0     0     0
            c2t9d0     ONLINE       0     0     0
            c2t10d0    ONLINE       0     0     0
            c2t3d0     ONLINE       0     0     0
            c2t8d0     ONLINE       0     0     0
            c2t4d0     ONLINE       0     0     0
            c2t11d0    ONLINE       0     0     0
          raidz2       DEGRADED     0     0     0
            c2t1d0     ONLINE       0     0     0
            c2t13d0    ONLINE       0     0     0
            c2t5d0     ONLINE       0     0     0
            spare      DEGRADED     0     0 85.9M
              c2t6d0   UNAVAIL      3     4     0  cannot open
              c2t12d0  ONLINE       0     0     0  1.65T resilvered
            c2t7d0     ONLINE       0     0     0
            c2t14d0    ONLINE       0     0     0
            c2t15d0    ONLINE       0     0     0
        spares
          c2t12d0      INUSE     currently in use

errors: No known data errors

Since the pool has already installed the hot spare, lets leave it in there permanently and replace the hot spare to save another re-silver.

Remove the failed drive from the zpool

First, we must remove the failed disk from the pool.

[root@hips ~]$ zpool detach backups c2t6d0

Unconfigure & remove the failed disk from the system

The current status is given by cfgadm:

[root@hips ~]$ cfgadm -a | grep c2t6d0
c2::dsk/c2t6d0                 disk         connected    configured   unknown

To remove it, use the unconfigure directive:

[root@hips ~]$ cfgadm -c unconfigure c2::dsk/c2t6d0

Verify that the disk is connected (but unconfigured):

[root@hips ~]$ cfgadm -a | grep c2t6d0
c2::dsk/c2t6d0                 disk         connected    unconfigured unknown

You may now physically remove the disk.

Add the disk as a hot spare to the zpool.

Verify that the new disk has been "configured" using the cfgadm command above.

zpool add backups spare c2t6d0

(This can take a little while.)

Verify that the zpool is back to normal.

[root@hips ~]$ zpool status backups
  pool: backups
 state: ONLINE
 scrub: resilver completed after 157h56m with 0 errors on Sun Mar 29 11:04:29 2015
config:

        NAME         STATE     READ WRITE CKSUM
        backups      ONLINE       0     0     0
          raidz2     ONLINE       0     0     0
            c2t2d0   ONLINE       0     0     0
            c2t9d0   ONLINE       0     0     0
            c2t10d0  ONLINE       0     0     0
            c2t3d0   ONLINE       0     0     0
            c2t8d0   ONLINE       0     0     0
            c2t4d0   ONLINE       0     0     0
            c2t11d0  ONLINE       0     0     0
          raidz2     ONLINE       0     0     0
            c2t1d0   ONLINE       0     0     0
            c2t13d0  ONLINE       0     0     0
            c2t5d0   ONLINE       0     0     0
            c2t12d0  ONLINE       0     0     0  1.65T resilvered
            c2t7d0   ONLINE       0     0     0
            c2t14d0  ONLINE       0     0     0
            c2t15d0  ONLINE       0     0     0
        spares
          c2t6d0     AVAIL   

errors: No known data errors

RIBS - Rsync Incremental Backup System

Source: http://www.mikerubel.org/computers/rsync_snapshots/

# Backup all the servers every night
18 1 * * * /root/ribs ALL daily

Ribs maintains a folder structure under /home/backup/(server-name)/ starting with a full daily ./daily.0 and differentials in daily.''n'' where ''n'' is days ago. The contents of these subdirectories will be the files that changed on that day - either deleted, created, or modified.

WIBS - (Windows) Incremental Backup System (Deprecated)

16 1 * * * /usr/local/bin/wibs-wrapper > /dev/null

Wibs is a (slightly) modified version of RIBS. To circumvent the limitations of windows (no SSH or rsync clients) hips keeps the \\(server-name)\c$ mounted under /smb/(server-name) with credentials saved in /root/backup.creds.

A wrapper script, wibs-wrapper exists to check that /smb/ mounts are properly mounted before backing up so that any dropped mappings will not damage our backups.

Bak-to-Nas (Deprecated?)

# Sync to the nas box every morning.
05 6 * * * /root/bak_to_nas /root/bak_to_nas_list > /dev/null
05 6 * * * /root/wibs_to_nas /root/wibs_to_nas_list > /dev/null

*_to_nas merely tar's and copies the daily backups to the NAS boxen.

SpaceWalk

See SpaceWalk

HIPs Solaris 10 Configurations

Shaun Meyer, June 2010

dns: hips.morningside.edu ( ip in NetworkRecords)

Installation

Full Install + OEM Support /var as a separate dataset.

`/etc/passwd:`

    root:x:0:0:Super-User:/root/:/usr/bin/bash

Root home changed to /root Shell changed to /usr/bin/bash

Crontab

    # HIPS / RIBS
    18 2 * * * /root/ribs ALL daily
    18 2 * * * /usr/local/bin/backup-dvserver

Rsync: Installed from src (/usr/local/src/rsync-*)

/usr/bin/backup-dvserver:

Modified to check exit code with -z instead of -n since a bug in our version of smbget reports inverse return code.

Additional configuration:

Installed xterm-color file according to mcwiki2 solaris 10 documentation to allow proper handling of xterm-color ${TERM} type.

Updates:

/usr/bin/updatemanager may be run remotely over SSH to configure for Oracle/Sun Solaris support code

Installed webstack for php.


CategoryServers CategoryDocumentation

Hips (last edited 2015-11-04 08:54:52 by colej)