Rsync Commands Guide

A comprehensive guide to rsync for backup, synchronization, and file transfer operations.

Table of Contents

• Introduction
• Basic Syntax and Options
• Common Use Cases
• Advanced Configuration
• Performance Optimization
• Security Configuration
• Backup Strategies
• Monitoring and Logging
• Troubleshooting
• Real-World Examples

---

Introduction

Rsync is a powerful utility for efficiently transferring and synchronizing files between systems. It uses delta-transfer algorithm to send only the differences between source and destination files.

Key Features

• Delta synchronization: Only transfers changed portions of files
• Compression: Reduces bandwidth usage
• Preservation: Maintains permissions, timestamps, and ownership
• Resume capability: Can resume interrupted transfers
• Remote synchronization: Works over SSH, rsh, or daemon mode

---

Basic Syntax and Options

General Syntax

rsync [OPTIONS] SOURCE DESTINATION

Essential Options

Option	Description	Example
-a, --archive	Archive mode (recursive, preserve everything)	rsync -a src/ dest/
-v, --verbose	Verbose output	rsync -av src/ dest/
-r, --recursive	Recurse into directories	rsync -r src/ dest/
-u, --update	Skip files newer on destination	rsync -au src/ dest/
-z, --compress	Compress during transfer	rsync -az src/ dest/
-h, --human-readable	Human readable output	rsync -avh src/ dest/
-P, --partial --progress	Show progress, keep partial files	rsync -avP src/ dest/
-n, --dry-run	Show what would be done	rsync -avn src/ dest/
--delete	Delete files not in source	rsync -av --delete src/ dest/
--exclude	Exclude files/directories	rsync -av --exclude='*.tmp' src/ dest/

Archive Mode Breakdown

Archive mode (-a) is equivalent to: -rlptgoD

• -r: recursive
• -l: copy symlinks as symlinks
• -p: preserve permissions
• -t: preserve modification times
• -g: preserve group
• -o: preserve owner
• -D: preserve device files and special files

---

Common Use Cases

1. Local Directory Synchronization

# Basic local sync
rsync -av /source/directory/ /destination/directory/

# Sync with deletion (mirror)
rsync -av --delete /source/ /destination/

# Exclude specific files
rsync -av --exclude='*.log' --exclude='tmp/' /source/ /destination/

# Dry run to test
rsync -avn --delete /source/ /destination/

2. Remote Synchronization via SSH

# Push to remote server
rsync -avz /local/path/ user@remote-server:/remote/path/

# Pull from remote server
rsync -avz user@remote-server:/remote/path/ /local/path/

# Using specific SSH port
rsync -avz -e "ssh -p 2222" /local/path/ user@server:/remote/path/

# Using SSH key
rsync -avz -e "ssh -i /path/to/key" /local/path/ user@server:/remote/path/

3. Bandwidth Limited Transfer

# Limit bandwidth to 1000 KB/s
rsync -av --bwlimit=1000 /source/ user@server:/destination/

# Limit with compression
rsync -avz --bwlimit=500 /source/ user@server:/destination/

---

Advanced Configuration

Configuration File Setup

Create /etc/rsyncd.conf for daemon mode:

# Global settings
uid = rsync
gid = rsync
use chroot = yes
max connections = 4
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
syslog facility = daemon

# Module definitions
[backup]
path = /backup
comment = Backup directory
read only = no
list = yes
uid = backup
gid = backup
auth users = backupuser
secrets file = /etc/rsyncd.secrets
hosts allow = 192.168.1.0/24

[logs]
path = /var/log
comment = Log files
read only = yes
list = yes
uid = nobody
gid = nobody
auth users = loguser
secrets file = /etc/rsyncd.secrets

Secrets File (/etc/rsyncd.secrets)

# Format: username:password
backupuser:SecurePassword123
loguser:AnotherPassword456

Set proper permissions:

chmod 600 /etc/rsyncd.secrets
chown root:root /etc/rsyncd.secrets

Starting Rsync Daemon

# Start daemon
rsync --daemon

# Start with specific config
rsync --daemon --config=/etc/rsyncd.conf

# SystemD service
systemctl enable rsync
systemctl start rsync

---

Performance Optimization

1. Compression Settings

# Default compression (level 6)
rsync -az source/ dest/

# No compression (faster for LAN)
rsync -av source/ dest/

# Maximum compression (slower, saves bandwidth)
rsync -av --compress-level=9 source/ dest/

# Skip compression for already compressed files
rsync -av --skip-compress=gz/jpg/mp[34]/7z/bz2 source/ dest/

2. Checksum Options

# Use checksums instead of mod-time & size
rsync -avc source/ dest/

# Whole file transfer (no delta-transfer)
rsync -avW source/ dest/

# Inplace updates (saves space)
rsync -av --inplace source/ dest/

3. Parallel Processing

# Using GNU parallel for multiple directories
parallel -j4 rsync -av {} user@server:/destination/ ::: dir1/ dir2/ dir3/ dir4/

# Multiple rsync processes
rsync -av dir1/ user@server:/dest/dir1/ &
rsync -av dir2/ user@server:/dest/dir2/ &
rsync -av dir3/ user@server:/dest/dir3/ &
wait

---

Security Configuration

1. SSH Key Authentication

# Generate SSH key pair
ssh-keygen -t rsa -b 4096 -f ~/.ssh/rsync_key

# Copy public key to remote server
ssh-copy-id -i ~/.ssh/rsync_key.pub user@server

# Use specific key for rsync
rsync -av -e "ssh -i ~/.ssh/rsync_key" source/ user@server:/dest/

2. Restricted SSH Commands

Create /home/backupuser/.ssh/authorized_keys:

command="rsync --server -vlogDtpre.iLs . /backup/",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2E...

3. Firewall Configuration

# Allow rsync daemon (port 873)
iptables -A INPUT -p tcp --dport 873 -s 192.168.1.0/24 -j ACCEPT

# For SSH-based rsync (port 22)
iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT

---

Backup Strategies

1. Incremental Backup with Hardlinks

#!/bin/bash
# incremental-backup.sh

BACKUP_ROOT="/backup"
SOURCE="/data"
DATE=$(date +%Y%m%d_%H%M%S)
CURRENT="$BACKUP_ROOT/current"
BACKUP_DIR="$BACKUP_ROOT/backup_$DATE"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Perform rsync with hardlinks to previous backup
if [ -d "$CURRENT" ]; then
    rsync -av --delete --link-dest="$CURRENT" "$SOURCE/" "$BACKUP_DIR/"
else
    rsync -av --delete "$SOURCE/" "$BACKUP_DIR/"
fi

# Update current symlink
rm -f "$CURRENT"
ln -s "$BACKUP_DIR" "$CURRENT"

echo "Backup completed: $BACKUP_DIR"

2. Rotating Backups

#!/bin/bash
# rotating-backup.sh

BACKUP_ROOT="/backup"
SOURCE="/data"
HOSTNAME=$(hostname)
DATE=$(date +%Y%m%d)

# Create daily backup
rsync -av --delete "$SOURCE/" "$BACKUP_ROOT/daily/$HOSTNAME-$DATE/"

# Weekly backup (Sunday)
if [ $(date +%u) -eq 7 ]; then
    rsync -av --delete "$SOURCE/" "$BACKUP_ROOT/weekly/$HOSTNAME-$DATE/"
fi

# Monthly backup (1st day of month)
if [ $(date +%d) -eq 01 ]; then
    rsync -av --delete "$SOURCE/" "$BACKUP_ROOT/monthly/$HOSTNAME-$DATE/"
fi

# Cleanup old backups
find "$BACKUP_ROOT/daily" -type d -mtime +7 -delete
find "$BACKUP_ROOT/weekly" -type d -mtime +28 -delete
find "$BACKUP_ROOT/monthly" -type d -mtime +365 -delete

3. Database Backup Integration

#!/bin/bash
# db-backup-rsync.sh

DB_NAME="myapp"
BACKUP_DIR="/tmp/db_backup"
REMOTE_SERVER="backup.example.com"
REMOTE_PATH="/backups/database"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Dump database
mysqldump -u root -p"$DB_PASS" "$DB_NAME" > "$BACKUP_DIR/${DB_NAME}_$(date +%Y%m%d).sql"

# Compress backup
gzip "$BACKUP_DIR/${DB_NAME}_$(date +%Y%m%d).sql"

# Sync to remote server
rsync -avz --delete "$BACKUP_DIR/" "user@$REMOTE_SERVER:$REMOTE_PATH/"

# Cleanup local backup
rm -rf "$BACKUP_DIR"

---

Monitoring and Logging

1. Progress Monitoring

# Real-time progress
rsync -av --progress source/ dest/

# Progress with statistics
rsync -av --progress --stats source/ dest/

# Itemized changes
rsync -av --itemize-changes source/ dest/

# Verbose with timestamps
rsync -av --verbose --log-file=/var/log/rsync.log source/ dest/

2. Custom Progress Script

#!/bin/bash
# rsync-with-progress.sh

SOURCE="$1"
DEST="$2"
LOGFILE="/var/log/rsync-$(date +%Y%m%d_%H%M%S).log"

# Function to log with timestamp
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOGFILE"
}

log "Starting sync from $SOURCE to $DEST"

# Run rsync with progress
rsync -av --progress --stats \
    --log-file="$LOGFILE" \
    "$SOURCE" "$DEST" 2>&1 | while read line; do
    log "$line"
done

log "Sync completed"

3. Email Notifications

#!/bin/bash
# rsync-with-notification.sh

SOURCE="$1"
DEST="$2"
EMAIL="admin@example.com"
LOGFILE="/tmp/rsync.log"

# Run rsync
if rsync -av --delete --stats "$SOURCE" "$DEST" > "$LOGFILE" 2>&1; then
    # Success
    SUBJECT="Rsync Success: $SOURCE -> $DEST"
    mail -s "$SUBJECT" "$EMAIL" < "$LOGFILE"
else
    # Failure
    SUBJECT="Rsync Failed: $SOURCE -> $DEST"
    mail -s "$SUBJECT" "$EMAIL" < "$LOGFILE"
fi

---

Troubleshooting

Common Issues and Solutions

1. Permission Denied

# Problem: Permission denied errors
# Solution: Check ownership and permissions

# Fix ownership
chown -R user:group /destination/

# Fix permissions
chmod -R 755 /destination/

# Use sudo for rsync
sudo rsync -av source/ dest/

2. SSH Connection Issues

# Test SSH connection first
ssh -v user@server

# Debug rsync SSH
rsync -av -e "ssh -v" source/ user@server:/dest/

# Use different SSH port
rsync -av -e "ssh -p 2222" source/ user@server:/dest/

3. Disk Space Issues

# Check available space
df -h /destination/

# Use --max-size to skip large files
rsync -av --max-size=100M source/ dest/

# Use --partial-dir for partial transfers
rsync -av --partial-dir=/tmp/rsync-partial source/ dest/

4. Network Timeouts

# Increase timeout
rsync -av --timeout=300 source/ user@server:/dest/

# Use contimeout for connection timeout
rsync -av --contimeout=60 source/ user@server:/dest/

Debug Mode

# Verbose debug output
rsync -avvv source/ dest/

# Debug with specific modules
rsync -av --debug=ALL source/ dest/

# Log to file
rsync -av --log-file=/var/log/rsync-debug.log source/ dest/

---

Real-World Examples

1. Web Server Content Sync

#!/bin/bash
# web-content-sync.sh

SOURCE="/var/www/html/"
SERVERS=("web1.example.com" "web2.example.com" "web3.example.com")
USER="deploy"
EXCLUDE_FILE="/etc/rsync-web-exclude.txt"

# Exclude file contents
cat > "$EXCLUDE_FILE" << EOF
*.log
cache/
sessions/
uploads/tmp/
.git/
EOF

for server in "${SERVERS[@]}"; do
    echo "Syncing to $server..."
    rsync -avz --delete \
        --exclude-from="$EXCLUDE_FILE" \
        "$SOURCE" "$USER@$server:/var/www/html/"
    
    if [ $? -eq 0 ]; then
        echo "✓ Success: $server"
    else
        echo "✗ Failed: $server"
    fi
done

2. Log File Aggregation

#!/bin/bash
# log-aggregation.sh

SERVERS=("app1" "app2" "app3" "db1")
LOG_CENTRAL="/var/log/central"
DATE=$(date +%Y%m%d)

for server in "${SERVERS[@]}"; do
    mkdir -p "$LOG_CENTRAL/$server/$DATE"
    
    # Sync logs from each server
    rsync -av --include="*.log" --include="*/" --exclude="*" \
        "root@$server:/var/log/" \
        "$LOG_CENTRAL/$server/$DATE/"
done

# Compress old logs
find "$LOG_CENTRAL" -name "*.log" -mtime +1 -exec gzip {} \;

3. Docker Registry Sync

#!/bin/bash
# docker-registry-sync.sh

PRIMARY_REGISTRY="/var/lib/docker/registry"
MIRROR_SERVERS=("mirror1.example.com" "mirror2.example.com")
USER="registry"

for mirror in "${MIRROR_SERVERS[@]}"; do
    echo "Syncing registry to $mirror..."
    
    # Stop registry service on mirror
    ssh "$USER@$mirror" "systemctl stop docker-registry"
    
    # Sync registry data
    rsync -av --delete \
        "$PRIMARY_REGISTRY/" \
        "$USER@$mirror:/var/lib/docker/registry/"
    
    # Start registry service on mirror
    ssh "$USER@$mirror" "systemctl start docker-registry"
    
    echo "Registry sync to $mirror completed"
done

4. Database Backup to Multiple Locations

#!/bin/bash
# multi-location-db-backup.sh

DB_NAME="production"
BACKUP_DIR="/tmp/db_backup"
LOCATIONS=(
    "backup1.example.com:/backups/db"
    "backup2.example.com:/backups/db"
    "s3://my-backup-bucket/database"
)

# Create backup
mkdir -p "$BACKUP_DIR"
pg_dump "$DB_NAME" | gzip > "$BACKUP_DIR/${DB_NAME}_$(date +%Y%m%d_%H%M%S).sql.gz"

# Sync to multiple locations
for location in "${LOCATIONS[@]}"; do
    if [[ "$location" == s3://* ]]; then
        # S3 sync (requires AWS CLI)
        aws s3 sync "$BACKUP_DIR/" "$location/"
    else
        # Regular rsync
        rsync -av "$BACKUP_DIR/" "$location/"
    fi
done

# Cleanup local backup
rm -rf "$BACKUP_DIR"

---

Performance Benchmarking

Speed Test Script

#!/bin/bash
# rsync-benchmark.sh

SOURCE="/test/data"
DEST="/test/backup"
SIZE="1G"

# Create test data
mkdir -p "$SOURCE"
dd if=/dev/zero of="$SOURCE/testfile" bs=1M count=1024

echo "Testing different rsync configurations..."

# Test 1: Basic sync
echo "Test 1: Basic sync"
time rsync -av "$SOURCE/" "$DEST-basic/"

# Test 2: With compression
echo "Test 2: With compression"
time rsync -avz "$SOURCE/" "$DEST-compressed/"

# Test 3: Whole file transfer
echo "Test 3: Whole file transfer"
time rsync -avW "$SOURCE/" "$DEST-wholefile/"

# Test 4: In-place updates
echo "Test 4: In-place updates"
time rsync -av --inplace "$SOURCE/" "$DEST-inplace/"

# Cleanup
rm -rf "$SOURCE" "$DEST"-*

---

Best Practices Summary

DO's

• ✅ Always test with --dry-run first
• ✅ Use archive mode (-a) for complete sync
• ✅ Include progress (-P) for long transfers
• ✅ Use compression (-z) over slow networks
• ✅ Set up proper SSH key authentication
• ✅ Monitor and log rsync operations
• ✅ Use exclude files for unwanted content
• ✅ Implement proper backup rotation

DON'Ts

• ❌ Don't forget trailing slashes (affects behavior)
• ❌ Don't run rsync as root unless necessary
• ❌ Don't sync running databases without proper dumps
• ❌ Don't ignore rsync exit codes in scripts
• ❌ Don't use --delete without understanding consequences
• ❌ Don't sync system directories carelessly
• ❌ Don't forget to test restore procedures

---

Quick Reference Commands

# Basic sync with archive mode
rsync -av source/ destination/

# Remote sync with compression and progress
rsync -avzP source/ user@server:/path/

# Mirror with deletion
rsync -av --delete source/ destination/

# Exclude files and show what would be done
rsync -avn --exclude='*.tmp' --delete source/ dest/

# Bandwidth limited sync
rsync -avz --bwlimit=1000 source/ user@server:/path/

# Resume interrupted transfer
rsync -avP --partial source/ destination/

# Sync only newer files
rsync -avu source/ destination/

# Itemized output showing changes
rsync -av --itemize-changes source/ destination/

---