Red Hat Enterprise Linux Guide
Table of Contents
- System Configuration
- Performance Tuning
- Network Configuration
- Storage Management
- Security Configuration
- Service Management
- Troubleshooting Methodologies
- Common Issues and Solutions
- Monitoring and Logging
- Advanced Troubleshooting
System Configuration
Initial System Setup
Red Hat Enterprise Linux requires proper initial configuration to ensure optimal performance and security. The primary configuration files are located in /etc/ directory and should be modified with appropriate tools and editors.
Hostname Configuration
Configure the system hostname using the hostnamectl command, which is part of systemd. This tool provides a unified interface for setting both static and transient hostnames.
# Set static hostname
hostnamectl set-hostname production-server.example.com
# Verify hostname configuration
hostnamectl status
# Set pretty hostname for display purposes
hostnamectl set-hostname "Production Web Server" --pretty
Time and Date Configuration
Proper time synchronization is critical for log correlation, authentication, and distributed systems. Red Hat uses chronyd as the default NTP client.
# Configure timezone
timedatectl set-timezone America/New_York
# Enable NTP synchronization
timedatectl set-ntp true
# Check time synchronization status
chrony sources -v
# Manual chrony configuration in /etc/chrony.conf
server 0.rhel.pool.ntp.org iburst
server 1.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
Locale and Keyboard Configuration
System locale affects date formats, number representations, and character encoding throughout the system.
# List available locales
localectl list-locales
# Set system locale
localectl set-locale LANG=en_US.UTF-8
# Configure keyboard layout
localectl set-keymap us
# Verify current settings
localectl status
Boot Configuration
GRUB Boot Issues
GRUB-related boot problems are common and can often be resolved through rescue modes or emergency procedures.
GRUB Not Found or Corrupted
When GRUB is missing or corrupted, the system cannot locate the bootloader. This typically occurs after disk operations or filesystem corruption.
# Boot from rescue media and chroot into system
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
chroot /mnt
# Reinstall GRUB
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg
# For UEFI systems
grub2-install --target=x86_64-efi --efi-directory=/boot/efi
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
Kernel Boot Parameters Issues
Incorrect kernel parameters can prevent successful boot. These can be edited temporarily at boot time or permanently in GRUB configuration.
# Temporary kernel parameter editing at GRUB menu
# Press 'e' at GRUB menu, edit linux line, press Ctrl+X to boot
# Common rescue parameters
linux ... single
linux ... init=/bin/bash
linux ... rd.break enforcing=0
# Permanent parameter changes
vim /etc/default/grub
# Modify GRUB_CMDLINE_LINUX
grub2-mkconfig -o /boot/grub2/grub.cfg
Filesystem Boot Issues
Filesystem corruption or misconfiguration can prevent successful system startup.
Root Filesystem Corruption
When the root filesystem is corrupted, the system cannot mount essential directories.
# Boot into rescue mode
# At GRUB menu, select rescue option or add rd.break to kernel line
# Check filesystem integrity
fsck -f /dev/sda2
# For ext4 filesystems
e2fsck -f -y /dev/sda2
# For XFS filesystems
xfs_repair /dev/sda2
# Mount filesystem read-only if fsck fails
mount -o ro /dev/sda2 /mnt
# Copy critical data before attempting repair
Fstab Configuration Errors
Incorrect /etc/fstab entries can prevent boot completion.
# Boot with rd.break parameter
mount -o remount,rw /sysroot
chroot /sysroot
# Check fstab syntax
cat /etc/fstab
# Comment out problematic entries
sed -i 's/^\/dev\/problem/#\/dev\/problem/' /etc/fstab
# Verify mount points exist
mkdir -p /mount/point
# Test fstab entries
mount -a
Network Connectivity Issues
Network problems can affect system functionality and require systematic diagnosis.
Interface Configuration Problems
Network interface issues often stem from configuration errors or hardware problems.
Interface Not Coming Up
When network interfaces fail to activate, check configuration and hardware status.
# Check interface status
ip link show
nmcli device status
# Bring interface up manually
ip link set eth0 up
nmcli device connect eth0
# Check for hardware issues
dmesg | grep -i eth0
ethtool eth0
# Verify cable connectivity
ethtool eth0 | grep "Link detected"
# Reset NetworkManager
systemctl restart NetworkManager
nmcli general reload
IP Configuration Issues
Incorrect IP addressing can prevent network communication.
# Check current IP configuration
ip addr show
nmcli connection show --active
# Static IP configuration troubleshooting
nmcli connection modify "System eth0" ipv4.addresses 192.168.1.100/24
nmcli connection modify "System eth0" ipv4.gateway 192.168.1.1
nmcli connection modify "System eth0" ipv4.dns 8.8.8.8
nmcli connection modify "System eth0" ipv4.method manual
# DHCP troubleshooting
dhclient -v eth0
nmcli connection modify "System eth0" ipv4.method auto
DNS Resolution Problems
DNS issues prevent hostname resolution and can affect application functionality.
DNS Server Configuration
Incorrect DNS server configuration prevents name resolution.
# Check DNS configuration
cat /etc/resolv.conf
nmcli device show eth0 | grep IP4.DNS
# Test DNS resolution
nslookup google.com
dig google.com
host google.com
# Configure DNS servers
nmcli connection modify "System eth0" ipv4.dns "8.8.8.8,8.8.4.4"
nmcli connection up "System eth0"
# Flush DNS cache
systemctl restart systemd-resolved
systemctl flush-dns # If available
DNS Service Issues
Local DNS services may fail or become misconfigured.
# Check local DNS service (if running bind)
systemctl status named
journalctl -u named
# Validate DNS zone files
named-checkzone example.com /var/named/example.com.zone
named-checkconf /etc/named.conf
# Test DNS server functionality
dig @localhost example.com
Storage and Filesystem Issues
Storage problems can cause data loss and system instability, requiring careful diagnosis and resolution.
Disk Space Issues
Insufficient disk space can cause application failures and system instability.
Identifying Space Usage
Understanding where disk space is consumed helps target cleanup efforts.
# Overall disk usage
df -h
df -i # Inode usage
# Directory space usage
du -sh /* | sort -hr
du -sh /var/* | sort -hr
# Find large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
find /var/log -name "*.log" -size +50M
# Check deleted but open files
lsof +L1
# Find files by age
find /tmp -type f -mtime +7 # Files older than 7 days
find /var/log -name "*.log" -mtime +30
Space Cleanup Procedures
Systematic cleanup helps reclaim disk space safely.
# Log file cleanup
journalctl --vacuum-time=7d
journalctl --vacuum-size=100M
logrotate -f /etc/logrotate.conf
# Package manager cleanup
yum clean all
dnf clean all
package-cleanup --oldkernels --count=2
# Temporary file cleanup
find /tmp -type f -atime +7 -delete
find /var/tmp -type f -atime +7 -delete
# Application-specific cleanup
# Database log cleanup
# Web server log rotation
# Application cache cleanup
Filesystem Corruption
Filesystem corruption requires careful analysis and repair procedures.
Detecting Corruption
Early detection of filesystem corruption can prevent data loss.
# Check filesystem health
fsck -n /dev/sda1 # Read-only check
tune2fs -l /dev/sda1 | grep -i error
# XFS filesystem check
xfs_info /dev/sda1
xfs_db -r /dev/sda1 # Read-only examination
# Monitor for I/O errors
dmesg | grep -i "i/o error"
grep -i "error" /var/log/messages
# SMART disk health
smartctl -a /dev/sda
smartctl -t short /dev/sda # Short self-test
Filesystem Repair
Repair procedures vary by filesystem type and corruption severity.
# Ext4 filesystem repair
umount /mount/point
e2fsck -f /dev/sda1
e2fsck -p /dev/sda1 # Automatic repair
# XFS filesystem repair
umount /mount/point
xfs_repair /dev/sda1
xfs_repair -n /dev/sda1 # No-modify mode
# LVM snapshot for safe repair
lvcreate -L 1G -s -n repair_snap /dev/vg/lv
# Perform repair on snapshot if successful
lvremove /dev/vg/repair_snap
Service and Application Issues
Application and service problems require understanding of service dependencies and configuration.
Service Start Failures
Services may fail to start due to configuration errors, resource constraints, or dependency issues.
Analyzing Service Failures
Systematic analysis helps identify the root cause of service failures.
# Service status analysis
systemctl status httpd
systemctl is-failed httpd
journalctl -u httpd -n 50
# Dependency analysis
systemctl list-dependencies httpd
systemctl list-dependencies --reverse httpd
# Configuration testing
httpd -t # Apache configuration test
nginx -t # Nginx configuration test
named-checkconf # BIND configuration test
# Resource constraint analysis
systemctl show httpd | grep -i limit
ulimit -a
cat /proc/PID/limits
Common Service Issues
Many service issues follow predictable patterns and have standard solutions.
Permission Issues
Incorrect file permissions can prevent service startup.
# Check service file permissions
ls -la /etc/systemd/system/myapp.service
ls -la /usr/lib/systemd/system/httpd.service
# Application file permissions
ls -la /var/www/html/
ls -ld /var/log/httpd/
# SELinux context issues
ls -Z /etc/httpd/conf/httpd.conf
restorecon -R /etc/httpd/
sealert -a /var/log/audit/audit.log
Port Conflicts
Services may fail if required ports are already in use.
# Check port usage
netstat -tulpn | grep :80
ss -tulpn | grep :80
lsof -i :80
# Kill process using port
fuser -k 80/tcp
# Check firewall blocking
firewall-cmd --list-all
iptables -L -n
Performance Issues
Performance problems require systematic analysis to identify bottlenecks and optimize system resources.
CPU Performance Issues
High CPU usage can affect system responsiveness and application performance.
Identifying CPU Bottlenecks
Understanding CPU usage patterns helps target optimization efforts.
# Real-time CPU monitoring
top -o %CPU
htop
atop 1 10
# CPU usage by process
ps aux --sort=-%cpu | head -20
pidstat -u 1 10
# System-wide CPU statistics
sar -u 1 10
mpstat -P ALL 1 10
# CPU context switches and interrupts
sar -w 1 10 # Context switches
cat /proc/interrupts
watch -n 1 cat /proc/interrupts
CPU Optimization Strategies
Various techniques can help optimize CPU performance.
# Process priority adjustment
nice -n -5 ./cpu-intensive-task
renice -10 -p PID
# CPU affinity setting
taskset -c 0,1 ./application
taskset -p 0x3 PID
# CPU governor optimization
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cpupower frequency-set -g performance
# Disable unnecessary services
systemctl disable unnecessary-service
systemctl mask unnecessary-service
Memory Performance Issues
Memory problems can cause swapping, application crashes, and system instability.
Memory Usage Analysis
Understanding memory consumption patterns helps identify issues.
# Memory usage overview
free -h
cat /proc/meminfo
# Process memory usage
ps aux --sort=-%mem | head -20
pmap -x PID
cat /proc/PID/smaps
# Memory allocation tracking
slabtop
cat /proc/slabinfo
# Swap usage analysis
swapon -s
cat /proc/swaps
Memory Optimization
Memory optimization involves tuning virtual memory parameters and managing application memory usage.
# Virtual memory tuning
echo 10 > /proc/sys/vm/swappiness
echo 1 > /proc/sys/vm/overcommit_memory
# Memory cleanup
echo 3 > /proc/sys/vm/drop_caches # Drop caches
sync && echo 1 > /proc/sys/vm/drop_caches # Page cache only
# Application memory limits
systemctl edit myapp
# Add:
[Service]
MemoryLimit=512M
MemoryAccounting=yes
# Huge pages configuration for databases
echo 1024 > /proc/sys/vm/nr_hugepages
hugeadm --pool-list
Monitoring and Logging
System Monitoring Implementation
Effective monitoring provides early warning of system issues and helps maintain optimal performance.
Monitoring Strategy
A comprehensive monitoring strategy covers all critical system components and provides actionable alerts.
Establish baseline metrics for normal system operation. This includes CPU usage, memory consumption, disk I/O, network traffic, and application-specific metrics. Baseline data helps identify anomalies and performance degradation.
Implement multi-layered monitoring that covers infrastructure, operating system, services, and applications. Each layer provides different insights and helps pinpoint issues at the appropriate level.
Define meaningful alerting thresholds based on business impact rather than arbitrary values. Consider alert fatigue and ensure that alerts require action rather than just notification.
Create monitoring dashboards that provide both high-level overviews and detailed drill-down capabilities. Visual representation of metrics helps identify trends and correlations.
Built-in Monitoring Tools
Red Hat Enterprise Linux provides numerous built-in tools for system monitoring and analysis.
System Activity Reporting (SAR)
SAR provides comprehensive system performance data collection and reporting.
# Enable SAR data collection
systemctl enable sysstat
systemctl start sysstat
# CPU utilization reporting
sar -u 1 10 # Every second for 10 intervals
sar -u -f /var/log/sa/sa01 # Historical data
# Memory usage reporting
sar -r 1 10 # Memory and swap usage
sar -R 1 10 # Memory statistics
# I/O activity reporting
sar -d 1 10 # Disk activity
sar -b 1 10 # I/O transfer rates
# Network activity reporting
sar -n DEV 1 10 # Network interface statistics
sar -n EDEV 1 10 # Network errors
# Generate daily reports
sar -A -f /var/log/sa/sa01 > daily_report.txt
Process and System Monitoring
Various tools provide real-time and historical process monitoring.
# Real-time process monitoring
top -d 1 -p PID1,PID2,PID3
htop -p PID1,PID2,PID3
# Process tree visualization
pstree -p
ps axjf
# System resource monitoring
vmstat 1 10 # Virtual memory statistics
iostat -x 1 10 # I/O statistics
netstat -s # Network statistics
# Continuous monitoring with watch
watch -n 1 'ps aux --sort=-%cpu | head -10'
watch -n 1 'free -h && df -h'
Log Management
Effective log management ensures that diagnostic information is available when needed while managing storage requirements.
Centralized Logging
Centralized logging aggregates logs from multiple systems for analysis and correlation.
# Rsyslog centralized logging configuration
# On log server (/etc/rsyslog.conf):
$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 0.0.0.0
# Template for organizing logs
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
# On client systems:
*.* @logserver.example.com:514
# Systemd journal forwarding
journalctl -o json | logger -t systemd-journal
Log Rotation and Retention
Proper log rotation prevents disk space exhaustion while maintaining historical data.
# Logrotate configuration example (/etc/logrotate.d/myapp)
/var/log/myapp/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
create 0644 myapp myapp
postrotate
/bin/systemctl reload myapp > /dev/null 2>&1 || true
endscript
}
# Test logrotate configuration
logrotate -d /etc/logrotate.d/myapp
logrotate -f /etc/logrotate.d/myapp
# Journal size management
journalctl --vacuum-time=30d
journalctl --vacuum-size=1G
# Configure journal retention in /etc/systemd/journald.conf
MaxRetentionSec=30day
SystemMaxUse=1G
Advanced Monitoring Techniques
Complex environments require sophisticated monitoring approaches and tools.
Custom Monitoring Scripts
Custom scripts can monitor application-specific metrics and business logic.
#!/bin/bash
# Application health check script
# Configuration
APP_URL="http://localhost:8080/health"
LOG_FILE="/var/log/app-monitor.log"
ALERT_THRESHOLD=5
RETRY_COUNT=3
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to check application health
check_app_health() {
local response_code
response_code=$(curl -s -o /dev/null -w "%{http_code}" "$APP_URL")
if [ "$response_code" = "200" ]; then
return 0
else
return 1
fi
}
# Function to send alert
send_alert() {
local message="$1"
# Send email alert
echo "$message" | mail -s "Application Alert" admin@example.com
# Log to syslog
logger -p daemon.crit "$message"
}
# Main monitoring loop
main() {
local failure_count=0
while true; do
if check_app_health; then
if [ $failure_count -gt 0 ]; then
log_message "Application recovered after $failure_count failures"
failure_count=0
fi
else
failure_count=$((failure_count + 1))
log_message "Application health check failed (attempt $failure_count)"
if [ $failure_count -ge $ALERT_THRESHOLD ]; then
send_alert "Application health check failed $failure_count times"
failure_count=0
fi
fi
sleep 60
done
}
main "$@"
Performance Baseline Establishment
Establishing performance baselines helps identify deviations and performance degradation.
#!/bin/bash
# Performance baseline collection script
BASELINE_DIR="/var/log/performance-baseline"
DATE=$(date +%Y%m%d)
DURATION=86400 # 24 hours in seconds
mkdir -p "$BASELINE_DIR"
# CPU baseline
sar -u 60 $((DURATION/60)) > "$BASELINE_DIR/cpu-baseline-$DATE.log" &
# Memory baseline
sar -r 60 $((DURATION/60)) > "$BASELINE_DIR/memory-baseline-$DATE.log" &
# I/O baseline
sar -d 60 $((DURATION/60)) > "$BASELINE_DIR/io-baseline-$DATE.log" &
# Network baseline
sar -n DEV 60 $((DURATION/60)) > "$BASELINE_DIR/network-baseline-$DATE.log" &
# Application-specific metrics
while true; do
echo "$(date): $(ps aux | grep myapp | grep -v grep | wc -l) processes" >> "$BASELINE_DIR/app-processes-$DATE.log"
echo "$(date): $(netstat -an | grep :8080 | grep ESTABLISHED | wc -l) connections" >> "$BASELINE_DIR/app-connections-$DATE.log"
sleep 300 # 5-minute intervals
done &
echo "Baseline collection started for 24 hours"
echo "Data will be stored in $BASELINE_DIR"
Advanced Troubleshooting
Kernel and System-Level Debugging
Advanced troubleshooting often requires kernel-level analysis and system debugging techniques.
Kernel Debugging Tools
Kernel issues require specialized tools and techniques for diagnosis.
Kernel Crash Analysis
System crashes generate kernel dumps that can be analyzed for root cause determination.
# Configure kdump for crash dumps
vim /etc/kdump.conf
# Configure dump location and options
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
# Start kdump service
systemctl enable kdump
systemctl start kdump
# Verify kdump configuration
kdumpctl showmem
cat /proc/cmdline | grep crashkernel
# Analyze crash dumps with crash utility
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore
# Common crash analysis commands
crash> bt # Backtrace
crash> ps # Process list at crash time
crash> log # Kernel log messages
crash> files PID # Open files for process
crash> vm PID # Virtual memory information
System Call Tracing
System call tracing helps identify application and kernel interaction issues.
# Trace system calls for a process
strace -p PID -o trace.log
# Trace specific system calls
strace -e trace=open,read,write -p PID
# Trace all processes of a program
strace -f -e trace=network ./program
# System call performance analysis
strace -c ./program # System call summary
strace -T ./program # Show time spent in each syscall
# Advanced tracing with perf
perf trace -p PID
perf trace -e syscalls:sys_enter_open
Advanced Diagnostic Techniques
Complex issues may require advanced diagnostic approaches and tools.
Dynamic Tracing with SystemTap
SystemTap provides dynamic instrumentation capabilities for advanced system analysis.
# Install SystemTap
yum install systemtap systemtap-runtime
debuginfo-install kernel
# Simple SystemTap script to trace file opens
cat > trace_opens.stp << 'EOF'
probe syscall.open {
printf("%s[%d]: open(%s)\n", execname(), pid(), filename)
}
EOF
# Run SystemTap script
stap trace_opens.stp
# SystemTap for performance analysis
cat > io_latency.stp << 'EOF'
global start_time, latencies
probe ioblock.request {
start_time[bio] = gettimeofday_us()
}
probe ioblock.end {
if (bio in start_time) {
latencies <<< gettimeofday_us() - start_time[bio]
delete start_time[bio]
}
}
probe timer.s(10) {
printf("I/O Latency Statistics:\n")
print(@hist_log(latencies))
delete latencies
}
EOF
stap io_latency.stp
BPF and eBPF Tools
Berkeley Packet Filter provides efficient kernel instrumentation with minimal overhead.
# Install BCC tools
yum install bcc-tools
# Trace block I/O latency
biolatency 10
# Monitor process creation
execsnoop
# Trace TCP connections
tcpconnect
# File system operation tracing
opensnoop
statsnoop
# Memory allocation tracing
mallocstacks
# CPU profiling
profile -F 99 -ag # Sample at 99Hz for all processes
Network Troubleshooting
Network issues can be complex and require understanding of multiple protocol layers.
Packet-Level Analysis
Deep network troubleshooting often requires packet capture and analysis.
Traffic Capture and Analysis
Packet capture provides detailed information about network communication.
# Basic packet capture
tcpdump -i eth0 -w capture.pcap
tcpdump -i any host 192.168.1.100
# Specific protocol capture
tcpdump -i eth0 tcp port 80
tcpdump -i eth0 udp port 53
# Advanced filtering
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'
tcpdump -i eth0 'icmp[icmptype] = icmp-echo'
# Capture with detailed output
tcpdump -i eth0 -v -n -x host 192.168.1.100
# Analysis with tshark (Wireshark command line)
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port
tshark -r capture.pcap -Y "tcp.analysis.retransmission"
Network Performance Analysis
Network performance issues require analysis of throughput, latency, and packet loss.
# Bandwidth testing
iperf3 -s # Server mode
iperf3 -c server_ip -t 60 # Client mode, 60-second test
# Network latency testing
ping -c 100 remote_host
hping3 -S -p 80 -c 100 remote_host # TCP SYN ping
# Path analysis
traceroute remote_host
mtr remote_host # Continuous traceroute
# Network interface statistics
ip -s link show eth0
cat /proc/net/dev
netstat -i
# Socket statistics
ss -s # Summary
ss -tuln # TCP and UDP listening sockets
ss -o state established # Established connections with options
Application Network Issues
Application-level network problems require understanding of application protocols and behavior.
HTTP/HTTPS Troubleshooting
Web applications have specific troubleshooting requirements.
# HTTP request testing
curl -v http://example.com
curl -I http://example.com # HEAD request only
curl -w "@curl-format.txt" http://example.com
# Create curl timing format file
cat > curl-format.txt << 'EOF'
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
EOF
# SSL/TLS troubleshooting
openssl s_client -connect example.com:443
openssl x509 -in certificate.crt -text -noout
# Load testing
ab -n 1000 -c 10 http://example.com/
siege -c 10 -t 1M http://example.com/
Database Connection Issues
Database connectivity problems require specific diagnostic approaches.
# MySQL/MariaDB connection testing
mysql -h database_host -u username -p -e "SELECT 1"
mysqladmin -h database_host -u username -p ping
# PostgreSQL connection testing
psql -h database_host -U username -d database -c "SELECT 1"
pg_isready -h database_host -p 5432
# Connection pool analysis
netstat -an | grep :3306 | grep ESTABLISHED | wc -l
ss -o state established '( dport = :3306 or sport = :3306 )'
# Database performance monitoring
mysqladmin -u root -p processlist
mysqladmin -u root -p extended-status | grep -E "(connections|queries)"
Storage Troubleshooting
Storage issues can cause data corruption, performance problems, and system instability.
Hardware-Level Storage Analysis
Storage hardware problems require low-level diagnostic tools.
SMART Monitoring
Self-Monitoring, Analysis, and Reporting Technology provides insight into disk health.
# Check SMART capability
smartctl -i /dev/sda
# Run SMART tests
smartctl -t short /dev/sda # Short self-test
smartctl -t long /dev/sda # Long self-test
smartctl -t conveyance /dev/sda # Conveyance test
# View SMART attributes
smartctl -A /dev/sda
smartctl -a /dev/sda # All SMART information
# Monitor SMART logs
smartctl -l error /dev/sda
smartctl -l selftest /dev/sda
# Automated SMART monitoring with smartd
vim /etc/smartd.conf
# Add monitoring configuration
/dev/sda -a -o on -S on -s (S/../.././02|L/../../6/03) -m admin@example.com
systemctl enable smartd
systemctl start smartd
I/O Performance Analysis
Understanding I/O patterns helps identify performance bottlenecks.
# I/O monitoring and analysis
iostat -x 1 10 # Extended statistics
iotop -o # I/O by process
pidstat -d 1 10 # Per-process I/O
# Block device statistics
cat /proc/diskstats
cat /sys/block/sda/stat
# I/O scheduler analysis
cat /sys/block/sda/queue/scheduler
echo deadline > /sys/block/sda/queue/scheduler
# I/O queue depth analysis
cat /sys/block/sda/queue/nr_requests
echo 128 > /sys/block/sda/queue/nr_requests
# Advanced I/O tracing
blktrace -d /dev/sda -o - | blkparse -i -
btrace /dev/sda # Real-time block I/O tracing
This comprehensive Red Hat manual covers the essential aspects of configuration, tuning, and troubleshooting. Each section provides practical examples and commands that can be directly applied in production environments. The manual serves as both a learning resource and a reference guide for system administrators working with Red Hat Enterprise Linux systems.2 Configuration
GRUB2 is the default bootloader for Red Hat Enterprise Linux. Proper configuration ensures reliable system startup and provides recovery options.
The main configuration file is /boot/grub2/grub.cfg, but this should never be edited directly. Instead, modify /etc/default/grub and regenerate the configuration.
# Edit GRUB defaults
vim /etc/default/grub
# Common GRUB parameters
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Red Hat Enterprise Linux"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"
# Regenerate GRUB configuration
grub2-mkconfig -o /boot/grub2/grub.cfg
# For UEFI systems
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
Kernel Parameters
Kernel parameters can be modified temporarily via /proc/sys/ or permanently through /etc/sysctl.conf and /etc/sysctl.d/.
# Temporary kernel parameter modification
echo 1 > /proc/sys/net/ipv4/ip_forward
# Permanent configuration in /etc/sysctl.d/99-custom.conf
net.ipv4.ip_forward = 1
vm.swappiness = 10
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
# Apply sysctl changes
sysctl -p /etc/sysctl.d/99-custom.conf
# Verify current values
sysctl -a | grep ip_forward
Performance Tuning
CPU Performance Tuning
CPU performance tuning involves optimizing processor usage, managing CPU affinity, and configuring power management settings.
CPU Governor Configuration
The CPU governor controls CPU frequency scaling. Different governors are appropriate for different workloads.
# Check available governors
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
# Set performance governor for maximum performance
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# Use cpupower tool for comprehensive CPU management
cpupower frequency-set -g performance
# Check current CPU frequency information
cpupower frequency-info
# Set CPU frequency limits
cpupower frequency-set -d 1.2GHz -u 3.4GHz
Process Priority and CPU Affinity
Managing process priorities and CPU affinity can significantly improve system performance for critical applications.
# Set process priority with nice
nice -n -10 ./critical-application
# Change priority of running process
renice -10 -p 1234
# Set CPU affinity for a process
taskset -c 0,1 ./application
# Check current CPU affinity
taskset -p 1234
# Use numactl for NUMA-aware process scheduling
numactl --cpunodebind=0 --membind=0 ./application
Memory Management Tuning
Memory tuning involves optimizing virtual memory settings, managing swap usage, and configuring memory allocation policies.
Virtual Memory Tuning
Key virtual memory parameters affect system performance and stability. These should be tuned based on workload characteristics.
# Key memory tuning parameters in /etc/sysctl.d/memory.conf
vm.swappiness = 10 # Reduce swap usage preference
vm.dirty_ratio = 15 # Percentage of memory for dirty pages
vm.dirty_background_ratio = 5 # Background writeback threshold
vm.vfs_cache_pressure = 50 # Inode/dentry cache reclaim pressure
vm.min_free_kbytes = 65536 # Minimum free memory to maintain
vm.zone_reclaim_mode = 0 # NUMA zone reclaim behavior
# Huge pages configuration for database workloads
vm.nr_hugepages = 1024 # Number of 2MB huge pages
kernel.shmmax = 137438953472 # Maximum shared memory segment size
kernel.shmall = 33554432 # Total shared memory pages
Memory Monitoring and Analysis
Understanding memory usage patterns is crucial for effective tuning.
# Detailed memory information
cat /proc/meminfo
# Memory usage by process
ps aux --sort=-%mem | head -10
# System memory statistics
vmstat 1 10
# Memory slab information
cat /proc/slabinfo
# Check for memory leaks
valgrind --tool=memcheck --leak-check=full ./application
Storage I/O Tuning
Storage performance directly impacts overall system performance. Proper I/O tuning involves optimizing disk schedulers, file systems, and I/O patterns.
I/O Scheduler Configuration
Different I/O schedulers are optimized for different storage types and workload patterns.
# Check current I/O scheduler
cat /sys/block/sda/queue/scheduler
# Change I/O scheduler temporarily
echo noop > /sys/block/sda/queue/scheduler
# Permanent I/O scheduler configuration in /etc/default/grub
GRUB_CMDLINE_LINUX="elevator=deadline"
# Configure I/O scheduler per device type
# For SSDs: noop or deadline
# For HDDs: cfq or deadline
# For virtual machines: noop
# I/O queue parameters
echo 256 > /sys/block/sda/queue/nr_requests
echo 0 > /sys/block/sda/queue/add_random
File System Tuning
File system mount options and parameters significantly affect I/O performance.
# Optimized ext4 mount options in /etc/fstab
/dev/sda1 /data ext4 defaults,noatime,barrier=0,data=writeback 0 2
# XFS tuning parameters
mount -o noatime,logbsize=256k,logdev=/dev/sdb1 /dev/sda1 /data
# File system block size optimization
mkfs.ext4 -b 4096 -E stride=32,stripe-width=64 /dev/sda1
# Check file system parameters
tune2fs -l /dev/sda1
# XFS performance analysis
xfs_info /dev/sda1
Network Configuration
Network Interface Configuration
Red Hat Enterprise Linux uses NetworkManager and traditional network scripts for interface configuration. Understanding both methods is essential for proper network management.
NetworkManager Configuration
NetworkManager provides dynamic network configuration with support for various connection types.
# List network connections
nmcli connection show
# Create static IP configuration
nmcli connection add type ethernet con-name "static-eth0" \
ifname eth0 ip4 192.168.1.100/24 gw4 192.168.1.1
# Configure DNS servers
nmcli connection modify static-eth0 ipv4.dns "8.8.8.8,8.8.4.4"
# Activate connection
nmcli connection up static-eth0
# Configure bonding interface
nmcli connection add type bond con-name "bond0" ifname bond0 \
mode active-backup ip4 192.168.1.200/24 gw4 192.168.1.1
# Add slave interfaces to bond
nmcli connection add type ethernet slave-type bond \
con-name "bond0-slave1" ifname eth0 master bond0
nmcli connection add type ethernet slave-type bond \
con-name "bond0-slave2" ifname eth1 master bond0
Traditional Network Scripts
Legacy network configuration using ifcfg files is still supported and sometimes necessary for specific configurations.
# Static interface configuration in /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eth0
UUID=12345678-1234-1234-1234-123456789abc
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.1.100
PREFIX=24
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
# Restart network service
systemctl restart network
Advanced Network Configuration
VLAN Configuration
Virtual LANs provide network segmentation and improved security.
# Create VLAN interface using NetworkManager
nmcli connection add type vlan con-name "vlan100" \
ifname eth0.100 dev eth0 id 100 ip4 192.168.100.1/24
# VLAN configuration using traditional method
# /etc/sysconfig/network-scripts/ifcfg-eth0.100
DEVICE=eth0.100
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.100.1
NETMASK=255.255.255.0
VLAN=yes
# Load 8021q module
modprobe 8021q
echo "8021q" >> /etc/modules-load.d/vlan.conf
Bridge Configuration
Network bridges are essential for virtualization and container networking.
# Create bridge using NetworkManager
nmcli connection add type bridge con-name "br0" ifname br0
nmcli connection modify br0 ipv4.addresses 192.168.1.50/24
nmcli connection modify br0 ipv4.gateway 192.168.1.1
nmcli connection modify br0 ipv4.dns 8.8.8.8
nmcli connection modify br0 ipv4.method manual
# Add interface to bridge
nmcli connection add type bridge-slave ifname eth0 master br0
# Traditional bridge configuration
# /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.1.50
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
Network Performance Tuning
Network performance tuning involves optimizing buffer sizes, interrupt handling, and protocol parameters.
# Network buffer tuning in /etc/sysctl.d/network.conf
net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
# Interrupt handling optimization
echo 2 > /proc/irq/24/smp_affinity
# Network interface ring buffer tuning
ethtool -G eth0 rx 4096 tx 4096
# Check network interface statistics
ethtool -S eth0
Storage Management
Logical Volume Management (LVM)
LVM provides flexible disk management with features like volume resizing, snapshots, and device migration.
LVM Basic Operations
Understanding LVM hierarchy is crucial: Physical Volumes (PV) → Volume Groups (VG) → Logical Volumes (LV).
# Create physical volume
pvcreate /dev/sdb /dev/sdc
# Create volume group
vgcreate data_vg /dev/sdb /dev/sdc
# Create logical volume
lvcreate -L 100G -n app_lv data_vg
# Extend logical volume
lvextend -L +50G /dev/data_vg/app_lv
# Extend file system after LV extension
resize2fs /dev/data_vg/app_lv # For ext4
xfs_growfs /mount/point # For XFS
# Create LVM snapshot
lvcreate -L 10G -s -n app_snap /dev/data_vg/app_lv
# Monitor LVM status
pvdisplay
vgdisplay
lvdisplay
Advanced LVM Features
LVM provides advanced features for enterprise storage management.
# LVM striping for performance
lvcreate -L 200G -i 2 -I 64 -n striped_lv data_vg
# LVM mirroring for redundancy
lvcreate -L 100G -m 1 -n mirror_lv data_vg
# Thin provisioning
lvcreate -L 1T -T data_vg/thin_pool
lvcreate -V 100G -T data_vg/thin_pool -n thin_lv
# Move physical extents between devices
pvmove /dev/sdb /dev/sdd
# Remove physical volume from volume group
vgreduce data_vg /dev/sdb
pvremove /dev/sdb
File System Management
File System Creation and Maintenance
Different file systems have specific use cases and maintenance requirements.
# Create ext4 file system with optimal settings
mkfs.ext4 -b 4096 -m 1 -O ^has_journal /dev/data_vg/app_lv
# Create XFS file system
mkfs.xfs -f -b size=4096 -d agcount=8 /dev/data_vg/app_lv
# File system check and repair
fsck -f /dev/data_vg/app_lv # Generic fsck
e2fsck -f /dev/data_vg/app_lv # ext4 specific
xfs_repair /dev/data_vg/app_lv # XFS specific
# Tune file system parameters
tune2fs -m 1 -c 0 -i 0 /dev/data_vg/app_lv
# File system usage analysis
df -h
du -sh /path/*
lsof +D /mount/point
Mount Options and Performance
Proper mount options significantly impact file system performance and behavior.
# Performance-oriented mount options in /etc/fstab
/dev/data_vg/app_lv /app ext4 defaults,noatime,nodiratime,barrier=0 0 2
/dev/data_vg/db_lv /database xfs defaults,noatime,logbsize=256k,nobarrier 0 2
# Temporary mount with specific options
mount -o noatime,nodiratime,barrier=0 /dev/data_vg/app_lv /app
# Bind mounts for directory access
mount --bind /original/path /new/path
# Check mount options
mount | grep /app
findmnt /app
Storage Monitoring and Troubleshooting
I/O Performance Analysis
Understanding storage performance characteristics is essential for troubleshooting and optimization.
# Real-time I/O monitoring
iostat -x 1 10
# Per-process I/O statistics
iotop -o
# Disk usage and performance
sar -d 1 10
# Advanced I/O tracing
blktrace -d /dev/sda -o - | blkparse -i -
# Storage latency analysis
ioping -c 10 /path/to/storage
# Benchmark storage performance
dd if=/dev/zero of=/tmp/testfile bs=1G count=1 oflag=direct
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
Security Configuration
Firewall Configuration
Red Hat Enterprise Linux uses firewalld as the default firewall management tool, providing both permanent and runtime configurations.
Firewalld Basic Configuration
Firewalld uses zones to manage different network trust levels and services to define allowed network traffic.
# Check firewalld status
systemctl status firewalld
firewall-cmd --state
# List all zones and their configurations
firewall-cmd --list-all-zones
# Set default zone
firewall-cmd --set-default-zone=public
# Add service to zone permanently
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
# Add custom port
firewall-cmd --permanent --zone=public --add-port=8080/tcp
# Create custom service definition
firewall-cmd --permanent --new-service=myapp
firewall-cmd --permanent --service=myapp --set-description="My Application"
firewall-cmd --permanent --service=myapp --set-short="MyApp"
firewall-cmd --permanent --service=myapp --add-port=9090/tcp
# Rich rules for complex configurations
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" service name="ssh" accept'
# Reload firewall configuration
firewall-cmd --reload
Advanced Firewall Configuration
Complex network environments require advanced firewall rules and zone management.
# Create custom zone
firewall-cmd --permanent --new-zone=dmz-servers
firewall-cmd --permanent --zone=dmz-servers --set-description="DMZ Server Zone"
# Interface assignment to zones
firewall-cmd --permanent --zone=dmz-servers --add-interface=eth1
# Port forwarding
firewall-cmd --permanent --zone=public --add-forward-port=port=2222:proto=tcp:toport=22:toaddr=192.168.1.100
# Masquerading for NAT
firewall-cmd --permanent --zone=public --add-masquerade
# Direct rules for iptables integration
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 3306 -s 192.168.1.0/24 -j ACCEPT
SELinux Configuration
Security-Enhanced Linux provides mandatory access control and should be properly configured rather than disabled.
SELinux Management
Understanding SELinux modes, contexts, and policies is crucial for system security.
# Check SELinux status
sestatus
getenforce
# Set SELinux mode temporarily
setenforce 0 # Permissive
setenforce 1 # Enforcing
# Permanent SELinux configuration in /etc/selinux/config
SELINUX=enforcing
SELINUXTYPE=targeted
# View SELinux contexts
ls -Z /var/www/html/
ps auxZ | grep httpd
# Set file contexts
semanage fcontext -a -t httpd_exec_t "/opt/myapp/bin(/.*)?"
restorecon -R /opt/myapp/bin
# SELinux boolean management
getsebool -a | grep httpd
setsebool -P httpd_can_network_connect on
# Troubleshoot SELinux denials
sealert -a /var/log/audit/audit.log
ausearch -m avc -ts recent
SELinux Policy Customization
Custom SELinux policies may be necessary for non-standard applications.
# Generate custom policy from audit logs
grep myapp /var/log/audit/audit.log | audit2allow -M myapp_policy
semodule -i myapp_policy.pp
# Create custom SELinux user
semanage user -a -R "staff_r system_r" -P user custom_user
# Port labeling for custom applications
semanage port -a -t http_port_t -p tcp 8080
# File context management
semanage fcontext -l | grep httpd
semanage fcontext -d -t httpd_exec_t "/old/path(/.*)?"
User and Access Management
User Account Security
Proper user account management is fundamental to system security.
# Create user with specific settings
useradd -m -s /bin/bash -G wheel,developers -c "Application User" appuser
# Set password policies in /etc/login.defs
PASS_MAX_DAYS 90
PASS_MIN_DAYS 7
PASS_MIN_LEN 8
PASS_WARN_AGE 14
# Account aging management
chage -M 90 -m 7 -W 14 username
chage -l username
# Lock and unlock accounts
usermod -L username # Lock
usermod -U username # Unlock
# Force password change
chage -d 0 username
# Monitor user login activity
last
lastb # Failed login attempts
who
w
SSH Security Configuration
SSH is the primary remote access method and requires proper security configuration.
# SSH daemon configuration in /etc/ssh/sshd_config
Port 2222
Protocol 2
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ClientAliveInterval 300
ClientAliveCountMax 2
MaxAuthTries 3
MaxStartups 10:30:100
# Generate SSH keys
ssh-keygen -t rsa -b 4096 -C "user@hostname"
ssh-keygen -t ed25519 -C "user@hostname"
# SSH key management
ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote-host
# SSH connection multiplexing in ~/.ssh/config
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600
Service Management
Systemd Service Management
Systemd is the init system and service manager for Red Hat Enterprise Linux, providing sophisticated service management capabilities.
Basic Service Operations
Understanding systemd service lifecycle and management commands is essential for system administration.
# Service status and control
systemctl status httpd
systemctl start httpd
systemctl stop httpd
systemctl restart httpd
systemctl reload httpd
# Enable/disable services
systemctl enable httpd # Start at boot
systemctl disable httpd # Don't start at boot
# Service dependency analysis
systemctl list-dependencies httpd
systemctl list-dependencies --reverse httpd
# Service configuration files
systemctl show httpd
systemctl cat httpd
systemctl edit httpd # Create override file
# Failed service analysis
systemctl --failed
systemctl reset-failed
Custom Service Creation
Creating custom systemd service units for applications and scripts.
# Custom service unit file /etc/systemd/system/myapp.service
[Unit]
Description=My Application Service
After=network.target
Requires=network.target
[Service]
Type=forking
User=appuser
Group=appgroup
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --daemon
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/myapp.pid
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
# Timer unit for scheduled tasks /etc/systemd/system/backup.timer
[Unit]
Description=Backup Timer
Requires=backup.service
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
# Reload systemd configuration
systemctl daemon-reload
systemctl enable myapp.service
systemctl start myapp.service
Advanced Systemd Features
Systemd provides advanced features for service management and system control.
# Service resource limits in service unit
[Service]
MemoryLimit=512M
CPUQuota=50%
TasksMax=100
PrivateTmp=true
NoNewPrivileges=true
# Socket activation
# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket
[Socket]
ListenStream=8080
Accept=false
[Install]
WantedBy=sockets.target
# Target management
systemctl get-default
systemctl set-default multi-user.target
systemctl isolate rescue.target
# Journal management
journalctl -u httpd
journalctl -f # Follow logs
journalctl --since "2023-01-01" --until "2023-01-02"
journalctl -p err # Error level and above
Troubleshooting Methodologies
Systematic Troubleshooting Approach
Effective troubleshooting requires a systematic methodology to identify and resolve issues efficiently.
Problem Identification and Analysis
The first step in troubleshooting is properly identifying and understanding the problem scope and impact.
Start by gathering comprehensive information about the issue. Document the symptoms, error messages, and affected systems or services. Determine when the problem started and identify any recent changes to the system configuration, software updates, or infrastructure modifications.
Establish the problem scope by identifying which systems, services, or users are affected. Determine if the issue is intermittent or persistent, and whether it affects all operations or specific functions. Understanding the business impact helps prioritize the troubleshooting effort.
Create a timeline of events leading to the problem. Review system logs, monitoring data, and change management records to identify potential triggers or contributing factors.
Root Cause Analysis
Once the problem is identified, systematic analysis helps determine the underlying cause rather than just addressing symptoms.
Use the "5 Whys" technique to drill down to root causes. For each symptom or immediate cause, ask "why" this occurred, continuing until you reach the fundamental issue.
Implement divide-and-conquer strategies by isolating system components. Test individual components separately to identify where the problem originates. This approach is particularly effective for complex systems with multiple interdependencies.
Correlation analysis involves examining relationships between different system metrics, log entries, and events. Look for patterns that might indicate causation rather than coincidence.
Documentation and Knowledge Management
Proper documentation ensures that troubleshooting knowledge is preserved and can be leveraged for future incidents.
Maintain detailed troubleshooting logs that include problem descriptions, investigation steps, findings, and resolution procedures. This documentation serves as a knowledge base for similar future issues.
Create runbooks for common problems and their solutions. Include step-by-step procedures, required tools, and verification steps to ensure consistency in problem resolution.
Post-incident reviews should analyze what worked well and what could be improved in the troubleshooting process. Use these insights to refine procedures and prevent similar issues.
Log Analysis and Interpretation
System logs provide crucial information for troubleshooting, but effective analysis requires understanding log formats, locations, and interpretation techniques.
System Log Analysis
Red Hat Enterprise Linux uses systemd journal and traditional syslog for system logging. Understanding both systems is essential for comprehensive log analysis.
# Journalctl advanced usage
journalctl --list-boots
journalctl -b -1 # Previous boot logs
journalctl -u sshd --since "1 hour ago"
journalctl -p warning..err # Warning through error levels
journalctl -f -u httpd # Follow specific service logs
# Traditional syslog analysis
tail -f /var/log/messages
grep -i error /var/log/messages
awk '/ERROR/ {print $1, $2, $3, $NF}' /var/log/messages
# Log rotation and management
logrotate -d /etc/logrotate.conf # Debug mode
logrotate -f /etc/logrotate.d/httpd # Force rotation
Application Log Analysis
Application logs often contain the most relevant information for troubleshooting application-specific issues.
# Web server log analysis
tail -f /var/log/httpd/access_log
grep "HTTP/1.1\" 5" /var/log/httpd/access_log # Server errors
awk '{print $1}' /var/log/httpd/access_log | sort | uniq -c | sort -nr # Top IP addresses
# Database log analysis
tail -f /var/log/mysql/error.log
grep -i "deadlock" /var/log/mysql/error.log
# Custom log parsing with awk and sed
awk '/ERROR/ {count++} END {print "Total errors:", count}' application.log
sed -n '1000,2000p' large_log_file.log # Extract specific line range
Performance Analysis Tools
Performance issues require specialized tools and techniques for identification and analysis.
System Performance Monitoring
Comprehensive performance analysis involves monitoring CPU, memory, I/O, and network resources.
# CPU analysis
top -p $(pgrep -d, processname) # Monitor specific process
htop # Interactive process viewer
mpstat 1 10 # CPU statistics
sar -u 1 10 # CPU utilization over time
# Memory analysis
free -h
cat /proc/meminfo
pmap -x PID # Process memory mapping
smem -t -k # Memory usage by process
# I/O analysis
iostat -x 1 10 # Extended I/O statistics
iotop -o # I/O by process
lsof +D /path # Files open in directory
# Network analysis
netstat -tulpn # Listening ports and connections
ss -tulpn # Socket statistics
nload # Network interface usage
tcpdump -i eth0 -n host 192.168.1.100 # Packet capture
Advanced Performance Analysis
Complex performance issues may require advanced analysis tools and techniques.
# System call tracing
strace -p PID # Trace system calls of running process
strace -c ./program # System call summary
# Performance profiling
perf record -g ./program
perf report
# Kernel tracing
ftrace # Function tracer
trace-cmd record -p function_graph ls
trace-cmd report
# Memory leak detection
valgrind --tool=memcheck --leak-check=full ./program
# Lock contention analysis
perf lock record ./program
perf lock report