linux-sysadmin-journey/projects/01-homelab-setup/monitoring-setup.md

System Monitoring and Maintenance Setup

Overview

This document tracks the implementation of system monitoring tools and maintenance procedures for our Ubuntu Server VM.

Monitoring Tools

1. htop Installation and Configuration ✅

# Install htop
sudo apt update
sudo apt install htop

# Basic usage
htop  # Interactive process viewer

Key htop Features

• F1 - Help
• F2 - Setup (customize display)
• F3 - Search processes
• F4 - Filter processes
• F5 - Tree view
• F6 - Sort by column
• F9 - Kill process
• F10 - Quit

Default Display Information

• CPU usage (per core)
• Memory usage
• Swap usage
• Load average
• Running processes
• System uptime

Customization Applied

# htop configuration is stored in
~/.config/htop/htoprc

2. Netdata Setup ✅

# Install required dependencies
sudo apt install curl

# Install Netdata
bash <(curl -Ss https://get.netdata.cloud/kickstart.sh)

# Allow Netdata web interface through firewall
sudo ufw allow 19999/tcp

Web Interface Access

• URL: http://169.254.0.2:19999
• Port: 19999
• Status: Active and accessible

Available Metrics

• Real-time CPU usage
• Memory utilization
• Disk operations
• Network traffic
• System load
• Active processes

3. Log Monitoring ✅

# Install logwatch for log analysis
sudo apt install logwatch

# Create and configure logwatch
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/logwatch.conf
sudo nano /etc/logwatch/conf/logwatch.conf

# Create required cache directory
sudo mkdir -p /var/cache/logwatch
sudo chown -R root:root /var/cache/logwatch
sudo chmod 750 /var/cache/logwatch

Basic Configuration

# /etc/logwatch/conf/logwatch.conf settings
Output = stdout
Format = text
MailTo = root
MailFrom = Logwatch
Detail = Low
Range = yesterday
Service = All

Usage Notes

• Empty output is normal for new systems with limited activity
• Logs will accumulate as the system is used
• Key log locations:

  /var/log/syslog        # General system logs
  /var/log/auth.log      # Authentication logs
  /var/log/kern.log      # Kernel logs
  /var/log/ufw.log       # Firewall logs
  

Verification Commands

# View recent logs directly
sudo tail -f /var/log/syslog
sudo tail -f /var/log/auth.log

# Generate logwatch report
sudo logwatch --output stdout --format text --range today --detail high

Maintenance Procedures

1. Backup Strategy ✅

• Define backup locations
  • System files: /backup/system/
  • Home directories: /backup/home/
• Select backup tools
  • Using rsync for efficient incremental backups
  • Custom backup script for automation
• Create backup schedule
  • Daily backups
  • 7-day retention policy

Backup Script

# Location: /usr/local/bin/system-backup.sh
# See script content above...

Manual Backup Commands

# Run system backup
sudo /usr/local/bin/system-backup.sh

# Verify backup contents
ls -l /backup/system/
ls -l /backup/home/

# Check backup logs
tail -f /var/log/syslog | grep backup

Backup Verification

• Test full system backup
• Verify backup contents
• Test file restoration

Automated Scheduling ✅

# Set up daily backup at 2 AM
sudo crontab -e

# Add the following line:
0 2 * * * /usr/local/bin/system-backup.sh >> /var/log/backup.log 2>&1

Backup Log

• Location: /var/log/backup.log
• Contains: Backup execution details and any errors
• Check status: sudo tail -f /var/log/backup.log

2. System Updates ✅

• Configure unattended-upgrades

  # Install and configure
  sudo apt install unattended-upgrades apt-listchanges
  sudo dpkg-reconfigure -plow unattended-upgrades
  

Update Configuration

# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};

# Additional settings
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";

3. Log Management ✅

• Configure log rotation
• Set retention policies
• Implement compression

Log Rotation Configuration

# /etc/logrotate.d/custom-logs
/var/log/backup.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}

/var/log/unattended-upgrades/unattended-upgrades.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}

Log Management Verification

# Test log rotation configuration
sudo logrotate -d /etc/logrotate.d/custom-logs

# View current log sizes
du -sh /var/log/*

# Monitor system logs
tail -f /var/log/syslog

Implementation Status

• Basic System Monitoring (htop)
• Performance Metrics (Netdata)
• Log Analysis (Logwatch)
• Backup System (rsync + cron)
• Maintenance Automation (unattended-upgrades + logrotate)

Commands and Configurations

Monitoring Commands

# System resource monitoring
htop
free -h
df -h
vmstat 1

# Process monitoring
ps aux
top

# Network monitoring
netstat -tulpn
ss -tulpn

Issues and Solutions

Document any issues encountered during setup and their solutions here.