linux-sysadmin-journey/projects/01-homelab-setup/monitoring-setup.md

# System Monitoring and Maintenance Setup

## Overview
This document tracks the implementation of system monitoring tools and maintenance procedures for our Ubuntu Server VM.

## Monitoring Tools

### 1. htop Installation and Configuration ✅
```bash
# Install htop
sudo apt update
sudo apt install htop

# Basic usage
htop  # Interactive process viewer
```

#### Key htop Features
- F1 - Help
- F2 - Setup (customize display)
- F3 - Search processes
- F4 - Filter processes
- F5 - Tree view
- F6 - Sort by column
- F9 - Kill process
- F10 - Quit

#### Default Display Information
- CPU usage (per core)
- Memory usage
- Swap usage
- Load average
- Running processes
- System uptime

#### Customization Applied
```bash
# htop configuration is stored in
~/.config/htop/htoprc
```

### 2. Netdata Setup ✅
```bash
# Install required dependencies
sudo apt install curl

# Install Netdata
bash <(curl -Ss https://get.netdata.cloud/kickstart.sh)

# Allow Netdata web interface through firewall
sudo ufw allow 19999/tcp
```

#### Web Interface Access
- URL: http://169.254.0.2:19999
- Port: 19999
- Status: Active and accessible

#### Available Metrics
- Real-time CPU usage
- Memory utilization
- Disk operations
- Network traffic
- System load
- Active processes

### 3. Log Monitoring ✅
```bash
# Install logwatch for log analysis
sudo apt install logwatch

# Create and configure logwatch
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/logwatch.conf
sudo nano /etc/logwatch/conf/logwatch.conf

# Create required cache directory
sudo mkdir -p /var/cache/logwatch
sudo chown -R root:root /var/cache/logwatch
sudo chmod 750 /var/cache/logwatch
```

#### Basic Configuration
```conf
# /etc/logwatch/conf/logwatch.conf settings
Output = stdout
Format = text
MailTo = root
MailFrom = Logwatch
Detail = Low
Range = yesterday
Service = All
```

#### Usage Notes
- Empty output is normal for new systems with limited activity
- Logs will accumulate as the system is used
- Key log locations:
  ```bash
  /var/log/syslog        # General system logs
  /var/log/auth.log      # Authentication logs
  /var/log/kern.log      # Kernel logs
  /var/log/ufw.log       # Firewall logs
  ```

#### Verification Commands
```bash
# View recent logs directly
sudo tail -f /var/log/syslog
sudo tail -f /var/log/auth.log

# Generate logwatch report
sudo logwatch --output stdout --format text --range today --detail high
```

## Maintenance Procedures

### 1. Backup Strategy ✅
- [x] Define backup locations
  - System files: `/backup/system/`
  - Home directories: `/backup/home/`
- [x] Select backup tools
  - Using `rsync` for efficient incremental backups
  - Custom backup script for automation
- [x] Create backup schedule
  - Daily backups
  - 7-day retention policy

#### Backup Script
```bash
# Location: /usr/local/bin/system-backup.sh
# See script content above...
```

#### Manual Backup Commands
```bash
# Run system backup
sudo /usr/local/bin/system-backup.sh

# Verify backup contents
ls -l /backup/system/
ls -l /backup/home/

# Check backup logs
tail -f /var/log/syslog | grep backup
```

#### Backup Verification
- [ ] Test full system backup
- [ ] Verify backup contents
- [ ] Test file restoration

#### Automated Scheduling ✅
```bash
# Set up daily backup at 2 AM
sudo crontab -e

# Add the following line:
0 2 * * * /usr/local/bin/system-backup.sh >> /var/log/backup.log 2>&1
```

#### Backup Log
- Location: `/var/log/backup.log`
- Contains: Backup execution details and any errors
- Check status: `sudo tail -f /var/log/backup.log`

### 2. System Updates ✅
- [x] Configure unattended-upgrades
  ```bash
  # Install and configure
  sudo apt install unattended-upgrades apt-listchanges
  sudo dpkg-reconfigure -plow unattended-upgrades
  ```

#### Update Configuration
```conf
# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};

# Additional settings
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
```

### 3. Log Management ✅
- [x] Configure log rotation
- [x] Set retention policies
- [x] Implement compression

#### Log Rotation Configuration
```conf
# /etc/logrotate.d/custom-logs
/var/log/backup.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}

/var/log/unattended-upgrades/unattended-upgrades.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}
```

#### Log Management Verification
```bash
# Test log rotation configuration
sudo logrotate -d /etc/logrotate.d/custom-logs

# View current log sizes
du -sh /var/log/*

# Monitor system logs
tail -f /var/log/syslog
```

## Implementation Status
- [x] Basic System Monitoring (htop)
- [x] Performance Metrics (Netdata)
- [x] Log Analysis (Logwatch)
- [x] Backup System (rsync + cron)
- [x] Maintenance Automation (unattended-upgrades + logrotate)

## Commands and Configurations

### Monitoring Commands
```bash
# System resource monitoring
htop
free -h
df -h
vmstat 1

# Process monitoring
ps aux
top

# Network monitoring
netstat -tulpn
ss -tulpn
```

## Issues and Solutions
Document any issues encountered during setup and their solutions here.