diff --git a/docs/INCIDENT_RECOVERY_2026-01-19.md b/docs/INCIDENT_RECOVERY_2026-01-19.md index 4a849cbe..8d9a9123 100644 --- a/docs/INCIDENT_RECOVERY_2026-01-19.md +++ b/docs/INCIDENT_RECOVERY_2026-01-19.md @@ -1,12 +1,15 @@ -# Incident Recovery Report - 2026-01-19 +# Incident Recovery Report - 2026-01-19/20 ## Executive Summary -**Status:** PARTIAL RECOVERY +**Status:** COMPLETE RECOVERY (Updated 2026-01-20) - Website: UP (https://agenticgovernance.digital/ responds HTTP 200) -- SSH Access: BROKEN (connection closes after authentication) -- Malware: REMOVED (PM2 and umami-deployment deleted) -- Root Cause: PM2 process manager running botnet malware +- SSH Access: WORKING (via fresh VPS reinstall) +- Malware: ELIMINATED (complete OS reinstall) +- Application: FULLY RESTORED +- Database: MIGRATED from local backup (134 documents) +- SSL: VALID (Let's Encrypt, expires April 2026) +- Root Cause: PM2 process manager running Exodus botnet malware --- @@ -249,6 +252,108 @@ This incident represents multiple failures: --- -**Report Date:** 2026-01-19 -**Status:** PARTIAL RECOVERY - Website up, SSH broken -**Next Action:** Re-enter rescue mode to fix SSH access +--- + +## COMPLETE RECOVERY - 2026-01-20 + +### What Was Done + +After multiple failed partial cleanup attempts, the decision was made to perform a **complete VPS reinstallation** as recommended in the remediation plan. + +#### Phase 1: VPS Reinstallation via OVH Manager +- User initiated complete OS reinstall from OVH Manager +- Fresh Ubuntu installation with new credentials +- All malware completely eliminated by full disk wipe + +#### Phase 2: System Setup +```bash +# Security tools +apt install -y fail2ban rkhunter chkrootkit + +# Daily security monitoring script +/usr/local/bin/daily-security-check.sh + +# MongoDB with log rotation +apt install -y mongodb-org +# Configured logrotate for /var/log/mongodb/ +``` + +#### Phase 3: Application Deployment +1. Created `/var/www/tractatus/` directory +2. Created production `.env` file with NODE_ENV=production +3. Deployed application via rsync from local (CLEAN source) +4. Installed dependencies including `@anthropic-ai/sdk` +5. Created systemd service (`/etc/systemd/system/tractatus.service`) +6. Configured nginx with SSL reverse proxy + +#### Phase 4: SSL Certificate +```bash +certbot --nginx -d agenticgovernance.digital +# Certificate valid until April 2026 +``` + +#### Phase 5: Database Migration +```bash +# Local: Export database +mongodump --db tractatus_dev --out ~/tractatus-backup + +# Transfer to VPS +rsync -avz ~/tractatus-backup/ ubuntu@vps:/tmp/tractatus-backup/ + +# VPS: Import to production +mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/ +# Result: 134 documents + 12 blog posts restored +``` + +#### Phase 6: Admin Setup +```bash +node scripts/fix-admin-user.js +node scripts/seed-projects.js +``` + +### Final System State (2026-01-20) + +**Services Running:** +- `tractatus.service` - Node.js application (port 9000) +- `nginx.service` - Web server with SSL +- `mongod.service` - MongoDB database +- `fail2ban.service` - Intrusion prevention + +**Services Explicitly BANNED:** +- PM2 - Never install (malware persistence vector) +- Docker - Never install (attack vector) +- PostgreSQL - Not needed (was for Umami) + +**Security Measures:** +- SSH key authentication only (password disabled) +- UFW firewall enabled +- fail2ban active +- Daily security scan at 3 AM UTC (`/usr/local/bin/daily-security-check.sh`) +- rkhunter and chkrootkit installed + +**Post-Recovery Improvements (same session):** +- Removed all Umami analytics references from codebase (29 HTML files) +- Deleted `/public/js/components/umami-tracker.js` +- Updated privacy policy to reflect "No Analytics" +- Added Research Papers section to landing page +- Created `/korero-counter-arguments.html` page +- Fixed Tailwind CSS to include emerald gradient classes + +### Verification Completed + +- [x] SSH access works with key authentication +- [x] Website responds correctly (HTTP 200) +- [x] SSL certificate valid +- [x] MongoDB running and accessible +- [x] All documents migrated (134 total) +- [x] Blog posts visible (12 posts) +- [x] Admin user functional +- [x] No PM2 installed +- [x] No Docker installed +- [x] Daily security scan configured + +--- + +**Report Date:** 2026-01-19 (initial) / 2026-01-20 (complete recovery) +**Status:** COMPLETE RECOVERY - All systems operational +**Next Action:** Resume normal development (/community project) diff --git a/docs/VPS_RECOVERY_REFERENCE.md b/docs/VPS_RECOVERY_REFERENCE.md new file mode 100644 index 00000000..7fb40679 --- /dev/null +++ b/docs/VPS_RECOVERY_REFERENCE.md @@ -0,0 +1,337 @@ +# VPS Recovery Reference - agenticgovernance.digital + +**Last Updated:** 2026-01-20 +**VPS:** vps-93a693da.vps.ovh.net +**IP:** 91.134.240.3 +**Provider:** OVH + +--- + +## Quick Reference + +### SSH Access +```bash +# Primary access (from theflow's machine) +ssh -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net + +# If host key changed (after reinstall) +ssh-keygen -f '/home/theflow/.ssh/known_hosts' -R 'vps-93a693da.vps.ovh.net' +ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net "echo 'Connected'" +``` + +### Service Management +```bash +# Check tractatus service +sudo systemctl status tractatus + +# Restart tractatus +sudo systemctl restart tractatus + +# Check all relevant services +sudo systemctl status tractatus nginx mongod fail2ban +``` + +### Deployment +```bash +# From local machine +cd ~/projects/tractatus +./scripts/deploy.sh +``` + +--- + +## If Server Is Compromised Again + +### RECOMMENDED: Complete Reinstall + +Based on experience with recurring compromises, **complete reinstall is recommended** over partial cleanup. + +#### Step 1: Backup Data (if accessible) +```bash +# From local machine - export current database +mongodump --db tractatus_dev --out ~/tractatus-backup-$(date +%Y%m%d) +``` + +#### Step 2: Reinstall via OVH Manager +1. Log into OVH Manager (https://www.ovh.com/manager/) +2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da +3. Click "Reinstall" button +4. Select: Ubuntu 22.04 LTS +5. Wait for completion (~10 minutes) +6. Check email for new root credentials + +#### Step 3: Initial System Setup +```bash +# SSH as root with new credentials +ssh root@91.134.240.3 + +# Update system +apt update && apt upgrade -y + +# Create ubuntu user +adduser ubuntu +usermod -aG sudo ubuntu + +# Setup SSH keys +mkdir -p /home/ubuntu/.ssh +cat > /home/ubuntu/.ssh/authorized_keys << 'EOF' +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy +EOF +chown -R ubuntu:ubuntu /home/ubuntu/.ssh +chmod 700 /home/ubuntu/.ssh +chmod 600 /home/ubuntu/.ssh/authorized_keys + +# Harden SSH +cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF' +PasswordAuthentication no +PermitRootLogin no +MaxAuthTries 3 +EOF +systemctl restart sshd + +# Install firewall +apt install -y ufw +ufw default deny incoming +ufw default allow outgoing +ufw allow 22/tcp +ufw allow 80/tcp +ufw allow 443/tcp +ufw --force enable + +# Install fail2ban +apt install -y fail2ban +systemctl enable fail2ban +systemctl start fail2ban +``` + +#### Step 4: Install Application Stack +```bash +# Node.js +curl -fsSL https://deb.nodesource.com/setup_20.x | bash - +apt install -y nodejs + +# MongoDB +curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor +echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list +apt update +apt install -y mongodb-org +systemctl enable mongod +systemctl start mongod + +# nginx +apt install -y nginx +systemctl enable nginx + +# certbot +apt install -y certbot python3-certbot-nginx + +# Security tools +apt install -y rkhunter chkrootkit +``` + +#### Step 5: Create Application Directory +```bash +mkdir -p /var/www/tractatus +chown ubuntu:ubuntu /var/www/tractatus +``` + +#### Step 6: Deploy Application (from local machine) +```bash +cd ~/projects/tractatus +./scripts/deploy.sh +``` + +#### Step 7: Create Production .env (on VPS) +```bash +cat > /var/www/tractatus/.env << 'EOF' +NODE_ENV=production +PORT=9000 +MONGODB_URI=mongodb://127.0.0.1:27017 +MONGODB_DB=tractatus +APP_URL=https://agenticgovernance.digital +SESSION_SECRET= +ANTHROPIC_API_KEY= +# Add other keys as needed +EOF +chmod 600 /var/www/tractatus/.env +``` + +#### Step 8: Install Dependencies (on VPS) +```bash +cd /var/www/tractatus +npm install +npm install @anthropic-ai/sdk # May need explicit install +``` + +#### Step 9: Create Systemd Service +```bash +cat > /etc/systemd/system/tractatus.service << 'EOF' +[Unit] +Description=Tractatus - Agentic Governance Framework +After=network.target mongod.service + +[Service] +Type=simple +User=ubuntu +WorkingDirectory=/var/www/tractatus +ExecStart=/usr/bin/node src/server.js +Restart=on-failure +RestartSec=10 +Environment=NODE_ENV=production + +[Install] +WantedBy=multi-user.target +EOF + +systemctl daemon-reload +systemctl enable tractatus +systemctl start tractatus +``` + +#### Step 10: Configure nginx +```bash +cat > /etc/nginx/sites-available/tractatus << 'EOF' +server { + listen 80; + server_name agenticgovernance.digital; + + location / { + proxy_pass http://127.0.0.1:9000; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection 'upgrade'; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_cache_bypass $http_upgrade; + } +} +EOF + +ln -sf /etc/nginx/sites-available/tractatus /etc/nginx/sites-enabled/ +rm -f /etc/nginx/sites-enabled/default +nginx -t && systemctl reload nginx +``` + +#### Step 11: SSL Certificate +```bash +certbot --nginx -d agenticgovernance.digital +``` + +#### Step 12: Restore Database +```bash +# From local machine +mongodump --db tractatus_dev --out ~/tractatus-backup +rsync -avz ~/tractatus-backup/ ubuntu@vps-93a693da.vps.ovh.net:/tmp/tractatus-backup/ + +# On VPS +mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/ +``` + +#### Step 13: Setup Admin User +```bash +cd /var/www/tractatus +node scripts/fix-admin-user.js +node scripts/seed-projects.js +``` + +#### Step 14: Daily Security Monitoring +```bash +cat > /usr/local/bin/daily-security-check.sh << 'EOF' +#!/bin/bash +LOG=/var/log/daily-security-check.log +echo "=== Security Check $(date) ===" >> $LOG + +# Check for banned software +if command -v pm2 &> /dev/null; then + echo "CRITICAL: PM2 DETECTED" >> $LOG +fi +if command -v docker &> /dev/null; then + echo "CRITICAL: Docker DETECTED" >> $LOG +fi +if [ -d "/home/ubuntu/.pm2" ]; then + echo "CRITICAL: .pm2 directory exists" >> $LOG +fi + +# Check unusual processes +ps aux | grep -E "(pm2|docker|umami)" | grep -v grep >> $LOG + +echo "Check complete" >> $LOG +EOF + +chmod +x /usr/local/bin/daily-security-check.sh +echo "0 3 * * * root /usr/local/bin/daily-security-check.sh" > /etc/cron.d/security-check +``` + +--- + +## BANNED Software + +**NEVER install these on this server:** + +| Software | Reason | +|----------|--------| +| PM2 | Malware persistence vector (Exodus botnet) | +| Docker | Attack vector (Umami compromise) | +| PostgreSQL | Only for Umami, not needed | +| Any analytics containers | Attack surface | + +--- + +## Attack History + +| Date | Attack Type | Root Cause | Resolution | +|------|-------------|------------|------------| +| 2025-12-09 | 83Kpps DNS flood | Docker/Umami Exodus botnet | Partial cleanup (FAILED) | +| 2026-01-18 | 171Kpps UDP flood | PM2 persistence | Partial cleanup (FAILED) | +| 2026-01-18 | 44Kpps UDP flood | PM2 persistence | Complete reinstall (SUCCESS) | + +--- + +## Malware Indicators + +**Exodus Botnet (Mirai variant):** +- C2 Server: 196.251.100.191 (South Africa) +- Persistence: PM2 process manager +- Attack types: UDP flood, DNS flood +- Target ports: Various (9007, 80, 53) + +**Signs of Compromise:** +- PM2 installed or .pm2 directory exists +- Docker installed +- High outbound UDP traffic +- OVH anti-hack alerts +- Unknown processes in `ps aux` + +--- + +## Verification Checklist + +After recovery, verify: + +- [ ] SSH works with key authentication +- [ ] `sudo systemctl status tractatus nginx mongod fail2ban` all active +- [ ] Website responds: `curl -I https://agenticgovernance.digital/` +- [ ] No PM2: `which pm2` returns nothing +- [ ] No Docker: `which docker` returns nothing +- [ ] No .pm2 directory: `ls -la /home/ubuntu/.pm2` not found +- [ ] UFW active: `sudo ufw status` +- [ ] fail2ban running: `sudo fail2ban-client status` + +--- + +## Related Documents + +- `docs/INCIDENT_RECOVERY_2026-01-19.md` - Full incident timeline and recovery details +- `docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md` - Original remediation plan +- `docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md` - Issues with OVH rescue mode + +--- + +## OVH Contact + +- Manager: https://www.ovh.com/manager/ +- Server: vps-93a693da.vps.ovh.net +- IP: 91.134.240.3 diff --git a/docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md b/docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md new file mode 100644 index 00000000..bafcf0e1 --- /dev/null +++ b/docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md @@ -0,0 +1,109 @@ +# OVH Rescue Mode - Known Issues and Workarounds + +**Created:** 2026-01-20 +**Updated:** 2026-01-20 +**VPS:** vps-93a693da.vps.ovh.net (91.134.240.3) +**Status:** RESOLVED - Complete reinstall successful + +--- + +## RESOLUTION (2026-01-20) + +After multiple failed partial cleanup attempts, the **complete reinstall option** was used: + +1. User accessed OVH Manager +2. Selected "Reinstall" option for the VPS +3. Chose Ubuntu 22.04 LTS +4. Fresh system was provisioned with new root credentials +5. Application was redeployed from clean local source + +**This is the recommended approach** for future compromises rather than attempting rescue mode cleanup. + +--- + +## FAILED APPROACH (Do NOT repeat) + +The following sequence has failed multiple times across sessions: + +1. ❌ `ssh root@91.134.240.3` - Fails with host key warning +2. ❌ `ssh-keygen -R '91.134.240.3'` - Removes old key +3. ❌ `ssh root@91.134.240.3` - Fails with "Too many authentication failures" +4. ❌ `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@91.134.240.3` - Connection closed after password prompt +5. ❌ Checking OVH secret link - Password appears expired or invalid +6. ❌ Re-requesting rescue mode - Unclear if this generates working credentials + +**Result:** Unable to SSH into rescue mode despite VPS showing "In rescue" status in OVH Manager. + +--- + +## ROOT CAUSE HYPOTHESES + +1. **Password expiration:** OVH rescue mode passwords expire after 30 days (or 7 days after first use). The secret link may be stale. + +2. **Rescue mode session expired:** The rescue environment itself may have timed out even though status shows "In rescue". + +3. **OVH anti-hack blocking SSH:** The anti-hack system that triggered rescue mode may also be blocking SSH access. + +4. **Incorrect password retrieval:** The OVH secret link format may have changed, or we're using the wrong credential source. + +--- + +## WHAT TO TRY NEXT SESSION + +### Option A: OVH KVM/VNC Console +- OVH Manager may have a web-based console (KVM/VNC/noVNC) +- This bypasses SSH entirely +- Look for "KVM" or "Console" or "VNC" button in OVH Manager + +### Option B: Fresh Rescue Mode Request +- In OVH Manager, explicitly DISABLE rescue mode first +- Wait for normal boot to complete (or fail) +- Then RE-ENABLE rescue mode +- This should generate a truly fresh password via email + +### Option C: Contact OVH Support +- If SSH continues to fail, open a support ticket +- Reference: Anti-hack incidents on this VPS +- Ask them to verify rescue mode SSH is accessible + +### Option D: Skip Rescue, Direct Reinstall +- OVH Manager may allow direct OS reinstall WITHOUT needing rescue mode access +- This would skip backup entirely but get a clean system +- Data loss acceptable if MongoDB backup exists elsewhere + +--- + +## QUESTIONS TO ASK USER + +1. Is there a KVM/VNC console option in OVH Manager? +2. Can you try disabling rescue mode and re-enabling it? +3. Do you have any MongoDB backups stored locally that we could restore from? +4. Is the OVH password coming from email or from the secret link? + +--- + +## OVH MANAGER NAVIGATION + +Based on OVH interface (may vary): + +``` +OVH Manager → Bare Metal Cloud → VPS → [Your VPS] → + - Dashboard: Shows status (rescue/normal) + - Boot: Shows boot mode, option to change + - Console/KVM: Web-based terminal (if available) + - Reinstall: Direct OS reinstall option +``` + +--- + +## SUCCESSFUL ACCESS REQUIREMENTS + +Before proceeding with backup/reinstall, we need ONE of: +- [ ] Working SSH access to rescue mode +- [ ] Working KVM/VNC console access +- [ ] Confirmation that direct reinstall (no backup) is acceptable + +--- + +**Last Updated:** 2026-01-20 +**Status:** BLOCKED - Cannot access rescue mode via SSH diff --git a/docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md b/docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md new file mode 100644 index 00000000..30fd43e8 --- /dev/null +++ b/docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md @@ -0,0 +1,465 @@ +# Comprehensive Remediation Plan: agenticgovernance.digital +## Date: 2026-01-19 +## Status: EXECUTED SUCCESSFULLY (2026-01-20) + +> **UPDATE 2026-01-20:** This remediation plan was successfully executed. Complete VPS reinstallation performed, all systems restored, security hardening applied. See `docs/INCIDENT_RECOVERY_2026-01-19.md` for full details. + +--- + +## Executive Summary + +The agenticgovernance.digital VPS (vps-93a693da.vps.ovh.net) has been compromised **three times** by the same botnet infrastructure. Each prior "recovery" was incomplete, leaving persistence mechanisms that allowed reinfection. + +**Recommendation: COMPLETE REINSTALL** + +Based on security industry best practices and the pattern of recurring compromise, partial cleanup is no longer viable. A complete OS reinstall is the only way to guarantee all malware is removed. + +--- + +## Attack History Analysis + +### Timeline + +| Date | Attack | Root Cause | Recovery Status | +|------|--------|------------|-----------------| +| 2025-12-09 | 83Kpps DNS flood (Exodus botnet via Docker/Umami) | Docker container compromise | **INCOMPLETE** - PM2, umami-deployment, cron jobs left | +| 2026-01-18 13:57 | 171Kpps UDP flood to 15.184.38.247:9007 | PM2 resurrected botnet processes | **INCOMPLETE** - SSH broken post-recovery | +| 2026-01-18 23:44 | 44Kpps UDP flood to 171.225.223.4:80 | Continued PM2 persistence | Server in rescue mode | +| 2026-01-19 (today) | OVH anti-hack triggered again | Unknown - likely same persistence | **CURRENT INCIDENT** | + +### What Was Missed in Each Recovery + +**December 2025 Recovery:** +- Docker removed ✓ +- PM2 process manager NOT removed ✗ +- `/home/ubuntu/umami-deployment/` NOT removed ✗ +- Ubuntu crontab NOT cleared ✗ +- PostgreSQL service NOT disabled ✗ + +**January 19 Recovery (earlier today):** +- PM2 removed ✓ +- umami-deployment removed ✓ +- PostgreSQL disabled ✓ +- **But server is in rescue mode AGAIN** = something else was missed + +### Malware Profile + +**Name:** Exodus Botnet (Mirai variant) +**C2 Server:** 196.251.100.191 (South Africa) +**Capabilities:** +- Multi-architecture binaries (x86, x86_64, ARM, MIPS, etc.) +- UDP/DNS flood attacks +- Self-replicating via PM2 process manager +- Persistence through system services + +**PM2 as Persistence Mechanism:** +- PM2's `resurrect` feature auto-restarts saved processes on boot +- Used by modern botnets like NodeCordRAT and Tsundere (2025) +- Survives manual process termination +- Creates systemd service (`pm2-ubuntu.service`) + +--- + +## Why Partial Cleanup Has Failed + +### Problem 1: Unknown Persistence Mechanisms +Each cleanup identified SOME persistence mechanisms but missed others. There may be: +- Modified system binaries (rootkits) +- Kernel modules +- Hidden cron jobs in unexpected locations +- Modified init scripts +- SSH backdoors (could explain broken SSH) + +### Problem 2: No Baseline for Comparison +Without knowing exactly what should exist on a clean system, we cannot verify complete removal. + +### Problem 3: Forensic Limitations in Rescue Mode +Rescue mode provides limited visibility into: +- Runtime state of malware +- Memory-resident components +- Kernel-level modifications + +### Expert Consensus +> "Reinstalling a computer after it has been compromised can be a painstaking process, but it is the best way to be certain that everything an attacker left behind has been found." - UC Berkeley Information Security Office + +> "Rootkits are difficult to remove, and the only 100% sure fire way to remove a rootkit from a device that has been infected is to wipe the device and reinstall the operating system." + +--- + +## Recommended Solution: Complete Reinstall + +### Phase 1: Data Backup (From Rescue Mode) + +**CRITICAL:** Before reinstalling, back up essential data: + +```bash +# 1. Boot into rescue mode via OVH Manager +# 2. Mount main disk +mount /dev/sdb1 /mnt/vps + +# 3. Create backup directory +mkdir -p /tmp/tractatus-backup + +# 4. Backup application code (verify hashes later) +tar -czf /tmp/tractatus-backup/app.tar.gz /mnt/vps/var/www/tractatus/ + +# 5. Backup MongoDB data +tar -czf /tmp/tractatus-backup/mongodb.tar.gz /mnt/vps/var/lib/mongodb/ + +# 6. Backup SSL certificates +tar -czf /tmp/tractatus-backup/ssl.tar.gz /mnt/vps/etc/letsencrypt/ + +# 7. Backup nginx config (for reference, will recreate) +cp /mnt/vps/etc/nginx/sites-available/tractatus /tmp/tractatus-backup/ + +# 8. Download backups to local machine +scp -r root@RESCUE_IP:/tmp/tractatus-backup/ ~/tractatus-recovery/ +``` + +**DO NOT BACKUP:** +- `/home/ubuntu/.pm2/` (malware) +- `/home/ubuntu/umami-deployment/` (malware) +- Any executables (may be compromised) +- `/var/lib/docker/` (attack vector) + +### Phase 2: VPS Reinstallation + +1. **Via OVH Manager:** + - Navigate to VPS management + - Select "Reinstall" option + - Choose: Ubuntu 22.04 LTS (or latest LTS) + - Wait for completion (~10 minutes) + +2. **Retrieve New Root Password:** + - Check email for new credentials + - Or use OVH password reset function + +### Phase 3: Fresh System Setup + +**Initial SSH Access:** +```bash +ssh root@91.134.240.3 +``` + +**Step 1: System Updates** +```bash +apt update && apt upgrade -y +``` + +**Step 2: Create Non-Root User** +```bash +adduser ubuntu +usermod -aG sudo ubuntu +``` + +**Step 3: SSH Hardening** +```bash +# Add authorized keys +mkdir -p /home/ubuntu/.ssh +cat > /home/ubuntu/.ssh/authorized_keys << 'EOF' +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ... theflow@the-flow +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy +EOF + +chown -R ubuntu:ubuntu /home/ubuntu/.ssh +chmod 700 /home/ubuntu/.ssh +chmod 600 /home/ubuntu/.ssh/authorized_keys + +# Harden SSH config +cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF' +PasswordAuthentication no +PermitRootLogin no +MaxAuthTries 3 +LoginGraceTime 20 +ClientAliveInterval 300 +ClientAliveCountMax 2 +EOF + +systemctl restart sshd +``` + +**Step 4: Firewall Configuration** +```bash +apt install -y ufw +ufw default deny incoming +ufw default allow outgoing +ufw allow 22/tcp comment 'SSH' +ufw allow 80/tcp comment 'HTTP' +ufw allow 443/tcp comment 'HTTPS' +# Block Docker ports (never needed) +ufw deny 2375/tcp comment 'Block Docker API' +ufw deny 2376/tcp comment 'Block Docker TLS' +ufw enable +``` + +**Step 5: Intrusion Prevention** +```bash +apt install -y fail2ban +cat > /etc/fail2ban/jail.local << 'EOF' +[sshd] +enabled = true +maxretry = 3 +bantime = 24h +findtime = 1h +EOF + +systemctl enable fail2ban +systemctl start fail2ban +``` + +**Step 6: Install Required Software** +```bash +# Node.js (via NodeSource) +curl -fsSL https://deb.nodesource.com/setup_20.x | bash - +apt install -y nodejs + +# MongoDB +curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor +echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list +apt update +apt install -y mongodb-org +systemctl enable mongod +systemctl start mongod + +# nginx +apt install -y nginx +systemctl enable nginx + +# certbot for SSL +apt install -y certbot python3-certbot-nginx +``` + +### Phase 4: Application Deployment + +**Step 1: Prepare Application Directory** +```bash +mkdir -p /var/www/tractatus +chown ubuntu:ubuntu /var/www/tractatus +``` + +**Step 2: Deploy from CLEAN Local Source** +```bash +# From local machine - deploy ONLY from verified clean source +cd ~/projects/tractatus +./scripts/deploy.sh --full +``` + +**Step 3: Restore MongoDB Data** +```bash +# If data integrity is verified +mongorestore --db tractatus ~/tractatus-recovery/mongodb/tractatus/ +``` + +**Step 4: SSL Certificate** +```bash +certbot --nginx -d agenticgovernance.digital +``` + +**Step 5: Create Systemd Service** +```bash +cat > /etc/systemd/system/tractatus.service << 'EOF' +[Unit] +Description=Tractatus Application +After=network.target mongod.service + +[Service] +Type=simple +User=ubuntu +WorkingDirectory=/var/www/tractatus +ExecStart=/usr/bin/node src/server.js +Restart=on-failure +RestartSec=10 +Environment=NODE_ENV=production + +[Install] +WantedBy=multi-user.target +EOF + +systemctl daemon-reload +systemctl enable tractatus +systemctl start tractatus +``` + +### Phase 5: Monitoring Setup + +**Step 1: Log Rotation for MongoDB** +```bash +cat > /etc/logrotate.d/mongodb << 'EOF' +/var/log/mongodb/*.log { + daily + rotate 7 + compress + missingok + notifempty + sharedscripts + postrotate + /bin/kill -SIGUSR1 $(cat /var/lib/mongodb/mongod.lock 2>/dev/null) 2>/dev/null || true + endscript +} +EOF +``` + +**Step 2: Install Rootkit Scanner** +```bash +apt install -y rkhunter chkrootkit lynis + +# Run initial scan +rkhunter --update +rkhunter --check --skip-keypress +chkrootkit +lynis audit system +``` + +**Step 3: Monitoring Script** +```bash +cat > /usr/local/bin/security-check.sh << 'EOF' +#!/bin/bash +# Daily security check +LOG=/var/log/security-check.log +echo "=== Security Check $(date) ===" >> $LOG + +# Check for unauthorized services +systemctl list-units --type=service --state=running | grep -v "systemd\|ssh\|nginx\|mongod\|tractatus\|fail2ban\|ufw" >> $LOG + +# Check for unusual network connections +netstat -tlnp | grep -v "127.0.0.1\|mongodb\|node\|nginx" >> $LOG + +# Check for PM2 (should never exist) +if command -v pm2 &> /dev/null; then + echo "WARNING: PM2 DETECTED - SHOULD NOT EXIST" >> $LOG +fi + +# Check for Docker (should never exist) +if command -v docker &> /dev/null; then + echo "WARNING: DOCKER DETECTED - SHOULD NOT EXIST" >> $LOG +fi +EOF +chmod +x /usr/local/bin/security-check.sh + +# Add to cron +echo "0 6 * * * root /usr/local/bin/security-check.sh" > /etc/cron.d/security-check +``` + +--- + +## What Must NEVER Exist on This Server + +| Component | Reason | +|-----------|--------| +| PM2 | Used for malware persistence | +| Docker | Attack vector (Umami compromise) | +| PostgreSQL | Only for Umami, not needed | +| Any analytics containers | Attack surface | +| Node packages outside app | Potential supply chain risk | + +**Verification Script:** +```bash +#!/bin/bash +ALERT=0 +if command -v pm2 &> /dev/null; then echo "ALERT: PM2 exists"; ALERT=1; fi +if command -v docker &> /dev/null; then echo "ALERT: Docker exists"; ALERT=1; fi +if [ -d "/home/ubuntu/.pm2" ]; then echo "ALERT: .pm2 directory exists"; ALERT=1; fi +if [ -d "/home/ubuntu/umami-deployment" ]; then echo "ALERT: umami-deployment exists"; ALERT=1; fi +if systemctl is-enabled postgresql &> /dev/null; then echo "ALERT: PostgreSQL enabled"; ALERT=1; fi +if [ $ALERT -eq 0 ]; then echo "Server is clean"; fi +``` + +--- + +## Post-Recovery Verification Checklist + +- [ ] SSH access works with key authentication +- [ ] Password authentication is disabled +- [ ] fail2ban is running and banning IPs +- [ ] UFW is enabled with correct rules +- [ ] nginx is serving the site +- [ ] tractatus service is running +- [ ] MongoDB is running and bound to 127.0.0.1 +- [ ] SSL certificate is valid +- [ ] No PM2 installed +- [ ] No Docker installed +- [ ] No PostgreSQL installed +- [ ] rkhunter scan is clean +- [ ] chkrootkit scan is clean +- [ ] Log rotation is configured +- [ ] Daily security check cron is active + +--- + +## Credentials to Rotate + +After reinstall, rotate all credentials: + +1. **MongoDB admin password** (if using authentication) +2. **Application secrets** in `.env` +3. **Session secrets** +4. **Any API keys** + +**Important:** Change passwords from a DIFFERENT machine, not the compromised server. + +--- + +## Long-Term Prevention + +1. **Never install Docker** - not needed for this application +2. **Never install PM2** - use systemd only +3. **Weekly security scans** - rkhunter, chkrootkit +4. **Monitor outbound traffic** - alert on unexpected destinations +5. **Keep system updated** - enable unattended-upgrades +6. **Review SSH logs weekly** - check for brute force patterns + +--- + +## OVH Support Communication Template + +``` +Subject: Request to restore VPS to normal mode after reinstallation + +Reference: [ref=1.2378332d] +Server: vps-93a693da.vps.ovh.net + +We have identified the cause of the anti-hack triggers: +- Compromised Docker container running botnet malware +- PM2 process manager persisting malicious processes + +We have completed a full OS reinstall and implemented: +- Hardened SSH configuration (key-only, no root) +- UFW firewall with minimal open ports +- fail2ban for intrusion prevention +- Removal of Docker and PM2 + +Please restore the VPS to normal boot mode. + +Thank you. +``` + +--- + +## Timeline Estimate + +| Phase | Duration | +|-------|----------| +| Backup data | 30 min | +| VPS reinstall | 10 min | +| System setup | 45 min | +| Application deployment | 30 min | +| Verification | 30 min | +| **Total** | ~2.5 hours | + +--- + +## References + +- [Mirai Malware Behavior & Mitigation (2025)](https://echoxec.com/mirai-malware-in-2025-variant-behavior-exploit-chains-and-mitigation-insights) +- [PM2 Malware Persistence (ThreatLabz)](https://www.zscaler.com/blogs/security-research/malicious-npm-packages-deliver-nodecordrat) +- [Linux Rootkit Detection Tools](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/) +- [OVH VPS Rescue Mode Guide](https://help.ovhcloud.com/csm/en-vps-rescue?id=kb_article_view&sysparm_article=KB0047656) +- [UC Berkeley - Reinstalling Compromised Systems](https://security.berkeley.edu/education-awareness/reinstalling-your-compromised-computer) +- [Common Malware Persistence Mechanisms](https://www.infosecinstitute.com/resources/malware-analysis/common-malware-persistence-mechanisms/) + +--- + +**Document Author:** Claude Code +**Date:** 2026-01-19 +**Status:** Ready for implementation +**Next Action:** User decision on proceeding with complete reinstall