docs: Complete VPS recovery documentation and attack reference
- Update INCIDENT_RECOVERY_2026-01-19.md with complete recovery status - Create VPS_RECOVERY_REFERENCE.md with step-by-step recovery guide - Update remediation plan to show executed status - Update OVH rescue mode doc with resolution notes Documents the successful complete reinstall approach after multiple failed partial cleanup attempts. Includes attack indicators, banned software list, and verification checklist for future incidents. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
TheFlow
parent
81cf409ce7
commit
d505f2d712
4 changed files with 1024 additions and 8 deletions
|
|
@ -1,12 +1,15 @@
|
|||
# Incident Recovery Report - 2026-01-19
|
||||
# Incident Recovery Report - 2026-01-19/20
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Status:** PARTIAL RECOVERY
|
||||
**Status:** COMPLETE RECOVERY (Updated 2026-01-20)
|
||||
- Website: UP (https://agenticgovernance.digital/ responds HTTP 200)
|
||||
- SSH Access: BROKEN (connection closes after authentication)
|
||||
- Malware: REMOVED (PM2 and umami-deployment deleted)
|
||||
- Root Cause: PM2 process manager running botnet malware
|
||||
- SSH Access: WORKING (via fresh VPS reinstall)
|
||||
- Malware: ELIMINATED (complete OS reinstall)
|
||||
- Application: FULLY RESTORED
|
||||
- Database: MIGRATED from local backup (134 documents)
|
||||
- SSL: VALID (Let's Encrypt, expires April 2026)
|
||||
- Root Cause: PM2 process manager running Exodus botnet malware
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -249,6 +252,108 @@ This incident represents multiple failures:
|
|||
|
||||
---
|
||||
|
||||
**Report Date:** 2026-01-19
|
||||
**Status:** PARTIAL RECOVERY - Website up, SSH broken
|
||||
**Next Action:** Re-enter rescue mode to fix SSH access
|
||||
---
|
||||
|
||||
## COMPLETE RECOVERY - 2026-01-20
|
||||
|
||||
### What Was Done
|
||||
|
||||
After multiple failed partial cleanup attempts, the decision was made to perform a **complete VPS reinstallation** as recommended in the remediation plan.
|
||||
|
||||
#### Phase 1: VPS Reinstallation via OVH Manager
|
||||
- User initiated complete OS reinstall from OVH Manager
|
||||
- Fresh Ubuntu installation with new credentials
|
||||
- All malware completely eliminated by full disk wipe
|
||||
|
||||
#### Phase 2: System Setup
|
||||
```bash
|
||||
# Security tools
|
||||
apt install -y fail2ban rkhunter chkrootkit
|
||||
|
||||
# Daily security monitoring script
|
||||
/usr/local/bin/daily-security-check.sh
|
||||
|
||||
# MongoDB with log rotation
|
||||
apt install -y mongodb-org
|
||||
# Configured logrotate for /var/log/mongodb/
|
||||
```
|
||||
|
||||
#### Phase 3: Application Deployment
|
||||
1. Created `/var/www/tractatus/` directory
|
||||
2. Created production `.env` file with NODE_ENV=production
|
||||
3. Deployed application via rsync from local (CLEAN source)
|
||||
4. Installed dependencies including `@anthropic-ai/sdk`
|
||||
5. Created systemd service (`/etc/systemd/system/tractatus.service`)
|
||||
6. Configured nginx with SSL reverse proxy
|
||||
|
||||
#### Phase 4: SSL Certificate
|
||||
```bash
|
||||
certbot --nginx -d agenticgovernance.digital
|
||||
# Certificate valid until April 2026
|
||||
```
|
||||
|
||||
#### Phase 5: Database Migration
|
||||
```bash
|
||||
# Local: Export database
|
||||
mongodump --db tractatus_dev --out ~/tractatus-backup
|
||||
|
||||
# Transfer to VPS
|
||||
rsync -avz ~/tractatus-backup/ ubuntu@vps:/tmp/tractatus-backup/
|
||||
|
||||
# VPS: Import to production
|
||||
mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/
|
||||
# Result: 134 documents + 12 blog posts restored
|
||||
```
|
||||
|
||||
#### Phase 6: Admin Setup
|
||||
```bash
|
||||
node scripts/fix-admin-user.js
|
||||
node scripts/seed-projects.js
|
||||
```
|
||||
|
||||
### Final System State (2026-01-20)
|
||||
|
||||
**Services Running:**
|
||||
- `tractatus.service` - Node.js application (port 9000)
|
||||
- `nginx.service` - Web server with SSL
|
||||
- `mongod.service` - MongoDB database
|
||||
- `fail2ban.service` - Intrusion prevention
|
||||
|
||||
**Services Explicitly BANNED:**
|
||||
- PM2 - Never install (malware persistence vector)
|
||||
- Docker - Never install (attack vector)
|
||||
- PostgreSQL - Not needed (was for Umami)
|
||||
|
||||
**Security Measures:**
|
||||
- SSH key authentication only (password disabled)
|
||||
- UFW firewall enabled
|
||||
- fail2ban active
|
||||
- Daily security scan at 3 AM UTC (`/usr/local/bin/daily-security-check.sh`)
|
||||
- rkhunter and chkrootkit installed
|
||||
|
||||
**Post-Recovery Improvements (same session):**
|
||||
- Removed all Umami analytics references from codebase (29 HTML files)
|
||||
- Deleted `/public/js/components/umami-tracker.js`
|
||||
- Updated privacy policy to reflect "No Analytics"
|
||||
- Added Research Papers section to landing page
|
||||
- Created `/korero-counter-arguments.html` page
|
||||
- Fixed Tailwind CSS to include emerald gradient classes
|
||||
|
||||
### Verification Completed
|
||||
|
||||
- [x] SSH access works with key authentication
|
||||
- [x] Website responds correctly (HTTP 200)
|
||||
- [x] SSL certificate valid
|
||||
- [x] MongoDB running and accessible
|
||||
- [x] All documents migrated (134 total)
|
||||
- [x] Blog posts visible (12 posts)
|
||||
- [x] Admin user functional
|
||||
- [x] No PM2 installed
|
||||
- [x] No Docker installed
|
||||
- [x] Daily security scan configured
|
||||
|
||||
---
|
||||
|
||||
**Report Date:** 2026-01-19 (initial) / 2026-01-20 (complete recovery)
|
||||
**Status:** COMPLETE RECOVERY - All systems operational
|
||||
**Next Action:** Resume normal development (/community project)
|
||||
|
|
|
|||
337
docs/VPS_RECOVERY_REFERENCE.md
Normal file
337
docs/VPS_RECOVERY_REFERENCE.md
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
# VPS Recovery Reference - agenticgovernance.digital
|
||||
|
||||
**Last Updated:** 2026-01-20
|
||||
**VPS:** vps-93a693da.vps.ovh.net
|
||||
**IP:** 91.134.240.3
|
||||
**Provider:** OVH
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### SSH Access
|
||||
```bash
|
||||
# Primary access (from theflow's machine)
|
||||
ssh -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net
|
||||
|
||||
# If host key changed (after reinstall)
|
||||
ssh-keygen -f '/home/theflow/.ssh/known_hosts' -R 'vps-93a693da.vps.ovh.net'
|
||||
ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net "echo 'Connected'"
|
||||
```
|
||||
|
||||
### Service Management
|
||||
```bash
|
||||
# Check tractatus service
|
||||
sudo systemctl status tractatus
|
||||
|
||||
# Restart tractatus
|
||||
sudo systemctl restart tractatus
|
||||
|
||||
# Check all relevant services
|
||||
sudo systemctl status tractatus nginx mongod fail2ban
|
||||
```
|
||||
|
||||
### Deployment
|
||||
```bash
|
||||
# From local machine
|
||||
cd ~/projects/tractatus
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## If Server Is Compromised Again
|
||||
|
||||
### RECOMMENDED: Complete Reinstall
|
||||
|
||||
Based on experience with recurring compromises, **complete reinstall is recommended** over partial cleanup.
|
||||
|
||||
#### Step 1: Backup Data (if accessible)
|
||||
```bash
|
||||
# From local machine - export current database
|
||||
mongodump --db tractatus_dev --out ~/tractatus-backup-$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
#### Step 2: Reinstall via OVH Manager
|
||||
1. Log into OVH Manager (https://www.ovh.com/manager/)
|
||||
2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da
|
||||
3. Click "Reinstall" button
|
||||
4. Select: Ubuntu 22.04 LTS
|
||||
5. Wait for completion (~10 minutes)
|
||||
6. Check email for new root credentials
|
||||
|
||||
#### Step 3: Initial System Setup
|
||||
```bash
|
||||
# SSH as root with new credentials
|
||||
ssh root@91.134.240.3
|
||||
|
||||
# Update system
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# Create ubuntu user
|
||||
adduser ubuntu
|
||||
usermod -aG sudo ubuntu
|
||||
|
||||
# Setup SSH keys
|
||||
mkdir -p /home/ubuntu/.ssh
|
||||
cat > /home/ubuntu/.ssh/authorized_keys << 'EOF'
|
||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy
|
||||
EOF
|
||||
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
|
||||
chmod 700 /home/ubuntu/.ssh
|
||||
chmod 600 /home/ubuntu/.ssh/authorized_keys
|
||||
|
||||
# Harden SSH
|
||||
cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF'
|
||||
PasswordAuthentication no
|
||||
PermitRootLogin no
|
||||
MaxAuthTries 3
|
||||
EOF
|
||||
systemctl restart sshd
|
||||
|
||||
# Install firewall
|
||||
apt install -y ufw
|
||||
ufw default deny incoming
|
||||
ufw default allow outgoing
|
||||
ufw allow 22/tcp
|
||||
ufw allow 80/tcp
|
||||
ufw allow 443/tcp
|
||||
ufw --force enable
|
||||
|
||||
# Install fail2ban
|
||||
apt install -y fail2ban
|
||||
systemctl enable fail2ban
|
||||
systemctl start fail2ban
|
||||
```
|
||||
|
||||
#### Step 4: Install Application Stack
|
||||
```bash
|
||||
# Node.js
|
||||
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
|
||||
apt install -y nodejs
|
||||
|
||||
# MongoDB
|
||||
curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
|
||||
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list
|
||||
apt update
|
||||
apt install -y mongodb-org
|
||||
systemctl enable mongod
|
||||
systemctl start mongod
|
||||
|
||||
# nginx
|
||||
apt install -y nginx
|
||||
systemctl enable nginx
|
||||
|
||||
# certbot
|
||||
apt install -y certbot python3-certbot-nginx
|
||||
|
||||
# Security tools
|
||||
apt install -y rkhunter chkrootkit
|
||||
```
|
||||
|
||||
#### Step 5: Create Application Directory
|
||||
```bash
|
||||
mkdir -p /var/www/tractatus
|
||||
chown ubuntu:ubuntu /var/www/tractatus
|
||||
```
|
||||
|
||||
#### Step 6: Deploy Application (from local machine)
|
||||
```bash
|
||||
cd ~/projects/tractatus
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
#### Step 7: Create Production .env (on VPS)
|
||||
```bash
|
||||
cat > /var/www/tractatus/.env << 'EOF'
|
||||
NODE_ENV=production
|
||||
PORT=9000
|
||||
MONGODB_URI=mongodb://127.0.0.1:27017
|
||||
MONGODB_DB=tractatus
|
||||
APP_URL=https://agenticgovernance.digital
|
||||
SESSION_SECRET=<generate-new-secret>
|
||||
ANTHROPIC_API_KEY=<get-from-credential-vault>
|
||||
# Add other keys as needed
|
||||
EOF
|
||||
chmod 600 /var/www/tractatus/.env
|
||||
```
|
||||
|
||||
#### Step 8: Install Dependencies (on VPS)
|
||||
```bash
|
||||
cd /var/www/tractatus
|
||||
npm install
|
||||
npm install @anthropic-ai/sdk # May need explicit install
|
||||
```
|
||||
|
||||
#### Step 9: Create Systemd Service
|
||||
```bash
|
||||
cat > /etc/systemd/system/tractatus.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Tractatus - Agentic Governance Framework
|
||||
After=network.target mongod.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ubuntu
|
||||
WorkingDirectory=/var/www/tractatus
|
||||
ExecStart=/usr/bin/node src/server.js
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
Environment=NODE_ENV=production
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable tractatus
|
||||
systemctl start tractatus
|
||||
```
|
||||
|
||||
#### Step 10: Configure nginx
|
||||
```bash
|
||||
cat > /etc/nginx/sites-available/tractatus << 'EOF'
|
||||
server {
|
||||
listen 80;
|
||||
server_name agenticgovernance.digital;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:9000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection 'upgrade';
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
ln -sf /etc/nginx/sites-available/tractatus /etc/nginx/sites-enabled/
|
||||
rm -f /etc/nginx/sites-enabled/default
|
||||
nginx -t && systemctl reload nginx
|
||||
```
|
||||
|
||||
#### Step 11: SSL Certificate
|
||||
```bash
|
||||
certbot --nginx -d agenticgovernance.digital
|
||||
```
|
||||
|
||||
#### Step 12: Restore Database
|
||||
```bash
|
||||
# From local machine
|
||||
mongodump --db tractatus_dev --out ~/tractatus-backup
|
||||
rsync -avz ~/tractatus-backup/ ubuntu@vps-93a693da.vps.ovh.net:/tmp/tractatus-backup/
|
||||
|
||||
# On VPS
|
||||
mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/
|
||||
```
|
||||
|
||||
#### Step 13: Setup Admin User
|
||||
```bash
|
||||
cd /var/www/tractatus
|
||||
node scripts/fix-admin-user.js
|
||||
node scripts/seed-projects.js
|
||||
```
|
||||
|
||||
#### Step 14: Daily Security Monitoring
|
||||
```bash
|
||||
cat > /usr/local/bin/daily-security-check.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
LOG=/var/log/daily-security-check.log
|
||||
echo "=== Security Check $(date) ===" >> $LOG
|
||||
|
||||
# Check for banned software
|
||||
if command -v pm2 &> /dev/null; then
|
||||
echo "CRITICAL: PM2 DETECTED" >> $LOG
|
||||
fi
|
||||
if command -v docker &> /dev/null; then
|
||||
echo "CRITICAL: Docker DETECTED" >> $LOG
|
||||
fi
|
||||
if [ -d "/home/ubuntu/.pm2" ]; then
|
||||
echo "CRITICAL: .pm2 directory exists" >> $LOG
|
||||
fi
|
||||
|
||||
# Check unusual processes
|
||||
ps aux | grep -E "(pm2|docker|umami)" | grep -v grep >> $LOG
|
||||
|
||||
echo "Check complete" >> $LOG
|
||||
EOF
|
||||
|
||||
chmod +x /usr/local/bin/daily-security-check.sh
|
||||
echo "0 3 * * * root /usr/local/bin/daily-security-check.sh" > /etc/cron.d/security-check
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## BANNED Software
|
||||
|
||||
**NEVER install these on this server:**
|
||||
|
||||
| Software | Reason |
|
||||
|----------|--------|
|
||||
| PM2 | Malware persistence vector (Exodus botnet) |
|
||||
| Docker | Attack vector (Umami compromise) |
|
||||
| PostgreSQL | Only for Umami, not needed |
|
||||
| Any analytics containers | Attack surface |
|
||||
|
||||
---
|
||||
|
||||
## Attack History
|
||||
|
||||
| Date | Attack Type | Root Cause | Resolution |
|
||||
|------|-------------|------------|------------|
|
||||
| 2025-12-09 | 83Kpps DNS flood | Docker/Umami Exodus botnet | Partial cleanup (FAILED) |
|
||||
| 2026-01-18 | 171Kpps UDP flood | PM2 persistence | Partial cleanup (FAILED) |
|
||||
| 2026-01-18 | 44Kpps UDP flood | PM2 persistence | Complete reinstall (SUCCESS) |
|
||||
|
||||
---
|
||||
|
||||
## Malware Indicators
|
||||
|
||||
**Exodus Botnet (Mirai variant):**
|
||||
- C2 Server: 196.251.100.191 (South Africa)
|
||||
- Persistence: PM2 process manager
|
||||
- Attack types: UDP flood, DNS flood
|
||||
- Target ports: Various (9007, 80, 53)
|
||||
|
||||
**Signs of Compromise:**
|
||||
- PM2 installed or .pm2 directory exists
|
||||
- Docker installed
|
||||
- High outbound UDP traffic
|
||||
- OVH anti-hack alerts
|
||||
- Unknown processes in `ps aux`
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After recovery, verify:
|
||||
|
||||
- [ ] SSH works with key authentication
|
||||
- [ ] `sudo systemctl status tractatus nginx mongod fail2ban` all active
|
||||
- [ ] Website responds: `curl -I https://agenticgovernance.digital/`
|
||||
- [ ] No PM2: `which pm2` returns nothing
|
||||
- [ ] No Docker: `which docker` returns nothing
|
||||
- [ ] No .pm2 directory: `ls -la /home/ubuntu/.pm2` not found
|
||||
- [ ] UFW active: `sudo ufw status`
|
||||
- [ ] fail2ban running: `sudo fail2ban-client status`
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- `docs/INCIDENT_RECOVERY_2026-01-19.md` - Full incident timeline and recovery details
|
||||
- `docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md` - Original remediation plan
|
||||
- `docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md` - Issues with OVH rescue mode
|
||||
|
||||
---
|
||||
|
||||
## OVH Contact
|
||||
|
||||
- Manager: https://www.ovh.com/manager/
|
||||
- Server: vps-93a693da.vps.ovh.net
|
||||
- IP: 91.134.240.3
|
||||
109
docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md
Normal file
109
docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
# OVH Rescue Mode - Known Issues and Workarounds
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Updated:** 2026-01-20
|
||||
**VPS:** vps-93a693da.vps.ovh.net (91.134.240.3)
|
||||
**Status:** RESOLVED - Complete reinstall successful
|
||||
|
||||
---
|
||||
|
||||
## RESOLUTION (2026-01-20)
|
||||
|
||||
After multiple failed partial cleanup attempts, the **complete reinstall option** was used:
|
||||
|
||||
1. User accessed OVH Manager
|
||||
2. Selected "Reinstall" option for the VPS
|
||||
3. Chose Ubuntu 22.04 LTS
|
||||
4. Fresh system was provisioned with new root credentials
|
||||
5. Application was redeployed from clean local source
|
||||
|
||||
**This is the recommended approach** for future compromises rather than attempting rescue mode cleanup.
|
||||
|
||||
---
|
||||
|
||||
## FAILED APPROACH (Do NOT repeat)
|
||||
|
||||
The following sequence has failed multiple times across sessions:
|
||||
|
||||
1. ❌ `ssh root@91.134.240.3` - Fails with host key warning
|
||||
2. ❌ `ssh-keygen -R '91.134.240.3'` - Removes old key
|
||||
3. ❌ `ssh root@91.134.240.3` - Fails with "Too many authentication failures"
|
||||
4. ❌ `ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@91.134.240.3` - Connection closed after password prompt
|
||||
5. ❌ Checking OVH secret link - Password appears expired or invalid
|
||||
6. ❌ Re-requesting rescue mode - Unclear if this generates working credentials
|
||||
|
||||
**Result:** Unable to SSH into rescue mode despite VPS showing "In rescue" status in OVH Manager.
|
||||
|
||||
---
|
||||
|
||||
## ROOT CAUSE HYPOTHESES
|
||||
|
||||
1. **Password expiration:** OVH rescue mode passwords expire after 30 days (or 7 days after first use). The secret link may be stale.
|
||||
|
||||
2. **Rescue mode session expired:** The rescue environment itself may have timed out even though status shows "In rescue".
|
||||
|
||||
3. **OVH anti-hack blocking SSH:** The anti-hack system that triggered rescue mode may also be blocking SSH access.
|
||||
|
||||
4. **Incorrect password retrieval:** The OVH secret link format may have changed, or we're using the wrong credential source.
|
||||
|
||||
---
|
||||
|
||||
## WHAT TO TRY NEXT SESSION
|
||||
|
||||
### Option A: OVH KVM/VNC Console
|
||||
- OVH Manager may have a web-based console (KVM/VNC/noVNC)
|
||||
- This bypasses SSH entirely
|
||||
- Look for "KVM" or "Console" or "VNC" button in OVH Manager
|
||||
|
||||
### Option B: Fresh Rescue Mode Request
|
||||
- In OVH Manager, explicitly DISABLE rescue mode first
|
||||
- Wait for normal boot to complete (or fail)
|
||||
- Then RE-ENABLE rescue mode
|
||||
- This should generate a truly fresh password via email
|
||||
|
||||
### Option C: Contact OVH Support
|
||||
- If SSH continues to fail, open a support ticket
|
||||
- Reference: Anti-hack incidents on this VPS
|
||||
- Ask them to verify rescue mode SSH is accessible
|
||||
|
||||
### Option D: Skip Rescue, Direct Reinstall
|
||||
- OVH Manager may allow direct OS reinstall WITHOUT needing rescue mode access
|
||||
- This would skip backup entirely but get a clean system
|
||||
- Data loss acceptable if MongoDB backup exists elsewhere
|
||||
|
||||
---
|
||||
|
||||
## QUESTIONS TO ASK USER
|
||||
|
||||
1. Is there a KVM/VNC console option in OVH Manager?
|
||||
2. Can you try disabling rescue mode and re-enabling it?
|
||||
3. Do you have any MongoDB backups stored locally that we could restore from?
|
||||
4. Is the OVH password coming from email or from the secret link?
|
||||
|
||||
---
|
||||
|
||||
## OVH MANAGER NAVIGATION
|
||||
|
||||
Based on OVH interface (may vary):
|
||||
|
||||
```
|
||||
OVH Manager → Bare Metal Cloud → VPS → [Your VPS] →
|
||||
- Dashboard: Shows status (rescue/normal)
|
||||
- Boot: Shows boot mode, option to change
|
||||
- Console/KVM: Web-based terminal (if available)
|
||||
- Reinstall: Direct OS reinstall option
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SUCCESSFUL ACCESS REQUIREMENTS
|
||||
|
||||
Before proceeding with backup/reinstall, we need ONE of:
|
||||
- [ ] Working SSH access to rescue mode
|
||||
- [ ] Working KVM/VNC console access
|
||||
- [ ] Confirmation that direct reinstall (no backup) is acceptable
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-20
|
||||
**Status:** BLOCKED - Cannot access rescue mode via SSH
|
||||
465
docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md
Normal file
465
docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md
Normal file
|
|
@ -0,0 +1,465 @@
|
|||
# Comprehensive Remediation Plan: agenticgovernance.digital
|
||||
## Date: 2026-01-19
|
||||
## Status: EXECUTED SUCCESSFULLY (2026-01-20)
|
||||
|
||||
> **UPDATE 2026-01-20:** This remediation plan was successfully executed. Complete VPS reinstallation performed, all systems restored, security hardening applied. See `docs/INCIDENT_RECOVERY_2026-01-19.md` for full details.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The agenticgovernance.digital VPS (vps-93a693da.vps.ovh.net) has been compromised **three times** by the same botnet infrastructure. Each prior "recovery" was incomplete, leaving persistence mechanisms that allowed reinfection.
|
||||
|
||||
**Recommendation: COMPLETE REINSTALL**
|
||||
|
||||
Based on security industry best practices and the pattern of recurring compromise, partial cleanup is no longer viable. A complete OS reinstall is the only way to guarantee all malware is removed.
|
||||
|
||||
---
|
||||
|
||||
## Attack History Analysis
|
||||
|
||||
### Timeline
|
||||
|
||||
| Date | Attack | Root Cause | Recovery Status |
|
||||
|------|--------|------------|-----------------|
|
||||
| 2025-12-09 | 83Kpps DNS flood (Exodus botnet via Docker/Umami) | Docker container compromise | **INCOMPLETE** - PM2, umami-deployment, cron jobs left |
|
||||
| 2026-01-18 13:57 | 171Kpps UDP flood to 15.184.38.247:9007 | PM2 resurrected botnet processes | **INCOMPLETE** - SSH broken post-recovery |
|
||||
| 2026-01-18 23:44 | 44Kpps UDP flood to 171.225.223.4:80 | Continued PM2 persistence | Server in rescue mode |
|
||||
| 2026-01-19 (today) | OVH anti-hack triggered again | Unknown - likely same persistence | **CURRENT INCIDENT** |
|
||||
|
||||
### What Was Missed in Each Recovery
|
||||
|
||||
**December 2025 Recovery:**
|
||||
- Docker removed ✓
|
||||
- PM2 process manager NOT removed ✗
|
||||
- `/home/ubuntu/umami-deployment/` NOT removed ✗
|
||||
- Ubuntu crontab NOT cleared ✗
|
||||
- PostgreSQL service NOT disabled ✗
|
||||
|
||||
**January 19 Recovery (earlier today):**
|
||||
- PM2 removed ✓
|
||||
- umami-deployment removed ✓
|
||||
- PostgreSQL disabled ✓
|
||||
- **But server is in rescue mode AGAIN** = something else was missed
|
||||
|
||||
### Malware Profile
|
||||
|
||||
**Name:** Exodus Botnet (Mirai variant)
|
||||
**C2 Server:** 196.251.100.191 (South Africa)
|
||||
**Capabilities:**
|
||||
- Multi-architecture binaries (x86, x86_64, ARM, MIPS, etc.)
|
||||
- UDP/DNS flood attacks
|
||||
- Self-replicating via PM2 process manager
|
||||
- Persistence through system services
|
||||
|
||||
**PM2 as Persistence Mechanism:**
|
||||
- PM2's `resurrect` feature auto-restarts saved processes on boot
|
||||
- Used by modern botnets like NodeCordRAT and Tsundere (2025)
|
||||
- Survives manual process termination
|
||||
- Creates systemd service (`pm2-ubuntu.service`)
|
||||
|
||||
---
|
||||
|
||||
## Why Partial Cleanup Has Failed
|
||||
|
||||
### Problem 1: Unknown Persistence Mechanisms
|
||||
Each cleanup identified SOME persistence mechanisms but missed others. There may be:
|
||||
- Modified system binaries (rootkits)
|
||||
- Kernel modules
|
||||
- Hidden cron jobs in unexpected locations
|
||||
- Modified init scripts
|
||||
- SSH backdoors (could explain broken SSH)
|
||||
|
||||
### Problem 2: No Baseline for Comparison
|
||||
Without knowing exactly what should exist on a clean system, we cannot verify complete removal.
|
||||
|
||||
### Problem 3: Forensic Limitations in Rescue Mode
|
||||
Rescue mode provides limited visibility into:
|
||||
- Runtime state of malware
|
||||
- Memory-resident components
|
||||
- Kernel-level modifications
|
||||
|
||||
### Expert Consensus
|
||||
> "Reinstalling a computer after it has been compromised can be a painstaking process, but it is the best way to be certain that everything an attacker left behind has been found." - UC Berkeley Information Security Office
|
||||
|
||||
> "Rootkits are difficult to remove, and the only 100% sure fire way to remove a rootkit from a device that has been infected is to wipe the device and reinstall the operating system."
|
||||
|
||||
---
|
||||
|
||||
## Recommended Solution: Complete Reinstall
|
||||
|
||||
### Phase 1: Data Backup (From Rescue Mode)
|
||||
|
||||
**CRITICAL:** Before reinstalling, back up essential data:
|
||||
|
||||
```bash
|
||||
# 1. Boot into rescue mode via OVH Manager
|
||||
# 2. Mount main disk
|
||||
mount /dev/sdb1 /mnt/vps
|
||||
|
||||
# 3. Create backup directory
|
||||
mkdir -p /tmp/tractatus-backup
|
||||
|
||||
# 4. Backup application code (verify hashes later)
|
||||
tar -czf /tmp/tractatus-backup/app.tar.gz /mnt/vps/var/www/tractatus/
|
||||
|
||||
# 5. Backup MongoDB data
|
||||
tar -czf /tmp/tractatus-backup/mongodb.tar.gz /mnt/vps/var/lib/mongodb/
|
||||
|
||||
# 6. Backup SSL certificates
|
||||
tar -czf /tmp/tractatus-backup/ssl.tar.gz /mnt/vps/etc/letsencrypt/
|
||||
|
||||
# 7. Backup nginx config (for reference, will recreate)
|
||||
cp /mnt/vps/etc/nginx/sites-available/tractatus /tmp/tractatus-backup/
|
||||
|
||||
# 8. Download backups to local machine
|
||||
scp -r root@RESCUE_IP:/tmp/tractatus-backup/ ~/tractatus-recovery/
|
||||
```
|
||||
|
||||
**DO NOT BACKUP:**
|
||||
- `/home/ubuntu/.pm2/` (malware)
|
||||
- `/home/ubuntu/umami-deployment/` (malware)
|
||||
- Any executables (may be compromised)
|
||||
- `/var/lib/docker/` (attack vector)
|
||||
|
||||
### Phase 2: VPS Reinstallation
|
||||
|
||||
1. **Via OVH Manager:**
|
||||
- Navigate to VPS management
|
||||
- Select "Reinstall" option
|
||||
- Choose: Ubuntu 22.04 LTS (or latest LTS)
|
||||
- Wait for completion (~10 minutes)
|
||||
|
||||
2. **Retrieve New Root Password:**
|
||||
- Check email for new credentials
|
||||
- Or use OVH password reset function
|
||||
|
||||
### Phase 3: Fresh System Setup
|
||||
|
||||
**Initial SSH Access:**
|
||||
```bash
|
||||
ssh root@91.134.240.3
|
||||
```
|
||||
|
||||
**Step 1: System Updates**
|
||||
```bash
|
||||
apt update && apt upgrade -y
|
||||
```
|
||||
|
||||
**Step 2: Create Non-Root User**
|
||||
```bash
|
||||
adduser ubuntu
|
||||
usermod -aG sudo ubuntu
|
||||
```
|
||||
|
||||
**Step 3: SSH Hardening**
|
||||
```bash
|
||||
# Add authorized keys
|
||||
mkdir -p /home/ubuntu/.ssh
|
||||
cat > /home/ubuntu/.ssh/authorized_keys << 'EOF'
|
||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ... theflow@the-flow
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy
|
||||
EOF
|
||||
|
||||
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
|
||||
chmod 700 /home/ubuntu/.ssh
|
||||
chmod 600 /home/ubuntu/.ssh/authorized_keys
|
||||
|
||||
# Harden SSH config
|
||||
cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF'
|
||||
PasswordAuthentication no
|
||||
PermitRootLogin no
|
||||
MaxAuthTries 3
|
||||
LoginGraceTime 20
|
||||
ClientAliveInterval 300
|
||||
ClientAliveCountMax 2
|
||||
EOF
|
||||
|
||||
systemctl restart sshd
|
||||
```
|
||||
|
||||
**Step 4: Firewall Configuration**
|
||||
```bash
|
||||
apt install -y ufw
|
||||
ufw default deny incoming
|
||||
ufw default allow outgoing
|
||||
ufw allow 22/tcp comment 'SSH'
|
||||
ufw allow 80/tcp comment 'HTTP'
|
||||
ufw allow 443/tcp comment 'HTTPS'
|
||||
# Block Docker ports (never needed)
|
||||
ufw deny 2375/tcp comment 'Block Docker API'
|
||||
ufw deny 2376/tcp comment 'Block Docker TLS'
|
||||
ufw enable
|
||||
```
|
||||
|
||||
**Step 5: Intrusion Prevention**
|
||||
```bash
|
||||
apt install -y fail2ban
|
||||
cat > /etc/fail2ban/jail.local << 'EOF'
|
||||
[sshd]
|
||||
enabled = true
|
||||
maxretry = 3
|
||||
bantime = 24h
|
||||
findtime = 1h
|
||||
EOF
|
||||
|
||||
systemctl enable fail2ban
|
||||
systemctl start fail2ban
|
||||
```
|
||||
|
||||
**Step 6: Install Required Software**
|
||||
```bash
|
||||
# Node.js (via NodeSource)
|
||||
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
|
||||
apt install -y nodejs
|
||||
|
||||
# MongoDB
|
||||
curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
|
||||
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list
|
||||
apt update
|
||||
apt install -y mongodb-org
|
||||
systemctl enable mongod
|
||||
systemctl start mongod
|
||||
|
||||
# nginx
|
||||
apt install -y nginx
|
||||
systemctl enable nginx
|
||||
|
||||
# certbot for SSL
|
||||
apt install -y certbot python3-certbot-nginx
|
||||
```
|
||||
|
||||
### Phase 4: Application Deployment
|
||||
|
||||
**Step 1: Prepare Application Directory**
|
||||
```bash
|
||||
mkdir -p /var/www/tractatus
|
||||
chown ubuntu:ubuntu /var/www/tractatus
|
||||
```
|
||||
|
||||
**Step 2: Deploy from CLEAN Local Source**
|
||||
```bash
|
||||
# From local machine - deploy ONLY from verified clean source
|
||||
cd ~/projects/tractatus
|
||||
./scripts/deploy.sh --full
|
||||
```
|
||||
|
||||
**Step 3: Restore MongoDB Data**
|
||||
```bash
|
||||
# If data integrity is verified
|
||||
mongorestore --db tractatus ~/tractatus-recovery/mongodb/tractatus/
|
||||
```
|
||||
|
||||
**Step 4: SSL Certificate**
|
||||
```bash
|
||||
certbot --nginx -d agenticgovernance.digital
|
||||
```
|
||||
|
||||
**Step 5: Create Systemd Service**
|
||||
```bash
|
||||
cat > /etc/systemd/system/tractatus.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Tractatus Application
|
||||
After=network.target mongod.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ubuntu
|
||||
WorkingDirectory=/var/www/tractatus
|
||||
ExecStart=/usr/bin/node src/server.js
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
Environment=NODE_ENV=production
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable tractatus
|
||||
systemctl start tractatus
|
||||
```
|
||||
|
||||
### Phase 5: Monitoring Setup
|
||||
|
||||
**Step 1: Log Rotation for MongoDB**
|
||||
```bash
|
||||
cat > /etc/logrotate.d/mongodb << 'EOF'
|
||||
/var/log/mongodb/*.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
missingok
|
||||
notifempty
|
||||
sharedscripts
|
||||
postrotate
|
||||
/bin/kill -SIGUSR1 $(cat /var/lib/mongodb/mongod.lock 2>/dev/null) 2>/dev/null || true
|
||||
endscript
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
**Step 2: Install Rootkit Scanner**
|
||||
```bash
|
||||
apt install -y rkhunter chkrootkit lynis
|
||||
|
||||
# Run initial scan
|
||||
rkhunter --update
|
||||
rkhunter --check --skip-keypress
|
||||
chkrootkit
|
||||
lynis audit system
|
||||
```
|
||||
|
||||
**Step 3: Monitoring Script**
|
||||
```bash
|
||||
cat > /usr/local/bin/security-check.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
# Daily security check
|
||||
LOG=/var/log/security-check.log
|
||||
echo "=== Security Check $(date) ===" >> $LOG
|
||||
|
||||
# Check for unauthorized services
|
||||
systemctl list-units --type=service --state=running | grep -v "systemd\|ssh\|nginx\|mongod\|tractatus\|fail2ban\|ufw" >> $LOG
|
||||
|
||||
# Check for unusual network connections
|
||||
netstat -tlnp | grep -v "127.0.0.1\|mongodb\|node\|nginx" >> $LOG
|
||||
|
||||
# Check for PM2 (should never exist)
|
||||
if command -v pm2 &> /dev/null; then
|
||||
echo "WARNING: PM2 DETECTED - SHOULD NOT EXIST" >> $LOG
|
||||
fi
|
||||
|
||||
# Check for Docker (should never exist)
|
||||
if command -v docker &> /dev/null; then
|
||||
echo "WARNING: DOCKER DETECTED - SHOULD NOT EXIST" >> $LOG
|
||||
fi
|
||||
EOF
|
||||
chmod +x /usr/local/bin/security-check.sh
|
||||
|
||||
# Add to cron
|
||||
echo "0 6 * * * root /usr/local/bin/security-check.sh" > /etc/cron.d/security-check
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Must NEVER Exist on This Server
|
||||
|
||||
| Component | Reason |
|
||||
|-----------|--------|
|
||||
| PM2 | Used for malware persistence |
|
||||
| Docker | Attack vector (Umami compromise) |
|
||||
| PostgreSQL | Only for Umami, not needed |
|
||||
| Any analytics containers | Attack surface |
|
||||
| Node packages outside app | Potential supply chain risk |
|
||||
|
||||
**Verification Script:**
|
||||
```bash
|
||||
#!/bin/bash
|
||||
ALERT=0
|
||||
if command -v pm2 &> /dev/null; then echo "ALERT: PM2 exists"; ALERT=1; fi
|
||||
if command -v docker &> /dev/null; then echo "ALERT: Docker exists"; ALERT=1; fi
|
||||
if [ -d "/home/ubuntu/.pm2" ]; then echo "ALERT: .pm2 directory exists"; ALERT=1; fi
|
||||
if [ -d "/home/ubuntu/umami-deployment" ]; then echo "ALERT: umami-deployment exists"; ALERT=1; fi
|
||||
if systemctl is-enabled postgresql &> /dev/null; then echo "ALERT: PostgreSQL enabled"; ALERT=1; fi
|
||||
if [ $ALERT -eq 0 ]; then echo "Server is clean"; fi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-Recovery Verification Checklist
|
||||
|
||||
- [ ] SSH access works with key authentication
|
||||
- [ ] Password authentication is disabled
|
||||
- [ ] fail2ban is running and banning IPs
|
||||
- [ ] UFW is enabled with correct rules
|
||||
- [ ] nginx is serving the site
|
||||
- [ ] tractatus service is running
|
||||
- [ ] MongoDB is running and bound to 127.0.0.1
|
||||
- [ ] SSL certificate is valid
|
||||
- [ ] No PM2 installed
|
||||
- [ ] No Docker installed
|
||||
- [ ] No PostgreSQL installed
|
||||
- [ ] rkhunter scan is clean
|
||||
- [ ] chkrootkit scan is clean
|
||||
- [ ] Log rotation is configured
|
||||
- [ ] Daily security check cron is active
|
||||
|
||||
---
|
||||
|
||||
## Credentials to Rotate
|
||||
|
||||
After reinstall, rotate all credentials:
|
||||
|
||||
1. **MongoDB admin password** (if using authentication)
|
||||
2. **Application secrets** in `.env`
|
||||
3. **Session secrets**
|
||||
4. **Any API keys**
|
||||
|
||||
**Important:** Change passwords from a DIFFERENT machine, not the compromised server.
|
||||
|
||||
---
|
||||
|
||||
## Long-Term Prevention
|
||||
|
||||
1. **Never install Docker** - not needed for this application
|
||||
2. **Never install PM2** - use systemd only
|
||||
3. **Weekly security scans** - rkhunter, chkrootkit
|
||||
4. **Monitor outbound traffic** - alert on unexpected destinations
|
||||
5. **Keep system updated** - enable unattended-upgrades
|
||||
6. **Review SSH logs weekly** - check for brute force patterns
|
||||
|
||||
---
|
||||
|
||||
## OVH Support Communication Template
|
||||
|
||||
```
|
||||
Subject: Request to restore VPS to normal mode after reinstallation
|
||||
|
||||
Reference: [ref=1.2378332d]
|
||||
Server: vps-93a693da.vps.ovh.net
|
||||
|
||||
We have identified the cause of the anti-hack triggers:
|
||||
- Compromised Docker container running botnet malware
|
||||
- PM2 process manager persisting malicious processes
|
||||
|
||||
We have completed a full OS reinstall and implemented:
|
||||
- Hardened SSH configuration (key-only, no root)
|
||||
- UFW firewall with minimal open ports
|
||||
- fail2ban for intrusion prevention
|
||||
- Removal of Docker and PM2
|
||||
|
||||
Please restore the VPS to normal boot mode.
|
||||
|
||||
Thank you.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Timeline Estimate
|
||||
|
||||
| Phase | Duration |
|
||||
|-------|----------|
|
||||
| Backup data | 30 min |
|
||||
| VPS reinstall | 10 min |
|
||||
| System setup | 45 min |
|
||||
| Application deployment | 30 min |
|
||||
| Verification | 30 min |
|
||||
| **Total** | ~2.5 hours |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Mirai Malware Behavior & Mitigation (2025)](https://echoxec.com/mirai-malware-in-2025-variant-behavior-exploit-chains-and-mitigation-insights)
|
||||
- [PM2 Malware Persistence (ThreatLabz)](https://www.zscaler.com/blogs/security-research/malicious-npm-packages-deliver-nodecordrat)
|
||||
- [Linux Rootkit Detection Tools](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/)
|
||||
- [OVH VPS Rescue Mode Guide](https://help.ovhcloud.com/csm/en-vps-rescue?id=kb_article_view&sysparm_article=KB0047656)
|
||||
- [UC Berkeley - Reinstalling Compromised Systems](https://security.berkeley.edu/education-awareness/reinstalling-your-compromised-computer)
|
||||
- [Common Malware Persistence Mechanisms](https://www.infosecinstitute.com/resources/malware-analysis/common-malware-persistence-mechanisms/)
|
||||
|
||||
---
|
||||
|
||||
**Document Author:** Claude Code
|
||||
**Date:** 2026-01-19
|
||||
**Status:** Ready for implementation
|
||||
**Next Action:** User decision on proceeding with complete reinstall
|
||||
Loading…
Add table
Reference in a new issue