tractatus/docs/VPS_RECOVERY_REFERENCE.md

# VPS Recovery Reference - agenticgovernance.digital

**Last Updated:** 2026-01-20
**VPS:** vps-93a693da.vps.ovh.net
**IP:** 91.134.240.3
**Provider:** OVH

---

## Quick Reference

### SSH Access
```bash
# Primary access (from theflow's machine)
ssh -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net

# If host key changed (after reinstall)
ssh-keygen -f '/home/theflow/.ssh/known_hosts' -R 'vps-93a693da.vps.ovh.net'
ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net "echo 'Connected'"
```

### Service Management
```bash
# Check tractatus service
sudo systemctl status tractatus

# Restart tractatus
sudo systemctl restart tractatus

# Check all relevant services
sudo systemctl status tractatus nginx mongod fail2ban
```

### Deployment
```bash
# From local machine
cd ~/projects/tractatus
./scripts/deploy.sh
```

---

## If Server Is Compromised Again

### RECOMMENDED: Complete Reinstall

Based on experience with recurring compromises, **complete reinstall is recommended** over partial cleanup.

#### Step 1: Backup Data (if accessible)
```bash
# From local machine - export current database
mongodump --db tractatus_dev --out ~/tractatus-backup-$(date +%Y%m%d)
```

#### Step 2: Reinstall via OVH Manager
1. Log into OVH Manager (https://www.ovh.com/manager/)
2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da
3. Click "Reinstall" button
4. Select: Ubuntu 22.04 LTS
5. Wait for completion (~10 minutes)
6. Check email for new root credentials

#### Step 3: Initial System Setup
```bash
# SSH as root with new credentials
ssh root@91.134.240.3

# Update system
apt update && apt upgrade -y

# Create ubuntu user
adduser ubuntu
usermod -aG sudo ubuntu

# Setup SSH keys
mkdir -p /home/ubuntu/.ssh
cat > /home/ubuntu/.ssh/authorized_keys << 'EOF'
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy
EOF
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
chmod 700 /home/ubuntu/.ssh
chmod 600 /home/ubuntu/.ssh/authorized_keys

# Harden SSH
cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF'
PasswordAuthentication no
PermitRootLogin no
MaxAuthTries 3
EOF
systemctl restart sshd

# Install firewall
apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw --force enable

# Install fail2ban
apt install -y fail2ban
systemctl enable fail2ban
systemctl start fail2ban
```

#### Step 4: Install Application Stack
```bash
# Node.js
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs

# MongoDB
curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list
apt update
apt install -y mongodb-org
systemctl enable mongod
systemctl start mongod

# nginx
apt install -y nginx
systemctl enable nginx

# certbot
apt install -y certbot python3-certbot-nginx

# Security tools
apt install -y rkhunter chkrootkit
```

#### Step 5: Create Application Directory
```bash
mkdir -p /var/www/tractatus
chown ubuntu:ubuntu /var/www/tractatus
```

#### Step 6: Deploy Application (from local machine)
```bash
cd ~/projects/tractatus
./scripts/deploy.sh
```

#### Step 7: Create Production .env (on VPS)
```bash
cat > /var/www/tractatus/.env << 'EOF'
NODE_ENV=production
PORT=9000
MONGODB_URI=mongodb://127.0.0.1:27017
MONGODB_DB=tractatus
APP_URL=https://agenticgovernance.digital
SESSION_SECRET=<generate-new-secret>
ANTHROPIC_API_KEY=<get-from-credential-vault>
# Add other keys as needed
EOF
chmod 600 /var/www/tractatus/.env
```

#### Step 8: Install Dependencies (on VPS)
```bash
cd /var/www/tractatus
npm install
npm install @anthropic-ai/sdk  # May need explicit install
```

#### Step 9: Create Systemd Service
```bash
cat > /etc/systemd/system/tractatus.service << 'EOF'
[Unit]
Description=Tractatus - Agentic Governance Framework
After=network.target mongod.service

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/www/tractatus
ExecStart=/usr/bin/node src/server.js
Restart=on-failure
RestartSec=10
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable tractatus
systemctl start tractatus
```

#### Step 10: Configure nginx
```bash
cat > /etc/nginx/sites-available/tractatus << 'EOF'
server {
    listen 80;
    server_name agenticgovernance.digital;

    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}
EOF

ln -sf /etc/nginx/sites-available/tractatus /etc/nginx/sites-enabled/
rm -f /etc/nginx/sites-enabled/default
nginx -t && systemctl reload nginx
```

#### Step 11: SSL Certificate
```bash
certbot --nginx -d agenticgovernance.digital
```

#### Step 12: Restore Database
```bash
# From local machine
mongodump --db tractatus_dev --out ~/tractatus-backup
rsync -avz ~/tractatus-backup/ ubuntu@vps-93a693da.vps.ovh.net:/tmp/tractatus-backup/

# On VPS
mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/
```

#### Step 13: Setup Admin User
```bash
cd /var/www/tractatus
node scripts/fix-admin-user.js
node scripts/seed-projects.js
```

#### Step 14: Daily Security Monitoring
```bash
cat > /usr/local/bin/daily-security-check.sh << 'EOF'
#!/bin/bash
LOG=/var/log/daily-security-check.log
echo "=== Security Check $(date) ===" >> $LOG

# Check for banned software
if command -v pm2 &> /dev/null; then
    echo "CRITICAL: PM2 DETECTED" >> $LOG
fi
if command -v docker &> /dev/null; then
    echo "CRITICAL: Docker DETECTED" >> $LOG
fi
if [ -d "/home/ubuntu/.pm2" ]; then
    echo "CRITICAL: .pm2 directory exists" >> $LOG
fi

# Check unusual processes
ps aux | grep -E "(pm2|docker|umami)" | grep -v grep >> $LOG

echo "Check complete" >> $LOG
EOF

chmod +x /usr/local/bin/daily-security-check.sh
echo "0 3 * * * root /usr/local/bin/daily-security-check.sh" > /etc/cron.d/security-check
```

---

## BANNED Software

**NEVER install these on this server:**

| Software | Reason |
|----------|--------|
| PM2 | Malware persistence vector (Exodus botnet) |
| Docker | Attack vector (Umami compromise) |
| PostgreSQL | Only for Umami, not needed |
| Any analytics containers | Attack surface |

---

## Attack History

| Date | Attack Type | Root Cause | Resolution |
|------|-------------|------------|------------|
| 2025-12-09 | 83Kpps DNS flood | Docker/Umami Exodus botnet | Partial cleanup (FAILED) |
| 2026-01-18 | 171Kpps UDP flood | PM2 persistence | Partial cleanup (FAILED) |
| 2026-01-18 | 44Kpps UDP flood | PM2 persistence | Complete reinstall (SUCCESS) |

---

## Malware Indicators

**Exodus Botnet (Mirai variant):**
- C2 Server: 196.251.100.191 (South Africa)
- Persistence: PM2 process manager
- Attack types: UDP flood, DNS flood
- Target ports: Various (9007, 80, 53)

**Signs of Compromise:**
- PM2 installed or .pm2 directory exists
- Docker installed
- High outbound UDP traffic
- OVH anti-hack alerts
- Unknown processes in `ps aux`

---

## Verification Checklist

After recovery, verify:

- [ ] SSH works with key authentication
- [ ] `sudo systemctl status tractatus nginx mongod fail2ban` all active
- [ ] Website responds: `curl -I https://agenticgovernance.digital/`
- [ ] No PM2: `which pm2` returns nothing
- [ ] No Docker: `which docker` returns nothing
- [ ] No .pm2 directory: `ls -la /home/ubuntu/.pm2` not found
- [ ] UFW active: `sudo ufw status`
- [ ] fail2ban running: `sudo fail2ban-client status`

---

## Related Documents

- `docs/INCIDENT_RECOVERY_2026-01-19.md` - Full incident timeline and recovery details
- `docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md` - Original remediation plan
- `docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md` - Issues with OVH rescue mode

---

## OVH Contact

- Manager: https://www.ovh.com/manager/
- Server: vps-93a693da.vps.ovh.net
- IP: 91.134.240.3