tractatus/docs/VPS_RECOVERY_REFERENCE.md

VPS Recovery Reference - agenticgovernance.digital

Last Updated: 2026-01-20 VPS: vps-93a693da.vps.ovh.net IP: 91.134.240.3 Provider: OVH

---

Quick Reference

SSH Access

# Primary access (from theflow's machine)
ssh -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net

# If host key changed (after reinstall)
ssh-keygen -f '/home/theflow/.ssh/known_hosts' -R 'vps-93a693da.vps.ovh.net'
ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net "echo 'Connected'"

Service Management

# Check tractatus service
sudo systemctl status tractatus

# Restart tractatus
sudo systemctl restart tractatus

# Check all relevant services
sudo systemctl status tractatus nginx mongod fail2ban

Deployment

# From local machine
cd ~/projects/tractatus
./scripts/deploy.sh

---

If Server Is Compromised Again

RECOMMENDED: Complete Reinstall

Based on experience with recurring compromises, complete reinstall is recommended over partial cleanup.

Step 1: Backup Data (if accessible)

# From local machine - export current database
mongodump --db tractatus_dev --out ~/tractatus-backup-$(date +%Y%m%d)

Step 2: Reinstall via OVH Manager

1. Log into OVH Manager (https://www.ovh.com/manager/)
2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da
3. Click "Reinstall" button
4. Select: Ubuntu 22.04 LTS
5. Wait for completion (~10 minutes)
6. Check email for new root credentials

Step 3: Initial System Setup

# SSH as root with new credentials
ssh root@91.134.240.3

# Update system
apt update && apt upgrade -y

# Create ubuntu user
adduser ubuntu
usermod -aG sudo ubuntu

# Setup SSH keys
mkdir -p /home/ubuntu/.ssh
cat > /home/ubuntu/.ssh/authorized_keys << 'EOF'
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy
EOF
chown -R ubuntu:ubuntu /home/ubuntu/.ssh
chmod 700 /home/ubuntu/.ssh
chmod 600 /home/ubuntu/.ssh/authorized_keys

# Harden SSH
cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF'
PasswordAuthentication no
PermitRootLogin no
MaxAuthTries 3
EOF
systemctl restart sshd

# Install firewall
apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw --force enable

# Install fail2ban
apt install -y fail2ban
systemctl enable fail2ban
systemctl start fail2ban

Step 4: Install Application Stack

# Node.js
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs

# MongoDB
curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list
apt update
apt install -y mongodb-org
systemctl enable mongod
systemctl start mongod

# nginx
apt install -y nginx
systemctl enable nginx

# certbot
apt install -y certbot python3-certbot-nginx

# Security tools
apt install -y rkhunter chkrootkit

Step 5: Create Application Directory

mkdir -p /var/www/tractatus
chown ubuntu:ubuntu /var/www/tractatus

Step 6: Deploy Application (from local machine)

cd ~/projects/tractatus
./scripts/deploy.sh

Step 7: Create Production .env (on VPS)

cat > /var/www/tractatus/.env << 'EOF'
NODE_ENV=production
PORT=9000
MONGODB_URI=mongodb://127.0.0.1:27017
MONGODB_DB=tractatus
APP_URL=https://agenticgovernance.digital
SESSION_SECRET=<generate-new-secret>
ANTHROPIC_API_KEY=<get-from-credential-vault>
# Add other keys as needed
EOF
chmod 600 /var/www/tractatus/.env

Step 8: Install Dependencies (on VPS)

cd /var/www/tractatus
npm install
npm install @anthropic-ai/sdk  # May need explicit install

Step 9: Create Systemd Service

cat > /etc/systemd/system/tractatus.service << 'EOF'
[Unit]
Description=Tractatus - Agentic Governance Framework
After=network.target mongod.service

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/www/tractatus
ExecStart=/usr/bin/node src/server.js
Restart=on-failure
RestartSec=10
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable tractatus
systemctl start tractatus

Step 10: Configure nginx

cat > /etc/nginx/sites-available/tractatus << 'EOF'
server {
    listen 80;
    server_name agenticgovernance.digital;

    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}
EOF

ln -sf /etc/nginx/sites-available/tractatus /etc/nginx/sites-enabled/
rm -f /etc/nginx/sites-enabled/default
nginx -t && systemctl reload nginx

Step 11: SSL Certificate

certbot --nginx -d agenticgovernance.digital

Step 12: Restore Database

# From local machine
mongodump --db tractatus_dev --out ~/tractatus-backup
rsync -avz ~/tractatus-backup/ ubuntu@vps-93a693da.vps.ovh.net:/tmp/tractatus-backup/

# On VPS
mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/

Step 13: Setup Admin User

cd /var/www/tractatus
node scripts/fix-admin-user.js
node scripts/seed-projects.js

Step 14: Daily Security Monitoring

cat > /usr/local/bin/daily-security-check.sh << 'EOF'
#!/bin/bash
LOG=/var/log/daily-security-check.log
echo "=== Security Check $(date) ===" >> $LOG

# Check for banned software
if command -v pm2 &> /dev/null; then
    echo "CRITICAL: PM2 DETECTED" >> $LOG
fi
if command -v docker &> /dev/null; then
    echo "CRITICAL: Docker DETECTED" >> $LOG
fi
if [ -d "/home/ubuntu/.pm2" ]; then
    echo "CRITICAL: .pm2 directory exists" >> $LOG
fi

# Check unusual processes
ps aux | grep -E "(pm2|docker|umami)" | grep -v grep >> $LOG

echo "Check complete" >> $LOG
EOF

chmod +x /usr/local/bin/daily-security-check.sh
echo "0 3 * * * root /usr/local/bin/daily-security-check.sh" > /etc/cron.d/security-check

---

BANNED Software

NEVER install these on this server:

Software	Reason
PM2	Malware persistence vector (Exodus botnet)
Docker	Attack vector (Umami compromise)
PostgreSQL	Only for Umami, not needed
Any analytics containers	Attack surface

---

Attack History

Date	Attack Type	Root Cause	Resolution
2025-12-09	83Kpps DNS flood	Docker/Umami Exodus botnet	Partial cleanup (FAILED)
2026-01-18	171Kpps UDP flood	PM2 persistence	Partial cleanup (FAILED)
2026-01-18	44Kpps UDP flood	PM2 persistence	Complete reinstall (SUCCESS)

---

Malware Indicators

Exodus Botnet (Mirai variant):

• C2 Server: 196.251.100.191 (South Africa)
• Persistence: PM2 process manager
• Attack types: UDP flood, DNS flood
• Target ports: Various (9007, 80, 53)

Signs of Compromise:

• PM2 installed or .pm2 directory exists
• Docker installed
• High outbound UDP traffic
• OVH anti-hack alerts
• Unknown processes in ps aux

---

Verification Checklist

After recovery, verify:

• SSH works with key authentication
• sudo systemctl status tractatus nginx mongod fail2ban all active
• Website responds: curl -I https://agenticgovernance.digital/
• No PM2: which pm2 returns nothing
• No Docker: which docker returns nothing
• No .pm2 directory: ls -la /home/ubuntu/.pm2 not found
• UFW active: sudo ufw status
• fail2ban running: sudo fail2ban-client status

---

Related Documents

• docs/INCIDENT_RECOVERY_2026-01-19.md - Full incident timeline and recovery details
• docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md - Original remediation plan
• docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md - Issues with OVH rescue mode

---

OVH Contact

• Manager: https://www.ovh.com/manager/
• Server: vps-93a693da.vps.ovh.net
• IP: 91.134.240.3