# VPS Recovery Reference - agenticgovernance.digital **Last Updated:** 2026-01-20 **VPS:** vps-93a693da.vps.ovh.net **IP:** 91.134.240.3 **Provider:** OVH --- ## Quick Reference ### SSH Access ```bash # Primary access (from theflow's machine) ssh -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net # If host key changed (after reinstall) ssh-keygen -f '/home/theflow/.ssh/known_hosts' -R 'vps-93a693da.vps.ovh.net' ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/tractatus_deploy ubuntu@vps-93a693da.vps.ovh.net "echo 'Connected'" ``` ### Service Management ```bash # Check tractatus service sudo systemctl status tractatus # Restart tractatus sudo systemctl restart tractatus # Check all relevant services sudo systemctl status tractatus nginx mongod fail2ban ``` ### Deployment ```bash # From local machine cd ~/projects/tractatus ./scripts/deploy.sh ``` --- ## If Server Is Compromised Again ### RECOMMENDED: Complete Reinstall Based on experience with recurring compromises, **complete reinstall is recommended** over partial cleanup. #### Step 1: Backup Data (if accessible) ```bash # From local machine - export current database mongodump --db tractatus_dev --out ~/tractatus-backup-$(date +%Y%m%d) ``` #### Step 2: Reinstall via OVH Manager 1. Log into OVH Manager (https://www.ovh.com/manager/) 2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da 3. Click "Reinstall" button 4. Select: Ubuntu 22.04 LTS 5. Wait for completion (~10 minutes) 6. Check email for new root credentials #### Step 3: Initial System Setup ```bash # SSH as root with new credentials ssh root@91.134.240.3 # Update system apt update && apt upgrade -y # Create ubuntu user adduser ubuntu usermod -aG sudo ubuntu # Setup SSH keys mkdir -p /home/ubuntu/.ssh cat > /home/ubuntu/.ssh/authorized_keys << 'EOF' ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy EOF chown -R ubuntu:ubuntu /home/ubuntu/.ssh chmod 700 /home/ubuntu/.ssh chmod 600 /home/ubuntu/.ssh/authorized_keys # Harden SSH cat > /etc/ssh/sshd_config.d/hardening.conf << 'EOF' PasswordAuthentication no PermitRootLogin no MaxAuthTries 3 EOF systemctl restart sshd # Install firewall apt install -y ufw ufw default deny incoming ufw default allow outgoing ufw allow 22/tcp ufw allow 80/tcp ufw allow 443/tcp ufw --force enable # Install fail2ban apt install -y fail2ban systemctl enable fail2ban systemctl start fail2ban ``` #### Step 4: Install Application Stack ```bash # Node.js curl -fsSL https://deb.nodesource.com/setup_20.x | bash - apt install -y nodejs # MongoDB curl -fsSL https://pgp.mongodb.com/server-7.0.asc | gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" > /etc/apt/sources.list.d/mongodb-org-7.0.list apt update apt install -y mongodb-org systemctl enable mongod systemctl start mongod # nginx apt install -y nginx systemctl enable nginx # certbot apt install -y certbot python3-certbot-nginx # Security tools apt install -y rkhunter chkrootkit ``` #### Step 5: Create Application Directory ```bash mkdir -p /var/www/tractatus chown ubuntu:ubuntu /var/www/tractatus ``` #### Step 6: Deploy Application (from local machine) ```bash cd ~/projects/tractatus ./scripts/deploy.sh ``` #### Step 7: Create Production .env (on VPS) ```bash cat > /var/www/tractatus/.env << 'EOF' NODE_ENV=production PORT=9000 MONGODB_URI=mongodb://127.0.0.1:27017 MONGODB_DB=tractatus APP_URL=https://agenticgovernance.digital SESSION_SECRET= ANTHROPIC_API_KEY= # Add other keys as needed EOF chmod 600 /var/www/tractatus/.env ``` #### Step 8: Install Dependencies (on VPS) ```bash cd /var/www/tractatus npm install npm install @anthropic-ai/sdk # May need explicit install ``` #### Step 9: Create Systemd Service ```bash cat > /etc/systemd/system/tractatus.service << 'EOF' [Unit] Description=Tractatus - Agentic Governance Framework After=network.target mongod.service [Service] Type=simple User=ubuntu WorkingDirectory=/var/www/tractatus ExecStart=/usr/bin/node src/server.js Restart=on-failure RestartSec=10 Environment=NODE_ENV=production [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable tractatus systemctl start tractatus ``` #### Step 10: Configure nginx ```bash cat > /etc/nginx/sites-available/tractatus << 'EOF' server { listen 80; server_name agenticgovernance.digital; location / { proxy_pass http://127.0.0.1:9000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_cache_bypass $http_upgrade; } } EOF ln -sf /etc/nginx/sites-available/tractatus /etc/nginx/sites-enabled/ rm -f /etc/nginx/sites-enabled/default nginx -t && systemctl reload nginx ``` #### Step 11: SSL Certificate ```bash certbot --nginx -d agenticgovernance.digital ``` #### Step 12: Restore Database ```bash # From local machine mongodump --db tractatus_dev --out ~/tractatus-backup rsync -avz ~/tractatus-backup/ ubuntu@vps-93a693da.vps.ovh.net:/tmp/tractatus-backup/ # On VPS mongorestore --db tractatus /tmp/tractatus-backup/tractatus_dev/ ``` #### Step 13: Setup Admin User ```bash cd /var/www/tractatus node scripts/fix-admin-user.js node scripts/seed-projects.js ``` #### Step 14: Daily Security Monitoring ```bash cat > /usr/local/bin/daily-security-check.sh << 'EOF' #!/bin/bash LOG=/var/log/daily-security-check.log echo "=== Security Check $(date) ===" >> $LOG # Check for banned software if command -v pm2 &> /dev/null; then echo "CRITICAL: PM2 DETECTED" >> $LOG fi if command -v docker &> /dev/null; then echo "CRITICAL: Docker DETECTED" >> $LOG fi if [ -d "/home/ubuntu/.pm2" ]; then echo "CRITICAL: .pm2 directory exists" >> $LOG fi # Check unusual processes ps aux | grep -E "(pm2|docker|umami)" | grep -v grep >> $LOG echo "Check complete" >> $LOG EOF chmod +x /usr/local/bin/daily-security-check.sh echo "0 3 * * * root /usr/local/bin/daily-security-check.sh" > /etc/cron.d/security-check ``` --- ## BANNED Software **NEVER install these on this server:** | Software | Reason | |----------|--------| | PM2 | Malware persistence vector (Exodus botnet) | | Docker | Attack vector (Umami compromise) | | PostgreSQL | Only for Umami, not needed | | Any analytics containers | Attack surface | --- ## Attack History | Date | Attack Type | Root Cause | Resolution | |------|-------------|------------|------------| | 2025-12-09 | 83Kpps DNS flood | Docker/Umami Exodus botnet | Partial cleanup (FAILED) | | 2026-01-18 | 171Kpps UDP flood | PM2 persistence | Partial cleanup (FAILED) | | 2026-01-18 | 44Kpps UDP flood | PM2 persistence | Complete reinstall (SUCCESS) | --- ## Malware Indicators **Exodus Botnet (Mirai variant):** - C2 Server: 196.251.100.191 (South Africa) - Persistence: PM2 process manager - Attack types: UDP flood, DNS flood - Target ports: Various (9007, 80, 53) **Signs of Compromise:** - PM2 installed or .pm2 directory exists - Docker installed - High outbound UDP traffic - OVH anti-hack alerts - Unknown processes in `ps aux` --- ## Verification Checklist After recovery, verify: - [ ] SSH works with key authentication - [ ] `sudo systemctl status tractatus nginx mongod fail2ban` all active - [ ] Website responds: `curl -I https://agenticgovernance.digital/` - [ ] No PM2: `which pm2` returns nothing - [ ] No Docker: `which docker` returns nothing - [ ] No .pm2 directory: `ls -la /home/ubuntu/.pm2` not found - [ ] UFW active: `sudo ufw status` - [ ] fail2ban running: `sudo fail2ban-client status` --- ## Related Documents - `docs/INCIDENT_RECOVERY_2026-01-19.md` - Full incident timeline and recovery details - `docs/plans/REMEDIATION_PLAN_AGENTICGOVERNANCE_20260119.md` - Original remediation plan - `docs/plans/OVH_RESCUE_MODE_KNOWN_ISSUES.md` - Issues with OVH rescue mode --- ## OVH Contact - Manager: https://www.ovh.com/manager/ - Server: vps-93a693da.vps.ovh.net - IP: 91.134.240.3