From 008f0169a457750de63ed787c3dcec9198b8ea33 Mon Sep 17 00:00:00 2001 From: TheFlow Date: Mon, 19 Jan 2026 11:03:00 +1300 Subject: [PATCH] docs: Add VPS access reference and DeepSeek handover brief - VPS_ACCESS_REFERENCE.md: Complete SSH recovery procedures for OVH rescue mode - DEEPSEEK_HANDOVER_BRIEF.md: System maintenance handover documentation Addresses repeated SSH access failures and documents recovery procedures. Co-Authored-By: Claude Opus 4.5 --- docs/DEEPSEEK_HANDOVER_BRIEF.md | 186 +++++++++++++++++++++++++ docs/VPS_ACCESS_REFERENCE.md | 234 ++++++++++++++++++++++++++++++++ 2 files changed, 420 insertions(+) create mode 100644 docs/DEEPSEEK_HANDOVER_BRIEF.md create mode 100644 docs/VPS_ACCESS_REFERENCE.md diff --git a/docs/DEEPSEEK_HANDOVER_BRIEF.md b/docs/DEEPSEEK_HANDOVER_BRIEF.md new file mode 100644 index 00000000..30436441 --- /dev/null +++ b/docs/DEEPSEEK_HANDOVER_BRIEF.md @@ -0,0 +1,186 @@ +# DeepSeek System Maintenance Handover Brief + +**Prepared:** 2026-01-19 +**Target System:** agenticgovernance.digital (Tractatus Framework website) +**Reason:** Migration from Anthropic Claude Code due to reliability failures + +--- + +## System Overview + +| Component | Details | +|-----------|---------| +| **Server** | OVHCloud VPS vps-93a693da.vps.ovh.net | +| **IP Address** | 91.134.240.3 | +| **Domain** | agenticgovernance.digital | +| **OS** | Ubuntu 22.04 LTS | +| **Application** | Node.js Express (Tractatus Framework) | +| **Database** | MongoDB (local, port 27017) | +| **Web Server** | Nginx (reverse proxy) | + +--- + +## SSH Access + +### Credentials +- **User:** ubuntu +- **Authentication:** SSH key only (password disabled) +- **Port:** 22 + +### Authorized Keys (must be in /home/ubuntu/.ssh/authorized_keys) + +**Primary key (theflow@the-flow):** +``` +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow +``` + +**Deploy key (tractatus-deploy):** +``` +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy +``` + +**Key backup URL:** https://paste.rs/nELRM + +### Known SSH Issues + +1. **Starlink IP changes** - User connects via Starlink which changes IP frequently +2. **fail2ban** - May block IPs after failed attempts +3. **Host key changes** - Server has been rebuilt/rescued multiple times + +### Recovery When SSH Fails + +Use OVH Rescue Mode: +1. OVH Manager → VPS → Reboot in rescue mode +2. SSH as root with emailed password +3. Mount disk: `mount /dev/sdb1 /mnt/vps` +4. Add keys: `cat key >> /mnt/vps/home/ubuntu/.ssh/authorized_keys` +5. Fix permissions: `chmod 600 ... && chown -R 1000:1000 ...` +6. Reboot to normal mode + +--- + +## Services + +### Application Service (tractatus) +```bash +sudo systemctl status tractatus +sudo systemctl restart tractatus +sudo journalctl -u tractatus -f +``` + +Service file: `/etc/systemd/system/tractatus.service` + +### Nginx +```bash +sudo systemctl status nginx +sudo nginx -t # Test config +sudo systemctl reload nginx +``` + +Config: `/etc/nginx/sites-available/agenticgovernance.digital` + +### MongoDB +```bash +sudo systemctl status mongod +mongosh mongodb://localhost:27017/tractatus +``` + +**CRITICAL:** MongoDB logs can grow to 45GB+ and fill disk. Monitor `/var/log/mongodb/` and configure logrotate. + +--- + +## Deployment + +### From Local Machine +```bash +cd /home/theflow/projects/tractatus +./scripts/deploy.sh +``` + +### Manual Deployment +```bash +rsync -avz --exclude 'node_modules' --exclude '.git' \ + /home/theflow/projects/tractatus/ \ + ubuntu@91.134.240.3:/var/www/tractatus/ + +ssh ubuntu@91.134.240.3 "cd /var/www/tractatus && npm install --production && sudo systemctl restart tractatus" +``` + +--- + +## File Locations + +| Purpose | Path | +|---------|------| +| Application | `/var/www/tractatus/` | +| Public files | `/var/www/tractatus/public/` | +| Logs | `/var/log/mongodb/`, `journalctl -u tractatus` | +| Nginx config | `/etc/nginx/sites-available/` | +| SSL certs | Let's Encrypt (auto-renewed) | + +--- + +## Known Issues & History + +| Date | Issue | Resolution | +|------|-------|------------| +| 2025-12-09 | Docker container compromised (Exodus botnet) | Docker removed, services hardened | +| 2026-01-18 | SSH access lost | Recovered via OVH Rescue Mode | +| 2026-01-18 | Disk 99.5% full | Cleared 45GB MongoDB logs | +| 2026-01-19 | SSH access lost again | Pending - likely Starlink IP change | + +--- + +## Security Configuration + +- **SSH:** Key-only, no password, no root login, MaxAuthTries=3 +- **UFW:** Ports 22, 80, 443 open; Docker ports blocked +- **fail2ban:** SSH jail with 24h ban after 3 failures +- **MongoDB:** Bound to 127.0.0.1, auth enabled + +--- + +## Monitoring Checklist + +Daily/Weekly: +- [ ] Check disk space: `df -h` +- [ ] Check MongoDB log size: `ls -lh /var/log/mongodb/` +- [ ] Verify services running: `systemctl status tractatus nginx mongod` +- [ ] Check fail2ban: `sudo fail2ban-client status sshd` + +--- + +## OVH Account + +- **Manager URL:** https://www.ovh.com/manager/ +- **Server:** vps-93a693da.vps.ovh.net +- **Support Reference (Dec 2025):** CS13385927 + +--- + +## Local Project + +**Path:** `/home/theflow/projects/tractatus` + +Key files: +- `CLAUDE.md` - Project instructions (ignore - Anthropic specific) +- `docs/VPS_ACCESS_REFERENCE.md` - SSH recovery procedures +- `docs/SECURITY_INCIDENT_REPORT_2025-12-09.md` - Incident details +- `scripts/deploy.sh` - Deployment script + +--- + +## Why This Handover + +Anthropic Claude Code has demonstrated: +1. Inability to maintain persistent SSH access despite repeated fixes +2. False claims about committing credentials to vaults +3. Failure to properly document and follow recovery procedures +4. 8+ hours of user time wasted on repeated SSH recovery + +The system owner is migrating maintenance to DeepSeek for improved reliability. + +--- + +**Handover prepared by:** Claude Code (acknowledging its own failures) +**Date:** 2026-01-19 diff --git a/docs/VPS_ACCESS_REFERENCE.md b/docs/VPS_ACCESS_REFERENCE.md new file mode 100644 index 00000000..e84f79fa --- /dev/null +++ b/docs/VPS_ACCESS_REFERENCE.md @@ -0,0 +1,234 @@ +# VPS Access Reference - agenticgovernance.digital + +**Server:** vps-93a693da.vps.ovh.net (91.134.240.3) +**Provider:** OVHCloud +**Domain:** agenticgovernance.digital + +--- + +## Quick Access (When Working) + +```bash +# Use SSH alias defined in ~/.ssh/config +ssh tractatus-vps + +# Or direct with deploy key +ssh -i ~/.ssh/tractatus_deploy_nopass ubuntu@91.134.240.3 + +# Or direct with main key +ssh -i ~/.ssh/id_rsa ubuntu@91.134.240.3 +``` + +--- + +## SSH Keys Authorized on Server + +| Key | File | Purpose | +|-----|------|---------| +| Main user key | `~/.ssh/id_rsa` | Primary access | +| Deploy key (no passphrase) | `~/.ssh/tractatus_deploy_nopass` | Automated deployments | + +**Public key fingerprints:** +``` +id_rsa: SHA256:IcvJSpZZ/0mn8j42weDKgiJPDJioSQfj/arutKmQ3ys +tractatus_deploy_nopass: SHA256:NGTIiqkLMCibVELpQ8BfSO+V3N3FTgATmB9KJi+VjZg +``` + +**Public key backup location:** https://paste.rs/nELRM (main key) + +--- + +## SSH Config (~/.ssh/config) + +``` +Host tractatus-vps + HostName 91.134.240.3 + User ubuntu + IdentityFile ~/.ssh/tractatus_deploy_nopass + IdentitiesOnly yes +``` + +--- + +## When SSH Access Breaks + +### Symptoms +- "Permission denied (publickey,password)" +- "Too many authentication failures" +- Host key changed warning + +### Diagnosis + +```bash +# Test with verbose output +ssh -v -o IdentitiesOnly=yes -i ~/.ssh/tractatus_deploy_nopass ubuntu@91.134.240.3 + +# Check if server is reachable +ping 91.134.240.3 + +# Check if web service is up +curl -I https://agenticgovernance.digital/ +``` + +### Recovery via OVH Rescue Mode + +**When to use:** SSH keys no longer accepted, server unresponsive, or need to recover from boot failure. + +#### Step 1: Enter Rescue Mode + +1. Log into OVH Manager: https://www.ovh.com/manager/ +2. Navigate to: Bare Metal Cloud → VPS → vps-93a693da +3. Click "Reboot in rescue mode" +4. Select "rescue-customer" image +5. OVH emails temporary root password to account email + +#### Step 2: Connect to Rescue Mode + +```bash +# Use the temporary password from email +ssh root@91.134.240.3 +``` + +#### Step 3: Mount Main Disk + +```bash +# Find the disk (usually /dev/sdb1) +lsblk + +# Mount it +mount /dev/sdb1 /mnt/vps + +# Verify +ls /mnt/vps/home/ubuntu +``` + +#### Step 4: Add SSH Keys + +**Method A: Direct authorized_keys edit** +```bash +# Create .ssh directory if needed +mkdir -p /mnt/vps/home/ubuntu/.ssh +chmod 700 /mnt/vps/home/ubuntu/.ssh + +# Add public key +echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCZ8BH+Bx4uO9DTatRZ/YF5xveP/bTyiAWj+qTF7I+ugxgL9/ejSlW1tSn5Seo4XHoEPD5wZCaWig7m1LMezrRq8fDWHbeXkZltK01xhAPU0L0+OvVZMZacW6+vkNfKcNG9vrxV+K/VTPkT+00TRqlHbP8ZWj0OWd92XAoTroKVYMt4L9e7QeJOJmRmHI0uFaJ0Ufexr2gmZyYhgL2p7PP3oiAvM0xlnTwygl06c3iwXpHKWNydOYPSDs3MkVnDjptmWgKv/J+QXksarwEpA4Csc2dLnco+8KrtocUUcAunz6NJfypA0yNWWzf+/OeffkJ2Rueoe8t/lVffXdI7eVuFkmDufE7XMk9YAE/8+XVqok4OV0Q+bjpH8mKlBA3rNobnWs6obBVJD8/5aphE8NdCR4cgIeRSwieFhfzCl+GBZNvs4yuBdKvQQIfCRAKqTgbuc03XERAef6lJUuJrDjwzvvp1Nd8L7AqJoQS6kYGyxXPf/6nWTZtpxoobdGnJ2FZK6OIpAlsWx9LnybMGy19VfaR9JZSAkLdWxGPb6acNUb2xaaqyuXPo4sWpBM27n1HeKMv/7Oh4WL4zrAxDKfN38k1JsjJJVEABuN/pEOb7BCDnTMLKXlTunZgynAZJ/Dxn+zOAyfzaYSNBotlpYy1zj1AmzvS31L7LJy/aSBHuWw== theflow@the-flow' >> /mnt/vps/home/ubuntu/.ssh/authorized_keys + +# Add deploy key +echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdJcKMabIVQRqKqNIpzxHNgxMZ8NOD+9gVCk6dY5uV0 tractatus-deploy' >> /mnt/vps/home/ubuntu/.ssh/authorized_keys + +# Fix permissions +chmod 600 /mnt/vps/home/ubuntu/.ssh/authorized_keys +chown -R 1000:1000 /mnt/vps/home/ubuntu/.ssh +``` + +**Method B: Fetch key from paste.rs** +```bash +curl -s https://paste.rs/nELRM >> /mnt/vps/home/ubuntu/.ssh/authorized_keys +chmod 600 /mnt/vps/home/ubuntu/.ssh/authorized_keys +chown -R 1000:1000 /mnt/vps/home/ubuntu/.ssh +``` + +#### Step 5: Exit Rescue Mode + +1. Return to OVH Manager +2. Click "Restart in normal mode" or "Reboot" +3. Wait 2-3 minutes for server to boot +4. Test SSH access + +--- + +## Common Issues + +### Host Key Changed + +```bash +# Remove old key +ssh-keygen -R 91.134.240.3 + +# Reconnect (will prompt to accept new key) +ssh tractatus-vps +``` + +### Disk Full (Prevents Services Starting) + +```bash +# Check disk usage +df -h + +# Common culprit: MongoDB logs +sudo rm -rf /var/log/mongodb/*.log* + +# Or journal logs +sudo journalctl --vacuum-size=500M +``` + +### Services Not Starting After Reboot + +```bash +# Check status +sudo systemctl status tractatus nginx mongod + +# Start services +sudo systemctl start mongod +sudo systemctl start tractatus +sudo systemctl start nginx + +# Enable for auto-start +sudo systemctl enable tractatus nginx mongod +``` + +--- + +## Deployment + +```bash +# From local tractatus project directory +./scripts/deploy.sh + +# Dry run first +./scripts/deploy.sh --dry-run +``` + +--- + +## Monitoring + +### Service Health +```bash +ssh tractatus-vps "sudo systemctl status tractatus nginx mongod --no-pager" +``` + +### Disk Space +```bash +ssh tractatus-vps "df -h" +``` + +### MongoDB Logs (check for growth) +```bash +ssh tractatus-vps "ls -lh /var/log/mongodb/" +``` + +--- + +## OVH References + +- **Manager:** https://www.ovh.com/manager/ +- **Server Name:** vps-93a693da.vps.ovh.net +- **IP:** 91.134.240.3 +- **Support Ticket (Dec 2025 incident):** CS13385927 + +--- + +## History + +| Date | Event | +|------|-------| +| 2025-12-09 | Security incident - Docker container compromised | +| 2025-12-09 | Recovery via rescue mode, Docker removed | +| 2026-01-18 | SSH access broken, recovered via rescue mode | +| 2026-01-18 | Disk space crisis (45GB MongoDB logs cleared) | +| 2026-01-19 | SSH access broken again | + +--- + +**Last Updated:** 2026-01-19