TheFlow b302960a61 docs: Complete VPS recovery documentation and attack reference

- Update INCIDENT_RECOVERY_2026-01-19.md with complete recovery status
- Create VPS_RECOVERY_REFERENCE.md with step-by-step recovery guide
- Update remediation plan to show executed status
- Update OVH rescue mode doc with resolution notes

Documents the successful complete reinstall approach after multiple
failed partial cleanup attempts. Includes attack indicators, banned
software list, and verification checklist for future incidents.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-20 12:06:32 +13:00

3.6 KiB

Raw Blame History

OVH Rescue Mode - Known Issues and Workarounds

Created: 2026-01-20 Updated: 2026-01-20 VPS: vps-93a693da.vps.ovh.net (91.134.240.3) Status: RESOLVED - Complete reinstall successful

RESOLUTION (2026-01-20)

After multiple failed partial cleanup attempts, the complete reinstall option was used:

User accessed OVH Manager
Selected "Reinstall" option for the VPS
Chose Ubuntu 22.04 LTS
Fresh system was provisioned with new root credentials
Application was redeployed from clean local source

This is the recommended approach for future compromises rather than attempting rescue mode cleanup.

FAILED APPROACH (Do NOT repeat)

The following sequence has failed multiple times across sessions:

❌ ssh root@91.134.240.3 - Fails with host key warning
❌ ssh-keygen -R '91.134.240.3' - Removes old key
❌ ssh root@91.134.240.3 - Fails with "Too many authentication failures"
❌ ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@91.134.240.3 - Connection closed after password prompt
❌ Checking OVH secret link - Password appears expired or invalid
❌ Re-requesting rescue mode - Unclear if this generates working credentials

Result: Unable to SSH into rescue mode despite VPS showing "In rescue" status in OVH Manager.

ROOT CAUSE HYPOTHESES

Password expiration: OVH rescue mode passwords expire after 30 days (or 7 days after first use). The secret link may be stale.
Rescue mode session expired: The rescue environment itself may have timed out even though status shows "In rescue".
OVH anti-hack blocking SSH: The anti-hack system that triggered rescue mode may also be blocking SSH access.
Incorrect password retrieval: The OVH secret link format may have changed, or we're using the wrong credential source.

WHAT TO TRY NEXT SESSION

Option A: OVH KVM/VNC Console

OVH Manager may have a web-based console (KVM/VNC/noVNC)
This bypasses SSH entirely
Look for "KVM" or "Console" or "VNC" button in OVH Manager

Option B: Fresh Rescue Mode Request

In OVH Manager, explicitly DISABLE rescue mode first
Wait for normal boot to complete (or fail)
Then RE-ENABLE rescue mode
This should generate a truly fresh password via email

Option C: Contact OVH Support

If SSH continues to fail, open a support ticket
Reference: Anti-hack incidents on this VPS
Ask them to verify rescue mode SSH is accessible

Option D: Skip Rescue, Direct Reinstall

OVH Manager may allow direct OS reinstall WITHOUT needing rescue mode access
This would skip backup entirely but get a clean system
Data loss acceptable if MongoDB backup exists elsewhere

QUESTIONS TO ASK USER

Is there a KVM/VNC console option in OVH Manager?
Can you try disabling rescue mode and re-enabling it?
Do you have any MongoDB backups stored locally that we could restore from?
Is the OVH password coming from email or from the secret link?

Based on OVH interface (may vary):

OVH Manager → Bare Metal Cloud → VPS → [Your VPS] →
  - Dashboard: Shows status (rescue/normal)
  - Boot: Shows boot mode, option to change
  - Console/KVM: Web-based terminal (if available)
  - Reinstall: Direct OS reinstall option

SUCCESSFUL ACCESS REQUIREMENTS

Before proceeding with backup/reinstall, we need ONE of:

Working SSH access to rescue mode
Working KVM/VNC console access
Confirmation that direct reinstall (no backup) is acceptable

Last Updated: 2026-01-20 Status: BLOCKED - Cannot access rescue mode via SSH

3.6 KiB Raw Blame History