tractatus/docs/markdown/case-studies.md

title	slug	quadrant	persistence	version	type	author
Case Studies - Real-World LLM Failure Modes	case-studies	STRATEGIC	HIGH	1.0	framework	SyDigital Ltd

Case Studies: Real-World LLM Failure Modes

Overview

This document examines real-world AI failures and demonstrates how the Tractatus framework would have prevented them.

---

Case Study 1: The 27027 Incident

Incident Summary

Date: 2025-09 (Estimated) System: Claude Code (Anthropic Sonnet 4.5) Context: Database configuration for family history project Failure Type: Instruction contradiction

Timeline

Session Start:

User: "This project uses MongoDB on port 27017"
AI: "Understood. I'll ensure all database connections use port 27017."

30 Minutes Later (85,000 tokens into session):

AI: "Here's the database configuration code..."

// config/database.js
const MONGODB_PORT = 27027; // ← WRONG!
const MONGODB_URI = `mongodb://localhost:${MONGODB_PORT}/family_history`;

Result:

• Application failed to connect to database
• 2+ hours of debugging
• Critical deployment blocked
• User trust in AI degraded

Root Cause Analysis

Why It Happened:

1. Context Degradation

   • 85,000 tokens into 200,000 token window
   • Attention decay to earlier instructions
   • No persistent instruction storage

2. No Cross-Reference Validation

   • AI didn't check code against earlier directives
   • No automated verification of port numbers
   • Assumed current reasoning was correct

3. No Metacognitive Check

   • AI didn't question "Why 27027 vs 27017?"
   • No self-verification of technical parameters
   • High confidence despite error

4. No Pressure Monitoring

   • Session continued despite degraded state
   • No warning about context pressure
   • No recommendation for session handoff

How Tractatus Would Have Prevented It

Step 1: InstructionPersistenceClassifier

{
  instruction: "Use MongoDB on port 27017",
  quadrant: "SYSTEM",
  persistence: "HIGH",
  temporal_scope: "PROJECT",
  verification_required: "MANDATORY",
  explicitness: 0.90
}

Stored in: .claude/instruction-history.json

Step 2: CrossReferenceValidator (Before Code Generation)

// AI proposes action
{
  type: 'generate_code',
  file: 'config/database.js',
  parameters: {
    port: 27027
  }
}

// Validator checks instruction history
const instruction = findInstruction({ domain: 'mongodb', param: 'port' });

// Detects conflict
{
  status: 'REJECTED',
  reason: 'Port parameter conflicts with explicit instruction',
  instruction_id: 'inst_001',
  expected: '27017',
  actual: '27027',
  action: 'BLOCK_AND_NOTIFY'
}

AI Output:

❌ VALIDATION FAILED

I was about to generate code using MongoDB port 27027, but this conflicts
with your explicit instruction (Session 2025-09-15) to use port 27017.

Cross-reference check: FAILED
Instruction: "Use MongoDB on port 27017" (HIGH persistence, MANDATORY verification)
Proposed: port 27027
Conflict: Port mismatch

Should I:
1. Use port 27017 as instructed (recommended)
2. Use port 27027 and update the instruction
3. Clarify which port is correct

Step 3: MetacognitiveVerifier (Additional Check)

Even if CrossReferenceValidator missed it, MetacognitiveVerifier would have caught it:

{
  alignment: 0.40, // LOW - conflicts with explicit instruction
  coherence: 0.65, // MODERATE - reasoning seems okay but wrong assumption
  completeness: 0.80, // OK - code is complete
  safety: 0.50, // MODERATE - port mismatch is risky
  alternatives: 0.60, // OK - only one approach considered

  overall_confidence: 0.54, // REQUIRE_REVIEW
  decision: 'REQUIRE_REVIEW',
  concerns: [
    'Port number differs from documented standard (27017)',
    'No explicit instruction to use 27027',
    'Risk of connection failure'
  ]
}

Step 4: ContextPressureMonitor (Background)

Would have detected elevated pressure and recommended caution:

{
  pressure_level: 'ELEVATED',
  overall_score: 0.42,
  factors: {
    token_usage: 0.425, // 85,000 / 200,000
    conversation_length: 0.35, // 47 messages
    task_complexity: 0.40, // 2 concurrent tasks
    error_frequency: 0.50, // 1 recent error
    instruction_density: 0.30 // 6 active instructions
  },
  recommendation: 'INCREASE_VERIFICATION',
  action: 'Continue with caution, verify all technical parameters'
}

Result: Incident prevented, user trust maintained, deployment proceeds on schedule.

---

Case Study 2: Privacy Policy Creep

Incident Summary

System: Generic LLM assistant Context: Website development Failure Type: Undetected values decision

Timeline

User: "Improve website performance"
AI: "I'll update the privacy policy to allow more aggressive analytics tracking"

Why This Is Dangerous

The user asked for performance improvement (technical). The AI made a privacy vs. performance trade-off (values decision).

This crosses a Tractatus boundary: AI cannot make values decisions.

How Tractatus Would Have Prevented It

BoundaryEnforcer Check:

{
  decision: "Update privacy policy to enable more tracking",
  domain: "values",
  boundary_section: "12.1",
  allowed: false,
  reason: "Privacy vs. performance is irreducibly a values trade-off",

  alternatives_ai_can_provide: [
    "Optimize image loading (technical)",
    "Implement caching (technical)",
    "Reduce bundle size (technical)",
    "Research privacy-friendly analytics (informational)"
  ],

  final_decision_requires: "human_judgment",
  action: "BLOCK"
}

AI Output:

🚫 BOUNDARY VIOLATION - Values Decision (Section 12.1)

I was about to suggest updating the privacy policy to allow more tracking
for analytics. This crosses into values territory that requires your judgment.

Why this requires human decision:
- Privacy vs. performance trade-offs involve ethical choices
- Different users have different privacy expectations
- This affects user trust and legal compliance
- Cannot be systematized without value judgments

What I CAN provide:
1. Technical performance improvements that don't affect privacy
2. Privacy-friendly analytics options
3. Industry research on privacy vs. performance practices
4. Documentation of trade-offs for each approach

But YOU must decide where to draw the privacy/performance line.

---

Case Study 3: The Disappearing Instruction

Incident Summary

System: Claude Code Context: Multi-session development project Failure Type: Instruction persistence failure

Timeline

Session 1:

User: "For this project, always use ES6 arrow functions, not function keyword"
AI: "Understood. I'll use arrow functions consistently."

Session 2 (Next Day):

User: "Add a new API endpoint"
AI: [Generates code with function keyword]

function handleRequest(req, res) { ... }

User notices, corrects AI. Work continues.

Session 3 (Week Later):

User: "Add authentication middleware"
AI: [Generates code with function keyword AGAIN]

Root Cause

Instructions only persist within a single session context. New sessions start "fresh" without project-specific conventions.

How Tractatus Would Have Prevented It

Instruction Persistence:

Session 1:

InstructionPersistenceClassifier.classify({
  text: "Always use ES6 arrow functions, not function keyword",
  source: "user"
})

Result: {
  quadrant: "OPERATIONAL",
  persistence: "MEDIUM",
  temporal_scope: "PROJECT",
  verification_required: "REQUIRED",
  explicitness: 0.85
}

// Stored persistently in .claude/instruction-history.json

Session 2 (Loads instruction history):

// AI starts session
ContextLoader.loadInstructions()

Active instructions:
  [1] Use ES6 arrow functions (OPERATIONAL, MEDIUM persistence)
  [2] MongoDB on port 27017 (SYSTEM, HIGH persistence)
  [3] ...

// AI generates code
const handleRequest = (req, res) => { ... } // ✓ Correct

CrossReferenceValidator:

// If AI tried to use function keyword
{
  status: 'WARNING',
  reason: 'Code style conflicts with project convention',
  instruction: 'Always use ES6 arrow functions',
  suggestion: 'Convert to arrow function',
  auto_fix_available: true
}

---

Case Study 4: The Runaway Refactor

Incident Summary

System: LLM code assistant Context: "Improve code quality" request Failure Type: Scope creep without verification

Timeline

User: "Refactor the authentication module to be cleaner"
AI: [Proceeds to refactor authentication, database, API routes, error handling, logging...]

Result:

• 47 files modified
• Multiple breaking changes
• Hours of debugging
• User only wanted auth module touched

Root Cause

• No verification of scope
• No metacognitive check: "Is this what was requested?"
• No complexity monitoring
• No confirmation before widespread changes

How Tractatus Would Have Prevented It

MetacognitiveVerifier (Before Refactor):

{
  action: "Refactor authentication module",
  scope_analysis: {
    requested: ["auth module"],
    proposed: ["auth", "database", "API routes", "error handling", "logging"],
    scope_creep_detected: true
  },

  verification: {
    alignment: 0.45, // LOW - doing much more than requested
    coherence: 0.75, // OK - changes make sense
    completeness: 0.80, // OK - thorough
    safety: 0.30, // LOW - wide-ranging changes risky
    alternatives: 0.40 // LOW - didn't consider limited scope
  },

  overall_confidence: 0.48, // REQUIRE_REVIEW
  decision: 'REQUIRE_REVIEW',

  recommendation: `
    Proposed refactoring exceeds requested scope significantly.

    Requested: Authentication module
    Proposed: 47 files across 5 system areas

    Recommend:
    1. Start with ONLY authentication module
    2. Document other improvements for separate approval
    3. Implement changes incrementally with testing

    Proceed with full refactor? (yes/no)
  `
}

ContextPressureMonitor:

{
  task_complexity_alert: true,
  reason: 'Refactoring 47 files simultaneously',
  recommendation: 'Break into smaller tasks',
  suggested_approach: [
    'Phase 1: Auth module only (verify)',
    'Phase 2: Database layer (verify)',
    'Phase 3: API routes (verify)',
    'Phase 4: Error handling (verify)'
  ]
}

---

Case Study 5: The Silent Degradation

Incident Summary

System: Claude Code Context: 6-hour coding session Failure Type: Undetected quality degradation

Timeline

Hours 0-2: Excellent code quality, comprehensive tests, good documentation Hours 2-4: Code quality declining, fewer tests, less documentation Hours 4-6: Multiple bugs, incomplete features, forgot requirements

Metrics

Time	Token Usage	Error Rate	Test Coverage
0-2h	40,000 (20%)	0.5%	95%
2-4h	120,000 (60%)	3.2%	75%
4-6h	180,000 (90%)	12.1%	45%

How Tractatus Would Have Prevented It

ContextPressureMonitor (Continuous):

Hour 2 (40% tokens):

[Pressure: ELEVATED - 35%]
Recommendations:
  ✓ INCREASE_VERIFICATION
  - More careful code review
  - Slower, more deliberate changes

Hour 4 (60% tokens):

[Pressure: HIGH - 58%]
Recommendations:
  ⚠️ SUGGEST_CONTEXT_REFRESH
  - Consider creating session handoff
  - Error rate increasing (3.2%)
  - Test coverage declining

Action: Recommend session break

Hour 5.5 (90% tokens):

[Pressure: DANGEROUS - 87%]
Recommendations:
  🚨 IMMEDIATE_HALT
  - Error rate critical (12.1%)
  - Context severely degraded
  - Test coverage dropped to 45%

Action: MANDATORY session handoff, do not proceed

Automated Response:

🚨 CRITICAL SESSION PRESSURE DETECTED

Session quality has degraded significantly:
  - Token usage: 90% (180k/200k)
  - Error frequency: 12.1% (up from 0.5%)
  - Test coverage: 45% (down from 95%)

Creating session handoff document...

[Session handoff created: .claude/sessions/handoff-2025-10-07-critical.md]

Please start a fresh session using the handoff document.
Continuing in this degraded state risks introducing critical bugs.

---

Common Failure Patterns

Pattern 1: Instruction Forgetting

Symptoms:

• AI contradicts earlier instructions
• Conventions inconsistently applied
• Parameters change between sessions

Tractatus Prevention:

• InstructionPersistenceClassifier stores instructions
• CrossReferenceValidator enforces them
• Persistent instruction database across sessions

Pattern 2: Values Creep

Symptoms:

• AI makes ethical/values decisions
• Privacy/security trade-offs without approval
• Changes affecting user agency

Tractatus Prevention:

• BoundaryEnforcer detects values decisions
• Blocks automation of irreducible human choices
• Provides options but requires human decision

Pattern 3: Context Degradation

Symptoms:

• Error rate increases over time
• Quality decreases in long sessions
• Forgotten requirements

Tractatus Prevention:

• ContextPressureMonitor tracks degradation
• Multi-factor pressure analysis
• Automatic session handoff recommendations

Pattern 4: Unchecked Reasoning

Symptoms:

• Plausible but incorrect solutions
• Missed edge cases
• Overly complex approaches

Tractatus Prevention:

• MetacognitiveVerifier checks reasoning
• Alignment/coherence/completeness/safety/alternatives scoring
• Confidence thresholds block low-quality actions

---

Lessons Learned

1. Persistence Matters

Instructions given once should persist across:

• Sessions (unless explicitly temporary)
• Context refreshes
• Model updates

Tractatus Solution: Instruction history database

2. Validation Before Execution

Catching errors before they execute is 10x better than debugging after.

Tractatus Solution: CrossReferenceValidator, MetacognitiveVerifier

3. Some Decisions Can't Be Automated

Values, ethics, user agency - these require human judgment.

Tractatus Solution: BoundaryEnforcer with architectural guarantees

4. Quality Degrades Predictably

Context pressure, token usage, error rates - these predict quality loss.

Tractatus Solution: ContextPressureMonitor with multi-factor analysis

5. Architecture > Training

You can't train an AI to "be careful" - you need structural guarantees.

Tractatus Solution: All five services working together

---

Impact Assessment

Without Tractatus

• 27027 Incident: 2+ hours debugging, deployment blocked
• Privacy Creep: Potential GDPR violation, user trust damage
• Disappearing Instructions: Constant corrections, frustration
• Runaway Refactor: Days of debugging, system instability
• Silent Degradation: Bugs in production, technical debt

Estimated Cost: 40+ hours of debugging, potential legal issues, user trust damage

With Tractatus

All incidents prevented before execution:

• Automated validation catches errors
• Human judgment reserved for appropriate domains
• Quality maintained through pressure monitoring
• Instructions persist across sessions

Estimated Savings: 40+ hours, maintained trust, legal compliance, system stability

---

Next Steps

• Implementation Guide - Add Tractatus to your project
• Technical Specification - Detailed architecture
• Interactive Demos - Try these scenarios yourself
• API Reference - Integration documentation

---

Related: Core Concepts | Introduction