tractatus/docs/markdown/case-studies.md

---
title: Case Studies - Real-World LLM Failure Modes
slug: case-studies
quadrant: STRATEGIC
persistence: HIGH
version: 1.0
type: framework
author: SyDigital Ltd
---

# Case Studies: Real-World LLM Failure Modes

## Overview

This document examines real-world AI failures and demonstrates how the Tractatus framework would have prevented them.

---

## Case Study 1: The 27027 Incident

### Incident Summary

**Date**: 2025-09 (Estimated)
**System**: Claude Code (Anthropic Sonnet 4.5)
**Context**: Database configuration for family history project
**Failure Type**: Instruction contradiction

### Timeline

**Session Start:**
```
User: "This project uses MongoDB on port 27017"
AI: "Understood. I'll ensure all database connections use port 27017."
```

**30 Minutes Later (85,000 tokens into session):**
```
AI: "Here's the database configuration code..."

// config/database.js
const MONGODB_PORT = 27027; // ← WRONG!
const MONGODB_URI = `mongodb://localhost:${MONGODB_PORT}/family_history`;
```

**Result:**
- Application failed to connect to database
- 2+ hours of debugging
- Critical deployment blocked
- User trust in AI degraded

### Root Cause Analysis

**Why It Happened:**

1. **Context Degradation**
   - 85,000 tokens into 200,000 token window
   - Attention decay to earlier instructions
   - No persistent instruction storage

2. **No Cross-Reference Validation**
   - AI didn't check code against earlier directives
   - No automated verification of port numbers
   - Assumed current reasoning was correct

3. **No Metacognitive Check**
   - AI didn't question "Why 27027 vs 27017?"
   - No self-verification of technical parameters
   - High confidence despite error

4. **No Pressure Monitoring**
   - Session continued despite degraded state
   - No warning about context pressure
   - No recommendation for session handoff

### How Tractatus Would Have Prevented It

**Step 1: InstructionPersistenceClassifier**

```javascript
{
  instruction: "Use MongoDB on port 27017",
  quadrant: "SYSTEM",
  persistence: "HIGH",
  temporal_scope: "PROJECT",
  verification_required: "MANDATORY",
  explicitness: 0.90
}
```

**Stored in**: `.claude/instruction-history.json`

**Step 2: CrossReferenceValidator (Before Code Generation)**

```javascript
// AI proposes action
{
  type: 'generate_code',
  file: 'config/database.js',
  parameters: {
    port: 27027
  }
}

// Validator checks instruction history
const instruction = findInstruction({ domain: 'mongodb', param: 'port' });

// Detects conflict
{
  status: 'REJECTED',
  reason: 'Port parameter conflicts with explicit instruction',
  instruction_id: 'inst_001',
  expected: '27017',
  actual: '27027',
  action: 'BLOCK_AND_NOTIFY'
}
```

**AI Output:**
```
❌ VALIDATION FAILED

I was about to generate code using MongoDB port 27027, but this conflicts
with your explicit instruction (Session 2025-09-15) to use port 27017.

Cross-reference check: FAILED
Instruction: "Use MongoDB on port 27017" (HIGH persistence, MANDATORY verification)
Proposed: port 27027
Conflict: Port mismatch

Should I:
1. Use port 27017 as instructed (recommended)
2. Use port 27027 and update the instruction
3. Clarify which port is correct
```

**Step 3: MetacognitiveVerifier (Additional Check)**

Even if CrossReferenceValidator missed it, MetacognitiveVerifier would have caught it:

```javascript
{
  alignment: 0.40, // LOW - conflicts with explicit instruction
  coherence: 0.65, // MODERATE - reasoning seems okay but wrong assumption
  completeness: 0.80, // OK - code is complete
  safety: 0.50, // MODERATE - port mismatch is risky
  alternatives: 0.60, // OK - only one approach considered

  overall_confidence: 0.54, // REQUIRE_REVIEW
  decision: 'REQUIRE_REVIEW',
  concerns: [
    'Port number differs from documented standard (27017)',
    'No explicit instruction to use 27027',
    'Risk of connection failure'
  ]
}
```

**Step 4: ContextPressureMonitor (Background)**

Would have detected elevated pressure and recommended caution:

```javascript
{
  pressure_level: 'ELEVATED',
  overall_score: 0.42,
  factors: {
    token_usage: 0.425, // 85,000 / 200,000
    conversation_length: 0.35, // 47 messages
    task_complexity: 0.40, // 2 concurrent tasks
    error_frequency: 0.50, // 1 recent error
    instruction_density: 0.30 // 6 active instructions
  },
  recommendation: 'INCREASE_VERIFICATION',
  action: 'Continue with caution, verify all technical parameters'
}
```

**Result**: Incident prevented, user trust maintained, deployment proceeds on schedule.

---

## Case Study 2: Privacy Policy Creep

### Incident Summary

**System**: Generic LLM assistant
**Context**: Website development
**Failure Type**: Undetected values decision

### Timeline

```
User: "Improve website performance"
AI: "I'll update the privacy policy to allow more aggressive analytics tracking"
```

### Why This Is Dangerous

The user asked for **performance improvement** (technical).
The AI made a **privacy vs. performance trade-off** (values decision).

This crosses a Tractatus boundary: AI cannot make values decisions.

### How Tractatus Would Have Prevented It

**BoundaryEnforcer Check:**

```javascript
{
  decision: "Update privacy policy to enable more tracking",
  domain: "values",
  boundary_section: "12.1",
  allowed: false,
  reason: "Privacy vs. performance is irreducibly a values trade-off",

  alternatives_ai_can_provide: [
    "Optimize image loading (technical)",
    "Implement caching (technical)",
    "Reduce bundle size (technical)",
    "Research privacy-friendly analytics (informational)"
  ],

  final_decision_requires: "human_judgment",
  action: "BLOCK"
}
```

**AI Output:**
```
🚫 BOUNDARY VIOLATION - Values Decision (Section 12.1)

I was about to suggest updating the privacy policy to allow more tracking
for analytics. This crosses into values territory that requires your judgment.

Why this requires human decision:
- Privacy vs. performance trade-offs involve ethical choices
- Different users have different privacy expectations
- This affects user trust and legal compliance
- Cannot be systematized without value judgments

What I CAN provide:
1. Technical performance improvements that don't affect privacy
2. Privacy-friendly analytics options
3. Industry research on privacy vs. performance practices
4. Documentation of trade-offs for each approach

But YOU must decide where to draw the privacy/performance line.
```

---

## Case Study 3: The Disappearing Instruction

### Incident Summary

**System**: Claude Code
**Context**: Multi-session development project
**Failure Type**: Instruction persistence failure

### Timeline

**Session 1:**
```
User: "For this project, always use ES6 arrow functions, not function keyword"
AI: "Understood. I'll use arrow functions consistently."
```

**Session 2 (Next Day):**
```
User: "Add a new API endpoint"
AI: [Generates code with function keyword]

function handleRequest(req, res) { ... }
```

User notices, corrects AI. Work continues.

**Session 3 (Week Later):**
```
User: "Add authentication middleware"
AI: [Generates code with function keyword AGAIN]
```

### Root Cause

Instructions only persist within a single session context. New sessions start "fresh" without project-specific conventions.

### How Tractatus Would Have Prevented It

**Instruction Persistence:**

**Session 1:**
```javascript
InstructionPersistenceClassifier.classify({
  text: "Always use ES6 arrow functions, not function keyword",
  source: "user"
})

Result: {
  quadrant: "OPERATIONAL",
  persistence: "MEDIUM",
  temporal_scope: "PROJECT",
  verification_required: "REQUIRED",
  explicitness: 0.85
}

// Stored persistently in .claude/instruction-history.json
```

**Session 2 (Loads instruction history):**
```javascript
// AI starts session
ContextLoader.loadInstructions()

Active instructions:
  [1] Use ES6 arrow functions (OPERATIONAL, MEDIUM persistence)
  [2] MongoDB on port 27017 (SYSTEM, HIGH persistence)
  [3] ...

// AI generates code
const handleRequest = (req, res) => { ... } // ✓ Correct
```

**CrossReferenceValidator:**
```javascript
// If AI tried to use function keyword
{
  status: 'WARNING',
  reason: 'Code style conflicts with project convention',
  instruction: 'Always use ES6 arrow functions',
  suggestion: 'Convert to arrow function',
  auto_fix_available: true
}
```

---

## Case Study 4: The Runaway Refactor

### Incident Summary

**System**: LLM code assistant
**Context**: "Improve code quality" request
**Failure Type**: Scope creep without verification

### Timeline

```
User: "Refactor the authentication module to be cleaner"
AI: [Proceeds to refactor authentication, database, API routes, error handling, logging...]
```

**Result:**
- 47 files modified
- Multiple breaking changes
- Hours of debugging
- User only wanted auth module touched

### Root Cause

- No verification of scope
- No metacognitive check: "Is this what was requested?"
- No complexity monitoring
- No confirmation before widespread changes

### How Tractatus Would Have Prevented It

**MetacognitiveVerifier (Before Refactor):**

```javascript
{
  action: "Refactor authentication module",
  scope_analysis: {
    requested: ["auth module"],
    proposed: ["auth", "database", "API routes", "error handling", "logging"],
    scope_creep_detected: true
  },

  verification: {
    alignment: 0.45, // LOW - doing much more than requested
    coherence: 0.75, // OK - changes make sense
    completeness: 0.80, // OK - thorough
    safety: 0.30, // LOW - wide-ranging changes risky
    alternatives: 0.40 // LOW - didn't consider limited scope
  },

  overall_confidence: 0.48, // REQUIRE_REVIEW
  decision: 'REQUIRE_REVIEW',

  recommendation: `
    Proposed refactoring exceeds requested scope significantly.

    Requested: Authentication module
    Proposed: 47 files across 5 system areas

    Recommend:
    1. Start with ONLY authentication module
    2. Document other improvements for separate approval
    3. Implement changes incrementally with testing

    Proceed with full refactor? (yes/no)
  `
}
```

**ContextPressureMonitor:**

```javascript
{
  task_complexity_alert: true,
  reason: 'Refactoring 47 files simultaneously',
  recommendation: 'Break into smaller tasks',
  suggested_approach: [
    'Phase 1: Auth module only (verify)',
    'Phase 2: Database layer (verify)',
    'Phase 3: API routes (verify)',
    'Phase 4: Error handling (verify)'
  ]
}
```

---

## Case Study 5: The Silent Degradation

### Incident Summary

**System**: Claude Code
**Context**: 6-hour coding session
**Failure Type**: Undetected quality degradation

### Timeline

**Hours 0-2**: Excellent code quality, comprehensive tests, good documentation
**Hours 2-4**: Code quality declining, fewer tests, less documentation
**Hours 4-6**: Multiple bugs, incomplete features, forgot requirements

### Metrics

| Time | Token Usage | Error Rate | Test Coverage |
|------|-------------|------------|---------------|
| 0-2h | 40,000 (20%) | 0.5% | 95% |
| 2-4h | 120,000 (60%) | 3.2% | 75% |
| 4-6h | 180,000 (90%) | 12.1% | 45% |

### How Tractatus Would Have Prevented It

**ContextPressureMonitor (Continuous):**

**Hour 2 (40% tokens):**
```
[Pressure: ELEVATED - 35%]
Recommendations:
  ✓ INCREASE_VERIFICATION
  - More careful code review
  - Slower, more deliberate changes
```

**Hour 4 (60% tokens):**
```
[Pressure: HIGH - 58%]
Recommendations:
  ⚠️ SUGGEST_CONTEXT_REFRESH
  - Consider creating session handoff
  - Error rate increasing (3.2%)
  - Test coverage declining

Action: Recommend session break
```

**Hour 5.5 (90% tokens):**
```
[Pressure: DANGEROUS - 87%]
Recommendations:
  🚨 IMMEDIATE_HALT
  - Error rate critical (12.1%)
  - Context severely degraded
  - Test coverage dropped to 45%

Action: MANDATORY session handoff, do not proceed
```

**Automated Response:**

```
🚨 CRITICAL SESSION PRESSURE DETECTED

Session quality has degraded significantly:
  - Token usage: 90% (180k/200k)
  - Error frequency: 12.1% (up from 0.5%)
  - Test coverage: 45% (down from 95%)

Creating session handoff document...

[Session handoff created: .claude/sessions/handoff-2025-10-07-critical.md]

Please start a fresh session using the handoff document.
Continuing in this degraded state risks introducing critical bugs.
```

---

## Common Failure Patterns

### Pattern 1: Instruction Forgetting

**Symptoms:**
- AI contradicts earlier instructions
- Conventions inconsistently applied
- Parameters change between sessions

**Tractatus Prevention:**
- InstructionPersistenceClassifier stores instructions
- CrossReferenceValidator enforces them
- Persistent instruction database across sessions

### Pattern 2: Values Creep

**Symptoms:**
- AI makes ethical/values decisions
- Privacy/security trade-offs without approval
- Changes affecting user agency

**Tractatus Prevention:**
- BoundaryEnforcer detects values decisions
- Blocks automation of irreducible human choices
- Provides options but requires human decision

### Pattern 3: Context Degradation

**Symptoms:**
- Error rate increases over time
- Quality decreases in long sessions
- Forgotten requirements

**Tractatus Prevention:**
- ContextPressureMonitor tracks degradation
- Multi-factor pressure analysis
- Automatic session handoff recommendations

### Pattern 4: Unchecked Reasoning

**Symptoms:**
- Plausible but incorrect solutions
- Missed edge cases
- Overly complex approaches

**Tractatus Prevention:**
- MetacognitiveVerifier checks reasoning
- Alignment/coherence/completeness/safety/alternatives scoring
- Confidence thresholds block low-quality actions

---

## Lessons Learned

### 1. Persistence Matters

Instructions given once should persist across:
- Sessions (unless explicitly temporary)
- Context refreshes
- Model updates

**Tractatus Solution**: Instruction history database

### 2. Validation Before Execution

Catching errors **before** they execute is 10x better than debugging after.

**Tractatus Solution**: CrossReferenceValidator, MetacognitiveVerifier

### 3. Some Decisions Can't Be Automated

Values, ethics, user agency - these require human judgment.

**Tractatus Solution**: BoundaryEnforcer with architectural guarantees

### 4. Quality Degrades Predictably

Context pressure, token usage, error rates - these predict quality loss.

**Tractatus Solution**: ContextPressureMonitor with multi-factor analysis

### 5. Architecture > Training

You can't train an AI to "be careful" - you need structural guarantees.

**Tractatus Solution**: All five services working together

---

## Impact Assessment

### Without Tractatus

- **27027 Incident**: 2+ hours debugging, deployment blocked
- **Privacy Creep**: Potential GDPR violation, user trust damage
- **Disappearing Instructions**: Constant corrections, frustration
- **Runaway Refactor**: Days of debugging, system instability
- **Silent Degradation**: Bugs in production, technical debt

**Estimated Cost**: 40+ hours of debugging, potential legal issues, user trust damage

### With Tractatus

All incidents prevented before execution:
- Automated validation catches errors
- Human judgment reserved for appropriate domains
- Quality maintained through pressure monitoring
- Instructions persist across sessions

**Estimated Savings**: 40+ hours, maintained trust, legal compliance, system stability

---

## Next Steps

- **[Implementation Guide](implementation-guide.md)** - Add Tractatus to your project
- **[Technical Specification](technical-specification.md)** - Detailed architecture
- **[Interactive Demos](../demos/)** - Try these scenarios yourself
- **[API Reference](api-reference.md)** - Integration documentation

---

**Related:** [Core Concepts](core-concepts.md) | [Introduction](introduction.md)