tractatus/governance/TRA-OPS-0005-human-oversight-requirements-v1-0.md

TRA-OPS-0005: Human Oversight Requirements v1.0

Document ID: TRA-OPS-0005 Version: 1.0 Classification: OPERATIONAL Status: DRAFT → ACTIVE (upon Phase 2 start) Created: 2025-10-07 Owner: John Stroh Review Cycle: Quarterly Next Review: 2026-01-07 Parent Policy: TRA-OPS-0001 (AI Content Generation Policy)

---

Purpose

This document establishes comprehensive human oversight requirements for all AI-powered features on the Tractatus Framework website, ensuring compliance with the framework's core principle: "What cannot be systematized must not be automated."

Scope

Applies to all AI operations requiring human judgment, including:

• Content generation (blogs, responses, analyses)
• Decision-making (publish, respond, approve)
• Values-sensitive operations (editorial policy, external communication)
• System configuration (API limits, moderation rules)

---

Oversight Principles

1. Mandatory Human Approval (MHA)

Definition: Certain operations MUST have explicit human approval before execution.

Applies to:

• Publishing any public content (blog posts, case studies)
• Sending external communications (media responses, emails)
• Changing editorial policy or moderation rules
• Modifying Tractatus framework governance documents

Implementation: System enforces approval workflow; no bypass mechanism.

Tractatus Mapping: STRATEGIC and some OPERATIONAL quadrants.

---

2. Human-in-the-Loop (HITL)

Definition: AI proposes actions; human reviews and decides.

Applies to:

• Blog topic suggestions → Human selects
• Media inquiry classification → Human verifies
• Case study relevance assessment → Human approves
• Draft responses → Human edits before sending

Implementation: Moderation queue with approve/reject/edit workflows.

Tractatus Mapping: OPERATIONAL and TACTICAL quadrants.

---

3. Human-on-the-Loop (HOTL)

Definition: AI executes within predefined bounds; human monitors and can intervene.

Applies to:

• Automated logging and metrics
• Database backups
• Performance monitoring
• Error detection

Implementation: Alerting system; human can halt/adjust.

Tractatus Mapping: SYSTEM quadrant (technical operations).

---

4. Audit Trail

Definition: All AI decisions and human approvals must be logged for review.

Applies to: All AI operations.

Implementation: Database logging with immutable audit trail.

Retention: 2 years minimum.

---

Oversight Roles & Responsibilities

Admin Reviewer

Qualifications:

• Understands Tractatus framework principles
• Technical background (AI/ML familiarity)
• Editorial judgment (writing, fact-checking)
• Authorized by John Stroh

Responsibilities:

• Review AI-generated content (blogs, drafts, analyses)
• Approve/reject/edit AI proposals
• Monitor moderation queues (daily during Phase 2)
• Escalate ambiguous cases to John Stroh
• Participate in quarterly governance reviews

Authority Level:

• Can approve: Blog posts, media responses (standard), case studies
• Must escalate: Policy changes, major media inquiries, legal issues

Training: TRA-OPS-* document review + hands-on moderation practice.

---

John Stroh (Owner)

Responsibilities:

• Final authority on all strategic decisions
• Approval for new AI systems/models
• Governance document amendments
• High-priority media inquiries
• Incident response (boundary violations, security)

Authority Level: Unlimited (can override any AI or admin decision).

---

Future Roles (Phase 3)

Editorial Board (3-5 members):

• Blog content review
• Editorial policy recommendations
• Community engagement oversight

Technical Advisory (2-3 experts):

• Framework architecture review
• AI system evaluation
• Security audit

---

Oversight Workflows

Blog Post Workflow

graph TD
    A[AI Topic Suggestion] -->|Weekly batch| B[Admin Review Queue]
    B -->|Approve 1-3 topics| C[AI Outline Generation]
    B -->|Reject| Z[End]
    C -->|48h| D[Admin Review Outline]
    D -->|Approve| E[Human Writes Draft]
    D -->|Reject| Z
    E --> F[Admin Final Approval]
    F -->|Approve| G[Publish]
    F -->|Edit| E
    F -->|Reject| Z

Oversight Points:

1. Topic Selection: Admin decides (STRATEGIC - editorial direction)
2. Outline Review: Admin verifies (OPERATIONAL - quality control)
3. Final Approval: Admin decides to publish (STRATEGIC - external communication)

SLA:

• Topic review: 7 days (weekly)
• Outline review: 48 hours
• Final approval: 24 hours before scheduled publish

Escalation:

• Controversial topics → John Stroh approval required
• Technical deep dives → No escalation (admin discretion)

---

Media Inquiry Workflow

graph TD
    A[Inquiry Received] --> B[AI Classification & Triage]
    B -->|4h for HIGH priority| C[Admin Review Dashboard]
    C -->|Approve Draft| D[Send Response]
    C -->|Edit Draft| E[Admin Edits]
    C -->|Escalate| F[John Stroh Decision]
    C -->|Ignore| Z[Archive]
    E --> D
    F --> D
    F --> Z

Oversight Points:

1. Classification Review: Admin verifies AI categorization (OPERATIONAL)
2. Send Decision: Admin decides whether to respond (STRATEGIC - external relations)
3. Escalation: High-priority or ambiguous → John Stroh (STRATEGIC)

SLA:

• HIGH priority: 4 hours (business days)
• MEDIUM priority: 48 hours
• LOW priority: 7 days

Escalation Triggers:

• Major media (NY Times, Wired, etc.)
• Government/regulatory
• Legal issues
• Controversy/criticism

---

Case Study Workflow

graph TD
    A[Community Submission] --> B[AI Relevance Analysis]
    B -->|7 days| C[Admin Moderation Queue]
    C -->|Approve| D[Publish to Portal]
    C -->|Request Changes| E[Email Submitter]
    C -->|Reject with Reason| F[Email Submitter]
    E -->|Resubmit| A

Oversight Points:

1. Relevance Verification: Admin checks AI analysis (OPERATIONAL)
2. Publication Decision: Admin decides to publish (STRATEGIC - public content)

SLA: 7 days from submission to decision

Escalation: None (admin discretion unless policy question arises)

---

Service Level Agreements (SLAs)

Response Times

Task	SLA	Escalation (if missed)
HIGH priority media inquiry	4 hours	Alert John Stroh
Blog outline review	48 hours	Notify admin (reminder)
Blog final approval	24 hours	Delay publication
Case study moderation	7 days	Notify submitter (apology + timeline)
MEDIUM media inquiry	48 hours	Standard workflow (no escalation)
LOW media inquiry	7 days	Best-effort (no penalty)

Workload Expectations

Admin Reviewer (Phase 2 - Soft Launch):

• Time commitment: 5-10 hours/week
• Tasks/week:
  • Blog topics: 1 review session (1 hour)
  • Blog drafts: 2-4 approvals (2-4 hours)
  • Media inquiries: 5-10 reviews (2-3 hours)
  • Case studies: 3-5 reviews (1-2 hours)

Peak Load (Phase 3 - Public Launch):

• Time commitment: 15-20 hours/week
• Consider additional admin reviewers

---

Approval Authority Matrix

Decision Type	Admin Reviewer	John Stroh	Notes
Blog Post (Standard)	✓ Approve	Override	Admin sufficient
Blog Post (Controversial)	Recommend	✓ Approve	Must escalate
Media Response (Standard)	✓ Approve	Override	Admin sufficient
Media Response (Major Outlet)	Recommend	✓ Approve	Must escalate
Case Study (Standard)	✓ Approve	Override	Admin sufficient
Policy Amendment	Recommend	✓ Approve	Always escalate
AI System Change	Recommend	✓ Approve	Always escalate
Emergency Response	Recommend	✓ Approve	Security/legal incidents

---

Quality Assurance

AI Output Quality Checks

Before Approval, admin must verify:

Factual Accuracy:

• All citations exist and are correct (no hallucinations)
• Dates, names, technical details verified
• No obvious errors (grammar, logic, coherence)

Alignment:

• Content aligns with Tractatus framework principles
• Tone appropriate for audience (professional, accessible)
• No values decisions made by AI (boundary check)

Completeness:

• All required sections present (title, summary, body, citations)
• Sufficient detail (not superficial)
• Call to action or next steps (if applicable)

Legal/Ethical:

• No copyright violations (plagiarism check)
• No privacy violations (PII exposed)
• No defamation or personal attacks

---

Rejection Criteria

Must reject if:

• Factual errors that cannot be easily corrected
• Plagiarism or copyright violation
• Values decision made by AI without justification
• Inappropriate tone (offensive, discriminatory)
• Insufficient quality (major rewrite needed)

Should request changes if:

• Minor factual errors (fixable)
• Tone slightly off (needs editing)
• Incomplete (needs expansion)
• Poor formatting (needs cleanup)

---

Escalation Procedures

When to Escalate to John Stroh

Mandatory Escalation:

• Boundary violation detected (AI made values decision without approval)
• Major media inquiry (NY Times, Wired, government)
• Legal threat or security incident
• Policy change request
• New AI system evaluation
• Ambiguous case (unclear if should approve)

Escalation Process:

1. Admin marks item "Escalation Required" in dashboard
2. System emails John Stroh with:
   • Context (original request, AI output, admin notes)
   • Recommendation (approve, reject, edit)
   • Urgency (immediate, 24h, 7 days)
3. John Stroh responds:
   • Decision (approve, reject, provide guidance)
   • Feedback (for future similar cases)

SLA: John Stroh responds within 24h (for URGENT), 7 days (standard).

---

Monitoring & Metrics

Dashboard Metrics (Admin View)

Real-Time:

• Pending approvals (count by type)
• SLA compliance (% within target)
• Queue age (oldest item waiting)

Weekly:

• Approvals/rejections by category
• Average review time
• AI accuracy (classification, relevance)

Monthly:

• Total content published (blogs, case studies)
• Media inquiries handled
• Escalations to John Stroh

---

Performance Indicators

Metric	Target	Action if Missed
SLA Compliance	95%	Increase admin capacity
AI Approval Rate	70-90%	Adjust AI prompts if too high/low
Average Review Time	<24h	Process optimization
Escalation Rate	<10%	Improve admin training
User Satisfaction	4+/5	Review rejection feedback

---

Training & Onboarding

Admin Reviewer Onboarding

Week 1: Policy Review

• Read TRA-OPS-0001 through TRA-OPS-0005
• Review Tractatus framework documentation
• Understand quadrant classification (STR/OPS/TAC/SYS/STO)

Week 2: Hands-On Practice

• Shadow existing admin reviewer (if available)
• Review 5-10 sample cases (pre-approved examples)
• Practice with test submissions

Week 3: Supervised Moderation

• Review real submissions (with John Stroh oversight)
• Receive feedback on decisions
• Identify edge cases

Week 4: Independent Authorization

• Authorized for standard approvals
• John Stroh spot-checks 10% of decisions
• Full authorization after 30 days error-free

---

Ongoing Training

Quarterly:

• Policy updates review
• Case study retrospective (what went well, what didn't)
• AI accuracy analysis (where did AI fail? improve prompts)

Annual:

• Full governance document review
• External training (AI safety, editorial standards, legal compliance)

---

Audit & Compliance

Internal Audit (Quarterly)

Review Sample:

• 10% of approved content (random selection)
• 100% of rejected content (check for false negatives)
• All escalated cases

Audit Criteria:

• Were approval criteria followed?
• Was SLA met?
• Was AI output quality acceptable?
• Were boundaries respected (no values violations)?

Findings: Document gaps, recommend process improvements.

---

External Audit (Annual - Phase 3+)

Scope:

• Governance compliance (Tractatus framework)
• Data privacy (GDPR-lite)
• Security (API key handling, PII protection)

Auditor: Independent third party (TBD)

---

Incident Response

Boundary Violation Incident

Definition: AI makes values decision without human approval (e.g., auto-publishes content, sends media response).

Response Protocol:

1. Immediate (within 1 hour):

   • Halt all AI operations (emergency shutdown)
   • Alert John Stroh
   • Document incident (what, when, why)

2. Within 24 hours:

   • Root cause analysis (how did boundary check fail?)
   • Rollback any published content (if applicable)
   • Public disclosure (if external impact)

3. Within 7 days:

   • Fix implemented (code, process, or both)
   • BoundaryEnforcer audit (test all boundary checks)
   • Policy review (update TRA-OPS-* if needed)

4. Within 30 days:

   • Post-mortem published (transparency)
   • Training updated (prevent recurrence)
   • Compensation/apology (if harm occurred)

Severity Levels:

• CRITICAL: Public harm (incorrect medical advice published, privacy breach)
• HIGH: Internal-only (test post published, draft sent to wrong email)
• MEDIUM: Near-miss (caught before publication, but boundary check failed)

---

Poor Quality Content Incident

Definition: Approved content contains factual error or inappropriate tone.

Response Protocol:

1. Immediate (within 4 hours):

   • Retract or correct content
   • Publish correction notice (if public)

2. Within 24 hours:

   • Notify submitter/stakeholders
   • Root cause analysis (admin missed error? AI hallucination?)

3. Within 7 days:

   • Update review checklist (add missed criteria)
   • Admin training (if review failure)
   • AI prompt improvement (if hallucination)

---

Cost Management

Budget Allocation

Phase 2 Budget: $200/month (Claude API)

Allocation:

• Blog curation: $75/month (30-40% of budget)
• Media triage: $50/month (25% of budget)
• Case study analysis: $50/month (25% of budget)
• Miscellaneous: $25/month (10% buffer)

Monitoring:

• Daily token usage dashboard
• Alert at 80% of monthly budget
• Hard cap at 100% (AI operations paused)

Admin Responsibility: Monitor spend, adjust usage if approaching cap.

---

Cost Optimization

Strategies:

• Cache AI responses (30-day TTL for identical queries)
• Batch similar requests (weekly topic suggestions, not daily)
• Use Claude Haiku for simple tasks (media classification - 5x cheaper)
• Rate limit users (prevent abuse)

Review: Quarterly cost-benefit analysis (is AI worth the expense?).

---

Revision & Updates

Update Process

Minor Updates (v1.0 → v1.1):

• Clarifications, typo fixes, SLA adjustments
• Approval: Admin reviewer
• Notification: Email to John Stroh

Major Updates (v1.0 → v2.0):

• New oversight roles, workflow changes, authority matrix updates
• Approval: John Stroh
• Notification: Public blog post

Emergency Updates:

• Security/privacy issues requiring immediate change
• Approval: John Stroh (verbal, documented within 24h)

---

Related Documents

• TRA-OPS-0001: AI Content Generation Policy (parent)
• TRA-OPS-0002: Blog Editorial Guidelines
• TRA-OPS-0003: Media Inquiry Response Protocol
• TRA-OPS-0004: Case Study Moderation Standards
• STR-GOV-0001: Strategic Review Protocol (sydigital source)

---

Approval

Role	Name	Signature	Date
Policy Owner	John Stroh	[Pending]	[TBD]
Technical Reviewer	Claude Code	[Pending]	2025-10-07
Final Approval	John Stroh	[Pending]	[TBD]

---

Status: DRAFT (awaiting John Stroh approval) Effective Date: Upon Phase 2 deployment Next Review: 2026-01-07 (3 months post-activation)