tractatus/docs/planning/PHASE_3_PROJECT_CONTEXT_PLAN.md

Phase 3: Project Context Awareness - Implementation Plan

Phase: 3 of Multi-Project Governance Implementation Status: PLANNING Dependencies: Phase 1 (Rule Manager) ✅, Phase 2 (AI Optimizer) ✅ Estimated Duration: 3-4 sessions

---

Executive Summary

Phase 3 implements project-aware governance, enabling the Tractatus system to manage rules across multiple projects with context-specific variable substitution. This transforms the system from single-project to multi-project capable.

Key Capabilities

1. Project Registry - Store and manage multiple projects
2. Variable Substitution - Replace ${VAR_NAME} with project-specific values
3. Context-Aware Rule Display - Show rules with values for current project
4. Project Switching - UI for selecting active project context
5. Variable Value Management - CRUD for project-specific variable values

---

Problem Statement

Current Limitations

From existing codebase analysis:

The current rule system supports:

• ✅ scope: 'UNIVERSAL' vs 'PROJECT_SPECIFIC'
• ✅ applicableProjects: ['tractatus', 'family-history']
• ✅ variables: ['DB_TYPE', 'DB_PORT']

But lacks:

• ❌ Project metadata storage (name, description, tech stack)
• ❌ Variable value storage per project
• ❌ Runtime variable substitution
• ❌ UI for project selection
• ❌ UI for variable value editing

Use Case Example

Rule: "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME} in ${ENVIRONMENT}."

Variables: DB_TYPE, DB_NAME, ENVIRONMENT

Project: tractatus

• DB_TYPE = "MongoDB"
• DB_NAME = "tractatus_dev"
• ENVIRONMENT = "development"

Rendered: "MongoDB port is 27017. MongoDB database MUST be named tractatus_dev in development."

Project: family-history

• DB_TYPE = "MongoDB"
• DB_NAME = "family_history"
• ENVIRONMENT = "production"

Rendered: "MongoDB port is 27017. MongoDB database MUST be named family_history in production."

---

Architecture Overview

Data Model

┌─────────────────┐         ┌──────────────────┐         ┌─────────────────┐
│  Project        │         │ GovernanceRule   │         │ VariableValue   │
├─────────────────┤         ├──────────────────┤         ├─────────────────┤
│ id (PK)         │◄────────│ applicableProjects│◄────────│ projectId (FK)  │
│ name            │         │ variables[]       │         │ variableName    │
│ description     │         │ scope             │         │ value           │
│ techStack       │         │ text (template)   │         │ description     │
│ repositoryUrl   │         │                   │         │ category        │
│ active          │         │                   │         │                 │
│ metadata        │         │                   │         │                 │
└─────────────────┘         └──────────────────┘         └─────────────────┘

Variable Substitution Flow

1. User selects project: "tractatus"
   └─> Frontend stores project ID in session/localStorage

2. Frontend requests rules for project
   └─> GET /api/admin/rules?project=tractatus

3. Backend finds applicable rules
   └─> Rules with scope=UNIVERSAL OR applicableProjects includes "tractatus"

4. Backend substitutes variables
   └─> For each rule with variables[], lookup values for project "tractatus"
   └─> Replace ${VAR_NAME} with actual value
   └─> Return rendered text + original template

5. Frontend displays rendered rules
   └─> Shows final text with substituted values
   └─> Provides "Edit Variables" option

---

Phase 3 Tasks Breakdown

3.1 Backend: Project Model & API

3.1.1 Create Project Model

File: src/models/Project.model.js

Schema:

{
  id: String (unique, e.g., 'tractatus', 'family-history'),
  name: String (e.g., 'Tractatus Framework'),
  description: String,
  techStack: {
    language: String (e.g., 'JavaScript'),
    framework: String (e.g., 'Node.js/Express'),
    database: String (e.g., 'MongoDB'),
    frontend: String (e.g., 'Vanilla JS')
  },
  repositoryUrl: String,
  active: Boolean,
  metadata: {
    defaultBranch: String,
    environment: String (dev/staging/prod),
    lastSynced: Date
  },
  createdAt: Date,
  updatedAt: Date
}

Indexes:

• id (unique)
• active
• name

---

3.1.2 Create VariableValue Model

File: src/models/VariableValue.model.js

Schema:

{
  projectId: String (FK to Project.id),
  variableName: String (e.g., 'DB_TYPE'),
  value: String (e.g., 'MongoDB'),
  description: String (what this variable represents),
  category: String (database/security/config/path/url),
  dataType: String (string/number/boolean/path/url),
  validationRules: {
    required: Boolean,
    pattern: String (regex),
    minLength: Number,
    maxLength: Number
  },
  createdAt: Date,
  updatedAt: Date
}

Indexes:

• projectId, variableName (compound unique)
• projectId
• variableName

---

3.1.3 Create Project API Endpoints

File: src/controllers/projects.controller.js

Endpoints:

1. GET /api/admin/projects

   • List all projects
   • Filter: ?active=true
   • Response: { projects: [...], total: N }

2. GET /api/admin/projects/:id

   • Get single project with all variable values
   • Response: { project: {...}, variables: [...] }

3. POST /api/admin/projects

   • Create new project
   • Body: { id, name, description, techStack, ... }
   • Validation: Unique ID, required fields

4. PUT /api/admin/projects/:id

   • Update project metadata
   • Body: { name, description, ... }

5. DELETE /api/admin/projects/:id

   • Soft delete (set active=false)
   • Cascade: Mark associated variable values as inactive

---

3.1.4 Create Variable API Endpoints

File: src/controllers/variables.controller.js

Endpoints:

1. GET /api/admin/projects/:projectId/variables

   • Get all variables for project
   • Response: { variables: [...], total: N }

2. GET /api/admin/variables/global

   • Get all unique variable names across all rules
   • Response: { variables: ['DB_TYPE', 'DB_NAME', ...] }

3. POST /api/admin/projects/:projectId/variables

   • Create/update variable value for project
   • Body: { variableName, value, description, category }
   • Upsert logic: Update if exists, create if not

4. PUT /api/admin/projects/:projectId/variables/:variableName

   • Update variable value
   • Body: { value, description, category }

5. DELETE /api/admin/projects/:projectId/variables/:variableName

   • Delete variable value
   • Soft delete or hard delete (TBD)

---

3.1.5 Enhance Rules API for Context

File: src/controllers/rules.controller.js (modify existing)

Enhanced Endpoints:

1. GET /api/admin/rules?project=:projectId
   • Filter rules by project
   • Logic: scope=UNIVERSAL OR applicableProjects includes projectId
   • Substitute variables with project-specific values
   • Return both text (template) and renderedText (substituted)

Response Format:

{
  "success": true,
  "rules": [
    {
      "id": "inst_019",
      "text": "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME}.",
      "renderedText": "MongoDB port is 27017. MongoDB database MUST be named tractatus_dev.",
      "variables": [
        {"name": "DB_TYPE", "value": "MongoDB"},
        {"name": "DB_NAME", "value": "tractatus_dev"}
      ],
      "scope": "UNIVERSAL",
      "applicableProjects": ["*"]
    }
  ]
}

---

3.1.6 Create Variable Substitution Service

File: src/services/VariableSubstitution.service.js

Methods:

class VariableSubstitutionService {
  /**
   * Substitute variables in rule text for specific project
   * @param {string} ruleText - Template text with ${VAR} placeholders
   * @param {string} projectId - Project ID
   * @returns {Promise<{renderedText: string, variables: Array}>}
   */
  async substituteVariables(ruleText, projectId) {
    // 1. Extract all ${VAR_NAME} from text
    // 2. Lookup values for projectId
    // 3. Replace placeholders with values
    // 4. Return rendered text + metadata
  }

  /**
   * Extract variable names from text
   * @param {string} text - Text with ${VAR} placeholders
   * @returns {Array<string>} - Variable names
   */
  extractVariables(text) {
    const regex = /\$\{([A-Z_]+)\}/g;
    const matches = [...text.matchAll(regex)];
    return [...new Set(matches.map(m => m[1]))];
  }

  /**
   * Get all unique variables across all rules
   * @returns {Promise<Array<{name: string, usageCount: number}>>}
   */
  async getAllVariables() {
    // Aggregate across all rules
    // Return unique variable names with usage count
  }

  /**
   * Validate variable values for project
   * @param {string} projectId
   * @returns {Promise<{complete: boolean, missing: Array<string>}>}
   */
  async validateProjectVariables(projectId) {
    // Find all rules for project
    // Extract required variables
    // Check if all have values
    // Return missing variables
  }
}

---

3.2 Frontend: Project Management UI

3.2.1 Create Project Manager Page

File: public/admin/project-manager.html

Features:

• Table listing all projects
• Columns: Name, ID, Tech Stack, Status, Actions
• Filter: Active/Inactive
• Search: By name or ID
• Actions: View, Edit, Delete
• "Add Project" button

Layout:

┌────────────────────────────────────────────────────────┐
│  Project Manager                        [+ Add Project]│
├────────────────────────────────────────────────────────┤
│  [Search...] [Filter: All ▼]                           │
├────────────────────────────────────────────────────────┤
│  Name            ID           Tech Stack   Status  ⚙️  │
│  Tractatus       tractatus    Node/Mongo   Active  ⋮  │
│  Family History  family-hist  Node/Mongo   Active  ⋮  │
└────────────────────────────────────────────────────────┘

---

3.2.2 Create Project Editor Component

File: public/js/admin/project-editor.js

Features:

• Form for project metadata (name, description, tech stack)
• Variable values editor (table)
• Variable discovery (scan rules for ${VAR})
• Validation (required fields, unique ID)
• Save/Cancel actions

Variable Values Table:

┌────────────────────────────────────────────────────────┐
│  Variable Values for: Tractatus                        │
├────────────────────────────────────────────────────────┤
│  Variable       Value             Category   Actions   │
│  DB_TYPE        MongoDB           database   [Edit]    │
│  DB_NAME        tractatus_dev     database   [Edit]    │
│  ENVIRONMENT    development       config     [Edit]    │
│  [+ Add Variable]                                      │
└────────────────────────────────────────────────────────┘

---

3.2.3 Create Project Selector Component

File: public/js/components/project-selector.js

Features:

• Dropdown in navbar (all admin pages)
• Shows current project
• Lists all active projects
• Stores selection in localStorage
• Emits event on change (for other components to react)

UI Location: Top navbar, next to logo

┌────────────────────────────────────────────────────────┐
│  Tractatus Admin  [Project: Tractatus ▼]    [Logout]  │
└────────────────────────────────────────────────────────┘

---

3.2.4 Enhance Rule Manager for Context

File: public/js/admin/rule-manager.js (modify existing)

Enhancements:

1. Add project filter dropdown
2. Show rendered text (with substituted variables)
3. Show template text in edit mode
4. Display variable values used
5. Highlight rules that apply to current project

Rule Card Enhancement:

┌────────────────────────────────────────────────────────┐
│  inst_019 | SYSTEM | HIGH                              │
│                                                         │
│  Rendered Text (for Tractatus):                        │
│  MongoDB port is 27017. MongoDB database MUST be       │
│  named tractatus_dev in development.                   │
│                                                         │
│  Template: ...${DB_TYPE}...${DB_NAME}...${ENVIRONMENT} │
│  Variables: DB_TYPE=MongoDB, DB_NAME=tractatus_dev     │
│                                                         │
│  [Edit] [Deactivate] [View Details]                   │
└────────────────────────────────────────────────────────┘

---

3.2.5 Create Variable Editor Modal

File: public/js/admin/variable-editor.js

Features:

• Modal for editing single variable value
• Field: Variable name (read-only if editing)
• Field: Value (input)
• Field: Description (textarea)
• Field: Category (dropdown)
• Validation: Required fields, data type validation
• Preview: Show how this affects related rules

---

3.3 Integration & Testing

3.3.1 Database Seeding

File: scripts/seed-projects.js

Seed Data:

1. Project: tractatus

   • Variables: DB_TYPE, DB_NAME, ENVIRONMENT, APP_PORT

2. Project: family-history

   • Variables: DB_TYPE, DB_NAME, ENVIRONMENT, S3_BUCKET

---

3.3.2 API Integration Tests

File: tests/integration/projects.test.js

Test Cases:

• ✅ Create project
• ✅ List projects
• ✅ Update project
• ✅ Delete project (soft delete)
• ✅ Add variable value
• ✅ Update variable value
• ✅ Get rules with variable substitution
• ✅ Validate project variables (missing detection)

---

3.3.3 Frontend Integration Tests

File: tests/frontend/project-context.test.js

Test Cases:

• ✅ Project selector displays projects
• ✅ Switching project updates localStorage
• ✅ Rule manager shows context-filtered rules
• ✅ Variable editor saves values
• ✅ Rendered text updates when variables change

---

3.4 Documentation

3.4.1 API Documentation

File: docs/api/PROJECTS_API.md

• Complete endpoint documentation
• Request/response examples
• Variable substitution algorithm
• Data models

---

3.4.2 User Guide

File: docs/USER_GUIDE_PROJECTS.md

• How to create projects
• How to manage variable values
• How to view context-aware rules
• Troubleshooting variable substitution

---

Task Priority & Sequence

Session 1: Backend Foundation

1. ✅ Create Project model
2. ✅ Create VariableValue model
3. ✅ Create VariableSubstitution service
4. ✅ Write unit tests for substitution logic

Session 2: Backend APIs

5. ✅ Create projects controller + routes
6. ✅ Create variables controller + routes
7. ✅ Enhance rules controller for context
8. ✅ Write API integration tests

Session 3: Frontend UI

9. ✅ Create project-manager.html
10. ✅ Create project-editor.js
11. ✅ Create project-selector component
12. ✅ Enhance rule-manager for context

Session 4: Testing & Documentation

13. ✅ Seed database with test projects
14. ✅ Frontend integration testing
15. ✅ Write API documentation
16. ✅ Write user guide

---

Success Criteria

Functional

• ✅ Can create/edit/delete projects
• ✅ Can set variable values per project
• ✅ Rules display with substituted values for selected project
• ✅ Variable substitution works correctly
• ✅ Missing variables are detected and reported

Technical

• ✅ All API endpoints tested (integration tests)
• ✅ Frontend components functional
• ✅ Variable substitution performance < 50ms
• ✅ Database indexes optimized for queries

User Experience

• ✅ Intuitive project switching
• ✅ Clear visual distinction: template vs rendered text
• ✅ Easy variable value editing
• ✅ Helpful error messages for missing variables

---

Technical Considerations

Variable Substitution Algorithm

Input:

{
  text: "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME}.",
  projectId: "tractatus"
}

Process:

1. Extract variables: ['DB_TYPE', 'DB_NAME']
2. Query: SELECT * FROM variableValues WHERE projectId='tractatus' AND variableName IN ('DB_TYPE', 'DB_NAME')
3. Build map: { DB_TYPE: 'MongoDB', DB_NAME: 'tractatus_dev' }
4. Replace: text.replace(/\$\{([A-Z_]+)\}/g, (match, name) => map[name] || match)
5. Return: { renderedText: "...", variables: [...] }

Edge Cases:

• ❓ Variable not defined for project → Keep placeholder ${VAR} or show error?
• ❓ Circular dependencies → Not possible with simple substitution
• ❓ Nested variables ${${VAR}} → Not supported
• ❓ Case sensitivity → Variables MUST be UPPER_SNAKE_CASE

---

Performance Optimization

Caching Strategy:

• Cache variable values per project in Redis (TTL: 5 minutes)
• Invalidate cache when variable values updated
• Cache rendered text per rule+project combination

Database Queries:

• Compound index on (projectId, variableName) for fast lookups
• Batch fetch all variables for project in single query
• Use MongoDB aggregation for efficient rule filtering

---

Security Considerations

Variable Value Validation:

• Sanitize input (prevent XSS in variable values)
• Validate data types (string, number, boolean)
• Pattern matching for paths, URLs
• Max length constraints

Access Control:

• Only authenticated admins can manage projects
• Project-level permissions (future: different admins for different projects)
• Audit log for variable value changes

---

Migration Strategy

Existing Rules

• Rules already have scope, applicableProjects, variables fields
• No schema changes needed
• Backward compatible: Rules without variables work as-is

Existing Data

• Create default "tractatus" project
• Migrate existing variable placeholders to VariableValue records
• Scan all rules for ${VAR} patterns
• Auto-create variable value entries (value = placeholder name initially)

---

Future Enhancements (Phase 4+)

Advanced Features

• Variable inheritance (global → project-specific overrides)
• Variable templates (reusable sets of variables)
• Variable validation rules (regex, enums, ranges)
• Variable dependencies (VAR_A requires VAR_B)
• Version control for variable values (track changes over time)
• Import/export variable configurations

Integration Features

• Auto-discovery of variables from git repositories
• Sync with .env files
• Integration with secret management (Vault, AWS Secrets Manager)
• Variable value encryption for sensitive data

---

Risks & Mitigation

Risk 1: Variable Namespace Collisions

Risk: Two projects use same variable name for different purposes Mitigation: Namespacing (e.g., TRACTATUS_DB_NAME vs FH_DB_NAME) or project-scoped lookups (already scoped)

Risk 2: Performance with Many Variables

Risk: Substitution slow with hundreds of variables Mitigation: Caching, batch queries, indexing

Risk 3: Missing Variable Values

Risk: Users forget to set values, rules show ${VAR} Mitigation: Validation service, UI warnings, required variable detection

---

Dependencies

External Libraries

• None (use existing stack)

Internal Dependencies

• Phase 1: Rule Manager UI ✅
• Phase 2: AI Optimizer ✅
• GovernanceRule model (already supports variables) ✅

---

Estimated Effort

Total: 12-16 hours (3-4 sessions)

Task Category	Hours
Backend models & services	3-4
Backend APIs & controllers	3-4
Frontend components	4-5
Testing	2-3
Documentation	1-2

---

Deliverables

Code

• Project model + migrations
• VariableValue model
• VariableSubstitution service
• Projects controller + routes
• Variables controller + routes
• Enhanced rules controller
• Project manager UI
• Project selector component
• Enhanced rule manager

Tests

• Unit tests (models, services)
• Integration tests (APIs)
• Frontend tests

Documentation

• API reference
• User guide
• Developer guide
• Migration guide

---

Next Steps

1. Review & Approve Plan - Get user confirmation
2. Session 1: Start Backend - Create models and services
3. Iterate & Test - Build incrementally with testing
4. Deploy & Validate - Seed data and verify functionality

---

Plan Created: 2025-10-11 Status: AWAITING APPROVAL Next: Begin Session 1 - Backend Foundation