tractatus/docs/planning/PHASE_3_PROJECT_CONTEXT_PLAN.md

# Phase 3: Project Context Awareness - Implementation Plan

**Phase**: 3 of Multi-Project Governance Implementation
**Status**: PLANNING
**Dependencies**: Phase 1 (Rule Manager) ✅, Phase 2 (AI Optimizer) ✅
**Estimated Duration**: 3-4 sessions

---

## Executive Summary

Phase 3 implements **project-aware governance**, enabling the Tractatus system to manage rules across multiple projects with context-specific variable substitution. This transforms the system from single-project to multi-project capable.

### Key Capabilities

1. **Project Registry** - Store and manage multiple projects
2. **Variable Substitution** - Replace `${VAR_NAME}` with project-specific values
3. **Context-Aware Rule Display** - Show rules with values for current project
4. **Project Switching** - UI for selecting active project context
5. **Variable Value Management** - CRUD for project-specific variable values

---

## Problem Statement

### Current Limitations

**From existing codebase analysis:**

The current rule system supports:
- ✅ `scope: 'UNIVERSAL'` vs `'PROJECT_SPECIFIC'`
- ✅ `applicableProjects: ['tractatus', 'family-history']`
- ✅ `variables: ['DB_TYPE', 'DB_PORT']`

**But lacks:**
- ❌ Project metadata storage (name, description, tech stack)
- ❌ Variable value storage per project
- ❌ Runtime variable substitution
- ❌ UI for project selection
- ❌ UI for variable value editing

### Use Case Example

**Rule**: "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME} in ${ENVIRONMENT}."

**Variables**: `DB_TYPE`, `DB_NAME`, `ENVIRONMENT`

**Project: tractatus**
- `DB_TYPE` = "MongoDB"
- `DB_NAME` = "tractatus_dev"
- `ENVIRONMENT` = "development"

**Rendered**: "MongoDB port is 27017. MongoDB database MUST be named tractatus_dev in development."

**Project: family-history**
- `DB_TYPE` = "MongoDB"
- `DB_NAME` = "family_history"
- `ENVIRONMENT` = "production"

**Rendered**: "MongoDB port is 27017. MongoDB database MUST be named family_history in production."

---

## Architecture Overview

### Data Model

```
┌─────────────────┐         ┌──────────────────┐         ┌─────────────────┐
│  Project        │         │ GovernanceRule   │         │ VariableValue   │
├─────────────────┤         ├──────────────────┤         ├─────────────────┤
│ id (PK)         │◄────────│ applicableProjects│◄────────│ projectId (FK)  │
│ name            │         │ variables[]       │         │ variableName    │
│ description     │         │ scope             │         │ value           │
│ techStack       │         │ text (template)   │         │ description     │
│ repositoryUrl   │         │                   │         │ category        │
│ active          │         │                   │         │                 │
│ metadata        │         │                   │         │                 │
└─────────────────┘         └──────────────────┘         └─────────────────┘
```

### Variable Substitution Flow

```
1. User selects project: "tractatus"
   └─> Frontend stores project ID in session/localStorage

2. Frontend requests rules for project
   └─> GET /api/admin/rules?project=tractatus

3. Backend finds applicable rules
   └─> Rules with scope=UNIVERSAL OR applicableProjects includes "tractatus"

4. Backend substitutes variables
   └─> For each rule with variables[], lookup values for project "tractatus"
   └─> Replace ${VAR_NAME} with actual value
   └─> Return rendered text + original template

5. Frontend displays rendered rules
   └─> Shows final text with substituted values
   └─> Provides "Edit Variables" option
```

---

## Phase 3 Tasks Breakdown

### 3.1 Backend: Project Model & API

#### 3.1.1 Create Project Model
**File**: `src/models/Project.model.js`

**Schema**:
```javascript
{
  id: String (unique, e.g., 'tractatus', 'family-history'),
  name: String (e.g., 'Tractatus Framework'),
  description: String,
  techStack: {
    language: String (e.g., 'JavaScript'),
    framework: String (e.g., 'Node.js/Express'),
    database: String (e.g., 'MongoDB'),
    frontend: String (e.g., 'Vanilla JS')
  },
  repositoryUrl: String,
  active: Boolean,
  metadata: {
    defaultBranch: String,
    environment: String (dev/staging/prod),
    lastSynced: Date
  },
  createdAt: Date,
  updatedAt: Date
}
```

**Indexes**:
- `id` (unique)
- `active`
- `name`

---

#### 3.1.2 Create VariableValue Model
**File**: `src/models/VariableValue.model.js`

**Schema**:
```javascript
{
  projectId: String (FK to Project.id),
  variableName: String (e.g., 'DB_TYPE'),
  value: String (e.g., 'MongoDB'),
  description: String (what this variable represents),
  category: String (database/security/config/path/url),
  dataType: String (string/number/boolean/path/url),
  validationRules: {
    required: Boolean,
    pattern: String (regex),
    minLength: Number,
    maxLength: Number
  },
  createdAt: Date,
  updatedAt: Date
}
```

**Indexes**:
- `projectId, variableName` (compound unique)
- `projectId`
- `variableName`

---

#### 3.1.3 Create Project API Endpoints
**File**: `src/controllers/projects.controller.js`

**Endpoints**:

1. **GET /api/admin/projects**
   - List all projects
   - Filter: `?active=true`
   - Response: `{ projects: [...], total: N }`

2. **GET /api/admin/projects/:id**
   - Get single project with all variable values
   - Response: `{ project: {...}, variables: [...] }`

3. **POST /api/admin/projects**
   - Create new project
   - Body: `{ id, name, description, techStack, ... }`
   - Validation: Unique ID, required fields

4. **PUT /api/admin/projects/:id**
   - Update project metadata
   - Body: `{ name, description, ... }`

5. **DELETE /api/admin/projects/:id**
   - Soft delete (set active=false)
   - Cascade: Mark associated variable values as inactive

---

#### 3.1.4 Create Variable API Endpoints
**File**: `src/controllers/variables.controller.js`

**Endpoints**:

1. **GET /api/admin/projects/:projectId/variables**
   - Get all variables for project
   - Response: `{ variables: [...], total: N }`

2. **GET /api/admin/variables/global**
   - Get all unique variable names across all rules
   - Response: `{ variables: ['DB_TYPE', 'DB_NAME', ...] }`

3. **POST /api/admin/projects/:projectId/variables**
   - Create/update variable value for project
   - Body: `{ variableName, value, description, category }`
   - Upsert logic: Update if exists, create if not

4. **PUT /api/admin/projects/:projectId/variables/:variableName**
   - Update variable value
   - Body: `{ value, description, category }`

5. **DELETE /api/admin/projects/:projectId/variables/:variableName**
   - Delete variable value
   - Soft delete or hard delete (TBD)

---

#### 3.1.5 Enhance Rules API for Context
**File**: `src/controllers/rules.controller.js` (modify existing)

**Enhanced Endpoints**:

1. **GET /api/admin/rules?project=:projectId**
   - Filter rules by project
   - Logic: `scope=UNIVERSAL OR applicableProjects includes projectId`
   - Substitute variables with project-specific values
   - Return both `text` (template) and `renderedText` (substituted)

**Response Format**:
```json
{
  "success": true,
  "rules": [
    {
      "id": "inst_019",
      "text": "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME}.",
      "renderedText": "MongoDB port is 27017. MongoDB database MUST be named tractatus_dev.",
      "variables": [
        {"name": "DB_TYPE", "value": "MongoDB"},
        {"name": "DB_NAME", "value": "tractatus_dev"}
      ],
      "scope": "UNIVERSAL",
      "applicableProjects": ["*"]
    }
  ]
}
```

---

#### 3.1.6 Create Variable Substitution Service
**File**: `src/services/VariableSubstitution.service.js`

**Methods**:

```javascript
class VariableSubstitutionService {
  /**
   * Substitute variables in rule text for specific project
   * @param {string} ruleText - Template text with ${VAR} placeholders
   * @param {string} projectId - Project ID
   * @returns {Promise<{renderedText: string, variables: Array}>}
   */
  async substituteVariables(ruleText, projectId) {
    // 1. Extract all ${VAR_NAME} from text
    // 2. Lookup values for projectId
    // 3. Replace placeholders with values
    // 4. Return rendered text + metadata
  }

  /**
   * Extract variable names from text
   * @param {string} text - Text with ${VAR} placeholders
   * @returns {Array<string>} - Variable names
   */
  extractVariables(text) {
    const regex = /\$\{([A-Z_]+)\}/g;
    const matches = [...text.matchAll(regex)];
    return [...new Set(matches.map(m => m[1]))];
  }

  /**
   * Get all unique variables across all rules
   * @returns {Promise<Array<{name: string, usageCount: number}>>}
   */
  async getAllVariables() {
    // Aggregate across all rules
    // Return unique variable names with usage count
  }

  /**
   * Validate variable values for project
   * @param {string} projectId
   * @returns {Promise<{complete: boolean, missing: Array<string>}>}
   */
  async validateProjectVariables(projectId) {
    // Find all rules for project
    // Extract required variables
    // Check if all have values
    // Return missing variables
  }
}
```

---

### 3.2 Frontend: Project Management UI

#### 3.2.1 Create Project Manager Page
**File**: `public/admin/project-manager.html`

**Features**:
- Table listing all projects
- Columns: Name, ID, Tech Stack, Status, Actions
- Filter: Active/Inactive
- Search: By name or ID
- Actions: View, Edit, Delete
- "Add Project" button

**Layout**:
```
┌────────────────────────────────────────────────────────┐
│  Project Manager                        [+ Add Project]│
├────────────────────────────────────────────────────────┤
│  [Search...] [Filter: All ▼]                           │
├────────────────────────────────────────────────────────┤
│  Name            ID           Tech Stack   Status  ⚙️  │
│  Tractatus       tractatus    Node/Mongo   Active  ⋮  │
│  Family History  family-hist  Node/Mongo   Active  ⋮  │
└────────────────────────────────────────────────────────┘
```

---

#### 3.2.2 Create Project Editor Component
**File**: `public/js/admin/project-editor.js`

**Features**:
- Form for project metadata (name, description, tech stack)
- Variable values editor (table)
- Variable discovery (scan rules for ${VAR})
- Validation (required fields, unique ID)
- Save/Cancel actions

**Variable Values Table**:
```
┌────────────────────────────────────────────────────────┐
│  Variable Values for: Tractatus                        │
├────────────────────────────────────────────────────────┤
│  Variable       Value             Category   Actions   │
│  DB_TYPE        MongoDB           database   [Edit]    │
│  DB_NAME        tractatus_dev     database   [Edit]    │
│  ENVIRONMENT    development       config     [Edit]    │
│  [+ Add Variable]                                      │
└────────────────────────────────────────────────────────┘
```

---

#### 3.2.3 Create Project Selector Component
**File**: `public/js/components/project-selector.js`

**Features**:
- Dropdown in navbar (all admin pages)
- Shows current project
- Lists all active projects
- Stores selection in localStorage
- Emits event on change (for other components to react)

**UI Location**: Top navbar, next to logo

```
┌────────────────────────────────────────────────────────┐
│  Tractatus Admin  [Project: Tractatus ▼]    [Logout]  │
└────────────────────────────────────────────────────────┘
```

---

#### 3.2.4 Enhance Rule Manager for Context
**File**: `public/js/admin/rule-manager.js` (modify existing)

**Enhancements**:
1. Add project filter dropdown
2. Show rendered text (with substituted variables)
3. Show template text in edit mode
4. Display variable values used
5. Highlight rules that apply to current project

**Rule Card Enhancement**:
```
┌────────────────────────────────────────────────────────┐
│  inst_019 | SYSTEM | HIGH                              │
│                                                         │
│  Rendered Text (for Tractatus):                        │
│  MongoDB port is 27017. MongoDB database MUST be       │
│  named tractatus_dev in development.                   │
│                                                         │
│  Template: ...${DB_TYPE}...${DB_NAME}...${ENVIRONMENT} │
│  Variables: DB_TYPE=MongoDB, DB_NAME=tractatus_dev     │
│                                                         │
│  [Edit] [Deactivate] [View Details]                   │
└────────────────────────────────────────────────────────┘
```

---

#### 3.2.5 Create Variable Editor Modal
**File**: `public/js/admin/variable-editor.js`

**Features**:
- Modal for editing single variable value
- Field: Variable name (read-only if editing)
- Field: Value (input)
- Field: Description (textarea)
- Field: Category (dropdown)
- Validation: Required fields, data type validation
- Preview: Show how this affects related rules

---

### 3.3 Integration & Testing

#### 3.3.1 Database Seeding
**File**: `scripts/seed-projects.js`

**Seed Data**:
1. **Project: tractatus**
   - Variables: DB_TYPE, DB_NAME, ENVIRONMENT, APP_PORT

2. **Project: family-history**
   - Variables: DB_TYPE, DB_NAME, ENVIRONMENT, S3_BUCKET

---

#### 3.3.2 API Integration Tests
**File**: `tests/integration/projects.test.js`

**Test Cases**:
- ✅ Create project
- ✅ List projects
- ✅ Update project
- ✅ Delete project (soft delete)
- ✅ Add variable value
- ✅ Update variable value
- ✅ Get rules with variable substitution
- ✅ Validate project variables (missing detection)

---

#### 3.3.3 Frontend Integration Tests
**File**: `tests/frontend/project-context.test.js`

**Test Cases**:
- ✅ Project selector displays projects
- ✅ Switching project updates localStorage
- ✅ Rule manager shows context-filtered rules
- ✅ Variable editor saves values
- ✅ Rendered text updates when variables change

---

### 3.4 Documentation

#### 3.4.1 API Documentation
**File**: `docs/api/PROJECTS_API.md`

- Complete endpoint documentation
- Request/response examples
- Variable substitution algorithm
- Data models

---

#### 3.4.2 User Guide
**File**: `docs/USER_GUIDE_PROJECTS.md`

- How to create projects
- How to manage variable values
- How to view context-aware rules
- Troubleshooting variable substitution

---

## Task Priority & Sequence

### Session 1: Backend Foundation
1. ✅ Create Project model
2. ✅ Create VariableValue model
3. ✅ Create VariableSubstitution service
4. ✅ Write unit tests for substitution logic

### Session 2: Backend APIs
5. ✅ Create projects controller + routes
6. ✅ Create variables controller + routes
7. ✅ Enhance rules controller for context
8. ✅ Write API integration tests

### Session 3: Frontend UI
9. ✅ Create project-manager.html
10. ✅ Create project-editor.js
11. ✅ Create project-selector component
12. ✅ Enhance rule-manager for context

### Session 4: Testing & Documentation
13. ✅ Seed database with test projects
14. ✅ Frontend integration testing
15. ✅ Write API documentation
16. ✅ Write user guide

---

## Success Criteria

### Functional
- ✅ Can create/edit/delete projects
- ✅ Can set variable values per project
- ✅ Rules display with substituted values for selected project
- ✅ Variable substitution works correctly
- ✅ Missing variables are detected and reported

### Technical
- ✅ All API endpoints tested (integration tests)
- ✅ Frontend components functional
- ✅ Variable substitution performance < 50ms
- ✅ Database indexes optimized for queries

### User Experience
- ✅ Intuitive project switching
- ✅ Clear visual distinction: template vs rendered text
- ✅ Easy variable value editing
- ✅ Helpful error messages for missing variables

---

## Technical Considerations

### Variable Substitution Algorithm

**Input**:
```javascript
{
  text: "MongoDB port is 27017. ${DB_TYPE} database MUST be named ${DB_NAME}.",
  projectId: "tractatus"
}
```

**Process**:
1. Extract variables: `['DB_TYPE', 'DB_NAME']`
2. Query: `SELECT * FROM variableValues WHERE projectId='tractatus' AND variableName IN ('DB_TYPE', 'DB_NAME')`
3. Build map: `{ DB_TYPE: 'MongoDB', DB_NAME: 'tractatus_dev' }`
4. Replace: `text.replace(/\$\{([A-Z_]+)\}/g, (match, name) => map[name] || match)`
5. Return: `{ renderedText: "...", variables: [...] }`

**Edge Cases**:
- ❓ Variable not defined for project → Keep placeholder `${VAR}` or show error?
- ❓ Circular dependencies → Not possible with simple substitution
- ❓ Nested variables `${${VAR}}` → Not supported
- ❓ Case sensitivity → Variables MUST be UPPER_SNAKE_CASE

---

### Performance Optimization

**Caching Strategy**:
- Cache variable values per project in Redis (TTL: 5 minutes)
- Invalidate cache when variable values updated
- Cache rendered text per rule+project combination

**Database Queries**:
- Compound index on `(projectId, variableName)` for fast lookups
- Batch fetch all variables for project in single query
- Use MongoDB aggregation for efficient rule filtering

---

### Security Considerations

**Variable Value Validation**:
- Sanitize input (prevent XSS in variable values)
- Validate data types (string, number, boolean)
- Pattern matching for paths, URLs
- Max length constraints

**Access Control**:
- Only authenticated admins can manage projects
- Project-level permissions (future: different admins for different projects)
- Audit log for variable value changes

---

## Migration Strategy

### Existing Rules
- Rules already have `scope`, `applicableProjects`, `variables` fields
- No schema changes needed
- Backward compatible: Rules without variables work as-is

### Existing Data
- Create default "tractatus" project
- Migrate existing variable placeholders to VariableValue records
- Scan all rules for `${VAR}` patterns
- Auto-create variable value entries (value = placeholder name initially)

---

## Future Enhancements (Phase 4+)

### Advanced Features
- Variable inheritance (global → project-specific overrides)
- Variable templates (reusable sets of variables)
- Variable validation rules (regex, enums, ranges)
- Variable dependencies (VAR_A requires VAR_B)
- Version control for variable values (track changes over time)
- Import/export variable configurations

### Integration Features
- Auto-discovery of variables from git repositories
- Sync with .env files
- Integration with secret management (Vault, AWS Secrets Manager)
- Variable value encryption for sensitive data

---

## Risks & Mitigation

### Risk 1: Variable Namespace Collisions
**Risk**: Two projects use same variable name for different purposes
**Mitigation**: Namespacing (e.g., `TRACTATUS_DB_NAME` vs `FH_DB_NAME`) or project-scoped lookups (already scoped)

### Risk 2: Performance with Many Variables
**Risk**: Substitution slow with hundreds of variables
**Mitigation**: Caching, batch queries, indexing

### Risk 3: Missing Variable Values
**Risk**: Users forget to set values, rules show `${VAR}`
**Mitigation**: Validation service, UI warnings, required variable detection

---

## Dependencies

### External Libraries
- None (use existing stack)

### Internal Dependencies
- Phase 1: Rule Manager UI ✅
- Phase 2: AI Optimizer ✅
- GovernanceRule model (already supports variables) ✅

---

## Estimated Effort

**Total**: 12-16 hours (3-4 sessions)

| Task Category | Hours |
|--------------|-------|
| Backend models & services | 3-4 |
| Backend APIs & controllers | 3-4 |
| Frontend components | 4-5 |
| Testing | 2-3 |
| Documentation | 1-2 |

---

## Deliverables

### Code
- [ ] Project model + migrations
- [ ] VariableValue model
- [ ] VariableSubstitution service
- [ ] Projects controller + routes
- [ ] Variables controller + routes
- [ ] Enhanced rules controller
- [ ] Project manager UI
- [ ] Project selector component
- [ ] Enhanced rule manager

### Tests
- [ ] Unit tests (models, services)
- [ ] Integration tests (APIs)
- [ ] Frontend tests

### Documentation
- [ ] API reference
- [ ] User guide
- [ ] Developer guide
- [ ] Migration guide

---

## Next Steps

1. **Review & Approve Plan** - Get user confirmation
2. **Session 1: Start Backend** - Create models and services
3. **Iterate & Test** - Build incrementally with testing
4. **Deploy & Validate** - Seed data and verify functionality

---

**Plan Created**: 2025-10-11
**Status**: AWAITING APPROVAL
**Next**: Begin Session 1 - Backend Foundation