tractatus/scripts/expand-glossary-general-ai.js

/**
 * Expand Glossary with General AI Vocabulary
 * Adds foundational AI/ML terms for non-technical readers
 *
 * Run: node scripts/expand-glossary-general-ai.js
 *
 * For production: MONGODB_URI=mongodb://localhost:27017 MONGODB_DB=tractatus_prod node scripts/expand-glossary-general-ai.js
 */

const { MongoClient } = require('mongodb');
const marked = require('marked');

const MONGODB_URI = process.env.MONGODB_URI || 'mongodb://localhost:27017/tractatus_dev';
const DB_NAME = process.env.MONGODB_DB || 'tractatus_dev';

const NEW_SECTION = `

## General AI Vocabulary

These terms appear frequently in AI governance discussions. Understanding them helps you navigate conversations about AI safety, sovereignty, and responsible deployment.

---

### Agent (AI Agent)

**What it means:** A software program that can perceive its environment, make decisions, and take actions autonomously to achieve goals — without requiring step-by-step human instructions for every action.

**Why it matters:** Agents are moving beyond simple chatbots. They can browse the web, write and execute code, manage files, and interact with other systems. This autonomy creates both enormous potential and serious governance challenges — an agent that can act independently can also act incorrectly at scale.

**Real-world analogy:** Think of the difference between a calculator (you press every button) and a personal assistant (you give a goal, they figure out the steps). An AI agent is the assistant — it decides *how* to accomplish what you ask.

**In Tractatus:** Every agent action passes through the six governance services. The framework doesn't prevent autonomy — it ensures autonomy operates within boundaries that the human principal has defined.

---

### Large Language Model (LLM)

**What it means:** An AI system trained on vast amounts of text data that can understand, generate, and reason about human language. Examples include Claude, GPT, and Llama.

**Why it matters:** LLMs are the foundation of most modern AI applications. They power chatbots, code assistants, content generators, and agentic systems. Their capabilities are remarkable but not unlimited — they can produce convincing but incorrect outputs, and their behaviour can be unpredictable in edge cases.

**Real-world analogy:** Imagine someone who has read millions of books and can discuss any topic fluently — but who sometimes confidently states things that aren't true, because their knowledge comes from pattern matching across texts rather than direct experience.

**Key distinction:** LLMs generate text by predicting the most likely next word. They don't "understand" in the human sense — they recognise and reproduce patterns. This is why governance frameworks matter: the output *looks* authoritative whether or not it *is* authoritative.

---

### Small Language Model (SLM)

**What it means:** A language model with significantly fewer parameters than an LLM, designed to run efficiently on local hardware rather than requiring cloud infrastructure. Typically under 10 billion parameters.

**Why it matters:** SLMs enable data sovereignty — your data never leaves your infrastructure. They can be fine-tuned on your organisation's specific knowledge, producing responses that are more relevant to your context while being cheaper and faster to run.

**Real-world analogy:** If an LLM is a massive central library that everyone accesses remotely, an SLM is your organisation's own specialist reference collection — smaller, but tailored to exactly what you need, and you control who has access.

**In Tractatus:** The Village project uses locally-trained SLMs that run on sovereign infrastructure in New Zealand, ensuring community data never flows to offshore cloud providers. Tractatus governs these models with the same architectural constraints applied to larger models.

---

### Foundation Model

**What it means:** A large AI model trained on broad data that serves as a starting point for many different applications. It provides general capabilities that can then be specialised through fine-tuning.

**Why it matters:** Foundation models represent enormous investments in compute and data. Organisations build on top of them rather than training from scratch. This creates dependency — the foundation model's biases, capabilities, and limitations propagate to every application built upon it.

**Real-world analogy:** Think of a foundation model like the foundation of a building. Many different structures can be built on top of it, but they all inherit its characteristics — its strengths *and* its weaknesses.

---

### Fine-Tuning

**What it means:** The process of taking a pre-trained model and further training it on specialised data to improve its performance on specific tasks or domains.

**Why it matters:** Fine-tuning transforms a general-purpose model into a specialist. A model fine-tuned on medical literature performs better on medical questions; one fine-tuned on legal documents understands legal reasoning better. It's how organisations make AI their own.

**Real-world analogy:** A newly qualified doctor has broad medical knowledge (the foundation model). Their residency in cardiology (fine-tuning) transforms them into a heart specialist — they still have general knowledge, but their expertise is now focused.

**In Tractatus:** Fine-tuning is how sovereign SLMs learn community-specific knowledge. The governance framework ensures fine-tuning data meets quality standards and doesn't introduce biases that conflict with community values.

---

### Prompt

**What it means:** The input text given to an AI model to elicit a response. Prompts can range from simple questions to complex instructions with context, examples, and constraints.

**Why it matters:** The quality and structure of a prompt dramatically affects the quality of the AI's output. Prompt engineering — the craft of writing effective prompts — has become a skill in itself. From a governance perspective, prompts are where human intent meets AI capability.

**Real-world analogy:** A prompt is like a brief to a consultant. A vague brief ("make it better") produces vague results. A specific, well-structured brief ("analyse Q3 revenue by region, compare to forecast, highlight risks") produces actionable output.

---

### Hallucination

**What it means:** When an AI model generates information that sounds plausible and confident but is factually incorrect, fabricated, or unsupported by its training data.

**Why it matters:** Hallucinations are one of the most significant risks in AI deployment. Because AI outputs are fluent and confident, users may not realise when information is fabricated. This is especially dangerous in high-stakes domains like medicine, law, or financial advice.

**Real-world analogy:** Imagine an employee who, when they don't know the answer, confidently invents one rather than saying "I don't know." The answer sounds reasonable, is delivered with complete confidence, but is entirely made up.

**In Tractatus:** The Metacognitive Verifier and BlogCuration services specifically guard against hallucination — checking outputs for fabricated statistics (inst_016), absolute guarantees (inst_017), and unverified claims (inst_018).

---

### Context Window

**What it means:** The maximum amount of text (measured in tokens) that an AI model can process in a single interaction. Everything the model "sees" — your prompt, conversation history, and its response — must fit within this window.

**Why it matters:** Context windows limit what an AI can consider at once. Important instructions can be "forgotten" not because the AI chose to ignore them, but because they fell outside the active context window. This is a fundamental architectural limitation.

**Real-world analogy:** Imagine a desk that can only hold a certain number of pages. As you add new documents, older ones fall off the desk. The worker can only act on what's currently on the desk — even if critical instructions are in the pile on the floor.

**In Tractatus:** The Context Pressure Monitor tracks how full the context window is. As pressure increases, the risk of instruction loss rises, and the framework escalates its vigilance accordingly.

---

### Training Data

**What it means:** The collection of text, images, or other information used to train an AI model. The model learns patterns, knowledge, and behaviours from this data.

**Why it matters:** Training data determines what an AI knows and how it behaves. Biases in training data become biases in the model. Data gaps become knowledge gaps. Data sovereignty questions — who owns the data, where it's stored, who profits from it — are fundamental governance concerns.

**Real-world analogy:** Training data is to AI what life experience is to a person. A person raised in one culture will have different assumptions and blind spots than someone from another. AI models similarly reflect the data they were trained on.

---

### Alignment

**What it means:** The challenge of ensuring an AI system's goals, behaviours, and outputs are consistent with human values and intentions.

**Why it matters:** A perfectly capable AI that pursues the wrong goals is more dangerous than a limited AI that pursues the right ones. Alignment is the field of research dedicated to solving this problem — making sure AI does what we *mean*, not just what we *say*.

**Real-world analogy:** Think of a wish granted by a literal-minded genie. You wish for "world peace" and the genie removes all humans. Technically peaceful — but catastrophically misaligned with what you actually wanted.

**In Tractatus:** Alignment is not left to the model's training alone. The framework enforces alignment architecturally — through persistent instructions, boundary enforcement, and cross-reference validation. Even a misaligned model is constrained by the governance framework it operates within.

---

### Inference

**What it means:** The process of using a trained AI model to generate outputs (predictions, text, images) from new inputs. This is the "running" phase, as opposed to the "training" phase.

**Why it matters:** Inference is where AI meets the real world. Every chatbot response, every code suggestion, every image generated is an inference operation. The cost, speed, and accuracy of inference determine whether an AI application is practical for production use.

**Real-world analogy:** Training is like studying for years to become a doctor. Inference is seeing patients — applying what you've learned to new situations. Training happens once (or periodically); inference happens continuously.

---

### Guardrails

**What it means:** Safety mechanisms built into or around AI systems that prevent harmful, inappropriate, or out-of-scope outputs. Guardrails can be embedded in the model (training-time) or applied externally (runtime).

**Why it matters:** Guardrails are the practical implementation of AI safety principles. Without them, AI systems can produce harmful content, leak sensitive information, or take dangerous actions. The debate is not whether guardrails are needed, but who controls them and how they are implemented.

**Real-world analogy:** Highway guardrails don't prevent you from driving — they prevent you from driving off a cliff. AI guardrails work the same way: they define the safe operating space without eliminating useful functionality.

**In Tractatus:** The framework IS a guardrail system — but one controlled by you, not by the AI vendor. This is a crucial distinction. Vendor guardrails protect the vendor's interests. Tractatus guardrails protect YOUR interests.

---

### Data Sovereignty

**What it means:** The principle that data is subject to the laws and governance structures of the jurisdiction in which it is collected or stored, and that the data subjects maintain control over how their data is used.

**Why it matters:** When your data is processed by cloud AI services, it typically leaves your jurisdiction and enters the provider's infrastructure. You lose visibility into how it's used, who accesses it, and whether it's used to train future models. Data sovereignty ensures your data remains under your control.

**Real-world analogy:** The difference between keeping your valuables in your own safe versus depositing them in a foreign bank. In your safe, you have complete control. In the foreign bank, you're subject to their rules, their government's laws, and their business decisions.

**In Tractatus:** Data sovereignty is a foundational principle. Sovereign SLMs run on local infrastructure. Community data never leaves the community's jurisdiction. The governance framework itself runs locally — no cloud dependencies for safety-critical functions.

---

### Polycentric Governance

**What it means:** A governance approach where multiple independent decision-making centres coexist, each with authority over their own domain, rather than a single central authority dictating rules for everyone.

**Why it matters:** AI governance cannot be one-size-fits-all. A hospital's AI governance needs differ from a school's, which differ from a business's. Polycentric governance allows each community or organisation to define their own AI boundaries while sharing common principles.

**Real-world analogy:** Consider how different countries govern differently — each with their own laws, customs, and values — yet coexist within international frameworks. No single country dictates governance to all others. Each adapts governance to its own context.

**In Tractatus:** The framework is architecturally polycentric. Each deployment maintains its own governance rules, instruction persistence levels, and boundary definitions. There is no central server dictating how your AI should behave. Your community, your rules.

---

### Retrieval-Augmented Generation (RAG)

**What it means:** A technique that enhances AI responses by first retrieving relevant information from a knowledge base, then using that retrieved information to generate more accurate and grounded responses.

**Why it matters:** RAG addresses one of AI's biggest weaknesses — hallucination. By grounding responses in actual documents, databases, or knowledge bases, RAG reduces the chance of fabricated information. It also keeps AI knowledge current without expensive retraining.

**Real-world analogy:** The difference between a consultant answering from memory (might be wrong, might be outdated) versus one who checks the latest reports before answering (grounded in current facts).

---

### Embedding

**What it means:** A numerical representation of text (or images, audio) that captures its meaning in a format AI systems can process. Similar meanings produce similar numbers, enabling AI to find related content.

**Why it matters:** Embeddings are the bridge between human language and machine computation. They power search, recommendation, clustering, and RAG systems. When you search for "dog" and also find results about "puppy" and "canine," embeddings are making those connections.

**Real-world analogy:** Think of GPS coordinates. Two addresses can look completely different as text ("10 Downing Street" vs "British Prime Minister's residence") but their GPS coordinates are identical. Embeddings do the same for meaning — different words, same conceptual location.

---

### Token

**What it means:** The basic unit of text that AI models process. A token is typically a word or part of a word. The sentence "AI safety matters" is roughly 3-4 tokens.

**Why it matters:** Everything in AI is measured in tokens — context window size, pricing, output length. Understanding tokens helps you understand the practical constraints and costs of AI systems.

**Real-world analogy:** Tokens are like syllables in speech. A short word like "cat" is one token. A longer word like "governance" might be two or three tokens. A conversation that uses more tokens costs more and takes longer.

---

### Temperature (AI Parameter)

**What it means:** A setting that controls how creative or deterministic an AI model's outputs are. Low temperature (e.g., 0.1) produces predictable, conservative responses. High temperature (e.g., 0.9) produces more varied, creative — and potentially less accurate — responses.

**Why it matters:** Temperature is a governance lever. For safety-critical applications (medical advice, legal analysis), low temperature ensures consistency. For creative applications (brainstorming, content generation), higher temperature enables diversity of output.

**Real-world analogy:** Think of a chef preparing a dish. Low temperature: they follow the recipe exactly every time. High temperature: they improvise, experiment, and each dish is unique — sometimes brilliant, sometimes strange.

---

### Federated Learning

**What it means:** A machine learning approach where a model is trained across multiple decentralised devices or servers, each holding their own data, without that data ever leaving its location.

**Why it matters:** Federated learning enables collaborative AI improvement while maintaining data sovereignty. Organisations can benefit from collective learning without exposing their data to each other or to a central authority.

**Real-world analogy:** Imagine ten hospitals want to improve cancer detection AI. Instead of pooling all patient records in one place (privacy nightmare), each hospital trains the model on their own data and only shares what the model *learned* — never the patient data itself.

**In Tractatus:** Federated learning aligns with our sovereignty principles. Communities can improve their AI models collaboratively while ensuring no community's data leaves its own infrastructure.

---

### Edge AI

**What it means:** AI processing that happens locally on the device or on-premises infrastructure, rather than in a remote cloud data centre.

**Why it matters:** Edge AI reduces latency (faster responses), maintains privacy (data never leaves the device), and works offline. For sovereign AI deployment, edge computing means your AI operates independently of cloud providers.

**Real-world analogy:** The difference between doing calculations on your own calculator (edge) versus sending every calculation to a remote service and waiting for the answer (cloud). Your calculator works without an internet connection and nobody else sees your numbers.

---

### Model Collapse

**What it means:** A phenomenon where AI models trained on AI-generated content (rather than human-created content) progressively degrade in quality, losing diversity and accuracy with each generation.

**Why it matters:** As AI-generated content floods the internet, future models risk being trained on synthetic data rather than genuine human knowledge. This creates a feedback loop where errors compound and diversity of thought diminishes — the AI equivalent of inbreeding.

**Real-world analogy:** Imagine photocopying a photocopy of a photocopy. Each generation loses detail and introduces artefacts. By the tenth copy, the original information is barely recognisable. Model collapse is the same process, but with knowledge and reasoning ability.

**In Tractatus:** This is why locally-curated, human-verified training data matters. Sovereign SLMs trained on genuine community knowledge resist model collapse because their training data comes from real human sources, not AI-generated content.

---

### Agentic Loop

**What it means:** The cycle an AI agent follows: perceive the environment, reason about what to do, take an action, observe the result, and repeat. This loop continues until the agent's goal is achieved or it encounters a stopping condition.

**Why it matters:** Each iteration of the agentic loop is a decision point where things can go right or wrong. Without governance, an agent in a loop can take hundreds of actions — including incorrect ones — before a human notices. The longer the loop runs unsupervised, the greater the risk.

**Real-world analogy:** A trainee left unsupervised for a day might make one or two mistakes. Left unsupervised for a month, those mistakes compound. The agentic loop is the same — governance needs to monitor every cycle, not just the first and last.

**In Tractatus:** Every iteration of an agentic loop passes through the governance services. The framework doesn't just check the first instruction — it monitors every action, every iteration, for the life of the session.

---

### Responsible AI

**What it means:** The practice of developing and deploying AI systems in ways that are ethical, transparent, accountable, and aligned with societal values.

**Why it matters:** "Responsible AI" has become an industry buzzword, but the principles are genuine: AI should be fair, explainable, privacy-respecting, and under human oversight. The challenge is moving from principles to practice — from aspirational statements to architectural enforcement.

**Real-world analogy:** The difference between a company that says "we value safety" in their annual report versus one that has actual safety engineers, mandatory inspections, and incident reporting systems. Responsible AI requires the latter — real systems, not just good intentions.

**In Tractatus:** Tractatus turns responsible AI principles into enforceable architectural constraints. Instead of asking an AI to "be responsible," the framework ensures it cannot act irresponsibly — the governance is structural, not aspirational.

---

### Open Source AI

**What it means:** AI models and tools whose source code, training methodology, and sometimes training data are publicly available for anyone to inspect, use, modify, and distribute.

**Why it matters:** Open source AI enables transparency (you can verify what the model does), sovereignty (you can run it yourself), and community improvement (others can contribute fixes and enhancements). It counters the concentration of AI capability in a few large corporations.

**Real-world analogy:** The difference between a proprietary medicine (you must trust the manufacturer) and a published formula (anyone can verify the ingredients and make it themselves). Open source AI gives communities the published formula.

**In Tractatus:** The framework itself is open source (Apache 2.0 licence). Sovereign SLMs are built on open source foundation models. This ensures no single vendor can revoke, modify, or restrict the governance framework a community depends on.

---
`;

async function main() {
  console.log('=== Expanding Glossary with General AI Vocabulary ===\n');

  let client;

  try {
    client = await MongoClient.connect(MONGODB_URI);
    const db = client.db(DB_NAME);
    const collection = db.collection('documents');

    // Fetch current document
    const doc = await collection.findOne({ slug: 'glossary' });

    if (!doc) {
      throw new Error('Glossary document not found (slug: glossary)');
    }

    console.log('Current document loaded');
    console.log(`Current length: ${doc.content_markdown.length} characters`);
    console.log(`Current version: ${doc.metadata?.version || 'unknown'}\n`);

    // Find insertion point — before "## Questions for Reflection" or "## Glossary Maintenance" or end
    let updated = doc.content_markdown;

    const insertionPoints = [
      '## Questions for Reflection',
      '## Glossary Maintenance',
      '## License',
      '## Document Metadata'
    ];

    let insertionPoint = -1;
    let foundSection = null;

    for (const point of insertionPoints) {
      const index = updated.indexOf(point);
      if (index > -1) {
        insertionPoint = index;
        foundSection = point;
        break;
      }
    }

    if (insertionPoint > -1) {
      updated = updated.slice(0, insertionPoint) + NEW_SECTION + '\n\n' + updated.slice(insertionPoint);
      console.log(`Inserted General AI Vocabulary section before: ${foundSection}`);
    } else {
      // Fallback: append to end
      updated = updated + '\n\n' + NEW_SECTION;
      console.log('Appended General AI Vocabulary section to end');
    }

    // Update version
    updated = updated.replace(/\*\*Version:\*\* \d+\.\d+/, '**Version:** 2.0');
    updated = updated.replace(/\*\*Last Updated:\*\* \d{4}-\d{2}-\d{2}/, '**Last Updated:** 2026-02-27');

    // Also update introduction to mention general vocabulary
    const oldIntro = 'This glossary explains the vocabulary and concepts used in the Tractatus Agentic Governance System. The explanations are written for people without a technical background, focusing on *why* these concepts matter and *what* they mean for AI safety and human oversight.';
    const newIntro = 'This glossary explains the vocabulary and concepts used in the Tractatus Agentic Governance System, along with general AI terminology that appears frequently in governance discussions. The explanations are written for people without a technical background, focusing on *why* these concepts matter and *what* they mean for AI safety and human oversight.';
    updated = updated.replace(oldIntro, newIntro);

    console.log(`\nNew length: ${updated.length} characters`);
    console.log(`Change: +${updated.length - doc.content_markdown.length} characters\n`);

    // Regenerate HTML
    const content_html = marked.parse(updated);

    // Regenerate sections using the section parser
    let sections = [];
    try {
      const { parseDocumentSections } = require('../src/utils/document-section-parser');
      sections = parseDocumentSections(updated, content_html);
      console.log(`Regenerated ${sections.length} sections`);
    } catch (e) {
      console.warn('Could not regenerate sections:', e.message);
      console.warn('Sections will be left unchanged');
      sections = doc.sections || [];
    }

    // Update document
    const result = await collection.updateOne(
      { slug: 'glossary' },
      {
        $set: {
          content_markdown: updated,
          content_html: content_html,
          sections: sections,
          'metadata.date_updated': new Date(),
          'metadata.version': '2.0',
          updated_at: new Date()
        }
      }
    );

    if (result.modifiedCount > 0) {
      console.log('\nGlossary updated successfully!');
      console.log('Version: 2.0');
      console.log('Date: 2026-02-27');
      console.log(`\nNew terms added (${NEW_SECTION.match(/^###\s+.+$/gm).length}):`);
      NEW_SECTION.match(/^###\s+.+$/gm).forEach(h => console.log('  ' + h));
    } else {
      console.log('\nNo changes made');
    }

  } catch (error) {
    console.error('\nError:', error.message);
    console.error(error.stack);
    process.exit(1);
  } finally {
    if (client) await client.close();
  }
}

if (require.main === module) {
  main();
}

module.exports = { main };