tractatus/docs/NEWSLETTER_SENDING_IMPLEMENTATION.md

# Newsletter Sending Implementation Plan

## Overview

✅ **COMPLETED**: Newsletter sending functionality with ProtonBridge email integration, template rendering, and admin controls.

**Email Provider**: ProtonBridge (replacing SendGrid)

---

## Dependencies Installed

```bash
npm install --save handlebars
npm install --save nodemailer
npm install --save html-to-text
```

**Package purposes:**
- `handlebars`: Template engine for email rendering (Mustache-compatible)
- `nodemailer`: SMTP client for ProtonBridge integration
- `html-to-text`: Generate plain-text versions from HTML

---

## Environment Variables

### Production (on VPS)

File: `/var/www/tractatus/.env`

```bash
# Email Service Configuration
EMAIL_ENABLED=true
EMAIL_PROVIDER=proton

# SMTP Configuration (ProtonBridge)
SMTP_HOST=127.0.0.1
SMTP_PORT=1026
SMTP_SECURE=false
SMTP_USER=your-tractatus-email@pm.me
SMTP_PASS=YOUR_BRIDGE_PASSWORD  # From ProtonBridge setup
EMAIL_FROM=your-tractatus-email@pm.me

# Newsletter URLs
NEWSLETTER_UNSUBSCRIBE_BASE_URL=https://agenticgovernance.digital/api/newsletter/unsubscribe
NEWSLETTER_PREFERENCES_BASE_URL=https://agenticgovernance.digital/newsletter/preferences
```

### Development (local)

File: `/home/theflow/projects/tractatus/.env`

```bash
# Email Service Configuration (DISABLED in dev)
EMAIL_ENABLED=false
```

**ProtonBridge Setup:**
See `docs/PROTONBRIDGE_SETUP.md` for complete installation guide.

---

## File Structure

```
src/
├── services/
│   └── email.service.js          # NEW - Email rendering & sending
├── controllers/
│   └── newsletter.controller.js  # MODIFY - Add send() function
├── routes/
│   └── newsletter.routes.js      # MODIFY - Add /send endpoint
└── utils/
    └── template-renderer.util.js  # NEW - Handlebars helpers

email-templates/
├── base-template.html            # EXISTS
├── research-updates-content.html # EXISTS
├── implementation-notes-content.html  # TODO
├── governance-discussions-content.html # TODO
└── project-updates-content.html       # TODO
```

---

## Implementation Steps

### Step 1: Create Email Service (`src/services/email.service.js`)

```javascript
const sgMail = require('@sendgrid/mail');
const Handlebars = require('handlebars');
const { convert: htmlToText } = require('html-to-text');
const fs = require('fs').promises;
const path = require('path');
const logger = require('../utils/logger.util');

// Initialize SendGrid
if (process.env.SENDGRID_API_KEY) {
  sgMail.setApiKey(process.env.SENDGRID_API_KEY);
} else {
  logger.warn('[EmailService] SENDGRID_API_KEY not set - email sending disabled');
}

class EmailService {
  constructor() {
    this.templatesPath = path.join(__dirname, '../../email-templates');
    this.templateCache = new Map();
  }

  /**
   * Load and compile template
   */
  async loadTemplate(templateName) {
    if (this.templateCache.has(templateName)) {
      return this.templateCache.get(templateName);
    }

    const templatePath = path.join(this.templatesPath, `${templateName}.html`);
    const templateContent = await fs.readFile(templatePath, 'utf8');
    const compiled = Handlebars.compile(templateContent);
    
    this.templateCache.set(templateName, compiled);
    return compiled;
  }

  /**
   * Render email from templates
   */
  async renderEmail(tier, variables) {
    try {
      // Load base template
      const baseTemplate = await this.loadTemplate('base-template');
      
      // Load content module for tier
      const contentTemplates = {
        'research': 'research-updates-content',
        'implementation': 'implementation-notes-content',
        'governance': 'governance-discussions-content',
        'project-updates': 'project-updates-content'
      };
      
      const contentTemplateName = contentTemplates[tier];
      if (!contentTemplateName) {
        throw new Error(`Unknown newsletter tier: ${tier}`);
      }
      
      const contentTemplate = await this.loadTemplate(contentTemplateName);
      
      // Render content module
      const renderedContent = contentTemplate(variables);
      
      // Render final email
      const finalHtml = baseTemplate({
        ...variables,
        content_body: renderedContent
      });
      
      // Generate plain text
      const plainText = htmlToText(finalHtml, {
        wordwrap: 80,
        ignoreImages: true,
        preserveNewlines: true
      });
      
      return { html: finalHtml, text: plainText };
    } catch (error) {
      logger.error('[EmailService] Template rendering error:', error);
      throw error;
    }
  }

  /**
   * Send email via SendGrid
   */
  async send({ to, subject, html, text, from = null }) {
    if (!process.env.SENDGRID_API_KEY) {
      logger.warn('[EmailService] Email sending skipped - no API key');
      return { success: false, error: 'Email service not configured' };
    }

    const fromAddress = from || {
      email: process.env.EMAIL_FROM_ADDRESS || 'hello@agenticgovernance.digital',
      name: process.env.EMAIL_FROM_NAME || 'Tractatus'
    };

    try {
      await sgMail.send({
        to,
        from: fromAddress,
        subject,
        html,
        text,
        trackingSettings: {
          clickTracking: { enable: true, enableText: false },
          openTracking: { enable: true }
        }
      });

      logger.info('[EmailService] Email sent successfully', { to, subject });
      return { success: true };
    } catch (error) {
      logger.error('[EmailService] Send error:', error);
      return { success: false, error: error.message };
    }
  }

  /**
   * Send newsletter to subscribers
   */
  async sendNewsletter({ tier, subject, previewText, variables, testMode = false, testEmail = null }) {
    const NewsletterSubscription = require('../models/NewsletterSubscription.model');

    try {
      // Get subscribers for tier
      let subscribers;
      if (testMode) {
        subscribers = [{ email: testEmail, name: 'Test User' }];
      } else {
        subscribers = await NewsletterSubscription.findByInterest(tier, { verified: true, active: true });
      }

      if (subscribers.length === 0) {
        return {
          success: true,
          message: 'No subscribers found for tier',
          sent: 0,
          failed: 0
        };
      }

      // Render base templates
      const results = {
        sent: 0,
        failed: 0,
        errors: []
      };

      // Send to each subscriber
      for (const subscriber of subscribers) {
        try {
          // Generate unsubscribe/preferences links
          const unsubscribeLink = `${process.env.NEWSLETTER_UNSUBSCRIBE_BASE_URL}?token=${subscriber.verification_token}`;
          const preferencesLink = `${process.env.NEWSLETTER_PREFERENCES_BASE_URL}?token=${subscriber.verification_token}`;

          // Render email with subscriber-specific data
          const { html, text } = await this.renderEmail(tier, {
            name: subscriber.name || 'there',
            tier_name: this.getTierDisplayName(tier),
            unsubscribe_link: unsubscribeLink,
            preferences_link: preferencesLink,
            ...variables
          });

          // Send
          const result = await this.send({
            to: subscriber.email,
            subject,
            html,
            text
          });

          if (result.success) {
            results.sent++;
          } else {
            results.failed++;
            results.errors.push({ email: subscriber.email, error: result.error });
          }

        } catch (error) {
          results.failed++;
          results.errors.push({ email: subscriber.email, error: error.message });
          logger.error('[EmailService] Failed to send to subscriber:', { email: subscriber.email, error });
        }
      }

      return {
        success: true,
        message: `Newsletter sent: ${results.sent} successful, ${results.failed} failed`,
        sent: results.sent,
        failed: results.failed,
        errors: results.errors.length > 0 ? results.errors : undefined
      };

    } catch (error) {
      logger.error('[EmailService] Newsletter send error:', error);
      throw error;
    }
  }

  getTierDisplayName(tier) {
    const names = {
      'research': 'Research Updates',
      'implementation': 'Implementation Notes',
      'governance': 'Governance Discussions',
      'project-updates': 'Project Updates'
    };
    return names[tier] || tier;
  }
}

module.exports = new EmailService();
```

### Step 2: Add Method to NewsletterSubscription Model

Add to `src/models/NewsletterSubscription.model.js`:

```javascript
/**
 * Find subscribers by interest tier
 */
static async findByInterest(interest, options = {}) {
  const collection = await getCollection('newsletter_subscriptions');
  
  const query = {
    interests: interest,
    active: options.active !== undefined ? options.active : true
  };
  
  if (options.verified !== undefined) {
    query.verified = options.verified;
  }

  return await collection
    .find(query)
    .sort({ subscribed_at: -1 })
    .limit(options.limit || 10000)
    .toArray();
}
```

### Step 3: Add Send Endpoint to Newsletter Controller

Add to `src/controllers/newsletter.controller.js`:

```javascript
const emailService = require('../services/email.service');

/**
 * Send newsletter (admin only)
 * POST /api/newsletter/admin/send
 */
exports.send = async (req, res) => {
  try {
    const { tier, subject, previewText, variables, testMode, testEmail } = req.body;

    // Validation
    if (!tier || !subject || !variables) {
      return res.status(400).json({
        success: false,
        error: 'Missing required fields: tier, subject, variables'
      });
    }

    if (testMode && !testEmail) {
      return res.status(400).json({
        success: false,
        error: 'Test email required when testMode is true'
      });
    }

    // Send newsletter
    const result = await emailService.sendNewsletter({
      tier,
      subject,
      previewText,
      variables,
      testMode: testMode === true,
      testEmail
    });

    logger.info('[Newsletter] Newsletter sent', { tier, sent: result.sent, failed: result.failed });

    res.json(result);

  } catch (error) {
    logger.error('[Newsletter] Send error:', error);
    res.status(500).json({
      success: false,
      error: 'Failed to send newsletter',
      details: process.env.NODE_ENV === 'development' ? error.message : undefined
    });
  }
};

/**
 * Preview newsletter (admin only)
 * POST /api/newsletter/admin/preview
 */
exports.preview = async (req, res) => {
  try {
    const { tier, variables } = req.body;

    if (!tier || !variables) {
      return res.status(400).json({
        success: false,
        error: 'Missing required fields: tier, variables'
      });
    }

    const { html } = await emailService.renderEmail(tier, {
      name: 'Preview User',
      tier_name: emailService.getTierDisplayName(tier),
      unsubscribe_link: '#',
      preferences_link: '#',
      ...variables
    });

    res.send(html);

  } catch (error) {
    logger.error('[Newsletter] Preview error:', error);
    res.status(500).json({
      success: false,
      error: 'Failed to generate preview',
      details: process.env.NODE_ENV === 'development' ? error.message : undefined
    });
  }
};
```

### Step 4: Add Routes

Add to `src/routes/newsletter.routes.js`:

```javascript
// Admin routes
router.post('/admin/send', requireAuth, requireAdmin, csrfProtection, asyncHandler(newsletterController.send));
router.post('/admin/preview', requireAuth, requireAdmin, csrfProtection, asyncHandler(newsletterController.preview));
```

### Step 5: Update Admin UI JavaScript

Create `public/js/admin/newsletter-sending.js`:

```javascript
// Newsletter sending functionality
document.getElementById('send-newsletter-form')?.addEventListener('submit', async (e) => {
  e.preventDefault();

  const tier = document.getElementById('newsletter-tier').value;
  const subject = document.getElementById('newsletter-subject').value;
  const preview = document.getElementById('newsletter-preview').value;
  const contentRaw = document.getElementById('newsletter-content').value;

  if (!tier || !subject || !contentRaw) {
    alert('Please fill all required fields');
    return;
  }

  let variables;
  try {
    variables = JSON.parse(contentRaw);
  } catch (error) {
    alert('Invalid JSON in content field');
    return;
  }

  const statusDiv = document.getElementById('send-status');
  statusDiv.className = 'bg-blue-50 border border-blue-200 text-blue-800 px-4 py-3 rounded';
  statusDiv.textContent = 'Sending newsletter...';
  statusDiv.classList.remove('hidden');

  try {
    const csrfToken = document.cookie
      .split('; ')
      .find(row => row.startsWith('csrf-token='))
      ?.split('=')[1];

    const response = await fetch('/api/newsletter/admin/send', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-CSRF-Token': csrfToken
      },
      body: JSON.stringify({
        tier,
        subject,
        previewText: preview,
        variables,
        testMode: false
      })
    });

    const result = await response.json();

    if (response.ok && result.success) {
      statusDiv.className = 'bg-green-50 border border-green-200 text-green-800 px-4 py-3 rounded';
      statusDiv.textContent = `Success! ${result.message}`;
    } else {
      throw new Error(result.error || 'Send failed');
    }

  } catch (error) {
    statusDiv.className = 'bg-red-50 border border-red-200 text-red-800 px-4 py-3 rounded';
    statusDiv.textContent = `Error: ${error.message}`;
  }
});

// Preview button
document.getElementById('preview-newsletter-btn')?.addEventListener('click', async () => {
  // Similar implementation - opens preview in new window
});

// Test send button
document.getElementById('test-newsletter-btn')?.addEventListener('click', async () => {
  // Similar implementation with testMode: true
});
```

---

## Testing Checklist

- [ ] Install dependencies
- [ ] Set up SendGrid account and API key
- [ ] Configure environment variables
- [ ] Test template rendering (preview endpoint)
- [ ] Test email sending (test mode with your email)
- [ ] Verify plain-text version generation
- [ ] Test unsubscribe/preferences links
- [ ] Send test newsletter to small group
- [ ] Monitor SendGrid dashboard for delivery rates
- [ ] Test across email clients (Gmail, Outlook, Apple Mail)

---

## Security Considerations

1. **Rate Limiting**: Add rate limiting to send endpoint (max 1 send per 5 minutes)
2. **Admin Only**: Ensure send endpoint requires authentication and admin role
3. **CSRF Protection**: Endpoint uses CSRF token
4. **Input Validation**: Validate tier, subject, and variables
5. **Error Handling**: Don't expose SendGrid errors to client
6. **Logging**: Log all send attempts with admin user ID

---

## Future Enhancements

1. **Scheduled Sending**: Use cron job or scheduler (e.g., `node-cron`)
2. **A/B Testing**: Send different versions to test groups
3. **Analytics Dashboard**: Track open rates, click rates, unsubscribes
4. **Email Templates Editor**: Visual editor for non-technical users
5. **Subscriber Segmentation**: Advanced targeting beyond just interests
6. **Bounce Handling**: Process SendGrid webhooks for bounces/spam reports

---

## Estimated Implementation Time

- **Email Service**: 2 hours
- **Controller & Routes**: 1 hour
- **Admin UI JavaScript**: 1 hour
- **Testing & Debugging**: 2 hours
- **Documentation**: 30 minutes

**Total**: ~6.5 hours

---

## Cost Estimate (SendGrid)

- Free tier: 100 emails/day
- Essentials: $19.95/month for 50,000 emails/month
- Pro: $89.95/month for 100,000 emails/month

**Recommendation**: Start with free tier, upgrade as subscriber base grows.