# iPaaS Flow Notification System

## Table of Contents

- [Overview](#overview)
- [Architecture Components](#architecture-components)
  - [1. Flow Completion Detection](#1-flow-completion-detection)
  - [2. Notification Service](#2-notification-service)
  - [3. Email System](#3-email-system)
  - [4. Database Models](#4-database-models)
- [Recipient Fallback Chain](#recipient-fallback-chain)
- [Default Behavior](#default-behavior)
- [Configuration](#configuration)
  - [Tenant-Level Settings](#tenant-level-settings)
  - [Flow-Level Recipients](#flow-level-recipients)
- [Queue Processing](#queue-processing)
- [Data Flow Diagram](#data-flow-diagram)
- [Developer Guidelines](#developer-guidelines)
  - [Adding Recipients](#adding-recipients)
  - [Testing Notifications](#testing-notifications)
  - [Debugging Issues](#debugging-issues)
- [Troubleshooting Guide](#troubleshooting-guide)
  - [Issue: Notifications Not Sending](#issue-notifications-not-sending)
  - [Issue: Wrong Recipients](#issue-wrong-recipients)
  - [Issue: Emails Queued But Not Delivered](#issue-emails-queued-but-not-delivered)
- [Related Documentation](#related-documentation)
- [Changelog](#changelog)

---

## Overview

The iPaaS notification system sends email alerts when flow executions complete with failures. This provides immediate visibility into data integration issues without requiring users to monitor flow dashboards constantly.

**Key Features**:
- ✅ Automatic failure detection on flow completion
- ✅ Three-tier recipient fallback system
- ✅ Enabled by default (safe, error-only notifications)
- ✅ Queue-based email delivery
- ✅ Tenant-isolated configuration
- ✅ Per-flow recipient customization

**Design Rationale**:
- Notifications only send when flows have failures (never on success)
- Only triggers if tenant is actively using iPaaS flows
- Safe to enable by default without spam risk
- Provides multiple configuration layers for flexibility

---

## Architecture Components

### 1. Flow Completion Detection

**Observer**: `App\Observers\FlowCompletedObserver`

Listens for `FlowCompleted` events and coordinates the notification process:

```php
class FlowCompletedObserver
{
    public function handle(FlowCompleted $event): void
    {
        // 1. Setup tenant connection
        setupTenantConnection($event->tenant_database);

        // 2. Update flow metrics
        $this->updateFlowMetrics($event->flow_id, $event->total_records, $event->tenant_database);

        // 3. Ensure metrics are finalized
        $this->ensureMetricsFinalized($event->flow_id, $event->tenant_database);

        // 4. Check for failures and send notifications
        $this->handleFlowNotifications($event);
    }
}
```

**Key Responsibilities**:
- Retrieves failure counts from `FlowMetricsService`
- Detects both normal failures (`failed_records > 0`) and critical failures (`failed_records === -1`)
- Enriches event data with metrics before notification

**Failure Types**:
- **Normal Failures**: Individual record processing failures (`failed_records > 0`)
- **Critical Failures**: System-level errors preventing flow execution (`failed_records === -1`)

### 2. Notification Service

**Service**: `Domain\Ipaas\Notifications\Services\FlowNotificationService`

Central orchestration service for notification logic:

```php
class FlowNotificationService
{
    public function handleFlowCompletion($event): void
    {
        // 1. Check if notifications enabled (default: true)
        if (!$this->areNotificationsEnabled($event->tenant_database)) {
            return;
        }

        // 2. Check if flow had failures
        if (!$this->hasFailures($event)) {
            return;
        }

        // 3. Resolve recipients (flow → custom → user ID 1)
        $recipients = $this->getFlowRecipients($event->flow_id, $event->tenant_database);

        // 4. Send notifications to all recipients
        $this->sendNotifications($recipients, $event);
    }
}
```

**Key Methods**:
- `areNotificationsEnabled()`: Checks tenant settings, **defaults to enabled**
- `hasFailures()`: Detects failures (normal or critical)
- `getFlowRecipients()`: Implements fallback chain
- `sendNotifications()`: Queues email for each recipient

### 3. Email System

**Mailable**: `App\Mail\FlowNotificationMail`

Generates notification emails with flow execution details:

**Queue Configuration**:
```php
// Queued to dedicated 'mail' queue
$this->onQueue('mail');
```

**Subject Lines**:
- Normal failures: `"🚨 Flow Execution Alert: {Flow Name} - {X} Records Failed"`
- Critical failures: `"🚨 CRITICAL Flow Failure: {Flow Name} - System Error"`

**Email Content**:
- Flow name and execution summary
- Start/end timestamps and duration
- Success/failure counts and success rate
- Direct link to flow editor
- Debug information (for test notifications)

**Template**: `resources/views/emails/flow-notification.blade.php`
- Professional HTML design
- Mobile-responsive layout
- Critical failure styling variations

### 4. Database Models

#### `FlowNotificationRecipient`
**Location**: `Domain\Ipaas\Notifications\Models\FlowNotificationRecipient`

Stores per-flow notification recipients:

```php
class FlowNotificationRecipient extends Model
{
    protected $connection = 'tenant_connection';

    protected $fillable = [
        'flow_id',
        'user_id',    // NULL for external emails
        'email',      // NULL for user recipients
        'is_external' // TRUE for external emails
    ];

    // Relationships
    public function flow(): BelongsTo;
    public function user(): BelongsTo;

    // Computed attributes
    public function getRecipientEmailAttribute(): string;
    public function getRecipientNameAttribute(): string;
}
```

**Constraints**:
- Unique per `(flow_id, user_id)` for user recipients
- Unique per `(flow_id, email)` for external recipients
- Check constraint ensures either `user_id` OR `email` is set (not both)

#### `TenantNotificationSettings`
**Location**: `Domain\Ipaas\Notifications\Models\TenantNotificationSettings`

Stores tenant-wide notification configuration:

```php
class TenantNotificationSettings extends Model
{
    protected $connection = 'tenant_connection';

    protected $fillable = [
        'tenant_id',
        'notifications_enabled', // Default: true
        'fallback_email'         // Optional custom global fallback
    ];

    // Helper methods
    public function areNotificationsEnabled(): bool;
    public function hasFallbackEmail(): bool;
    public function getFallbackEmail(): ?string;
}
```

---

## Recipient Fallback Chain

The system implements a three-tier fallback strategy to ensure notifications always reach someone:

```
┌─────────────────────────────────────────────┐
│  1. Flow-Specific Recipients                │
│     Table: flow_notification_recipients     │
│     Query: WHERE flow_id = X                │
└──────────────┬──────────────────────────────┘
               │
               ↓ (if empty)
┌─────────────────────────────────────────────┐
│  2. Tenant Custom Global Email              │
│     Table: tenant_notification_settings     │
│     Field: fallback_email                   │
└──────────────┬──────────────────────────────┘
               │
               ↓ (if null)
┌─────────────────────────────────────────────┐
│  3. User ID 1 (Ultimate Fallback)           │
│     Table: users                            │
│     Query: WHERE id = 1                     │
└──────────────┬──────────────────────────────┘
               │
               ↓
         Email Sent ✉️
```

**Implementation** (`FlowNotificationService::getFlowRecipients()`):

```php
// 1. Check for flow-specific recipients
$recipients = FlowNotificationRecipient::forFlow($flowId)->get();

if ($recipients->isEmpty()) {
    // 2. Check for tenant custom email
    $settings = TenantNotificationSettings::getForTenant($tenantId);
    if ($settings && $settings->hasFallbackEmail()) {
        return collect([new FlowNotificationRecipient([
            'email' => $settings->getFallbackEmail(),
            'is_external' => true
        ])]);
    }

    // 3. Ultimate fallback to user ID 1
    $user = User::find(1);
    if ($user && $user->email) {
        return collect([new FlowNotificationRecipient([
            'user_id' => 1,
            'email' => $user->email,
            'is_external' => false
        ])]);
    }
}
```

**Logging**:
Each fallback tier logs its resolution for debugging:
- `"📋 Found flow-specific recipients"` - Flow recipients found
- `"📧 Using tenant global custom fallback email"` - Custom email used
- `"📧 Using user ID 1 as ultimate fallback recipient"` - User ID 1 used
- `"⚠️ No fallback recipient available"` - All fallbacks exhausted (rare)

---

## Default Behavior

### Enabled by Default

**Rationale**: Notifications are **enabled by default** when no tenant settings exist because:

1. **Error-Only**: Notifications never send on successful flow executions
2. **Usage-Gated**: Only triggers if tenant actively uses iPaaS flows
3. **Immediate Value**: Provides instant failure visibility without configuration
4. **Safe Default**: No risk of notification spam

**Implementation**:
```php
// If no settings exist, default to enabled
$isEnabled = $settings ? $settings->areNotificationsEnabled() : true;
```

**Override**: Tenants can explicitly disable notifications by inserting a row in `tenant_notification_settings` with `notifications_enabled = false`.

### No Configuration Required

**Out-of-Box Behavior**:
- ✅ Notifications enabled
- ✅ Falls back to user ID 1
- ✅ No manual setup required

**Result**: Flow failures immediately notify the tenant owner without any configuration steps.

---

## Configuration

### Tenant-Level Settings

**Optional Configuration**: Create/update row in `tenant_notification_settings`

#### Disable Notifications
```sql
INSERT INTO tenant_notification_settings
  (tenant_id, notifications_enabled, created_at, updated_at)
VALUES
  ('1762279505', FALSE, NOW(), NOW());
```

#### Set Custom Global Fallback Email
```sql
INSERT INTO tenant_notification_settings
  (tenant_id, notifications_enabled, fallback_email, created_at, updated_at)
VALUES
  ('1762279505', TRUE, 'admin@company.com', NOW(), NOW())
ON DUPLICATE KEY UPDATE
  fallback_email = 'admin@company.com';
```

**Use Case**: Route all flow failure notifications to a team email or integration monitoring system.

### Flow-Level Recipients

**Configuration**: Add rows to `flow_notification_recipients`

#### Add Tenant User as Recipient
```sql
INSERT INTO flow_notification_recipients
  (flow_id, user_id, is_external, created_at, updated_at)
VALUES
  (123, 5, FALSE, NOW(), NOW());
```

#### Add External Email as Recipient
```sql
INSERT INTO flow_notification_recipients
  (flow_id, email, is_external, created_at, updated_at)
VALUES
  (123, 'partner@external.com', TRUE, NOW(), NOW());
```

**Use Case**:
- Notify specific users responsible for maintaining flows
- Send alerts to external partners or vendors
- Route critical flow failures to on-call systems

**UI Management**:
- Available via `POST /ipaas/flows/{flow}/recipients` endpoint
- Controller: `FlowNotificationRecipientsController`
- Routes defined in `routes/tenant.php`

---

## Queue Processing

### Queue Architecture

**Queue**: `mail` (dedicated queue for email notifications)

**Configuration**:
```php
// config/queue.php
'default' => env('QUEUE_CONNECTION', 'redis'),

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
    ],
],
```

**Mailable Queue Assignment**:
```php
// FlowNotificationMail constructor
$this->onQueue('mail');
```

### Worker Requirements

**Critical**: Queue workers MUST be running to process email notifications.

**Local Development**:
```bash
php artisan queue:work redis --queue=mail --tries=3 --timeout=90
```

**Production (Supervisor)**:
```ini
[program:suitex-mail-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work redis --queue=mail --tries=3 --timeout=90
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/path/to/storage/logs/mail-worker.log
```

**Horizon Alternative**:
```php
// config/horizon.php
'environments' => [
    'production' => [
        'mail' => [
            'connection' => 'redis',
            'queue' => ['mail'],
            'processes' => 3,
            'tries' => 3,
            'timeout' => 90,
        ],
    ],
],
```

### Queue Monitoring

**Check worker status**:
```bash
ps aux | grep "queue:work" | grep -v grep
```

**Check queue length**:
```bash
redis-cli LLEN "queues:mail"
```

**Monitor queue in real-time**:
```bash
redis-cli MONITOR | grep mail
```

---

## Data Flow Diagram

### Flow with Failures → Email Notification

```
┌──────────────────────────────────────────────────────┐
│  ProcessFlowPage Job                                 │
│  - Processes final page of flow                      │
│  - Detects completion (counter ready)                │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  event(new FlowCompleted(...))                       │
│  - flow_id, run_id, total_records                    │
│  - tenant_database, completed_at                     │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  FlowCompletedObserver::handle()                     │
│  1. Update flow metrics                              │
│  2. Ensure metrics finalized                         │
│  3. Call handleFlowNotifications()                   │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  FlowMetricsService::getFlowMetric()                 │
│  - Retrieves failed_records count                    │
│  - Retrieves successful_records count                │
│  - Returns flow metrics                              │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  FlowNotificationService::handleFlowCompletion()     │
│  1. Check notifications enabled (default: true)      │
│  2. Check has failures (failed_records > 0)          │
│  3. Get recipients (fallback chain)                  │
│  4. For each recipient: queue email                  │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  Mail::to($email)->queue(FlowNotificationMail)       │
│  - Serializes mailable                               │
│  - Pushes job to 'mail' queue in Redis               │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  Redis Queue ('mail')                                │
│  - Job stored, waiting for worker                    │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
┌──────────────────────────────────────────────────────┐
│  Queue Worker                                        │
│  php artisan queue:work redis --queue=mail          │
│  - Polls Redis for jobs                              │
│  - Deserializes FlowNotificationMail                 │
│  - Renders email template                            │
│  - Sends via SMTP                                    │
└─────────────────────┬────────────────────────────────┘
                      │
                      ↓
                Email Delivered ✉️
```

---

## Developer Guidelines

### Adding Recipients

**Via UI** (Recommended):
```javascript
// POST /ipaas/flows/{flow}/recipients
{
    "user_id": 5,        // For tenant users
    "email": null,
    "is_external": false
}

// OR for external emails
{
    "user_id": null,
    "email": "partner@external.com",
    "is_external": true
}
```

**Via Eloquent**:
```php
use Domain\Ipaas\Notifications\Models\FlowNotificationRecipient;

// Add user recipient
FlowNotificationRecipient::create([
    'flow_id' => 123,
    'user_id' => 5,
    'is_external' => false
]);

// Add external email
FlowNotificationRecipient::create([
    'flow_id' => 123,
    'email' => 'partner@external.com',
    'is_external' => true
]);
```

**User Deletion Cascade**:
- User recipients automatically removed when user is deleted
- Handled by `UserObserver`
- External emails persist independently

### Testing Notifications

#### Built-in Test Endpoint

**Endpoint**: `POST /ipaas/settings/notifications/test`

**Request**:
```json
{
  "email": "test@example.com"
}
```

**Controller**: `TenantNotificationSettingsController::testNotification()`

**What It Does**:
- Creates mock flow (id: 0) with test data
- Generates test event (2 failed, 8 successful, 10 total)
- Queues notification email
- Returns success response

**Requirements**:
- Authenticated user
- Tenant notifications enabled (or default enabled)
- Queue worker running

#### Via Tinker

```php
php artisan tinker

// Setup tenant context
setupTenantConnection('sx_db_1762279505');

use Domain\Ipaas\Flows\Models\Flow;
use App\Mail\FlowNotificationMail;
use Illuminate\Support\Facades\Mail;

// Create test flow
$flow = new Flow();
$flow->id = 999;
$flow->name = 'Test Flow';

// Create test event
$event = (object) [
    'flow_id' => 999,
    'tenant_database' => 'sx_db_1762279505',
    'started_at' => now()->subMinutes(5)->toIso8601String(),
    'completed_at' => now()->toIso8601String(),
    'total_records' => 100,
    'successful_records' => 95,
    'failed_records' => 5,
    'run_id' => 'test-' . time()
];

// Queue test email
Mail::to('your-email@example.com')
    ->queue(new FlowNotificationMail($flow, $event, 'Test User'));

echo "Email queued! Check worker logs.\n";
```

#### Unit Tests

**Test File**: `tests/Unit/Domain/Ipaas/Notifications/Services/FlowNotificationServiceTest.php`

**Key Test Cases**:
- ✅ Notifications disabled when settings say so
- ✅ No mail when no failures
- ✅ Mail queued when failures exist
- ✅ External recipient path
- ✅ Fallback recipient when no flow recipients
- ✅ Multiple recipients fan-out
- ✅ Critical failure semantics

**Run Tests**:
```bash
./vendor/bin/pest tests/Unit/Domain/Ipaas/Notifications/Services/FlowNotificationServiceTest.php
```

### Debugging Issues

#### Check Logs

**Notification Processing**:
```bash
tail -f storage/logs/laravel.log | grep -E "🔔|📧|✉️"
```

**Log Markers**:
- `🔔 FlowNotificationService: Processing flow completion`
- `✅ No notification settings found for tenant, defaulting to ENABLED`
- `📋 Found flow-specific recipients`
- `📧 Using tenant global custom fallback email`
- `📧 Using user ID 1 as ultimate fallback recipient`
- `✉️ Flow notification email dispatched to queue`

**Error Logs**:
```bash
grep -i "notification\|failed to send" storage/logs/laravel.log
```

#### Verify Queue State

**Check queued jobs**:
```bash
# Queue length
redis-cli LLEN "queues:mail"

# View jobs
redis-cli LRANGE "queues:mail" 0 -1
```

**Check failed jobs**:
```bash
redis-cli KEYS "*failed*"
```

#### Verify Configuration

**Check tenant settings**:
```sql
SELECT * FROM tenant_notification_settings
WHERE tenant_id = '1762279505';
```

**Check flow recipients**:
```sql
SELECT * FROM flow_notification_recipients
WHERE flow_id = 123;
```

**Check user ID 1**:
```sql
SELECT id, email, name FROM users WHERE id = 1;
```

---

## Troubleshooting Guide

### Issue: Notifications Not Sending

**Symptom**: No emails received when flows fail

**Possible Causes**:

1. **Queue worker not running**
   ```bash
   # Check worker status
   ps aux | grep "queue:work" | grep -v grep

   # If no workers, start one
   php artisan queue:work redis --queue=mail --tries=3 --timeout=90 &
   ```

2. **Notifications explicitly disabled**
   ```sql
   -- Check settings
   SELECT * FROM tenant_notification_settings WHERE tenant_id = 'YOUR_TENANT_ID';

   -- If notifications_enabled = 0, enable them
   UPDATE tenant_notification_settings
   SET notifications_enabled = 1
   WHERE tenant_id = 'YOUR_TENANT_ID';
   ```

3. **No failures detected**
   ```bash
   # Check logs for failure detection
   grep "Flow completed without failures" storage/logs/laravel.log

   # Verify metrics service is recording failures
   grep "failed_records" storage/logs/laravel.log
   ```

4. **No recipients found**
   ```bash
   # Check logs
   grep "No recipients configured" storage/logs/laravel.log

   # Verify user ID 1 exists
   mysql -e "SELECT id, email FROM users WHERE id = 1" YOUR_DATABASE
   ```

### Issue: Wrong Recipients

**Symptom**: Notifications going to unexpected addresses

**Resolution**:

1. **Check flow-specific recipients**:
   ```sql
   SELECT fr.*, u.email as user_email, u.name
   FROM flow_notification_recipients fr
   LEFT JOIN users u ON fr.user_id = u.id
   WHERE fr.flow_id = YOUR_FLOW_ID;
   ```

2. **Check tenant fallback email**:
   ```sql
   SELECT fallback_email
   FROM tenant_notification_settings
   WHERE tenant_id = 'YOUR_TENANT_ID';
   ```

3. **Verify fallback chain in logs**:
   ```bash
   grep -E "Using tenant global|Using user ID 1" storage/logs/laravel.log
   ```

### Issue: Emails Queued But Not Delivered

**Symptom**: Jobs in queue but emails never arrive

**Resolution**:

1. **Check queue worker is processing mail queue**:
   ```bash
   ps aux | grep "queue:work" | grep "mail"
   ```

2. **Check for failed jobs**:
   ```bash
   php artisan queue:failed
   ```

3. **Check mail configuration**:
   ```bash
   php artisan config:show mail

   # Verify SMTP settings
   # - mailers.smtp.host
   # - mailers.smtp.port
   # - mailers.smtp.username
   # - from.address
   ```

4. **Test mail directly**:
   ```bash
   php artisan tinker
   >>> Mail::raw('Test', fn($msg) => $msg->to('test@example.com')->subject('Test'));
   ```

5. **Check mail service logs**:
   - MailTrap: Check inbox at https://mailtrap.io
   - SMTP logs: Check SMTP server logs for connection/auth errors

---

## Related Documentation

- **Design Document**: `docs/designs/ipaas-notification-system.md` - Complete specification and requirements
- **Fix Summary**: `docs/debug/ipaas-notification-fix-summary.md` - Issue resolution and fixes applied
- **Debug Analysis**: `docs/debug/ipaas-notification-debug-analysis.md` - Detailed debugging investigation
- **Test Suite**: `tests/Unit/Domain/Ipaas/Notifications/Services/FlowNotificationServiceTest.php`
- **Mail Test**: `tests/Unit/App/Mail/FlowNotificationMailTest.php`

---

## Changelog

| Date | Author | Change |
|------|--------|--------|
| 2025-09-08 | Quinn Johns | Initial implementation with tenant isolation |
| 2025-09-08 | Quinn Johns | Created database migrations for notification tables |
| 2025-09-08 | Quinn Johns | Implemented three-tier fallback chain |
| 2025-09-17 | Quinn Johns | Design review: Identified issues with fail-closed and missing fallback |
| 2025-11-11 | AI Agent | Fixed default behavior (fail-closed → enabled by default) |
| 2025-11-11 | AI Agent | Added user ID 1 as ultimate fallback in recipient chain |
| 2025-11-11 | AI Agent | Fixed critical failure notifications: ProcessFlow now dispatches FlowCompleted event on API errors |
| 2025-11-11 | AI Agent | Created architecture documentation |

---

**Last Updated**: November 11, 2025
**Maintainer**: Quinn Johns (quinn@suitedynamics.io)