# Import Fixes: Derived Wave Execution After Parent Completion

**Date:** 2025-09-29
**Status:** Design Phase
**Priority:** Critical
**Impact:** Transaction line imports fail to execute after parent records complete

---

## Executive Summary

The wave coordination system prematurely marks jobs as complete when the last main type wave (e.g., SalesOrder with dependency_level = -1) finishes, without checking for pending derived transaction line waves (dependency_level = -2). This causes TransactionLine imports to never execute, despite being correctly created by the BatchJobCompletedListener.

**Root Cause:** Job completion logic in `WaveCoordinator::handleMainTypeWaveProgression()` only checks for additional main waves, not derived waves, before marking the job complete and stopping wave coordination.

---

## Problem Statement

### Current Behavior

1. **Parent records import successfully** (e.g., SalesOrder batches complete)
2. **Derived waves are created correctly** (TransactionLine batches created with dependency_level = -2)
3. **Wave coordination stops prematurely** (job marked complete after last main wave)
4. **Derived waves never dispatch** (monitoring flag deleted before derived waves can execute)

### Impact

- TransactionLine records are never imported
- Jobs appear "complete" in the UI but are actually incomplete
- Data integrity issues (parent records exist without their line items)
- Silent failure mode (no errors logged, just missing data)

### Affected Components

- `WaveCoordinator.php` - Main coordination logic
- `BatchJobCompletedListener.php` - Derived wave creation (works correctly)
- Wave monitoring system - Premature deactivation
- Job completion detection - Incorrect completion criteria

---

## Root Cause Analysis

### Critical Code Path: Premature Job Completion

**Location:** `src/App/Services/ImportJobs/WaveCoordinator.php`
**Method:** `handleMainTypeWaveProgression()`
**Lines:** 1806-1830

```php
// CURRENT PROBLEMATIC CODE
if ($waveCompletion['completed']) {
    // Final wave is fully complete - job is complete
    Log::info('Final main type wave completed, marking job as complete', [
        'job_id' => $jobId,
        'final_wave' => $currentWave,
        'progress' => $waveCompletion['progress']
    ]);

    // ❌ PROBLEM: Stops monitoring without checking for derived waves
    Cache::forget("wave_monitoring_active_{$jobId}");

    $monitoringData = Cache::get("wave_monitoring_{$jobId}");
    JobCacheManager::put("wave_monitoring_{$jobId}", array_merge($monitoringData, [
        'status' => 'completed',  // ❌ Job marked complete prematurely
        'completed_at' => Carbon::now()
    ]), 'final_wave_completion', $jobId);

    return [
        'status' => 'all_waves_completed',
        'message' => 'All dependency and main type waves completed',  // ❌ INCORRECT
        'job_id' => $jobId,
        'final_wave' => $currentWave,
        'progress' => $waveCompletion['progress']
    ];
}
```

**The Problem:**
- Checks: "Is there a next main wave (dependency_level = -1)?" → No
- Assumes: "No more waves, job complete!" → **WRONG**
- Forgets to check: "Are there derived waves (dependency_level = -2)?" → **NEVER CHECKED**
- Deletes `wave_monitoring_active_{$jobId}` → **STOPS ALL COORDINATION**

### Race Condition Timeline

```
Event Timeline:
─────────────────────────────────────────────────────────────────

1. [T+0s]   SalesOrder Wave 5 reaches 100% completion
            └─> BatchJobCompletedListener triggered

2. [T+1s]   BatchJobCompletedListener updates wave_coordination
            └─> Wave 5 status = 'completed'

3. [T+2s]   WaveCoordinator::checkAndTriggerNextWave() called
            └─> Checks for next main wave (dependency_level = -1)
            └─> No next main wave found

4. [T+3s]   handleMainTypeWaveProgression() executes
            ❌ Assumes job complete
            ❌ Deletes wave_monitoring_active flag
            ❌ Marks job status = 'completed'

5. [T+4s]   BatchJobCompletedListener checks parent completion
            ✅ Parent at 100% completion threshold
            ✅ Creates derived TransactionLine waves
            ✅ Sets wave_monitoring_active = true (again)
            ✅ Calls coordinator->checkAndTriggerNextWave()

6. [T+5s]   Coordinator checks job status
            ❌ Job already marked 'completed'
            ❌ May skip processing or
            ❌ Derived check bypassed by completion logic

7. [T+6s]   Derived waves sit in 'pending' status forever
            ❌ Never dispatched
            ❌ No errors logged
            ❌ Silent failure
```

### Why Existing Derived Wave Logic Doesn't Help

The coordinator DOES have derived wave detection logic (lines 1102-1148):

```php
// ✅ This logic EXISTS and WORKS
$nextDerivedWave = DB::connection($this->connectionName)
    ->table('wave_batches')
    ->where('wave_batches.job_id', $jobId)
    ->where('wave_batches.record_type_id', -13)
    ->where('wave_batches.status', 'pending')
    ->join('wave_coordination', function ($join) use ($jobId) {
        $join->on('wave_coordination.wave_number', '=', 'wave_batches.wave_number')
             ->where('wave_coordination.job_id', '=', $jobId)
             ->where('wave_coordination.dependency_level', '=', -2);
    })
    ->orderBy('wave_batches.wave_number')
    ->value('wave_batches.wave_number');
```

**But this is bypassed because:**
1. Job completion happens FIRST (step 4 in timeline)
2. Derived waves created AFTER (step 5 in timeline)
3. By the time derived check could run, job is already "complete"
4. `wave_monitoring_active` flag deleted prevents periodic coordinator ticks

---

## Design Solution

### Principle: Complete Only When ALL Waves Complete

**New Completion Criteria:**
- ✅ All dependency waves (dependency_level >= 0) complete
- ✅ All main type waves (dependency_level = -1) complete
- ✅ All derived waves (dependency_level = -2) complete **← NEW CHECK**
- ✅ No pending waves exist in wave_coordination table

### Solution Architecture

```
┌─────────────────────────────────────────────────────────────┐
│         Main Wave Completes (dependency_level = -1)         │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
        ┌────────────────────────────────────────┐
        │  Check: Next Main Wave Exists?         │
        │  (dependency_level = -1, wave_number > current) │
        └─────────┬──────────────────────┬───────┘
                  │                      │
              YES │                      │ NO
                  ▼                      ▼
        ┌──────────────────┐   ┌─────────────────────────────┐
        │ Dispatch Next    │   │ ✨ NEW: Check Derived Waves │
        │ Main Wave        │   │ (dependency_level = -2)     │
        └──────────────────┘   └──────────┬──────────────────┘
                                           │
                         ┌─────────────────┴─────────────────┐
                         │                                   │
                    YES  │                               NO  │
                         ▼                                   ▼
              ┌────────────────────────┐        ┌──────────────────────┐
              │ Keep Monitoring Active │        │ Safe to Complete Job │
              │ Dispatch Derived Wave  │        │ No More Work         │
              └────────────────────────┘        └──────────────────────┘
```

---

## Implementation Details

### Change #1: Add Derived Wave Check Before Job Completion ⭐ CRITICAL

**File:** `src/App/Services/ImportJobs/WaveCoordinator.php`
**Method:** `handleMainTypeWaveProgression()`
**Location:** Lines 1806-1830 (replace existing completion logic)

```php
// After checking for next main wave and finding none...
// BEFORE marking job complete, check for derived waves

if ($waveCompletion['completed']) {
    // ✨ NEW: Check if there are any derived waves (dependency_level = -2)
    $hasDerivedWaves = DB::connection($this->connectionName)
        ->table('wave_coordination')
        ->where('job_id', $jobId)
        ->where('dependency_level', -2) // Derived waves (TransactionLine)
        ->exists();

    if ($hasDerivedWaves) {
        // Derived waves exist - check their status
        $derivedPendingCount = DB::connection($this->connectionName)
            ->table('wave_coordination')
            ->where('job_id', $jobId)
            ->where('dependency_level', -2)
            ->whereIn('status', ['pending', 'dispatching', 'processing'])
            ->count();

        $derivedCompletedCount = DB::connection($this->connectionName)
            ->table('wave_coordination')
            ->where('job_id', $jobId)
            ->where('dependency_level', -2)
            ->where('status', 'completed')
            ->count();

        if ($derivedPendingCount > 0) {
            Log::info('Main waves complete but derived waves still pending', [
                'job_id' => $jobId,
                'final_main_wave' => $currentWave,
                'derived_pending' => $derivedPendingCount,
                'derived_completed' => $derivedCompletedCount
            ]);

            // ✅ Keep monitoring active for derived waves
            JobCacheManager::refreshIfNeeded(
                "wave_monitoring_active_{$jobId}",
                true,
                'derived_waves_pending',
                $jobId
            );

            // ✅ Attempt to dispatch next pending derived wave
            try {
                $derivedResult = $this->checkAndDispatchDerivedWaves($jobId);
                if ($derivedResult['status'] === 'derived_wave_triggered') {
                    // Update monitoring to track derived wave
                    $monitoringData = Cache::get("wave_monitoring_{$jobId}") ?? [];
                    $monitoringData['current_wave'] = $derivedResult['next_wave'];
                    $monitoringData['current_wave_type'] = 'derived';
                    $monitoringData['main_waves_completed'] = true;
                    JobCacheManager::put(
                        "wave_monitoring_{$jobId}",
                        $monitoringData,
                        'derived_wave_dispatched',
                        $jobId
                    );

                    return $derivedResult;
                }
            } catch (\Throwable $e) {
                Log::warning('Failed to dispatch derived wave after main completion', [
                    'job_id' => $jobId,
                    'error' => $e->getMessage(),
                    'trace' => $e->getTraceAsString()
                ]);
            }

            // Return status indicating derived waves are pending
            return [
                'status' => 'main_complete_derived_pending',
                'message' => 'Main type waves complete, derived waves pending',
                'job_id' => $jobId,
                'final_main_wave' => $currentWave,
                'derived_pending_count' => $derivedPendingCount,
                'derived_completed_count' => $derivedCompletedCount
            ];
        } else if ($derivedCompletedCount > 0) {
            // All derived waves are complete
            Log::info('All main and derived waves completed successfully', [
                'job_id' => $jobId,
                'final_main_wave' => $currentWave,
                'derived_completed' => $derivedCompletedCount
            ]);
        }
    }

    // ✅ Only mark complete if:
    // 1. No derived waves exist, OR
    // 2. All derived waves are completed
    Log::info('All waves completed, marking job as complete', [
        'job_id' => $jobId,
        'final_main_wave' => $currentWave,
        'has_derived_waves' => $hasDerivedWaves,
        'all_waves_complete' => true
    ]);

    // Safe to complete now
    Cache::forget("wave_monitoring_active_{$jobId}");

    $monitoringData = Cache::get("wave_monitoring_{$jobId}");
    JobCacheManager::put("wave_monitoring_{$jobId}", array_merge($monitoringData ?? [], [
        'status' => 'completed',
        'completed_at' => Carbon::now(),
        'all_waves_completed' => true,
        'derived_waves_completed' => $hasDerivedWaves
    ]), 'final_wave_completion', $jobId);

    return [
        'status' => 'all_waves_completed',
        'message' => 'All dependency, main type, and derived waves completed',
        'job_id' => $jobId,
        'final_wave' => $currentWave,
        'progress' => $waveCompletion['progress'],
        'derived_waves_included' => $hasDerivedWaves
    ];
}
```

### Change #2: Extract Derived Wave Check to Reusable Method

**File:** `src/App/Services/ImportJobs/WaveCoordinator.php`
**Location:** Add new protected method (around line 1150)

```php
/**
 * Check and dispatch pending derived waves (dependency_level = -2)
 *
 * This method handles TransactionLine waves that are created after
 * parent records (SalesOrder, Invoice, etc.) complete.
 *
 * @param string $jobId The import job ID
 * @return array Status of derived wave dispatch attempt
 */
protected function checkAndDispatchDerivedWaves(string $jobId): array
{
    Log::debug('Checking for pending derived waves', [
        'job_id' => $jobId,
        'method' => __METHOD__
    ]);

    // Find the lowest pending wave_number that has TransactionLine (-13) batches
    $nextDerivedWave = DB::connection($this->connectionName)
        ->table('wave_batches')
        ->select('wave_batches.wave_number')
        ->where('wave_batches.job_id', $jobId)
        ->where('wave_batches.record_type_id', -13) // TransactionLine
        ->where('wave_batches.status', 'pending')
        ->join('wave_coordination', function ($join) use ($jobId) {
            $join->on('wave_coordination.wave_number', '=', 'wave_batches.wave_number')
                 ->where('wave_coordination.job_id', '=', $jobId)
                 ->where('wave_coordination.dependency_level', '=', -2);
        })
        ->orderBy('wave_batches.wave_number')
        ->value('wave_batches.wave_number');

    if ($nextDerivedWave === null) {
        Log::debug('No pending derived waves found', [
            'job_id' => $jobId
        ]);
        return [
            'status' => 'no_derived_waves',
            'message' => 'No pending derived waves found',
            'job_id' => $jobId
        ];
    }

    // Check if derived waves are ready to dispatch
    $derivedReady = Cache::get("derived_lines_ready_{$jobId}", false);

    if (!$derivedReady) {
        // Fallback: Check DB for presence of -13 batches
        $hasDerived = DB::connection($this->connectionName)
            ->table('wave_batches')
            ->where('job_id', $jobId)
            ->where('record_type_id', -13)
            ->exists();

        if ($hasDerived) {
            $derivedReady = true;
            Cache::put("derived_lines_ready_{$jobId}", true, 7200);
            Log::info('Derived-ready flag restored from DB fallback', [
                'job_id' => $jobId
            ]);
        }
    }

    if (!$derivedReady) {
        Log::warning('Derived waves exist but not marked ready', [
            'job_id' => $jobId,
            'wave_number' => $nextDerivedWave,
            'derived_ready_flag' => $derivedReady
        ]);
        return [
            'status' => 'derived_not_ready',
            'message' => 'Derived waves exist but initialization not complete',
            'job_id' => $jobId,
            'wave_number' => $nextDerivedWave
        ];
    }

    // All checks passed - dispatch the derived wave
    Log::info('Dispatching derived TransactionLine wave', [
        'job_id' => $jobId,
        'wave_number' => (int) $nextDerivedWave,
        'wave_type' => 'derived'
    ]);

    $this->dispatchWave($jobId, (int) $nextDerivedWave);

    return [
        'status' => 'derived_wave_triggered',
        'message' => "Derived wave {$nextDerivedWave} triggered",
        'job_id' => $jobId,
        'next_wave' => (int) $nextDerivedWave,
        'wave_type' => 'derived'
    ];
}
```

### Change #3: Update checkAndTriggerNextWave to Use New Method

**File:** `src/App/Services/ImportJobs/WaveCoordinator.php`
**Method:** `checkAndTriggerNextWave()`
**Location:** Lines 1102-1154 (replace existing derived wave logic)

```php
// Derived waves progression: dispatch TransactionLine waves when pending (separate level -2)
try {
    $derivedResult = $this->checkAndDispatchDerivedWaves($jobId);

    if ($derivedResult['status'] === 'derived_wave_triggered') {
        // Update monitoring to track the dispatched derived wave
        $monitoringData['current_wave'] = $derivedResult['next_wave'];
        $monitoringData['current_wave_type'] = 'derived';
        JobCacheManager::refreshIfNeeded(
            "wave_monitoring_{$jobId}",
            $monitoringData,
            'wave_monitoring_update',
            $jobId
        );

        return $derivedResult;
    } else if ($derivedResult['status'] === 'derived_not_ready') {
        Log::debug('Derived waves not ready, will retry on next tick', [
            'job_id' => $jobId,
            'result' => $derivedResult
        ]);
    }
} catch (\Throwable $e) {
    Log::warning('Derived waves progression check failed', [
        'job_id' => $jobId,
        'error' => $e->getMessage(),
        'trace' => $e->getTraceAsString()
    ]);
}
```

### Change #4: Add Derived Wave Completion Check

**File:** `src/App/Services/ImportJobs/WaveCoordinator.php`
**Method:** `checkAndTriggerNextWave()`
**Location:** After derived wave progression check (around line 1150)

```php
// After attempting to dispatch derived waves, check if a derived wave just completed
// and if there are more derived waves to dispatch
if ($hasMainTypeWaves) {
    // Check if current wave is a derived wave that's complete
    $currentWaveInfo = DB::connection($this->connectionName)
        ->table('wave_coordination')
        ->where('job_id', $jobId)
        ->where('wave_number', $currentWave)
        ->first();

    if ($currentWaveInfo && $currentWaveInfo->dependency_level == -2) {
        $currentWaveCompletion = $this->checkWaveCompletion($jobId, $currentWave);

        if ($currentWaveCompletion['completed']) {
            Log::info('Derived wave completed, checking for next derived wave', [
                'job_id' => $jobId,
                'completed_wave' => $currentWave,
                'progress' => $currentWaveCompletion['progress']
            ]);

            // Check for next derived wave
            $derivedResult = $this->checkAndDispatchDerivedWaves($jobId);

            if ($derivedResult['status'] === 'derived_wave_triggered') {
                return $derivedResult;
            } else if ($derivedResult['status'] === 'no_derived_waves') {
                // All derived waves complete - mark job as complete
                Log::info('All derived waves completed, marking job as complete', [
                    'job_id' => $jobId,
                    'final_derived_wave' => $currentWave
                ]);

                Cache::forget("wave_monitoring_active_{$jobId}");

                $monitoringData = Cache::get("wave_monitoring_{$jobId}");
                JobCacheManager::put("wave_monitoring_{$jobId}", array_merge($monitoringData ?? [], [
                    'status' => 'completed',
                    'completed_at' => Carbon::now(),
                    'all_derived_waves_completed' => true
                ]), 'derived_waves_completion', $jobId);

                return [
                    'status' => 'all_waves_completed',
                    'message' => 'All waves including derived waves completed',
                    'job_id' => $jobId,
                    'final_wave' => $currentWave,
                    'wave_type' => 'derived'
                ];
            }
        }
    }
}
```

---

## Testing Strategy

### Unit Tests

**File:** `tests/Unit/Services/ImportJobs/WaveCoordinatorDerivedTest.php`

```php
<?php

use Domain\RecordTypes\Models\RecordType;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;

describe('WaveCoordinator Derived Wave Handling', function () {
    beforeEach(function () {
        // Setup test database
        config(['database.default' => 'tenant_connection']);
        DB::purge('tenant_connection');
        DB::reconnect('tenant_connection');

        // Create tables
        Schema::connection('tenant_connection')->create('wave_coordination', function ($table) {
            $table->id();
            $table->string('job_id');
            $table->integer('wave_number');
            $table->integer('dependency_level');
            $table->integer('total_batches')->default(0);
            $table->integer('completed_batches')->default(0);
            $table->integer('failed_batches')->default(0);
            $table->string('status')->default('pending');
            $table->timestamp('dispatched_at')->nullable();
            $table->timestamp('completed_at')->nullable();
            $table->timestamps();
        });

        Schema::connection('tenant_connection')->create('wave_batches', function ($table) {
            $table->id();
            $table->string('job_id');
            $table->integer('wave_number');
            $table->string('batch_id');
            $table->integer('record_type_id');
            $table->integer('batch_number');
            $table->string('status')->default('pending');
            $table->text('chunk_ids_json')->nullable();
            $table->timestamps();
        });

        Log::spy();
        Cache::flush();
    });

    it('does not complete job when derived waves exist', function () {
        $jobId = 'test_job_123';

        // Create main wave (completed)
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 1,
            'dependency_level' => -1, // Main type
            'total_batches' => 10,
            'completed_batches' => 10,
            'status' => 'completed',
            'completed_at' => now()
        ]);

        // Create derived wave (pending)
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 2,
            'dependency_level' => -2, // Derived
            'total_batches' => 5,
            'completed_batches' => 0,
            'status' => 'pending'
        ]);

        DB::connection('tenant_connection')->table('wave_batches')->insert([
            'job_id' => $jobId,
            'wave_number' => 2,
            'batch_id' => 'derived_batch_1',
            'record_type_id' => -13, // TransactionLine
            'batch_number' => 1,
            'status' => 'pending'
        ]);

        Cache::put("derived_lines_ready_{$jobId}", true, 7200);
        Cache::put("wave_monitoring_{$jobId}", [
            'current_wave' => 1,
            'total_waves' => 2
        ], 7200);

        $coordinator = new \App\Services\ImportJobs\WaveCoordinator();
        $result = $coordinator->checkAndTriggerNextWave($jobId);

        expect($result['status'])->not->toBe('all_waves_completed');
        expect($result['status'])->toBeIn(['derived_wave_triggered', 'main_complete_derived_pending']);
        expect(Cache::has("wave_monitoring_active_{$jobId}"))->toBeTrue();
    });

    it('dispatches derived wave after main wave completion', function () {
        $jobId = 'test_job_456';

        // Main wave complete
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 1,
            'dependency_level' => -1,
            'total_batches' => 5,
            'completed_batches' => 5,
            'status' => 'completed'
        ]);

        // Derived wave pending
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 2,
            'dependency_level' => -2,
            'total_batches' => 3,
            'completed_batches' => 0,
            'status' => 'pending'
        ]);

        DB::connection('tenant_connection')->table('wave_batches')->insert([
            'job_id' => $jobId,
            'wave_number' => 2,
            'batch_id' => 'derived_batch_1',
            'record_type_id' => -13,
            'batch_number' => 1,
            'status' => 'pending'
        ]);

        Cache::put("derived_lines_ready_{$jobId}", true, 7200);

        $coordinator = new \App\Services\ImportJobs\WaveCoordinator();
        $result = $coordinator->checkAndTriggerNextWave($jobId);

        expect($result['status'])->toBe('derived_wave_triggered');
        expect($result['next_wave'])->toBe(2);
        expect($result['wave_type'])->toBe('derived');
    });

    it('completes job only after all derived waves finish', function () {
        $jobId = 'test_job_789';

        // Main wave complete
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 1,
            'dependency_level' => -1,
            'total_batches' => 5,
            'completed_batches' => 5,
            'status' => 'completed'
        ]);

        // Derived wave also complete
        DB::connection('tenant_connection')->table('wave_coordination')->insert([
            'job_id' => $jobId,
            'wave_number' => 2,
            'dependency_level' => -2,
            'total_batches' => 3,
            'completed_batches' => 3,
            'status' => 'completed'
        ]);

        Cache::put("wave_monitoring_{$jobId}", [
            'current_wave' => 2,
            'total_waves' => 2
        ], 7200);

        $coordinator = new \App\Services\ImportJobs\WaveCoordinator();
        $result = $coordinator->checkAndTriggerNextWave($jobId);

        expect($result['status'])->toBe('all_waves_completed');
        expect($result['derived_waves_included'] ?? false)->toBeTrue();
        expect(Cache::has("wave_monitoring_active_{$jobId}"))->toBeFalse();
    });
});
```

### Integration Tests

**File:** `tests/Integration/Jobs/ImportJobs/DerivedWaveExecutionTest.php`

```php
<?php

describe('Derived Wave Execution Integration', function () {
    it('executes transaction lines after parent records complete', function () {
        // 1. Setup job with SalesOrder and TransactionLine
        // 2. Dispatch dependency waves
        // 3. Dispatch main SalesOrder wave
        // 4. Wait for completion
        // 5. Verify TransactionLine wave created
        // 6. Verify TransactionLine wave dispatched
        // 7. Verify job completes after all waves

        // Detailed test implementation here...
    });
});
```

### Manual Testing Checklist

- [ ] Import job with SalesOrder + TransactionLine
- [ ] Verify main waves execute
- [ ] Verify derived waves are created
- [ ] Verify derived waves are dispatched (not stuck in pending)
- [ ] Verify all transaction lines are imported
- [ ] Verify job completes only after derived waves finish
- [ ] Check logs for proper progression messages
- [ ] Verify monitoring UI shows derived wave progress

---

## Rollout Plan

### Phase 1: Code Changes (This PR)

1. Implement all changes in `WaveCoordinator.php`
2. Add comprehensive unit tests
3. Add integration tests
4. Update design documentation

### Phase 2: Testing (Dev/Staging)

1. Deploy to development environment
2. Run test imports with transaction records
3. Verify derived waves execute correctly
4. Monitor logs for proper progression
5. Test edge cases (no derived waves, multiple derived waves, etc.)

### Phase 3: Production Deployment

1. Deploy during maintenance window
2. Monitor first few imports closely
3. Verify transaction lines importing correctly
4. Check for any performance impacts

### Phase 4: Validation

1. Review completed imports
2. Verify data integrity (parent + line records)
3. Monitor for any issues
4. Document any edge cases discovered

---

## Monitoring & Alerts

### Key Metrics to Track

- **Derived wave dispatch rate**: Waves with dependency_level = -2 being dispatched
- **Derived wave completion rate**: Percentage of derived waves completing successfully
- **Job completion accuracy**: Jobs not completing prematurely
- **Transaction line import success**: TransactionLine records being created

### Log Messages to Monitor

```
✅ Success Indicators:
- "Main waves complete but derived waves still pending"
- "Dispatching derived TransactionLine wave"
- "All main and derived waves completed successfully"

❌ Failure Indicators:
- "Final main type wave completed, marking job as complete" (without derived check)
- "Derived waves exist but not marked ready"
- Jobs completing with pending derived waves
```

### Alerts to Configure

- **Alert:** Job marked complete with pending derived waves
- **Alert:** Derived waves stuck in pending for > 10 minutes
- **Alert:** High failure rate on derived wave dispatch
- **Alert:** Transaction line import count drop

---

## Edge Cases & Considerations

### Edge Case 1: No Derived Waves Needed

**Scenario:** Parent records imported but no transaction lines needed
**Handling:** Job should complete normally (existing behavior works)

### Edge Case 2: Derived Waves Created Late

**Scenario:** Main wave completes before derived waves are created
**Handling:** Periodic coordinator ticks will pick up and dispatch derived waves

### Edge Case 3: Multiple Derived Waves

**Scenario:** Large import creates multiple derived waves
**Handling:** Each derived wave progresses independently, job completes after all finish

### Edge Case 4: Derived Wave Creation Fails

**Scenario:** Error creating derived waves
**Handling:** Main waves complete, job may complete without derived waves (log warning)

### Edge Case 5: Mixed Record Types

**Scenario:** Import has multiple transaction types (SalesOrder, Invoice, etc.)
**Handling:** Each creates its own derived waves, all must complete before job finishes

---

## Backward Compatibility

### No Breaking Changes

- ✅ Existing wave coordination logic unchanged for non-transaction imports
- ✅ Dependency wave handling unchanged
- ✅ Main wave handling enhanced but compatible
- ✅ Only adds new checks, doesn't remove existing functionality

### Migration Path

- ✅ No database migrations required
- ✅ No configuration changes required
- ✅ Works with existing jobs mid-flight
- ✅ Safe to deploy incrementally

---

## Success Criteria

### Definition of Done

- [ ] All code changes implemented and reviewed
- [ ] Unit tests passing (>95% coverage on new code)
- [ ] Integration tests passing
- [ ] Manual testing completed successfully
- [ ] Performance impact assessed (should be minimal)
- [ ] Documentation updated
- [ ] Deployed to staging and validated
- [ ] Deployed to production and monitored

### Validation Metrics

- **100%** of transaction line imports execute after parent completion
- **0** jobs completing prematurely with pending derived waves
- **<1%** derived wave dispatch failures
- **<5s** additional time for derived wave checks (negligible impact)

---

## References

### Related Files

- `src/App/Services/ImportJobs/WaveCoordinator.php` - Main coordination logic
- `src/App/Listeners/ImportJobs/BatchJobCompletedListener.php` - Derived wave creation
- `docs/designs/import-hardening.md` - Import hardening design
- `docs/designs/parent-first-import.md` - Parent-first transaction approach
- `docs/designs/WAVE_COORDINATION_DESIGN.md` - Wave coordination system

### Related Issues

- TransactionLine records not importing
- Jobs appearing complete but missing data
- Wave coordination stopping prematurely
- Silent failures in derived wave execution

---

## Appendix: Code Snippets

### Current Problematic Logic

```php
// Lines 1806-1830 in WaveCoordinator.php
// PROBLEM: Only checks for main waves, ignores derived waves
if ($waveCompletion['completed']) {
    Log::info('Final main type wave completed, marking job as complete', [...]);
    Cache::forget("wave_monitoring_active_{$jobId}"); // ❌ STOPS MONITORING
    // Marks job complete without checking derived waves
}
```

### Fixed Logic (Summary)

```php
if ($waveCompletion['completed']) {
    // ✅ NEW: Check for derived waves before completing
    $hasDerivedWaves = DB::connection($this->connectionName)
        ->table('wave_coordination')
        ->where('job_id', $jobId)
        ->where('dependency_level', -2)
        ->exists();

    if ($hasDerivedWaves) {
        // Keep monitoring active, dispatch derived waves
        // DON'T mark job complete yet
    } else {
        // Safe to complete - no derived waves
        Cache::forget("wave_monitoring_active_{$jobId}");
        // Mark job complete
    }
}
```

---

**End of Design Document**