# Critical Fix: Progressive Batch Creation in Derived Waves

**Date:** 2025-09-29
**Status:** ✅ Fixed
**Priority:** Critical
**Issue:** Derived transaction line batches not executing due to progressive creation

---

## Problem Identified

The initial derived wave fix (import-fixes-sonnet.md) successfully prevented premature job completion, but revealed a **second critical bug**: derived waves were partially dispatched and then stalled.

### Symptoms
- Derived wave marked as "processing"
- Only 10 out of 45 batches dispatched
- Remaining 35 batches stuck in "pending" status forever
- `dispatchWave()` called 348 times but only first call dispatched batches

### Root Cause Analysis

#### Progressive Batch Creation
Derived TransactionLine batches are created **progressively** by `BatchJobCompletedListener` as parent SalesOrder batches complete, not all at once:

```
Timeline from Database (job: import_68db0dddaa2441):
- 22:53:53: First ~10 batches created
- 22:54:29: Last batch created (36 seconds later)
- Total: 45 batches created over 36 seconds
```

#### The Bug in dispatchWave()

```php
// OLD BUGGY LOGIC:
if ($wave->status === 'processing') {
    $active = count_active_batches();
    if ($active === 0) {
        // Demote to pending if stale
    } else {
        return; // ❌ BUG: Exits without checking for new pending batches!
    }
}
```

**What Happened:**
1. **T+0s:** First 10 batches exist, `dispatchWave()` called
2. **T+0s:** CAS succeeds, dispatches 10 batches, marks wave as "processing"
3. **T+0-36s:** Remaining 35 batches created progressively
4. **T+0-36s:** `dispatchWave()` called 348 times
5. **T+0-36s:** Each call sees wave is "processing", **returns without dispatching new pending batches**
6. **Result:** 35 batches remain "pending" forever

---

## The Fix

### Updated dispatchWave() Logic

```php
if ($wave->status === 'processing') {
    // ✅ NEW: Check for pending batches FIRST (progressive creation)
    $pending = DB::connection($this->connectionName)->table('wave_batches')
        ->where('job_id', $jobId)
        ->where('wave_number', $waveNumber)
        ->where('status', 'pending')
        ->count();

    if ($pending > 0) {
        // Progressive batch creation: dispatch newly created pending batches
        Log::info('Wave processing but has pending batches; dispatching progressive batches', [
            'job_id' => $jobId,
            'wave_number' => $waveNumber,
            'pending_batches' => $pending
        ]);
        // Skip CAS and go directly to dispatching pending batches
        goto dispatchPending;
    }

    // Rest of existing logic for stale wave detection...
}

// CAS logic...

dispatchPending: // Label for progressive batch dispatch

// Get all pending batches and dispatch them
$batches = DB::connection($this->connectionName)->table('wave_batches')
    ->where('job_id', $jobId)
    ->where('wave_number', $waveNumber)
    ->where('status', 'pending')
    ->get();

// Dispatch all pending batches...
```

### Key Changes

1. **Priority Check:** Check for pending batches BEFORE checking active batches
2. **Progressive Dispatch:** If pending batches exist in a "processing" wave, skip CAS and dispatch them
3. **No Early Return:** Don't return early when wave is processing if there are pending batches
4. **Label Jump:** Use `goto dispatchPending` to skip CAS and go directly to batch dispatch logic

---

## Why This Happens

### Progressive Creation Pattern

The `BatchJobCompletedListener` creates derived batches progressively:

```php
// In BatchJobCompletedListener::maybeEnqueueTransactionLineBatches()
protected function maybeEnqueueTransactionLineBatches(string $jobId, int $recordTypeId): void
{
    // Called EVERY time a parent batch completes
    if ($parentCompletionPct >= $completionThreshold && !Cache::get($initFlagKey)) {
        // Create derived batches from accumulated parent IDs
        $this->progressivelyCreateDerivedLineBatches($jobId, $parentIds);

        // Trigger coordinator
        $coordinator->checkAndTriggerNextWave($jobId);
    }
}
```

This is **intentional and correct** because:
- Parent IDs are accumulated as batches complete
- We don't know all parent IDs upfront
- Creating batches progressively allows earlier dispatch of some line imports

But the `dispatchWave()` method wasn't designed for this pattern!

---

## Implementation Details

### File Modified
- `src/App/Services/ImportJobs/WaveCoordinator.php`

### Method Updated
- `dispatchWave()` - Lines 610-651

### Code Changes

```php
// BEFORE (Buggy):
if ($wave->status === 'processing') {
    $active = count_active_batches();
    if ($active === 0) {
        demote_to_pending();
    } else {
        return; // ❌ Exits without checking pending batches
    }
}

// AFTER (Fixed):
if ($wave->status === 'processing') {
    $pending = count_pending_batches();

    if ($pending > 0) {
        // ✅ Dispatch progressive batches
        goto dispatchPending;
    }

    $active = count_active_batches();
    if ($active === 0) {
        demote_to_pending();
    } else {
        return; // Only return if no pending batches
    }
}
```

---

## Testing

### Validation Steps

1. **Check Progressive Creation:**
```sql
-- Verify batches created over time
SELECT batch_id, status, created_at
FROM wave_batches
WHERE job_id LIKE 'import_68db0dddaa%'
  AND wave_number = 7
ORDER BY created_at;
```

2. **Monitor Dispatch Logs:**
```
✅ "Wave processing but has pending batches; dispatching progressive batches"
✅ Pending batch count logged
✅ All batches eventually marked as "dispatched"
```

3. **Verify Completion:**
```sql
-- All batches should be dispatched/completed
SELECT status, count(*)
FROM wave_batches
WHERE job_id LIKE 'import_68db0dddaa%'
  AND wave_number = 7
GROUP BY status;

-- Expected: All in "dispatched", "processing", or "completed"
-- No "pending" batches remaining
```

### Expected Log Output

```
[INFO] Wave processing but has pending batches; dispatching progressive batches
  {job_id: "import_xxx", wave_number: 7, pending_batches: 35}

[DEBUG] Wave dispatched successfully
  {job_id: "import_xxx", wave_number: 7, batches_dispatched: 35}
```

---

## Impact Assessment

### Before Fix
- ❌ Only first batch of progressive creation dispatched
- ❌ Subsequent batches stuck in "pending" forever
- ❌ Derived waves never complete
- ❌ TransactionLine records not imported

### After Fix
- ✅ All progressively created batches dispatched
- ✅ Derived waves complete successfully
- ✅ TransactionLine records imported correctly
- ✅ Jobs complete after all waves (main + derived) finish

### Performance
- **No performance impact:** Only adds one DB query when wave is "processing"
- **Reduces redundant calls:** 348 calls now handled efficiently
- **Faster completion:** Batches dispatched immediately as they're created

---

## Related Issues

### Issue #1: Premature Job Completion (Fixed)
- **Problem:** Jobs completed when main waves finished, ignoring derived waves
- **Fix:** Check for derived waves before marking job complete
- **Document:** `docs/designs/import-fixes-sonnet.md`

### Issue #2: Progressive Batch Stalling (This Fix)
- **Problem:** Progressive batch creation caused partial wave dispatch
- **Fix:** Check for pending batches even when wave is "processing"
- **Document:** This document

### Combined Effect
Both fixes are **required** for derived waves to work correctly:
1. Fix #1 ensures the job doesn't complete prematurely
2. Fix #2 ensures all derived batches are dispatched

---

## Future Considerations

### Alternative Approach: Batch All Upfront
Instead of progressive creation, create all derived batches at once:

**Pros:**
- Simpler dispatch logic
- No partial wave dispatch issues

**Cons:**
- Must wait for ALL parent batches to complete
- Higher latency before first line batch can start
- Memory overhead of holding all parent IDs

**Decision:** Keep progressive creation because:
- Lower latency (can start importing lines sooner)
- Better memory usage (stream parent IDs)
- Fix is simple and adds minimal overhead

### Monitoring Recommendations

Add alerts for:
- Waves stuck in "processing" with pending batches > 0 for > 5 minutes
- Derived waves with pending batch count not decreasing
- Multiple `dispatchWave()` calls (>100) for same wave

---

## Conclusion

The progressive batch creation pattern is **correct and beneficial** for performance, but revealed a bug in the `dispatchWave()` method that assumed all batches exist when the wave first starts.

The fix is simple: **check for pending batches before returning early** when a wave is already "processing", and dispatch any newly created batches.

**Status:** ✅ Fixed and tested
**PHPStan:** ✅ Passing (Level 5)
**Ready for:** Production deployment