# Duplicate Prevention System

## Overview

The Duplicate Prevention System is designed to prevent duplicate processing of records and pages across multiple flow runs. It uses a combination of Redis and database storage to track processed items and ensure idempotency.

## Architecture

### Storage Layers

1. **Redis Layer**: Fast, in-memory storage for active deduplication
2. **Database Layer**: Persistent storage for audit trails and long-term tracking

### Key Components

- **FlowDuplicatePrevention**: Tracks processed records at the record level
- **FlowCompletionLocks**: Ensures flow completion events are only dispatched once
- **FlowPagePrevention**: Prevents duplicate page processing
- **CleanupFlowDataService**: Manages cleanup of expired entries

## Key Pattern Handling (Latest)

### Problem
The original implementation used `SCAN` operations for Redis key cleanup, but `SCAN` commands do not automatically handle Redis key prefixes. This caused `phpredis` to fail to find keys while `predis` succeeded, leading to inconsistent cleanup behavior.

Additionally, even after switching to `KEYS` operations, there was a **double-prefix issue** where:
1. **Key Discovery**: `KEYS` command returned keys with full prefixes (e.g., `suitex_database_flow:33:...`)
2. **Key Deletion**: `DEL` command expected unprefixed keys, causing deletion to fail with `deleted_count: 0`

### Solution
Implemented a **production-safe SCAN solution with KEYS fallback**:

1. **Production-safe SCAN implementation** with comprehensive retry logic and error handling
2. **Automatic fallback to KEYS** when SCAN operations fail consistently
3. **Proper prefix handling** for both SCAN and KEYS operations
4. **Batch processing** to minimize Redis blocking and memory usage

#### Before (SCAN with manual prefix handling + double-prefix deletion)
```php
// Get Redis prefix for SCAN operations (Redis client doesn't auto-handle prefixes in SCAN)
$config = config('database.redis.' . $connection);
$prefix = $config['prefix'] ?? config('database.redis.options.prefix', 'suitex_database_');

// Normalize patterns with prefix for SCAN operations
$scanPatterns = array_map(function($pattern) use ($prefix) {
    if (str_starts_with($pattern, $prefix)) {
        return $pattern;
    }
    return $prefix . $pattern;
}, $basePatterns);

foreach ($scanPatterns as $index => $pattern) {
    $cursor = 0; // CRITICAL BUG: Should be null, not 0 - causes immediate false returns
    $keys = [];

    do {
        [$cursor, $matchedKeys] = $redis->scan($cursor, ['match' => $pattern, 'count' => 100]);
        if (!empty($matchedKeys)) {
            $keys = array_merge($keys, $matchedKeys);
        }
    } while ($cursor != 0);
}

// Delete keys directly - caused double-prefix issue
$deleted = $redis->del($batch); // Keys already had prefix, causing deletion to fail
```

#### After (Production-safe SCAN with KEYS fallback + proper prefix handling)
```php
// Primary: Production-safe SCAN implementation
private function scanRedisKeys($redis, string $pattern, string $connection): array
{
    // Get Redis prefix for SCAN operations
    $config = config('database.redis.' . $connection);
    $prefix = $config['prefix'] ?? config('database.redis.options.prefix', 'suitex_database_');
    $prefixedPattern = $prefix . $pattern;

    // Enable retry mode for phpredis
    if (method_exists($redis, 'setOption')) {
        $redis->setOption(\Redis::OPT_SCAN, \Redis::SCAN_RETRY);
    }

    $keys = [];
    $cursor = null; // CRITICAL: Must be null, not 0, for proper SCAN initialization
    $consecutiveFalseResults = 0;

    do {
        $result = $redis->scan($cursor, [
            'match' => $prefixedPattern,
            'count' => 500 // Production-safe batch size
        ]);

        if ($result === false) {
            $consecutiveFalseResults++;

            // Throw exception for fallback after 3 attempts
            if ($consecutiveFalseResults >= 3) {
                throw new \Exception("SCAN operation consistently returning false");
            }

            usleep(10000); // 10ms delay before retry
            continue;
        }

        [$cursor, $matchedKeys] = $result;
        if (!empty($matchedKeys)) {
            $keys = array_merge($keys, $matchedKeys);
        }

    } while ($cursor != 0 || ($consecutiveFalseResults > 0 && $consecutiveFalseResults < 3));

    return array_unique($keys); // Remove duplicates
}

// Fallback: Use KEYS if SCAN fails
try {
    $keys = $this->scanRedisKeys($redis, $pattern, $connection);
} catch (\Exception $e) {
    Log::warning('⚠️ SCAN operation failed, falling back to KEYS', [
        'pattern' => $pattern,
        'error' => $e->getMessage()
    ]);

    // Fallback to KEYS
    $keys = $redis->keys($pattern);
}

// Strip Redis prefix from keys before deletion
$config = config('database.redis.' . $connection);
$prefix = $config['prefix'] ?? config('database.redis.options.prefix', 'suitex_database_');
$unprefixedBatch = array_map(function($key) use ($prefix) {
    return str_starts_with($key, $prefix) ? substr($key, strlen($prefix)) : $key;
}, $batch);

// Delete keys without prefix - Redis client will add prefix automatically
$deleted = $redis->del($unprefixedBatch);
```

### Benefits
- **Production-safe**: Non-blocking SCAN operations prevent Redis performance issues
- **Driver-agnostic**: Works consistently with both `phpredis` and `predis`
- **Reliable fallback**: Automatic fallback to KEYS when SCAN fails
- **Memory efficient**: Batch processing (500 keys) minimizes memory usage
- **Comprehensive error handling**: Detailed logging and retry mechanisms
- **Proper deletion**: Eliminates double-prefix issue that caused deletion failures
- **Scalable**: Handles large key sets without blocking Redis operations
- **Correct initialization**: Fixed critical cursor initialization bug

### ⚠️ Critical SCAN Gotcha Fixed
**Problem**: Initializing SCAN cursor to `0` instead of `null`
```php
// ❌ WRONG - Causes immediate false returns
$cursor = 0;
$result = $redis->scan($cursor, ['match' => $pattern]);

// ✅ CORRECT - Proper SCAN initialization
$cursor = null;
$result = $redis->scan($cursor, ['match' => $pattern]);
```

**Why this matters**:
- Redis SCAN expects the initial cursor to be `null`
- Starting with `0` causes SCAN to return `false` immediately
- This was the primary cause of "SCAN unreliable" issues
- The fix enables proper SCAN functionality across all Redis drivers

### Technical Details

#### Key Discovery Flow
1. **Pattern Definition**: Define base patterns without manual prefix handling
   ```php
   $basePatterns = [
       "flow:{$flowId}:processed_record:*",
       "flow:{$flowId}:processed_page:*",
       "flow:{$flowId}:completion_lock"
   ];
   ```

2. **SCAN Operation**: Production-safe iterative scanning with automatic fallback
   ```php
   try {
       $keys = $this->scanRedisKeys($redis, $pattern, $connection);
       // Non-blocking SCAN with retry logic
   } catch (\Exception $e) {
       $keys = $redis->keys($pattern); // Fallback to KEYS
   }
   // Returns: ["suitex_database_flow:33:processed_record:hash1", ...]
   ```

#### Key Deletion Flow
1. **Prefix Extraction**: Get Redis prefix from configuration
   ```php
   $prefix = $config['prefix'] ?? 'suitex_database_';
   ```

2. **Prefix Stripping**: Remove prefix from keys before deletion
   ```php
   $unprefixedBatch = array_map(function($key) use ($prefix) {
       return str_starts_with($key, $prefix) ? substr($key, strlen($prefix)) : $key;
   }, $batch);
   ```

3. **Deletion**: Delete unprefixed keys (Redis client adds prefix automatically)
   ```php
   $deleted = $redis->del($unprefixedBatch);
   ```

#### Example Transformation
```
Input keys: ["suitex_database_flow:33:processed_record:hash1"]
Prefix: "suitex_database_"
Unprefixed keys: ["flow:33:processed_record:hash1"]
Redis DEL result: 1 (successful deletion)
```

## Redis Key Cleanup Issues (Latest)

### Problem Description
The cleanup service was failing to find Redis keys when using the `phpredis` driver, while working correctly with `predis`. This was due to differences in how the two drivers handle Redis key prefixes in `SCAN` operations.

### Root Cause
- **Critical Bug**: SCAN cursor initialized to `0` instead of `null`, causing immediate failures
- **Redis SCAN Gotcha**: SCAN expects initial cursor to be `null`; using `0` causes operations to return `false` immediately
- **`KEYS` command**: Automatically handles Redis key prefixes through Laravel's Redis client
- **`SCAN` command**: Does NOT automatically handle Redis key prefixes, requiring manual prefix management
- **Driver differences**: `phpredis` and `predis` handle `SCAN` operations differently, leading to inconsistent behavior

### Impact
- Incomplete cleanup of Redis keys
- Potential memory leaks from accumulated keys
- Inconsistent behavior between different Redis drivers

## Redis Key Cleanup Driver-Agnostic Approach (Latest)

### Solution Overview
Replaced all `SCAN` operations with `KEYS` operations to ensure consistent behavior across Redis drivers.

### Implementation Details

1. **Pattern Definition**: Define base patterns without manual prefix handling
   ```php
   $basePatterns = [
       "flow:{$flowId}:processed_record:*",  // Record-level deduplication
       "flow:{$flowId}:processed_page:*",    // Page-level deduplication
       "flow:{$flowId}:completion_lock",     // Completion lock
       "flow:{$flowId}:start_time",          // Flow timing
       "subflow_state:*:{$flowId}",         // Subflow state
       "subflow_item:*:{$flowId}:*"         // Subflow item idempotency
   ];
   ```

2. **Key Discovery**: Use `KEYS` command for each pattern
   ```php
   foreach ($basePatterns as $pattern) {
       $keys = $redis->keys($pattern);
       // Process found keys...
   }
   ```

3. **Key Deletion**: Delete keys in batches
   ```php
   $batches = array_chunk($keys, 100);
   foreach ($batches as $batch) {
       $deleted = $redis->del($batch);
       $totalDeleted += $deleted;
   }
   ```

### Benefits
- **Consistent behavior**: Same results regardless of Redis driver
- **Simplified maintenance**: No need to handle driver-specific prefix logic
- **Improved reliability**: `KEYS` command is more predictable than `SCAN`
- **Better performance**: Single command per pattern instead of cursor iteration

### Trade-offs
- **Memory usage**: `KEYS` loads all matching keys into memory at once
- **Blocking**: `KEYS` can block Redis during execution (acceptable for cleanup operations)
- **Pattern complexity**: Limited to simple glob patterns supported by `KEYS`

## Immediate Priorities

### ✅ Completed
- **Production-safe SCAN**: Implemented non-blocking SCAN operations with comprehensive retry logic
- **KEYS fallback mechanism**: Added automatic fallback to KEYS when SCAN operations fail
- **Redis cleanup optimization**: Developed hybrid approach using SCAN-first with KEYS fallback
- **Key pattern handling**: Proper prefix handling for both SCAN and KEYS operations
- **Key deletion fix**: Resolved double-prefix issue that prevented successful key deletion
- **Service reliability**: Ensured consistent cleanup behavior across Redis drivers
- **Complete solution**: Both key discovery and deletion now work correctly with `phpredis` and `predis`

### 🔄 In Progress
- **Unit test coverage**: Improving test coverage for the cleanup service
- **Performance monitoring**: Monitoring cleanup performance in production

### 📈 Future Optimizations

Based on production log analysis, the following optimizations have been identified for future implementation:

#### 1. **Pattern Optimization**
**Issue**: Many generic wildcard patterns return false unnecessarily, leading to inefficient SCAN operations.

**Current Approach**:
```php
// Too many failed pattern attempts
$patterns = [
    "flow:33:run:*:jobs_expected",     // ❌ Should be specific run ID
    "flow:33:run:*:jobs_completed",    // ❌ Should be specific run ID
    "subflow_state:*:33"               // ❌ Rarely has matches
];
```

**Recommended Improvement**:
```php
// Dynamic pattern generation based on actual run IDs
private function generateOptimizedPatterns(int $flowId, ?string $runId = null): array
{
    $patterns = $this->getStaticPatterns($flowId);

    if ($runId) {
        $patterns = array_merge($patterns, $this->getRunSpecificPatterns($flowId, $runId));
    } else {
        // Dynamically discover run IDs first
        $runIds = $this->discoverRunIds($flowId);
        foreach ($runIds as $discoveredRunId) {
            $patterns = array_merge($patterns, $this->getRunSpecificPatterns($flowId, $discoveredRunId));
        }
    }

    return $patterns;
}

private function discoverRunIds(int $flowId): array
{
    // Use lightweight SCAN to find run IDs first
    $redis = Redis::connection();
    $cursor = null;
    $runIds = [];

    do {
        $result = $redis->scan($cursor, [
            'match' => "suitex_database_flow:{$flowId}:run:*",
            'count' => 100
        ]);

        if ($result !== false) {
            [$cursor, $keys] = $result;
            foreach ($keys as $key) {
                if (preg_match('/flow:' . $flowId . ':run:([^:]+):/', $key, $matches)) {
                    $runIds[] = $matches[1];
                }
            }
        }
    } while ($cursor != 0);

    return array_unique($runIds);
}
```

#### 2. **Connection Optimization**
**Issue**: Operations check all connections even when certain connections consistently have no keys.

**Current Behavior**:
```
🔍 Found keys using SCAN {"connection":"default"}
⚠️ SCAN returned false {"connection":"cache","keys_found":0}
```

**Recommended Improvement**:
```php
private function getOptimalConnections(int $flowId): array
{
    // Check which connections actually have keys for this flow
    $activeConnections = [];
    foreach (['default', 'cache'] as $connection) {
        if ($this->connectionHasKeys($connection, $flowId)) {
            $activeConnections[] = $connection;
        }
    }
    return $activeConnections ?: ['default']; // Fallback to default
}

private function connectionHasKeys(string $connection, int $flowId): bool
{
    try {
        $redis = Redis::connection($connection);
        $cursor = null;
        $result = $redis->scan($cursor, [
            'match' => "suitex_database_flow:{$flowId}:*",
            'count' => 1
        ]);

        return $result !== false && !empty($result[1]);
    } catch (\Exception $e) {
        return false;
    }
}
```

#### 3. **SCAN Retry Logic Enhancement**
**Issue**: SCAN retries occur even for patterns that are unlikely to have matches.

**Current Behavior**:
```
⚠️ SCAN returned false, retrying {"consecutive_false":1}
🔍 Searched Redis keys {"keys_found":0}
```

**Recommended Improvement**:
```php
private function scanRedisKeysWithSmartRetry($redis, string $pattern, string $connection): array
{
    // Skip retry for patterns that consistently return false
    if ($this->isCommonEmptyPattern($pattern)) {
        $result = $redis->scan(null, [
            'match' => $this->addPrefix($pattern, $connection),
            'count' => 500
        ]);

        if ($result === false) {
            Log::debug('⏭️ Skipping retries for common empty pattern', [
                'pattern' => $pattern,
                'connection' => $connection
            ]);
            return [];
        }

        return $result[1] ?? [];
    }

    // Use existing robust retry logic for other patterns
    return $this->scanRedisKeys($redis, $pattern, $connection);
}

private function isCommonEmptyPattern(string $pattern): bool
{
    $emptyPatterns = [
        'flow:*:completion_lock',
        'flow:*:start_time',
        'subflow_state:*:*',
    ];

    foreach ($emptyPatterns as $emptyPattern) {
        if (fnmatch($emptyPattern, $pattern)) {
            return true;
        }
    }

    return false;
}
```

#### 4. **Logging Efficiency**
**Issue**: Verbose logging creates 1.28 log entries per key cleaned (32 entries for 25 keys).

**Recommended Improvement**:
```php
private function logScanAttempt(string $pattern, string $connection, array $result): void
{
    // Only log retries for patterns likely to have matches
    if ($result === false && !$this->isCommonEmptyPattern($pattern)) {
        Log::debug('⚠️ SCAN returned false, retrying', [
            'pattern' => $pattern,
            'connection' => $connection
        ]);
    }

    // Aggregate logging for batch operations
    if (!empty($result[1])) {
        Log::debug('🔍 SCAN batch found', [
            'pattern' => $pattern,
            'connection' => $connection,
            'batch_size' => count($result[1])
        ]);
    }
}
```

#### 5. **Performance Metrics Collection**
**Recommended Addition**: Add metrics collection for optimization tracking.

```php
private function collectPerformanceMetrics(array $scanResults): void
{
    $metrics = [
        'total_patterns_scanned' => count($scanResults),
        'successful_patterns' => count(array_filter($scanResults, fn($r) => $r['keys_found'] > 0)),
        'empty_patterns' => count(array_filter($scanResults, fn($r) => $r['keys_found'] === 0)),
        'average_keys_per_pattern' => array_sum(array_column($scanResults, 'keys_found')) / count($scanResults),
        'scan_efficiency' => count(array_filter($scanResults, fn($r) => $r['keys_found'] > 0)) / count($scanResults) * 100
    ];

    Log::info('📊 Cleanup performance metrics', $metrics);
}
```

### 📊 Current Performance Baseline

**From Production Analysis (Flow ID 33)**:
- ✅ **Total Redis Keys Cleaned**: 25 keys
- ✅ **Database Entries Cleaned**: 11 entries
- ✅ **Success Rate**: 100% for key discovery and deletion
- ✅ **SCAN Success**: 6 successful operations found keys
- ⚠️ **SCAN Attempts**: 32 total attempts (26 returned false as expected)
- ⚠️ **Log Efficiency**: 1.28 log entries per key cleaned
- ✅ **Zero Fallbacks**: No KEYS fallback operations needed

**Performance Score**: 9/10 - Excellent functionality with room for pattern optimization

### 🎯 Implementation Priority

1. **High Priority**: Pattern optimization - Will reduce unnecessary SCAN operations by ~60%
2. **Medium Priority**: Connection optimization - Will reduce redundant connection checks
3. **Low Priority**: Logging efficiency - Cosmetic improvement for cleaner logs

### 📋 Planned
- **Cleanup scheduling**: Implement automated cleanup scheduling
- **Metrics collection**: Add cleanup metrics and monitoring
- **Error handling**: Improve error handling and recovery mechanisms

## Usage Examples

### Manual Cleanup
```bash
# Dry run to see what would be cleaned up
php artisan flow:cleanup-expired --dry-run

# Actual cleanup
php artisan flow:cleanup-expired
```

### Programmatic Cleanup
```php
use App\Services\CleanupFlowDataService;

$service = new CleanupFlowDataService();
$result = $service->cleanupFlowData($flowId);

// Result contains cleanup statistics
echo "Redis keys deleted: " . $result['redis_keys_deleted'];
echo "Database entries deleted: " . $result['database_entries_deleted'];
```

## Configuration

### Redis Settings
```php
// config/database.php
'redis' => [
    'default' => [
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', 6379),
        'database' => env('REDIS_DB', 0),
        'prefix' => env('REDIS_PREFIX', 'suitex_database_'),
    ],
    'cache' => [
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', 6379),
        'database' => env('REDIS_CACHE_DB', 1),
        'prefix' => env('REDIS_PREFIX', 'suitex_database_'),
    ],
],
```

### Cleanup Settings
```php
// config/flow.php
'cleanup' => [
    'batch_size' => 100,
    'max_execution_time' => 300, // 5 minutes
    'log_level' => 'info',
],
```

## Monitoring and Maintenance

### Key Metrics
- Redis key count per flow
- Cleanup execution time
- Keys deleted per cleanup run
- Error rates and types

### Regular Maintenance
- Monitor Redis memory usage
- Review cleanup logs for errors
- Adjust cleanup frequency based on usage patterns
- Verify cleanup effectiveness across different Redis drivers

## Troubleshooting

### Common Issues

1. **Keys not found during cleanup**
   - Verify Redis connection and authentication
   - Check key pattern matching
   - Ensure Redis prefix configuration is correct

2. **Cleanup performance issues**
   - Reduce batch size if memory usage is high
   - Increase cleanup frequency to reduce key accumulation
   - Monitor Redis performance during cleanup operations

3. **Driver-specific issues**
   - Test with both `phpredis` and `predis` drivers
   - Verify Redis version compatibility
   - Check driver-specific configuration requirements

### Debug Commands
```bash
# Check Redis connection
php artisan tinker --execute="Redis::connection()->ping()"

# List Redis keys
php artisan tinker --execute="Redis::connection()->keys('*')"

# Test key patterns
php artisan tinker --execute="Redis::connection()->keys('flow:*')"
```

## Future Enhancements

### Planned Features
- **Intelligent cleanup**: Prioritize cleanup based on key age and usage patterns
- **Distributed cleanup**: Support for multi-instance cleanup coordination
- **Cleanup analytics**: Detailed reporting on cleanup effectiveness and patterns
- **Adaptive scheduling**: Dynamic cleanup scheduling based on system load

### Research Areas
- **Alternative cleanup strategies**: Exploring Redis modules and advanced cleanup techniques
- **Performance optimization**: Investigating batch processing and parallel cleanup
- **Monitoring integration**: Integration with APM and monitoring systems