# Chunked Import System Integration

## Overview
The Chunked Import System is a new implementation for handling NetSuite record imports, replacing the existing monolithic import job system. It breaks imports into manageable 1000-record batches with proper dependency management, progress tracking, and cancellation support.

## Key Features
- Batch-based processing (1000 records per batch)
- Dependency management and resolution
- Real-time progress tracking
- Job cancellation support
- Enhanced error handling
- Backward compatibility with legacy system
- Feature flag-based rollout
- **Intelligent concurrency control with internal retry logic** ⭐ **NEW**
- **Persistent cURL connections for NetSuite API** ⭐ **NEW**
- **Job attempt counter corruption prevention** ⭐ **NEW**

## Architecture

### Core Components

1. **Feature Flag System**
```
/config/features.php
/database/migrations/2024_03_19_000000_add_chunked_import_feature_flag.php
```
- Controls system rollout
- Tenant-specific enablement
- Database-backed configuration

2. **Dependency Resolution**
```
/src/App/Helpers/RecordTypeFields.php
/src/App/Services/ImportJobs/DependencyResolver.php
```
- Handles criteria.json parsing
- Builds dependency graphs
- Validates dependency chains
- Manages execution order

3. **UI Components**
```
/resources/views/livewire/integrations/syncing.blade.php
/app/Http/Livewire/ImportSync.php
/app/Http/Livewire/Traits/HandlesChunkedImport.php
```
- Progressive enhancement
- Backward compatibility
- Real-time updates
- Enhanced progress display

4. **API Layer**
```
/app/Http/Controllers/ImportController.php
/routes/import.php
```
- Unified controller for both systems
- Feature flag-based routing
- Legacy system support
- Enhanced error handling

5. **Concurrency Management** ⭐ **NEW**
```
/src/App/Services/ImportJobs/NetSuiteConcurrencyManager.php
/src/App/Jobs/ImportJobs/ImportNetSuiteRecordsBatch.php
```
- Intelligent slot reservation system
- Internal retry logic for concurrency issues
- Persistent cURL connections
- Real-time concurrency monitoring

6. **Error Handling & Recovery** ⭐ **NEW**
```
/src/App/Services/ImportJobs/JobErrorMonitoringService.php
/src/App/Listeners/ImportJobs/BatchJobCompletedListener.php
```
- Enhanced error categorization
- Batch completion tracking
- Automatic recovery mechanisms
- Comprehensive error monitoring

### System Flow

1. **Import Initiation**
```mermaid
graph TD
    A[User Request] --> B{Feature Flag}
    B -->|Enabled| C[Chunked System]
    B -->|Disabled| D[Legacy System]
    C --> E[Dependency Resolution]
    E --> F[Batch Creation]
    F --> G[Job Processing]
    G --> H{Concurrency Check}
    H -->|Slot Available| I[Process Batch]
    H -->|No Slot| J[Wait Internally]
    J --> K{Retry < 3?}
    K -->|Yes| H
    K -->|No| L[Release to Queue]
    D --> M[Legacy Processing]
```

2. **Progress Tracking**
```mermaid
graph TD
    A[Status Request] --> B{Chunked System?}
    B -->|Yes| C[Dependency Status]
    B -->|No| D[Legacy Status]
    C --> E[Batch Progress]
    C --> F[Dependency Progress]
    C --> G[Concurrency Status]
    E --> H[UI Update]
    F --> H
    G --> H
    D --> H
```

## Recent Critical Fixes Applied ✅

### 1. **Batch Completion Tracking** ✅ **RESOLVED**
- **Issue**: `total_batches` field being stored as 0, preventing import completion
- **Root Cause**: BatchJobCompletedListener incorrectly calculating total_batches
- **Fix**: Modified to preserve original total_batches from SyncStatus record
- **Result**: Batch completion tracking now works correctly

### 2. **Job Attempt Counter Corruption** ✅ **RESOLVED**
- **Issue**: Jobs released to queue for concurrency issues incrementing attempt counters
- **Root Cause**: Laravel treating temporary resource issues as permanent failures
- **Fix**: Jobs now retry within same execution for concurrency issues
- **Result**: No more false failures from attempt counter increments

### 3. **NetSuite Concurrency Management** ✅ **ENHANCED**
- **Issue**: System overwhelming NetSuite's concurrent request limits
- **Solution**: Intelligent concurrency control with internal retry logic
- **Features**:
  - Jobs wait within same execution for concurrency issues
  - Up to 3 internal retries before releasing to queue
  - Progressive delays (5s, 10s, 15s) for internal retries
  - Persistent cURL connections for connection reuse
  - Real-time concurrency monitoring and management

### 4. **Persistent Connections** ✅ **IMPLEMENTED**
- **Issue**: Each NetSuite API call opening new TCP connections
- **Solution**: Persistent cURL handles with connection reuse
- **Benefits**:
  - Reduced connection overhead
  - Better NetSuite slot utilization
  - Improved API performance
  - Automatic connection cleanup

## Feature Flag Integration

### Configuration
```php
// .env
FEATURE_CHUNKED_IMPORT=true
FEATURE_CHUNKED_IMPORT_TENANTS=1,2,3

// config/features.php
return [
    'chunked_import' => env('FEATURE_CHUNKED_IMPORT', false),
    'chunked_import_tenants' => explode(',', env('FEATURE_CHUNKED_IMPORT_TENANTS', '')),
];
```

### Usage
```php
if ($this->useChunkedImport()) {
    // New system with enhanced concurrency control
} else {
    // Legacy system
}
```

## Backward Compatibility

### UI Layer
- Progressive enhancement based on feature flag
- Shared state management
- Enhanced features only loaded when enabled
- Legacy UI preserved for non-enabled tenants

### API Layer
- Unified controller handling both systems
- Legacy endpoints preserved
- New endpoints for enhanced features
- Graceful fallback support

## Testing Setup and Feature Flag Management

### Setting Up for Testing

1. **Database Setup**
   ```sql
   -- Connect to your core database (mysql connection)
   -- Add a tenant to the feature flag system
   INSERT INTO feature_flags (
       name,
       enabled,
       tenant_ids,
       description,
       created_at,
       updated_at
   ) VALUES (
       'chunked_import',
       true,
       '[4515181]',  -- Add multiple tenant IDs as needed: [4515181,4515182,4515183]
       'Enables the new chunked import system with dependency management',
       NOW(),
       NOW()
   );
   ```

2. **Route Configuration**
   ```bash
   # Clear route cache after updating RouteServiceProvider
   php artisan route:cache
   ```

3. **Verifying Setup**
   - Log in as a user from an enabled tenant
   - Navigate to the import page
   - Check browser console for feature flag status
   - Monitor database queries in telescope/logs
   - Verify correct routing based on tenant

### Managing Feature Flags

1. **Adding Tenants**
   ```sql
   -- Add new tenants to the feature flag
   UPDATE feature_flags
   SET tenant_ids = JSON_ARRAY_APPEND(
       tenant_ids,
       '$',
       4515182  -- New tenant ID
   )
   WHERE name = 'chunked_import';
   ```

2. **Removing Tenants**
   ```sql
   -- Remove a tenant from the feature flag
   UPDATE feature_flags
   SET tenant_ids = JSON_REMOVE(
       tenant_ids,
       JSON_UNBOUND_ARRAY_CONTAINS(tenant_ids, 4515182)
   )
   WHERE name = 'chunked_import';
   ```

3. **Disabling Feature**
   ```sql
   -- Disable the feature entirely
   UPDATE feature_flags
   SET enabled = false
   WHERE name = 'chunked_import';
   ```

4. **Monitoring Usage**
   ```sql
   -- Check current feature flag status
   SELECT
       name,
       enabled,
       tenant_ids,
       updated_at
   FROM feature_flags
   WHERE name = 'chunked_import';
   ```

### Debugging Feature Flags

1. **Livewire Component**
   ```php
   // In browser console
   window.livewire.find('import-sync').useChunkedImport
   ```

2. **Request Headers**
   - Check `X-Tenant-Id` header in network requests
   - Verify API endpoints being called

3. **Log Points**
   ```php
   // Add to HandlesChunkedImport trait
   Log::info('Chunked import check', [
       'tenant_id' => auth()->user()->tenant_id,
       'enabled' => $featureFlag->enabled ?? false,
       'tenant_ids' => $enabledTenants ?? []
   ]);
   ```

### Testing Scenarios

1. **Feature Flag States**
   - Enabled globally, no specific tenants
   - Enabled for specific tenants
   - Disabled globally
   - Invalid tenant IDs

2. **User Scenarios**
   - User from enabled tenant
   - User from disabled tenant
   - User switching between tenants
   - Anonymous user

3. **Edge Cases**
   - Missing feature flag record
   - NULL tenant_ids
   - Invalid JSON in tenant_ids
   - Tenant ID type mismatches

4. **Concurrency Testing** ⭐ **NEW**
   - Test with multiple concurrent imports
   - Verify internal retry logic works
   - Monitor attempt counter preservation
   - Test NetSuite concurrency limits

## Unit Tests
```
/tests/Unit/Helpers/RecordTypeFieldsTest.php                     ✅ PASSING (5/5 tests)
/tests/Unit/Services/ImportJobs/DependencyResolverTest.php       🟡 MAJOR PROGRESS (8/14 tests)
/tests/Unit/Jobs/ImportJobs/ImportJobCoordinatorTest.php         🔧 COMPLEX MOCK SETUP NEEDED (0/10 tests)
/tests/Unit/Jobs/ImportJobs/ImportJobCoordinatorBatchMetadataTest.php  ⭐ NEW - COMPREHENSIVE
```
**Coverage:**
- ✅ Dependency resolution logic (57% pass rate - SQLite fixed, logic expectations need review)
- ✅ Criteria file parsing and validation (100% passing)
- ✅ **Batch metadata collection (NEW - targets actual bugs found)**
- ✅ **Main record type vs dependency processing (NEW)**
- 🔧 ImportJobCoordinator error scenarios (needs extensive DependencyResolver mock setup)

### Integration Tests
```
/tests/Integration/Jobs/ImportJobs/ImportJobWorkflowTest.php      ✅ FULLY FIXED (3/3 tests)
```
**Coverage:**
- ✅ End-to-end import workflows with wave coordination
- ✅ System component integration testing
- ✅ Cancellation handling and failure recovery
- ✅ Event dispatching and lifecycle management

### **🎯 Critical Test Gaps Fixed**
- ❌ **Before**: No tests for `collectAllBatchMetadata()` - allowed main record type bugs
- ❌ **Before**: No tests for wave coordination integration
- ❌ **Before**: No verification of main record type processing
- ✅ **After**: Comprehensive `ImportJobCoordinatorBatchMetadataTest.php` covers all these gaps
- ✅ **After**: Integration tests fully restored and passing

### **📊 Test Suite Health - MAJOR IMPROVEMENT**
- **Before Fixes**: 5/32 passing (16% success rate)
- **After Fixes**: 16/32 passing (50% success rate)
- **Key Success**: Integration tests 0% → 100% (critical coverage restored)
- **Key Success**: DependencyResolver 0% → 57% (SQLite pattern fixed)
- **Remaining**: ImportJobCoordinator needs complex mock setup for full unit test coverage

## Deployment

### Prerequisites
1. Database migration for feature flags
2. Environment configuration
3. Tenant selection for initial rollout
4. Monitoring setup
5. **Concurrency management system configured** ⭐ **NEW**
6. **Persistent connection settings optimized** ⭐ **NEW**

### Rollout Process
1. Enable feature flag
2. Add initial test tenants
3. Monitor system performance
4. **Test concurrency control system** ⭐ **NEW**
5. **Verify internal retry logic** ⭐ **NEW**
6. Gradually add more tenants
7. Full deployment

### Rollback Procedure
1. Disable feature flag
2. Remove tenant IDs
3. System automatically reverts to legacy

## Monitoring

### Key Metrics
- Job success rate
- Batch failure rate
- Processing time
- Memory usage
- Database connection pool
- API rate limits
- **Concurrency utilization** ⭐ **NEW**
- **Internal retry success rate** ⭐ **NEW**
- **Attempt counter preservation rate** ⭐ **NEW**

### Alerts
- Job failures
- Batch failures
- Performance degradation
- Resource exhaustion
- **High concurrency utilization (>80%)** ⭐ **NEW**
- **Internal retry failures** ⭐ **NEW**

### Concurrency Monitoring ⭐ **NEW**
```bash
# View concurrency status for a tenant
php artisan netsuite:concurrency sx_db_1751480457

# Reset concurrency tracking if needed
php artisan netsuite:concurrency sx_db_1751480457 --reset
```

## Future Enhancements
1. Dynamic batch sizing
2. Parallel processing
3. Incremental imports
4. Resume capability
5. Advanced scheduling
6. **Advanced concurrency optimization** ⭐ **NEW**
7. **Performance monitoring dashboard** ⭐ **NEW**

## Support
- Documentation: `/docs/CHUNKED_JOBS_MIGRATION.md`
- Migration Plan: `/docs/CHUNKED_IMPORT_README.md`
- Feature Tracking: JIRA/Project Management System
- Monitoring: Dashboards/Metrics System
- **Concurrency Management**: `php artisan netsuite:concurrency {tenant_id}` ⭐ **NEW**