# API Node Configuration JSON Optimization - Developer Implementation Guide

## Problem Synopsis

**Current Issue**: The `initialResponse` property stored in `request_config` JSON contains full external API responses, creating unnecessarily large configuration objects. Some responses exceed 1MB when only field paths are needed for UI operations.

**Root Cause**: Large JSON objects are loaded every time an API Node configuration form is accessed, even though only field paths are typically needed for UI operations.

**Solution**: Separate `initialResponse` data into dedicated database column with lazy loading, reducing configuration JSON size by 90%+ while maintaining all existing functionality.

**Target Outcome**: Reduced configuration JSON size with zero functional regressions.

## Technical Architecture

**Approach**: Separate database column with Laravel model accessors for transparent data access.

**Storage Strategy**:
- `initial_response` LONGTEXT column for large response data
- Existing `request_config` JSON retains configuration metadata only
- Lazy loading via model accessors when response data is actually needed

**Data Access Pattern**:
- Model accessors handle transparent retrieval/storage
- Response data loaded only for field generation and split data processing
- No changes to existing API interfaces

## Files Requiring Changes

### Database Layer

#### Migration File (New)
```php
// database/migrations/tenants/YYYY_MM_DD_HHMMSS_add_initial_response_to_api_nodes.php
```
**Changes Required**:
- Add `initial_response` LONGTEXT column to `api_nodes` table
- Create data migration script to move existing data
- Add rollback logic to restore original structure

#### ApiNode Model
```php
// src/Domain/Ipaas/Nodes/Models/ApiNode.php
```
**Changes Required**:
- Add `initial_response` to `$fillable` array
- Implement `getInitialResponseAttribute()` accessor
- Implement `setInitialResponseAttribute()` mutator
- Add lazy loading logic for response data
- Handle backward compatibility for existing code

### Application Layer

#### ApiNodeForm Livewire Component
```php
// src/App/Livewire/Ipaas/Nodes/ApiNodeForm.php
```
**Changes Required**:
- Update `initialResponse` loading from new column
- Update save logic to use new storage pattern
- Modify configuration saving to exclude response data
- Ensure field generation continues working with new data access

#### CreateApiNode Action
```php
// src/Domain/Ipaas/Nodes/Actions/CreateApiNode.php
```
**Changes Required**:
- Update to handle `initial_response` column during node creation
- Modify configuration processing to separate response data
- Ensure proper data validation and storage

#### UpdateApiNode Action
```php
// src/Domain/Ipaas/Nodes/Actions/UpdateApiNode.php
```
**Changes Required**:
- Update save logic for separated storage
- Modify configuration update to handle new data structure
- Maintain backward compatibility during transition

### Helper Functions

#### IpaasHelper
```php
// src/App/Helpers/IpaasHelper.php
```
**Changes Required**:
- Verify `getKeyFiltered()` function works with new data access pattern
- No direct changes needed - function works with processed arrays

## Implementation Tasks

### Task 1: Database Schema Changes
**Dependencies**: None
**Components**:
1. Create migration file for `initial_response` column
2. Write data migration script to move existing `initialResponse` data
3. Test migration on staging data

### Task 2: Model Layer Updates
**Dependencies**: Task 1 completed
**Components**:
1. Update `ApiNode` model with new field and accessors
2. Implement lazy loading logic for response data
3. Add caching for frequently accessed response data
4. Create backward compatibility layer for transition period

### Task 3: Application Code Updates
**Dependencies**: Task 2 completed
**Components**:
1. Update `ApiNodeForm` to use new data access pattern
2. Modify `CreateApiNode` and `UpdateApiNode` actions
3. Update configuration save/load logic
4. Ensure field generation continues working correctly

### Task 4: Data Migration Execution
**Dependencies**: Tasks 1-3 completed and tested
**Components**:
1. Run data migration in staging environment
2. Validate data integrity and completeness
3. Update `request_config` JSON to remove `initialResponse` property
4. Test all affected functionality

## Technical Implementation Details

### Database Migration Script

```php
<?php
// database/migrations/tenants/YYYY_MM_DD_HHMMSS_add_initial_response_to_api_nodes.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    public function up(): void
    {
        // Add new column
        Schema::connection('tenant_connection')->table('api_nodes', function (Blueprint $table) {
            $table->longText('initial_response')->nullable();
        });

        // Migrate existing data
        $this->migrateExistingData();
    }

    public function down(): void
    {
        Schema::connection('tenant_connection')->table('api_nodes', function (Blueprint $table) {
            $table->dropColumn('initial_response');
        });
    }

    private function migrateExistingData(): void
    {
        $nodes = DB::connection('tenant_connection')
            ->table('api_nodes')
            ->whereNotNull('request_config')
            ->get();

        foreach ($nodes as $node) {
            $config = json_decode($node->request_config, true);
            if (isset($config['initialResponse'])) {
                DB::connection('tenant_connection')
                    ->table('api_nodes')
                    ->where('id', $node->id)
                    ->update([
                        'initial_response' => json_encode($config['initialResponse'])
                    ]);

                // Remove from request_config
                unset($config['initialResponse']);
                DB::connection('tenant_connection')
                    ->table('api_nodes')
                    ->where('id', $node->id)
                    ->update([
                        'request_config' => json_encode($config)
                    ]);
            }
        }
    }
};
```

### Model Implementation

```php
// src/Domain/Ipaas/Nodes/Models/ApiNode.php

class ApiNode extends Model
{
    // Add initial_response to fillable array
    // Cast initial_response as array for automatic JSON handling

    // Create accessor for backward compatibility:
    // - Return the initial_response data when accessed

    // Override request_config accessor:
    // - Decode existing JSON config
    // - For backward compatibility, include initialResponse in config if not already present
    // - Return merged configuration

    // Override request_config mutator:
    // - Extract initialResponse from incoming config data
    // - Store initialResponse in separate column
    // - Remove initialResponse from config before storing
    // - Store cleaned config as JSON
}
```

### Application Layer Changes

#### ApiNodeForm Updates
```php
// src/App/Livewire/Ipaas/Nodes/ApiNodeForm.php

public function mount($record, $firstNode)
{
    // In existing record loading logic:
    // - Load initialResponse from new initial_response column instead of config JSON
    // - Set this->initialResponse from record->initial_response
    // - Maintain all other existing loading logic
}

public function preview($requestConfig)
{
    // In existing preview logic:
    // - When firstNode is true, store API result in record->initial_response
    // - Update this->initialResponse for UI display
    // - Continue using IpaasHelper::getKeyFiltered for field generation
}

public function save($requestConfig, $paginationConfig = null)
{
    // In existing save logic:
    // - For first nodes, save initialResponse data to separate column
    // - Remove initialResponse from requestConfig before saving
    // - Ensure requestConfig no longer contains response data
    // - Maintain all other validation and save logic
}
```

## Acceptance Criteria

### Functional Verification
- [ ] `IpaasHelper::getKeyFiltered()` generates identical field lists from new data source
- [ ] Field dropdown populations in `api-node-form.blade.php` remain unchanged
- [ ] Split data processing in `save()` method works with separated storage
- [ ] Preview execution updates both `previewResult` and `initial_response` correctly
- [ ] Form mounting loads `initialResponse` from new column when available
- [ ] Backward compatibility maintained during transition period

### Data Integrity
- [ ] Migration script successfully moves all `initialResponse` data to new column
- [ ] Zero data loss during migration process (validated by count and hash checks)
- [ ] `request_config` JSON size reduced by 80%+ for affected nodes
- [ ] New `initial_response` column contains valid JSON arrays/objects

### Code Quality
- [ ] Model accessors handle null/empty values gracefully
- [ ] Error handling preserves existing behavior
- [ ] Laravel coding standards maintained throughout changes
- [ ] No breaking changes to public method signatures
- [ ] Proper use of `tenant_connection` for all database operations

### Testing Requirements
- [ ] Unit tests for model accessors/mutators
- [ ] Integration tests for form save/load operations
- [ ] Regression tests for field generation functionality
- [ ] Migration tests with various data scenarios

## Technical Risks

### Data Migration Risks
**Risk**: Incomplete or corrupted data migration
**Mitigation**:
- Test migration script on copy of production data
- Implement data validation checksums
- Create comprehensive rollback procedures

**Risk**: Long migration time causing downtime
**Mitigation**:
- Run migration during maintenance window
- Use batched processing for large datasets
- Prepare blue-green deployment if needed

### Code Integration Risks
**Risk**: Breaking field generation functionality
**Mitigation**:
- Extensive testing of `getKeyFiltered()` with new data access
- Validate all field dropdown populations
- Test with various API response structures

## Testing Strategy

### Unit Testing (Pest Format)
```php
describe('ApiNode Model', function () {
    it('returns array from initial_response accessor', function () {
        // Create ApiNode with JSON data in initial_response column
        // Verify that accessor returns proper array format
        // Assert that array contains expected test data
    });

    it('includes initial_response in request_config for compatibility', function () {
        // Create ApiNode with data in both columns
        // Access request_config property
        // Verify initialResponse is included for backward compatibility
        // Assert data matches what's stored in separate column
    });

    it('handles json encoding in initial_response mutator', function () {
        // Create new ApiNode instance
        // Set initial_response with array data
        // Verify that mutator properly JSON encodes the data
        // Assert stored attribute is valid JSON string
    });
});
```

### Integration Testing (Pest Format)
```php
describe('ApiNodeForm Integration', function () {
    it('loads form with separated storage', function () {
        // Create ApiNode with data in separate initial_response column
        // Mount Livewire component with test node
        // Assert that form loads initialResponse from new column
        // Verify UI displays correct data from separated storage
    });

    it('updates both storage locations during preview', function () {
        // Setup Livewire component for first node
        // Mock API response with test data
        // Call preview method with configuration
        // Assert both record.initial_response and component.initialResponse are updated
        // Verify data consistency between storage locations
    });

    it('separates initial_response from config during save', function () {
        // Create test ApiNode and mount component
        // Set initialResponse data in component
        // Call save method with mixed configuration data
        // Verify initialResponse is stored in separate column
        // Assert request_config no longer contains initialResponse
    });
});
```

### Migration Testing (Pest Format)
```php
describe('Migration Tests', function () {
    it('migrates existing initialResponse data to new column', function () {
        // Insert test data with initialResponse in request_config JSON
        // Run migration command
        // Verify data is moved to new initial_response column
        // Assert original request_config no longer contains initialResponse
        // Confirm data integrity and JSON validity
    });

    it('handles empty initialResponse data', function () {
        // Insert node with request_config that has no initialResponse
        // Run migration
        // Verify initial_response column remains null
        // Assert other config data is preserved
    });

    it('handles corrupted JSON data gracefully', function () {
        // Insert node with invalid JSON in request_config
        // Run migration without throwing exceptions
        // Verify migration completes successfully
        // Assert corrupted data is handled appropriately
    });
});
```