# Epic 3: Gap Analysis Against V10 Design Document

**Date:** 2026-03-12
**Scope:** Comparison of Epic 3 files (`design-spec.md`, `tech-spec.md`, `jira-ticket.md`) against `docs/designs/data-sync/V10.md`.

---

## Overall Assessment

Epic 3 correctly identifies the need for a mock server to avoid exhausting live NetSuite governance limits during development, and makes reasonable infrastructure choices (NestJS, Cloud Run, Cloud SQL for persistence). The cursor-based polling endpoint accurately simulates NetSuite's `Search.runPaged()` constraints (1,000 result limit, lastModifiedDate + internalId sorting). The Chaos Interceptor concept directly supports testing V10's adaptive throttling requirements.

However, the mock server has several structural gaps that would prevent it from supporting the integration and end-to-end test scenarios defined in V10 Appendix Q2. Most critically, it lacks marker field support for circular update prevention testing, uses the wrong payload format for the write endpoint, and is missing the two-phase polling pattern that V10 specifies.

Additionally, the `design-spec.md` acceptance criteria are copy-pasted from Epic 1 and test JSON schema validation -- not mock server behavior.

---

## Gap 1: design-spec.md Acceptance Criteria Are from Epic 1 (Critical)

**V10 Reference:** Not a V10 gap per se, but a document integrity issue.

**Current State:** The three acceptance criteria in `design-spec.md` (Scenarios 1-3) are verbatim copies of Epic 1's acceptance criteria. They test:
- Valid payload acceptance against the canonical JSON schema
- Normalization rejection of un-normalized NetSuite data types
- Attribution metadata enforcement (`sourceSystem`, `writeId`)

These test the **schema validator**, not the **mock server**. The `jira-ticket.md` has the correct acceptance criteria (State Storage, Governance Exhaustion, Cursor Pagination).

**Impact:** A developer using `design-spec.md` as the source of truth would build schema validation tests instead of mock server behavioral tests. The Epic would be marked "done" without proving the mock server actually works.

**Required Change:** Replace the `design-spec.md` acceptance criteria with the ones from `jira-ticket.md` (or an expanded set). The correct scenarios for a mock server are:
1. State persistence and simulated ingestion (write → persist → return internalId)
2. Governance exhaustion (concurrent requests → 429 beyond threshold)
3. Cursor pagination (polling query → sorted, paginated results ≤ 1,000)
4. Idempotency (duplicate writeId → return existing record, not create new)

**Affected Files:** `design-spec.md` (Acceptance Criteria section).

---

## Gap 2: Write Endpoint Accepts Canonical Envelope Instead of RESTlet Format (High)

**V10 Reference:** The SuiteX → NetSuite write path (lines 456-472):

> Issues RESTlet or REST/SOAP calls to NetSuite with idempotency keys derived from `eventId`, `writeId`, and record identifiers, and sets the appropriate NetSuite marker fields (`custbody_suitex_write_id` and `custbody_suitex_write_source`).

The canonical event envelope (Epic 1's schema) is the format for messages on the `events.raw` and `events.merged` Pub/Sub topics. It is NOT the format of an outbound RESTlet request to NetSuite. SuiteX's NetSuite Writer consumer translates `MergedEvent` data into a RESTlet-compatible request payload that includes:
- The record's field values (in NetSuite's expected format)
- `custbody_suitex_write_id` = UUID
- `custbody_suitex_write_source` = `'suitex'`

**Current State:** Epic 3 states: "These endpoints must accept and validate the Epic 1 Canonical JSON payloads." The write endpoint expects the canonical envelope wrapper (with `eventId`, `accountId`, `schemaVersion`, etc.) rather than what SuiteX would actually send to a NetSuite RESTlet.

**Impact:** The mock server cannot test the actual SuiteX → NetSuite write path. The NetSuite Writer consumer would need to be coded against a different format than what the mock server accepts, requiring either:
- A translation layer that won't be tested until the real NetSuite integration
- Rewriting the mock server later when the actual format is needed

**Required Change:** Define a separate RESTlet request schema that the mock server accepts. This schema should represent what NetSuite's RESTlet would expect:

```json
{
  "recordType": "project",
  "operation": "create|update|delete",
  "fields": {
    "companyname": "Acme Corp",
    "custbody_suitex_write_id": "uuid",
    "custbody_suitex_write_source": "suitex"
  }
}
```

The mock can still validate that the field values within `fields` conform to Epic 1 normalization rules (booleans, ISO dates, sorted multiselects), but the outer wrapper should simulate the RESTlet interface, not the Pub/Sub envelope.

**Affected Files:** `design-spec.md` (section A), `tech-spec.md` (Write Endpoint), `jira-ticket.md` (Write Endpoint).

---

## Gap 3: No Marker Field Support for Circular Update Prevention Testing (High)

**V10 Reference:** Circular Update Prevention appendix, Section 3 (lines 2777-2803):

> SuiteX writes to NetSuite must mark the internal NetSuite record with:
> - `custbody_suitex_write_id` (free-text)
> - `custbody_suitex_write_source` (list/enum: suitex, batch, other)

V10 Appendix M7 Polling Suppression (lines 2040-2055): The polling layer checks these marker fields before emitting events. If `writeSource === 'suitex'`, the polling event is suppressed.

V10 Appendix Q2 End-to-End Tests (lines 2664-2669): Five circular update prevention test scenarios require marker field behavior.

**Current State:** The mock server's `MockRecord` entity has no fields for `custbody_suitex_write_id` or `custbody_suitex_write_source`. When the polling endpoint returns records, there is no way to test whether SuiteX's poller correctly suppresses SuiteX-originated writes based on marker fields.

**Impact:** V10's end-to-end testing checklist (Q2) explicitly requires testing:
- SuiteX write → NetSuite → Polling suppresses event
- No infinite loops in bidirectional sync

Without marker field support in the mock, these tests cannot be run against the mock server.

**Required Change:** Add marker fields to the `MockRecord` entity:

```typescript
@Entity()
export class MockRecord {
  // ... existing fields ...

  @Column({ nullable: true })
  suitexWriteId: string | null;

  @Column({ nullable: true })
  suitexWriteSource: string | null; // 'suitex' | 'batch' | null
}
```

The write endpoint must persist incoming marker fields. The polling endpoint must return them so SuiteX's poller can test suppression logic.

**Affected Files:** `tech-spec.md` (Entity Model), `design-spec.md` (section B).

---

## Gap 4: Polling Uses Single-Phase Instead of V10's Two-Phase Pattern (Medium)

**V10 Reference:** Appendix M1 (lines 1876-1880):

> Two-phase polling:
> 1. **Discovery phase**: Query for changed record IDs and timestamps
> 2. **Detail phase**: Fetch full field values for changed records (batched)

**Current State:** Epic 3 defines a single search endpoint that returns full records (including `payload`) in one call. V10's polling architecture separates discovery (lightweight: just IDs and timestamps) from detail fetch (heavier: full field values), specifically to minimize governance cost.

**Impact:** SuiteX's polling pipeline will be built with a two-phase architecture. If the mock server only offers a single-phase endpoint, the discovery phase has no endpoint to test against, and the detail fetch phase (which V10 recommends using SOAP `getList` for efficiency -- Appendix M3) has no mock equivalent.

**Required Change:** Add a second polling endpoint that returns only discovery-level data:

**Discovery Endpoint:** `POST /mock-netsuite/search/:recordType`
- Returns: `[{ internalId, lastModifiedDate }]` (no full payload)
- Sorted by `lastModifiedDate ASC, internalId ASC`
- Limited to 1,000 results

**Detail Fetch Endpoint:** `POST /mock-netsuite/record/:recordType`
- Accepts: `{ ids: ["123", "456", ...] }` (up to 1,000 IDs, simulating SOAP `getList`)
- Returns: Full records with all fields including marker fields

**Affected Files:** `tech-spec.md` (section B), `design-spec.md` (section C), `jira-ticket.md` (Search Endpoint).

---

## Gap 5: No Deleted Record Detection Endpoint (Medium)

**V10 Reference:** Appendix M6 (lines 1996-2018): Deleted records don't update `lastmodifieddate`, so they are invisible to the standard polling cursor. V10 specifies using the `getDeleted` SOAP API to detect deletions. Appendix Q2 End-to-End Tests (line 2660): "Deleted record detection (`getDeleted` + reconciliation)."

**Current State:** The mock server has no endpoint to simulate `getDeleted`. Records can be created and updated but never deleted, and there is no way to test SuiteX's delete detection pipeline.

**Required Change:** Add a delete endpoint and a `getDeleted` query endpoint:

**Delete Endpoint:** `DELETE /mock-netsuite/record/:recordType/:internalId`
- Soft-deletes the record (marks as deleted with a `deletedAt` timestamp)

**getDeleted Endpoint:** `POST /mock-netsuite/getDeleted/:recordType`
- Accepts: `{ deletedAfter: "ISO8601" }`
- Returns: `[{ internalId, deletedDate }]` for records deleted after the given timestamp

Add `deletedAt` to the `MockRecord` entity:

```typescript
@Column({ type: 'datetime', nullable: true })
deletedAt: Date | null;
```

**Affected Files:** `tech-spec.md` (new endpoint section), `design-spec.md` (section C addendum).

---

## Gap 6: Governance Simulation Missing Tier Configuration and Error Variety (Medium)

**V10 Reference:** Appendix L1 (lines 1725-1730) defines three NetSuite tiers with different concurrency limits:

| Tier | Concurrent Requests | Requests/Hour |
|------|---------------------|---------------|
| Standard | 1 (can burst to 5) | ~1,000 |
| Premium | 5 (can burst to 10) | ~5,000 |
| Premium Plus | 10 (can burst to 25) | ~10,000 |

Appendix O3-O4 (lines 2331-2363) defines additional error types SuiteX must handle:
- `401 Unauthorized` (auth failure -- 3 retries)
- `403 Forbidden` (permanent -- 0 retries)
- `SSS_USAGE_LIMIT_EXCEEDED` (governance budget -- wait until next day)

V10 line 1734: "Retry-After header may be present (often not)."

**Current State:** The Chaos Interceptor has a single hardcoded concurrency threshold (e.g., 5) and only generates 429 and 503 responses. It cannot simulate:
- Different NetSuite tiers (Standard vs Premium vs Premium Plus)
- Burst concurrency (NetSuite allows brief bursts above the sustained limit)
- Auth failures (401/403)
- Daily governance exhaustion (distinct from per-request rate limiting)
- `Retry-After` header on 429 responses

**Required Change:** Make the Chaos Interceptor configurable via environment variables:

```env
MOCK_NS_TIER=premium               # standard | premium | premium_plus
MOCK_NS_SUSTAINED_LIMIT=5          # Per-tier sustained concurrency
MOCK_NS_BURST_LIMIT=10             # Per-tier burst allowance
MOCK_NS_DAILY_BUDGET=5000          # Requests/day; 0 = unlimited
MOCK_NS_AUTH_FAILURE_RATE=0         # Percentage of requests returning 401
MOCK_NS_RETRY_AFTER_ENABLED=false  # Whether to include Retry-After header on 429
MOCK_NS_TRANSIENT_FAILURE_RATE=5   # Percentage chance of 503
```

Add support for:
1. **Burst mode**: Allow brief spikes above `SUSTAINED_LIMIT` up to `BURST_LIMIT`, then enforce 429
2. **Daily budget**: Track cumulative requests; return `SSS_USAGE_LIMIT_EXCEEDED`-style response when budget exhausted
3. **Auth errors**: Configurable rate of 401 responses to test SuiteX's auth error retry queue
4. **Retry-After header**: Optionally include on 429 responses

**Affected Files:** `tech-spec.md` (Chaos Engineering section), `design-spec.md` (section D).

---

## Gap 7: `internalId` Format Does Not Match NetSuite (Low)

**V10 Reference:** Throughout V10, NetSuite `recordId` and `internalid` values are numeric strings: `"12345"` (lines 193, 682, 1902, 2225). NetSuite internal IDs are always integers.

**Current State:** The tech-spec shows `internalId: 'NS-9932'` -- a prefixed alphanumeric string. This does not match NetSuite's actual ID format.

**Impact:** SuiteX code that parses or validates NetSuite internal IDs (e.g., integer casting, numeric comparison for cursor pagination) would pass against the mock but fail against real NetSuite. The cursor-based pagination logic (Appendix N3: "Multiple changes within same second processed in ID order") depends on numeric comparison of internal IDs.

**Required Change:** Auto-generate numeric `internalId` values (e.g., auto-incrementing integers cast to strings: `"1001"`, `"1002"`, etc.). This matches NetSuite's behavior and ensures cursor pagination with `internalId > lastInternalId` works correctly with numeric comparison.

**Affected Files:** `tech-spec.md` (Entity Model, behavioral note).

---

## Gap 8: No UE Event Emission Simulation (Low — Future Enhancement)

**V10 Reference:** V10 Appendix Q2 Integration Tests (lines 2650-2651):
- "UE event emission (create, update, delete)"
- "SuiteX → NetSuite write (via RESTlet)"

V10 Appendix Q2 End-to-End Tests (lines 2664-2665):
- "SuiteX write → NetSuite → UE suppresses event"

In V10's architecture, when any record is created/updated in NetSuite, the UE script fires. For SuiteX-originated writes (marker fields present), the UE suppresses event emission. For other writes, the UE emits an event to `events.raw`.

**Current State:** The mock server is entirely passive -- it accepts writes and returns data, but never publishes anything to Pub/Sub. It cannot simulate the UE behavior that would follow a write.

**Impact:** End-to-end testing of the full circular update prevention loop (SuiteX → mock NetSuite → UE event → SuiteX suppresses) is not possible with the mock alone. This isn't blocking for Epic 3's scope but limits the mock's utility for later integration testing.

**Required Change (Optional for MVP):** Add an optional UE simulation mode where the mock server publishes events to `events.raw` after writes, applying the same suppression logic as V10's UE scripts:

- If `custbody_suitex_write_source === 'suitex'`: Do NOT publish event (UE suppressed)
- Otherwise: Publish a change event to `events.raw` with `source: 'netsuite-ue'` and appropriate attribution metadata

This can be gated behind a feature flag (`MOCK_UE_EMISSION_ENABLED=false` by default) and implemented in a later iteration.

**Affected Files:** `tech-spec.md` (new section), `design-spec.md` (optional deliverable).

---

## Summary of Required Changes

| Gap | Severity | Files Affected | Change Type |
|-----|----------|----------------|-------------|
| 1. Wrong acceptance criteria in design-spec | **Critical** | design-spec | Replace with mock server scenarios |
| 2. Write endpoint format mismatch | **High** | all three files | Define RESTlet request format |
| 3. No marker field support | **High** | tech-spec, design-spec | Add fields to entity and endpoints |
| 4. Single-phase instead of two-phase polling | **Medium** | all three files | Add discovery + detail endpoints |
| 5. No deleted record detection | **Medium** | tech-spec, design-spec | Add delete and getDeleted endpoints |
| 6. Governance simulation too simplistic | **Medium** | tech-spec, design-spec | Add tier config and error variety |
| 7. Non-numeric internalId format | **Low** | tech-spec | Use auto-increment integers |
| 8. No UE event emission simulation | **Low** | tech-spec, design-spec | Optional future enhancement |

---

## Corrected MockRecord Entity

Incorporating gaps 3, 5, and 7:

```typescript
@Entity()
export class MockRecord {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  recordType: string;

  @Column({ unique: true })
  internalId: number; // Auto-increment integer, matching NetSuite's numeric IDs

  @Column({ type: 'datetime' })
  lastModifiedDate: Date;

  @Column({ type: 'json' })
  payload: Record<string, unknown>;

  @Column()
  writeId: string;

  @Column({ nullable: true })
  suitexWriteId: string | null; // custbody_suitex_write_id marker

  @Column({ nullable: true })
  suitexWriteSource: string | null; // custbody_suitex_write_source marker

  @Column({ type: 'datetime', nullable: true })
  deletedAt: Date | null; // Soft delete for getDeleted simulation

  @CreateDateColumn()
  createdAt: Date;
}
```

## Corrected Endpoint Map

| Endpoint | Method | Purpose | V10 Equivalent |
|---|---|---|---|
| `/mock-netsuite/restlet/:recordType` | POST | Write record (create/update) | RESTlet ingestion |
| `/mock-netsuite/search/:recordType` | POST | Discovery: return IDs + timestamps | Polling discovery (M2) |
| `/mock-netsuite/record/:recordType` | POST | Detail fetch: return full records by IDs | SOAP getList / record.load (M3) |
| `/mock-netsuite/record/:recordType/:id` | DELETE | Soft-delete a record | Record deletion |
| `/mock-netsuite/getDeleted/:recordType` | POST | Return deleted record IDs since timestamp | SOAP getDeleted (M6) |