LOE: 33 pts (XL) | Rationale: Eight structural gaps spanning acceptance criteria correctness, endpoint contract format, entity model completeness, two-phase polling architecture, delete detection, governance configurability, internalId type correctness, and future UE emission documentation. Three gaps require new endpoint definitions with TypeORM entity changes. The write endpoint format mismatch is the highest contract risk — the NetSuite Writer consumer will be built against this spec. Marker fields and two-phase polling are foundational dependencies for circular update prevention and delete detection tests. Governance simulation extensibility is required for full adaptive throttling test coverage.

# Jira Task: Refactor

**Title:** `[DATA-SYNC][Epic 3]` Resolve Mock NetSuite Server Gap Analysis — Entity, Endpoints, Governance, and Contract Alignment
**Priority:** High
**Story Points:** 33
**Assignee:** TBD

## Context

A gap analysis comparing the Epic 3 mock server specification against the [Design Doc](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook) revealed eight structural issues. If left unresolved, the mock server will diverge from the actual NetSuite API contract, making integration tests invalid and requiring rework of both the mock server and the NetSuite Writer consumer before real NetSuite integration can begin.

The most critical gap — acceptance criteria copied verbatim from Epic 1's schema validation suite — would allow the Epic to be marked "done" while none of the mock server's behavioral guarantees have been tested. The write endpoint format mismatch compounds this risk: the NetSuite Writer consumer will be coded against the wrong interface if the spec is not corrected first.

### Current System Behavior

- **Current Pattern:** The mock server write endpoint is specified to accept the canonical Pub/Sub envelope format (with `eventId`, `accountId`, `schemaVersion`) rather than a RESTlet request payload. The `MockRecord` entity lacks marker fields (`suitexWriteId`, `suitexWriteSource`, `deletedAt`) required for circular update prevention and delete detection testing. Polling is single-phase (full records returned in one call). No deleted record detection endpoints exist. The governance interceptor has a hardcoded concurrency threshold and produces only 429 and 503 responses. `internalId` uses a prefixed alphanumeric string format (`'NS-9932'`) rather than a numeric integer, which will break cursor pagination comparisons. No UE event emission capability is documented.

## Description

Correct the mock server specification across all applicable areas to align with the [Design Doc](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook)'s NetSuite API simulation requirements. This covers eight distinct gaps: acceptance criteria replacement, write endpoint contract correction, entity model additions, two-phase polling endpoint split, soft-delete and getDeleted endpoint addition, governance simulation configurability, internalId type correction, and UE emission future enhancement documentation.

All changes target the mock server implementation and its specification. The goal is a mock server that correctly simulates NetSuite's RESTlet write interface, two-phase polling architecture, delete detection API, and governance tier behavior — enabling SuiteX's polling pipeline, NetSuite Writer consumer, and adaptive throttling logic to be tested end-to-end before real NetSuite credentials are available.

### Areas to Review

- [Design Doc — SuiteX sync pipeline to NetSuite](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#suitesync-sync-pipeline-to-netsuite) (RESTlet write path)
- [Design Doc — M1. Polling Strategy](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m1-polling-strategy) (polling discovery phase)
- [Design Doc — M3. Detail Fetch Pattern](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m3-detail-fetch-pattern) (SOAP getList simulation)
- [Design Doc — M6. Deleted Record Detection](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m6-deleted-record-detection)
- [Design Doc — M7. Polling Deduplication](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m7-polling-deduplication) (circular update prevention / polling suppression)
- [Design Doc — L1. API Request Limits](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#l1-api-request-limits) (NetSuite tier concurrency limits)
- [Design Doc — O3. Transient vs Permanent Errors](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#o3-transient-vs-permanent-errors) and [O4. Governance-Induced Failures](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#o4-governance-induced-failures) (error types: 401, 403, SSS_USAGE_LIMIT_EXCEEDED)
- [Design Doc — M2. Discovery Query Pattern](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m2-discovery-query-pattern) (cursor filter logic and compound OR query)
- [Design Doc — N3. Polling Ordering Guarantees](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#n3-polling-ordering-guarantees)
- [Design Doc — Q2. Testing Checklist](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#q2-testing-checklist) (end-to-end test scenarios)

## Deliverables

1. Acceptance criteria replaced with four mock server behavioral scenarios (write → persist → numeric internalId; concurrent requests beyond limit → 429; polling with cursor → sorted paginated result ≤ 1,000 records; duplicate writeId → idempotent response).
2. Write endpoint redefined to accept RESTlet request format (`recordType`, `operation`, `fields`) instead of the canonical Pub/Sub envelope. Field values inside `fields` must still conform to Epic 1 normalization rules (booleans, ISO dates, sorted multiselects). `custbody_suitex_write_id` and `custbody_suitex_write_source` documented as required fields.
3. `MockRecord` entity updated with `suitexWriteId: string | null`, `suitexWriteSource: string | null`, and `deletedAt: Date | null` columns. Write endpoint spec updated to extract and persist these values from the incoming RESTlet payload.
4. Single-phase polling endpoint replaced with two endpoints: a discovery endpoint (`POST /mock-netsuite/search/:recordType`) returning only `[{ internalId, lastModifiedDate }]` sorted and capped at 1,000 records, and a detail fetch endpoint (`POST /mock-netsuite/record/:recordType`) accepting `{ ids: string[] }` and returning full records including marker fields.
5. Two new endpoints added: `DELETE /mock-netsuite/record/:recordType/:internalId` (soft-delete, sets `deletedAt`) and `POST /mock-netsuite/getDeleted/:recordType` (returns `[{ internalId, deletedDate }]` for records deleted after a given timestamp). Discovery endpoint updated to exclude soft-deleted records.
6. Governance interceptor extended with eight configurable environment variables covering tier, sustained/burst concurrency limits, daily budget, auth failure rate (401), forbidden rate (403), Retry-After header, and transient failure rate. Both 401 and 403 are permanent errors per [Design Doc O3](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#o3-transient-vs-permanent-errors) — the mock simulates them to test SuiteX's error classification and dead-letter routing, not retry logic. `SSS_USAGE_LIMIT_EXCEEDED` response format documented. Preset configurations for Standard, Premium, and Premium Plus tiers documented in a reference table.
7. `internalId` column type changed to auto-incrementing integer (`number`). All example data updated to numeric string format. Cursor pagination documentation updated to specify numeric (not lexicographic) comparison.
8. "Future Enhancement: UE Event Emission Simulation" section added, clearly marked out of scope for MVP, documenting the `MOCK_UE_EMISSION_ENABLED=false` feature flag and suppression logic.

## Acceptance Criteria

- [ ] `POST /mock-netsuite/restlet/:recordType` with a valid RESTlet payload persists the record and returns a numeric `internalId`.
- [ ] Concurrent requests beyond the configured sustained concurrency limit return `429 Too Many Requests`.
- [ ] `POST /mock-netsuite/search/:recordType` with a cursor returns a sorted, paginated result set of at most 1,000 records containing only `internalId` and `lastModifiedDate`. The cursor uses a compound OR filter per [Design Doc M2](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m2-discovery-query-pattern): `(lastModifiedDate > afterDate) OR (lastModifiedDate == afterDate AND internalId > afterInternalId)`.
- [ ] A duplicate `writeId` on a write request returns the existing record's `internalId` without creating a new record (idempotency).
- [ ] Write endpoint accepts a RESTlet request body with `recordType`, `operation`, and `fields` as top-level keys; no `eventId`, `accountId`, or `schemaVersion` fields are present in the write endpoint definition.
- [ ] `fields` object includes `custbody_suitex_write_id` and `custbody_suitex_write_source` as documented fields; field values conform to Epic 1 normalization rules.
- [ ] `MockRecord` entity includes `suitexWriteId: string | null`, `suitexWriteSource: string | null`, and `deletedAt: Date | null`.
- [ ] Discovery endpoint returns only `internalId` and `lastModifiedDate` per record — no `payload`, no marker fields.
- [ ] Discovery endpoint results are sorted `lastModifiedDate ASC, internalId ASC` and exclude records where `deletedAt IS NOT NULL`.
- [ ] Detail fetch endpoint accepts `{ ids: string[] }` (up to 1,000 IDs) and returns full records including `suitexWriteId`, `suitexWriteSource`, and `payload`.
- [ ] Detail fetch endpoint returns soft-deleted records when queried by ID (allows reconciliation logic to read final state before deletion confirmation).
- [ ] `DELETE /mock-netsuite/record/:recordType/:internalId` sets `deletedAt` to the current timestamp and returns `204 No Content`; returns `404` if the record does not exist.
- [ ] `POST /mock-netsuite/getDeleted/:recordType` accepts `{ deletedAfter: "ISO8601" }` and returns `[{ internalId, deletedDate }]` for all records deleted after the given timestamp.
- [ ] Eight governance environment variables are defined and documented: `MOCK_NS_TIER`, `MOCK_NS_SUSTAINED_LIMIT`, `MOCK_NS_BURST_LIMIT`, `MOCK_NS_DAILY_BUDGET`, `MOCK_NS_AUTH_FAILURE_RATE`, `MOCK_NS_FORBIDDEN_RATE`, `MOCK_NS_RETRY_AFTER_ENABLED`, `MOCK_NS_TRANSIENT_FAILURE_RATE`. Both 401 and 403 are permanent errors per [Design Doc O3](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#o3-transient-vs-permanent-errors) (do not retry); the mock simulates them to test SuiteX's error classification and dead-letter routing.
- [ ] Burst mode behavior specified: requests above `SUSTAINED_LIMIT` but below `BURST_LIMIT` succeed; requests above `BURST_LIMIT` return 429.
- [ ] When `MOCK_NS_DAILY_BUDGET` is exhausted, responses return an `SSS_USAGE_LIMIT_EXCEEDED`-style body; `DAILY_BUDGET=0` disables the limit.
- [ ] When `MOCK_NS_RETRY_AFTER_ENABLED=true`, 429 responses include a `Retry-After: <seconds>` header.
- [ ] `MockRecord` entity shows `internalId: number` with auto-increment behavior; no prefixed alphanumeric examples (e.g., `'NS-9932'`) remain.
- [ ] A "Future Enhancement" section (clearly marked out of scope for MVP) documents `MOCK_UE_EMISSION_ENABLED=false` and UE suppression logic.

## Validation & Testing

1. Start the mock server with default configuration. Send `POST /mock-netsuite/restlet/project` with a valid RESTlet payload (including `custbody_suitex_write_id` and `custbody_suitex_write_source`). Verify the response contains a numeric `internalId` (e.g., `"1001"`, not `"NS-1001"`).
2. Send the same request a second time with the identical `custbody_suitex_write_id`. Verify the response returns the same `internalId` as the first request and no duplicate record is created in the database.
3. Send `POST /mock-netsuite/search/project` with `{ "afterDate": "2000-01-01T00:00:00Z" }`. Verify the response contains only `internalId` and `lastModifiedDate` per record — no `payload`, no `suitexWriteId`.
4. Send `POST /mock-netsuite/record/project` with `{ "ids": ["1001"] }`. Verify the response includes `internalId`, `lastModifiedDate`, `suitexWriteId`, `suitexWriteSource`, and `payload`.
5. Send `DELETE /mock-netsuite/record/project/1001`. Verify `204 No Content`. Send `POST /mock-netsuite/search/project` and confirm `internalId: "1001"` is absent from discovery results. Send `POST /mock-netsuite/record/project` with `{ "ids": ["1001"] }` and verify the full record (including `deletedAt` timestamp) is still returned by the detail fetch endpoint. Send `POST /mock-netsuite/getDeleted/project` with `{ "deletedAfter": "2000-01-01T00:00:00Z" }` and confirm `"1001"` appears in the response.
6. Set `MOCK_NS_SUSTAINED_LIMIT=1` and `MOCK_NS_BURST_LIMIT=3`. Send 4 concurrent requests to any endpoint. Verify requests 1–3 succeed and request 4 returns `429`.
7. Set `MOCK_NS_DAILY_BUDGET=5`. Send 6 requests sequentially. Verify the 6th response contains an `SSS_USAGE_LIMIT_EXCEEDED`-style error body.
8. Set `MOCK_NS_RETRY_AFTER_ENABLED=true` and trigger a 429. Verify the response includes a `Retry-After` header with a numeric seconds value.
9. Confirm that a write request containing a canonical Pub/Sub envelope (with top-level `eventId` or `accountId`) is rejected or returns a validation error — it must not be silently accepted.

## Technical Notes

### Implementation Guidance

**RESTlet Request Schema (write endpoint payload):**

```json
{
  "recordType": "project",
  "operation": "create | update | delete",
  "fields": {
    "companyname": "Acme Corp",
    "custbody_suitex_write_id": "550e8400-e29b-41d4-a716-446655440000",
    "custbody_suitex_write_source": "suitex"
  }
}
```

Field values inside `fields` must conform to Epic 1 normalization rules: booleans as `true`/`false` (not `"T"`/`"F"`), dates as ISO 8601 strings, multiselect values sorted lexicographically. The outer wrapper simulates the RESTlet interface — not the Pub/Sub envelope.

---

**Final `MockRecord` Entity:**

```typescript
@Entity()
export class MockRecord {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  recordType: string;

  @Column({ unique: true })
  internalId: number;

  @Column({ type: 'datetime' })
  lastModifiedDate: Date;

  @Column({ type: 'json' })
  payload: Record<string, unknown>;

  @Column()
  writeId: string;

  @Column({ nullable: true })
  suitexWriteId: string | null;

  @Column({ nullable: true })
  suitexWriteSource: string | null;

  @Column({ type: 'datetime', nullable: true })
  deletedAt: Date | null;

  @CreateDateColumn()
  createdAt: Date;
}
```

`internalId` is stored as an integer and serialized as a string in API responses (e.g., `"1001"`) to match NetSuite's actual response format. Cursor pagination comparisons (`internalId > lastInternalId`) must use numeric comparison, not lexicographic.

---

**Final Endpoint Map:**

| Endpoint | Method | Purpose | Design Doc Reference |
|---|---|---|---|
| `/mock-netsuite/restlet/:recordType` | POST | Write record (create/update/delete) | [SuiteX sync pipeline to NetSuite](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#suitesync-sync-pipeline-to-netsuite) |
| `/mock-netsuite/search/:recordType` | POST | Discovery: IDs + timestamps only | [M1. Polling Strategy](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m1-polling-strategy) |
| `/mock-netsuite/record/:recordType` | POST | Detail fetch by ID list | [M3. Detail Fetch Pattern](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m3-detail-fetch-pattern) |
| `/mock-netsuite/record/:recordType/:id` | DELETE | Soft-delete | [M6. Deleted Record Detection](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m6-deleted-record-detection) |
| `/mock-netsuite/getDeleted/:recordType` | POST | Return deleted IDs since timestamp | [M6. Deleted Record Detection](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m6-deleted-record-detection) |

---

**Discovery Endpoint Request/Response:**

```json
// POST /mock-netsuite/search/:recordType
// Request
{
  "afterDate": "2026-01-01T00:00:00Z",
  "afterInternalId": "1050"
}

// Response
[
  { "internalId": "1051", "lastModifiedDate": "2026-01-02T10:00:00Z" },
  { "internalId": "1052", "lastModifiedDate": "2026-01-02T10:00:01Z" }
]
```

**Cursor compound OR filter** ([Design Doc M2](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#m2-discovery-query-pattern)): The mock must apply the cursor parameters as:
`(lastModifiedDate > afterDate) OR (lastModifiedDate == afterDate AND internalId > afterInternalId)`.
This ensures records modified in the same second as the watermark timestamp are not skipped — they are returned if their `internalId` is greater than the cursor's `afterInternalId`.

---

**Detail Fetch Endpoint Request/Response:**

```json
// POST /mock-netsuite/record/:recordType
// Request
{ "ids": ["1051", "1052"] }

// Response
[
  {
    "internalId": "1051",
    "lastModifiedDate": "2026-01-02T10:00:00Z",
    "suitexWriteId": "550e8400-e29b-41d4-a716-446655440000",
    "suitexWriteSource": "suitex",
    "payload": { "companyname": "Acme Corp" }
  }
]
```

---

**getDeleted Endpoint Request/Response:**

```json
// POST /mock-netsuite/getDeleted/:recordType
// Request
{ "deletedAfter": "2026-01-01T00:00:00Z" }

// Response
[
  { "internalId": "1045", "deletedDate": "2026-01-03T15:22:00Z" },
  { "internalId": "1047", "deletedDate": "2026-01-04T09:11:00Z" }
]
```

---

**Governance Environment Variable Schema:**

```env
MOCK_NS_TIER=premium               # standard | premium | premium_plus
MOCK_NS_SUSTAINED_LIMIT=5          # Sustained concurrency limit
MOCK_NS_BURST_LIMIT=10             # Brief burst allowance above sustained limit
MOCK_NS_DAILY_BUDGET=5000          # Cumulative requests/day; 0 = unlimited
MOCK_NS_AUTH_FAILURE_RATE=0        # % of requests returning 401 (permanent error per Design Doc §O3; integer 0–100)
MOCK_NS_FORBIDDEN_RATE=0          # % of requests returning 403 (permanent error per Design Doc §O3; integer 0–100)
MOCK_NS_RETRY_AFTER_ENABLED=false  # Include Retry-After header on 429
MOCK_NS_TRANSIENT_FAILURE_RATE=5   # % chance of 503 per request
```

See [Design Doc O3](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#o3-transient-vs-permanent-errors) for permanent error classification.

**Tier Preset Reference:**

| Tier | `SUSTAINED_LIMIT` | `BURST_LIMIT` | `DAILY_BUDGET` |
|------|-------------------|---------------|----------------|
| standard | 1 | 5 | 1,000 |
| premium | 5 | 10 | 5,000 |
| premium_plus | 10 | 25 | 10,000 |

**`SSS_USAGE_LIMIT_EXCEEDED` Response (simulated):**

```json
{
  "type": "error.com.netsuite.platform.core.governanceExceededException",
  "title": "SSS_USAGE_LIMIT_EXCEEDED",
  "status": 429,
  "o:errorDetails": [
    { "detail": "Script exceeded governance limits.", "type": "error" }
  ]
}
```

---

**Future Enhancement: UE Event Emission Simulation** *(Out of scope for MVP)*

Controlled by `MOCK_UE_EMISSION_ENABLED=false`. When enabled, the mock server publishes a change event to `events.raw` with `source: 'netsuite-ue'` after every write — except when `custbody_suitex_write_source === 'suitex'`, in which case the event is suppressed to simulate NetSuite's User Event script behavior. This capability is required for full end-to-end testing of the circular update prevention loop described in [Design Doc Q2](https://github.com/SuiteDynamics/SuiteX/wiki/Design-%7C-NetSuite-Sync-%E2%80%94-Design,-Requirements,-Deliverables,-and-Runbook#q2-testing-checklist). It must not be implemented in MVP; the flag must be present and default to `false` so that suppression logic can be added without a breaking change.