# Canonical Envelope Schema — Gap Resolution Spec

**Task:** fix/canonical-envelope-schema-gaps  
**Date:** 2026-03-17  
**Status:** Authoritative — supersedes any conflicting description in prior Epic 1 documentation

This document is the prose companion to the corrected `schemas/envelope/v1.schema.json`. It provides extended rationale, worked examples, and implementation guidance for the four gaps that require explanatory prose beyond what can be expressed inline in a JSON Schema file.

---

## Gap 4 — source vs sourceSystem: Distinguishing the Two Attribution Fields

### Problem

The original envelope schema defined `source` and `sourceSystem` without documentation explaining how they differ. Downstream consumers risk conflating them, producing incorrect circular-update suppression logic or wrong conflict resolution priority decisions.

### Definition

| Field | Semantic | Question Answered | Type |
|-------|----------|-------------------|------|
| `source` | **Physical emission pathway** — which system component pushed this event onto the backbone | *How did this event enter the pipeline?* | Enum: `reconciliation`, `netsuite-ue`, `netsuite-poll`, `suitex` |
| `sourceSystem` | **Logical business actor** — which domain authority made the change | *Who owns the authoritative state for this write?* | Enum: `suitex`, `netsuite`, `workflow`, `user` |

### Conflict Resolution Priority (source enum ordering)

The `source` enum is ordered from **highest to lowest conflict resolution priority**:

1. `reconciliation` — nightly diff / backfill jobs; highest authority as they represent ground-truth reconciliation against NetSuite
2. `netsuite-ue` — User Event scripts firing synchronously; second-highest as they are triggered at the moment of NetSuite write
3. `netsuite-poll` — poller-detected changes; lower because they may lag real state
4. `suitex` — SuiteX-originated writes; lowest because SuiteX is the consumer, not the NetSuite source of truth

When the merge orchestrator receives two events for the same `(accountId, recordType, recordId)` with overlapping field sets and the same `baseVersion`, the event with the higher-priority `source` value wins that field's value.

### Worked Examples

**Example 1: SuiteX user updates a project field**

```json
{
  "source": "suitex",
  "sourceSystem": "user"
}
```

A SuiteX user saved a change via the SuiteX web UI. The SuiteX application emitted the event directly onto the backbone (`source = suitex`). The change was initiated by a human user (`sourceSystem = user`).

**Example 2: NetSuite UE detects a SuiteX-originated write echoing back**

```json
{
  "source": "netsuite-ue",
  "sourceSystem": "suitex"
}
```

The SuiteX sync worker previously wrote a record to NetSuite. NetSuite's User Event script fired and emitted an event. The physical emitter is the UE script (`source = netsuite-ue`), but the logical author of the original change was SuiteX (`sourceSystem = suitex`). The merge orchestrator uses `sourceSystem = suitex` + `writeId` to detect circular echo and suppress this event.

**Example 3: Reconciliation job corrects a drift**

```json
{
  "source": "reconciliation",
  "sourceSystem": "netsuite"
}
```

A nightly reconciliation job detected drift between NetSuite's current state and SuiteX's projection. It emits a reconciliation event that reflects NetSuite's authoritative value. The emitter is the reconciliation process (`source = reconciliation`). The authoritative business state is NetSuite's (`sourceSystem = netsuite`).

**Example 4: Workflow automation changes a field**

```json
{
  "source": "netsuite-ue",
  "sourceSystem": "workflow"
}
```

A NetSuite workflow script updated a record. The UE script detected and emitted the event (`source = netsuite-ue`). The logical actor is a NetSuite workflow automation (`sourceSystem = workflow`), not a human user.

---

## Gap 5 — patternProperties: NetSuite Custom Field Prefix Reference Table

### Problem

The original design documented `patternProperties` as permitting only `custentity_` fields. Four additional prefix families (`custbody_`, `custcol_`, `custitem_`, `custevent_`) are valid in NetSuite and must be accepted by the canonical envelope. Consumers building emitters or validators without this table may strip or reject valid custom fields.

### The Five NetSuite Custom Field Prefix Families

| Prefix | NetSuite Custom Field Type | Applies To | Stage 2 Validation Key |
|--------|---------------------------|------------|------------------------|
| `custentity_` | Entity Custom Field | Customers, Vendors, Employees, Contacts | `field_metadata.field_id LIKE 'custentity_%'` |
| `custbody_` | Transaction Body Custom Field | Transactions (Invoices, Purchase Orders, Projects) | `field_metadata.field_id LIKE 'custbody_%'` |
| `custcol_` | Transaction Column / Line-Item Custom Field | Transaction lines (sublists) | `field_metadata.field_id LIKE 'custcol_%'` |
| `custitem_` | Item Custom Field | Inventory Items, Non-Inventory Items | `field_metadata.field_id LIKE 'custitem_%'` |
| `custevent_` | CRM Event Custom Field | Phone Calls, Tasks, Events | `field_metadata.field_id LIKE 'custevent_%'` |

### Record-Type-to-Prefix Mapping (Guidance, not enforcement)

The Stage 1 JSON Schema permits all five prefix families on every record type. Stage 2 enforcement via `field_metadata` constrains which specific field IDs are valid per `(accountId, recordType)`. The table below is reference guidance for emitter developers:

| Record Type | Primary Prefix Families | Notes |
|-------------|------------------------|-------|
| `project` | `custentity_`, `custbody_` | Projects are entity records in NetSuite |
| `projecttask` | `custbody_`, `custcol_` | Tasks have body and line-item custom fields |
| `customer` | `custentity_` | Customers are entity records |
| `vendor` | `custentity_` | Vendors are entity records |
| `employee` | `custentity_` | Employees are entity records |
| `inventoryitem` | `custitem_` | Items use the item prefix family |
| `phonecall` / `task` / `event` | `custevent_` | CRM activity records |

### JSON Schema Pattern Used

```
^cust(body|col|entity|item|record|event)[a-zA-Z0-9_]*$
```

Note: `custrecord_` is included in the regex as NetSuite also supports custom record type fields. It is not one of the canonical five families for standard emitters but is accepted to avoid false rejections.

---

## Gap 8 — field_metadata DDL: Constraints, Partial Index, and Placement Note

### Problem

The original `field_metadata` migration was created in the tenant-scoped migration directory and is missing:
1. Explicit NOT NULL constraints on `is_synced` and `is_readonly`
2. A corrected default for `is_synced` (should default to `false` — opt-in sync, not opt-out)
3. A partial index to accelerate the most frequent production query
4. A placement annotation clarifying whether this table belongs to the root/landlord DB or the tenant DB

### Corrected DDL Specification

```sql
CREATE TABLE field_metadata (
    account_id      VARCHAR(255)    NOT NULL,
    record_type     VARCHAR(100)    NOT NULL,
    field_id        VARCHAR(255)    NOT NULL,
    field_type      VARCHAR(50)     NOT NULL,
    is_synced       BOOLEAN         NOT NULL DEFAULT FALSE,
    is_readonly     BOOLEAN         NOT NULL DEFAULT FALSE,
    normalization_rule JSON         NULL,
    conflict_policy VARCHAR(50)     NOT NULL DEFAULT 'manual',
    created_at      TIMESTAMP       NULL,
    updated_at      TIMESTAMP       NULL,

    PRIMARY KEY (account_id, record_type, field_id)
);

-- Partial index: accelerates the Stage 2 semantic validator's primary query
-- which filters for active sync fields only: WHERE is_synced = TRUE
-- This index covers the query pattern:
--   SELECT * FROM field_metadata
--   WHERE account_id = ? AND record_type = ? AND is_synced = TRUE
CREATE INDEX idx_field_metadata_synced
    ON field_metadata (account_id, record_type)
    WHERE is_synced = TRUE;
```

### Key Constraint Notes

| Column | Current Migration | Corrected Spec | Rationale |
|--------|------------------|----------------|-----------|
| `is_synced` | `default(true)` | `NOT NULL DEFAULT FALSE` | Opt-in sync is safer: new fields are not synced until explicitly configured. Opt-out (`DEFAULT TRUE`) risks silently syncing fields that have no `normalization_rule` defined. |
| `is_readonly` | `default(false)` | `NOT NULL DEFAULT FALSE` | Already correct semantics; explicit NOT NULL added for documentation clarity. |
| `conflict_policy` | `default('last-write-wins')` | `DEFAULT 'manual'` | `last-write-wins` is a dangerous default that silently overwrites data. New fields should require explicit policy assignment. `manual` signals that no automated resolution is configured, forcing operators to review field conflicts. |

### Placement Note: Root/Landlord Migration, NOT Tenant Migration

**Current state:** The `field_metadata` migration resides in `database/migrations/tenants/` and uses `Schema::connection('tenant_connection')`.

**Design authority position:** `field_metadata` is a **landlord/root-level table** that stores per-account field configuration. It should be:
- Stored in the **root (landlord) database**, not per-tenant databases
- Accessible cross-tenant by the sync infrastructure for account routing decisions
- Managed by root-level migrations (NOT `database/migrations/tenants/`)

**Impact:** Moving this table from tenant to root scope is a **breaking schema change** requiring:
1. A new root-level migration creating `field_metadata` with the corrected DDL
2. A tenant-level migration dropping the tenant-scoped table after data migration
3. Updates to `FieldMetadata` model to remove `UsesTenantConnection` and use the default connection
4. Updates to `FieldMetadataCacheObserver` cache key to include tenant discriminator (`account_id` is already in the key, so cache invalidation logic is unaffected)

**This migration correction is OUT OF SCOPE for this spec task** and must be tracked as a separate Epic 1 remediation ticket. It is documented here as a known placement defect for downstream implementors.

### Partial Index Implementation in Laravel

MySQL/MariaDB (used in production) does **not** support partial indexes natively. The WHERE clause in the DDL above is the specification target for PostgreSQL or Cloud SQL PostgreSQL variants. For MySQL, the equivalent performance optimization is to create a non-partial composite index:

```php
$table->index(['account_id', 'record_type', 'is_synced'], 'idx_field_metadata_synced');
```

This is less selective than a true partial index but achieves similar query performance for the `is_synced = TRUE` hot path.

---

## Gap 9 — Stage 2 Outbound Validation: is_readonly Field Stripping

### Problem

The current `ValidateSemanticCustomFieldsAction` validates all custom fields in `changes` against `field_metadata`. However, it has no rule for fields where `is_readonly = true`. The V10 design specifies that readonly fields must be stripped from outbound `changes` payloads rather than dead-lettered, because readonly fields are NetSuite-managed computed fields (e.g., system totals) that SuiteX must never attempt to write back.

Without this rule, a `changes` payload containing a readonly field would either:
- Pass through silently and cause a NetSuite write rejection (hard failure at the RESTlet layer)
- Be routed to the Dead Letter Queue incorrectly (treating a configuration issue as a data error)

### Corrected Validation Rule

**Location:** Stage 2 (`ValidateSemanticCustomFieldsAction`) — outbound path only (SuiteX → NetSuite direction).

**Rule:** Before performing any type or policy validation, scan `changes` for any fields where `field_metadata.is_readonly = true`. Strip those fields from `changes` and emit a LOG WARN entry. Do NOT dead-letter the event.

**Edge Case:** If stripping all `is_readonly = true` fields results in an empty `changes` object (`{}`), the event is still **valid**. An empty `changes` object after stripping is a no-op event — it must NOT be dead-lettered. The correct handling is:
1. Log at INFO level: "Event {eventId} has empty changes after readonly field strip — treating as no-op"
2. Return the validated payload object with empty `changes`
3. Allow downstream consumers to decide whether to discard or record the no-op

### Pseudocode Specification

```
function applyReadonlyFieldStrip(changes, fieldMetadataMap):
    strippedFields = []

    for each (fieldId, value) in changes:
        meta = fieldMetadataMap[fieldId]
        if meta is not null and meta.is_readonly == true:
            remove fieldId from changes
            strippedFields.append(fieldId)

    if strippedFields is not empty:
        LOG WARN "Stripped {count} readonly fields from changes for {recordType}/{recordId}: [{fieldIds}]. These fields are NetSuite-managed and cannot be written back."

    if changes is empty:
        LOG INFO "Event {eventId} has empty changes after readonly field strip — treating as no-op. Not dead-lettered."

    return changes  // may be empty; caller must not dead-letter empty result
```

### What NOT to Do

| Incorrect Handling | Why It Is Wrong |
|---------------------|-----------------|
| Dead-letter event when `is_readonly` field found in `changes` | is_readonly is a configuration state, not a data error. The event itself is valid. |
| Dead-letter event when `changes` is empty after strip | Empty changes is a valid no-op. Dead-lettering it floods the DLQ with false positives. |
| Silently pass readonly fields through to the NetSuite writer | NetSuite RESTlet will reject write attempts on computed/readonly fields, causing avoidable write failures. |
| Reject event with `SchemaValidationException` for readonly fields | Schema validation applies to structure and types, not to the readonly policy. Policy enforcement is a business rule, not a schema violation. |

### Test Scenarios Required (for `/test-writer` handoff)

| # | # | Scenario | Input | Expected Output |
|---|----------|-------|-----------------|
| 1 | No readonly fields in changes | changes = {custentity_foo: "bar"}, is_readonly=false | No strip; changes unchanged |
| 2 | One readonly field stripped | changes = {custentity_foo: "bar", custentity_sys: "computed"}, custentity_sys is_readonly=true | changes = {custentity_foo: "bar"}; LOG WARN emitted |
| 3 | All fields readonly → empty changes | changes = {custentity_sys: "computed"}, all is_readonly=true | changes = {}; LOG INFO emitted; NOT dead-lettered |
| 4 | Unknown field (not in field_metadata) | changes = {custentity_unknown: "x"} | Handled by existing Fail-Closed type check; is_readonly strip only applies to known fields |

---

## Downstream PHP Migration Required (Out of Scope — For Handoff)

The schema corrections in this task introduce **breaking changes to all existing PHP test fixtures** that use `eventType` or `schemaVersion: v1.1`. The following files require PHP-level migration as a separate follow-up task:

### Files Requiring Update

| File | Change Required | Occurrences |
|------|----------------|-------------|
| `tests/Unit/Domain/Events/Actions/EventSchemaValidationTest.php` | Rename `eventType` → `operation` in all payload fixtures; update `schemaVersion` from `v1.1` → `v1`; add `orderingKey` to all fixtures | ~12 occurrences |
| `src/App/Models/JobEvent.php` | Review `eventType` references — may be unrelated to envelope schema (legacy field name) | 6 occurrences |

### Impact Assessment

Running `php artisan test tests/Unit/Domain/Events/` **will produce failures** after this schema correction because:

1. All payload fixtures use `'eventType' => 'update'` — now rejected as `additionalProperties: false` rejects unknown fields
2. All payload fixtures use `'schemaVersion' => 'v1.1'` — now rejected as `schemaVersion` enum is `["v1"]`
3. No payload fixture includes `orderingKey` — now required and will fail `required` array validation

**Recommended Next Step:** Delegate to `/feature-implementer` or `/test-writer` with this document as the specification input to update all test fixtures and verify the test suite returns to GREEN.