# Progress Tracker: Epic 1 — Canonical JSON Schema & Internal ACL
**Branch:** `feat/epic1-canonical-schema-acl`
**Base:** `staging`
**Created:** 2026-03-04
**Last Updated:** 2026-03-11 (enhancement — `actorId` required field added to envelope schema v1.1 for audit logging)

---

## Task Context
Implement the "Contract First" Canonical JSON Schema and Anti-Corruption Layer (ACL) as a prerequisite phase for the integration between SuiteX and NetSuite. This is an internal SuiteX module — no external server is built in this Epic.

## Objectives
1. Define strict JSON Schema contracts for all events crossing the NetSuite–SuiteX boundary.
2. Build a two-stage validation engine (Stage 1: Structural JSON, Stage 2: Semantic DB).
3. Persist rejected events to a Dead Letter Queue (DLQ) for forensic analysis.
4. Validate the engine with a comprehensive Pest test suite.

---

## Progress Log

| # | Date | Phase | Action | Status |
|---|------|-------|--------|--------|
| 1 | 2026-03-04 | Planning | Designed and approved implementation plan | ✅ Done |
| 2 | 2026-03-04 | Planning | Aligned with AI Rules (DDD, Tenancy, Security) | ✅ Done |
| 3 | 2026-03-04 | Execution | Installed `justinrainbow/json-schema` via Composer | ✅ Done |
| 4 | 2026-03-04 | Execution | Created `schemas/envelope/v1.schema.json` | ✅ Done |
| 5 | 2026-03-04 | Execution | Created `schemas/payloads/project/v1.schema.json` | ✅ Done |
| 6 | 2026-03-04 | Execution | Created `schemas/payloads/project-task/v1.schema.json` | ✅ Done |
| 7 | 2026-03-04 | Execution | Created migration `create_field_metadata_table` (Composite PK) | ✅ Done |
| 8 | 2026-03-04 | Execution | Created migration `create_dead_letter_events_table` (DLQ) | ✅ Done |
| 9 | 2026-03-04 | Execution | Created `Domain\Events\Exceptions\SchemaValidationException` | ✅ Done |
| 10 | 2026-03-04 | Execution | Created `Domain\Events\Models\DeadLetterEvent` | ✅ Done |
| 11 | 2026-03-04 | Execution | Created `Domain\Metadata\Models\FieldMetadata` | ✅ Done |
| 12 | 2026-03-04 | Execution | Created `Domain\Events\Actions\ValidateCanonicalEnvelopeAction` (Stage 1) | ✅ Done |
| 13 | 2026-03-04 | Execution | Created `Domain\Events\Actions\ValidateSemanticCustomFieldsAction` (Stage 2) | ✅ Done |
| 14 | 2026-03-04 | Verification | All 3 Pest test scenarios passed | ✅ Done |
| 15 | 2026-03-05 | Remediation | Task-Architect audit: 3 gaps fixed in Epic 1 | ✅ Done |
| 16 | 2026-03-05 | Remediation | GAP-E1-01: Added full SQLite `beforeEach` setup to test file | ✅ Done |
| 17 | 2026-03-05 | Remediation | GAP-E1-02: Created `Domain\Events\Support\RecordTypeRegistry` | ✅ Done |
| 18 | 2026-03-05 | Remediation | GAP-E1-02: Refactored `ValidateCanonicalEnvelopeAction` to use Registry | ✅ Done |
| 19 | 2026-03-05 | Verification | PHPStan Level 5 — FieldMetadata composite PK suppressed with justification | ✅ Done |
| 20 | 2026-03-05 | Verification | PR Review executed via `/pr-reviewer` workflow | ✅ Done |
| 21 | 2026-03-05 | Bug Fix | **BUG:** `field_metadata` test schema mismatch (Cursor Bot report) | ✅ Done |
| 22 | 2026-03-05 | Bug Fix | CAUSE: `expected_type`/`on_conflict` invented; REAL: `field_type`, `is_synced`, `is_readonly`, `normalization_rule`, `conflict_policy` | ✅ Done |
| 23 | 2026-03-05 | Improvement | Heuristic added to `docs/ai-rules/400-testing.md` §6 (Schema-From-Memory anti-pattern) | ✅ Done |
| 24 | 2026-03-05 | Bug Fix | **BUG (Medium):** `ValidateSemanticCustomFieldsAction` hardcoded alias mapping instead of using `RecordTypeRegistry` | ✅ Done |
| 25 | 2026-03-05 | Bug Fix | FIX: Replaced hardcode with `RecordTypeRegistry::resolveSchemaName()`. `$recordType` → `$schemaName`. PHPStan ✅ Tests 3/3 ✅ | ✅ Done |
| 26 | 2026-03-05 | Bug Fix | **BUG (High):** `uses()->in('Unit')` applied `RefreshDatabase` to all 83 Unit tests globally | ✅ Done |
| 27 | 2026-03-05 | Bug Fix | FIX: Removed `->in('Unit')`. Corrected canonical example in `400-testing.md`. Added §7 heuristic. task-architect TDD Step 3 updated | ✅ Done |
| 28 | 2026-03-05 | Bug Fix | **BUG (Low):** `routeToDeadLetterQueue` copy-pasted in both Actions (DRY violation) | ✅ Done |
| 29 | 2026-03-05 | Bug Fix | FIX: Extracted to `Domain\\Events\\Support\\DeadLetterRouter::dispatch()`. PHPStan ✅ Tests 3/3 ✅ | ✅ Done |
| 30 | 2026-03-05 | Bug Fix | **BUG (Medium):** `DeadLetterEvent` missing tenant connection trait per spec | ✅ Done |
| 31 | 2026-03-05 | Bug Fix | FIX: Added `App\\Traits\\UsesTenantConnection` to `DeadLetterEvent`. | ✅ Done |
| 32 | 2026-03-06 | Bug Fix | **BUG (Medium):** `str_starts_with('cust')` matches standard fields like `customer` causing false positives | ✅ Done |
| 33 | 2026-03-06 | Bug Fix | FIX: Replaced with strict regex `preg_match('/^cust(body|col|entity|item|record|event)/i')`. PHPStan ✅ Tests 3/3 ✅ | ✅ Done |
| 34 | 2026-03-06 | Bug Fix | **BUG (Low):** `RecordTypeRegistry::supportedTypes()` is defined but never called (dead code) | ✅ Done |
| 35 | 2026-03-06 | Bug Fix | FIX: Removed method `supportedTypes()` completely. PHPStan ✅ Tests 3/3 ✅ | ✅ Done |
| 36 | 2026-03-06 | Bug Fix | **BUG (High):** Tenant migrations `field_metadata` and `dead_letter_events` in wrong directory/connection | ✅ Done |
| 37 | 2026-03-06 | Bug Fix | FIX: Moved to `migrations/tenants/` and added `Schema::connection('tenant_connection')`. | ✅ Done |
| 38 | 2026-03-06 | Bug Fix | **BUG (Medium):** `project-task` JSON schema regex `^cust[a-zA-Z0-9_]+$` is too broad (allows 'customer') | ✅ Done |
| 39 | 2026-03-06 | Bug Fix | FIX: Replaced with strict semantic regex `^cust(body|col|entity|item|record|event)[a-zA-Z0-9_]*$` | ✅ Done |
| 40 | 2026-03-06 | Verification | **BUG (False Positive):** Removed `ignoreErrors` for `projecttasks` in `phpstan.neon` breaks CI | ✅ Done |
| 41 | 2026-03-06 | Verification | VERDICT: False Positive. `phpstan analyse src/Domain/Projects/Models/Project.php` passed 0 errors. Relationship inferred dynamically. | ✅ Done |
| 42 | 2026-03-06 | Bug Fix | **BUG (Low):** Redundant `routeToDeadLetterQueue` wrappers remain in Action classes | ✅ Done |
| 43 | 2026-03-06 | Bug Fix | FIX: Removed methods and delegated directly to `DeadLetterRouter::dispatch()`. PHPStan ✅ Tests 3/3 ✅ | ✅ Done |
| 44 | 2026-03-06 | Bug Fix | **BUG (Low):** Redundant `$connection` property in `FieldMetadata` model | ✅ Done |
| 45 | 2026-03-06 | Bug Fix | FIX: Removed `$connection` property, relying solely on `UsesTenantConnection` trait. | ✅ Done |
| 46 | 2026-03-09 | Bug Fix | **BUG (Medium):** Project schema regex `^custentity_` rejects valid `custbody_*` custom fields | ✅ Done |
| 47 | 2026-03-09 | Bug Fix | FIX: Aligned `project` regex with `project-task`. Added Scenario 4 to `EventSchemaValidationTest`. PHPStan ✅ | ✅ Done |
| 48 | 2026-03-09 | Bug Fix | **BUG (Medium):** N+1 queries in semantic custom field validation in `ValidateSemanticCustomFieldsAction` | ✅ Done |
| 49 | 2026-03-09 | Bug Fix | FIX: Implemented bulk-fetch strategy using `whereIn`. Added N+1 regression test (Scenario 5). PHPStan ✅ | ✅ Done |
| 50 | 2026-03-09 | Bug Fix | **BUG (Medium):** Semantic validator silently accepts unknown field types (Fail-Open) | ✅ Done |
| 51 | 2026-03-09 | Bug Fix | FIX: Switched to Fail-Closed strategy. Added Scenario 6 (Unknown Type Rejection). Added heuristic to `400-testing.md`. | ✅ Done |
| 52 | 2026-03-09 | Bug Fix | **BUG (Medium):** Catch block misses `TypeError` from `json_encode` failure in Stage 1 | ✅ Done |
| 53 | 2026-03-09 | Bug Fix | FIX: Expanded `catch (\Exception)` to `catch (\Throwable)` in Stage 1 & 2. Added Scenario 7. PHPStan ✅ | ✅ Done |
| 54 | 2026-03-09 | Bug Fix | **BUG (High):** Scenario 7 test used non-existent hook and missed real `Throwable` code path | ✅ Done |
| 55 | 2026-03-09 | Bug Fix | FIX: Refactored Scenario 7 with Mockery to trigger real `TypeError` & verify DLQ dispatch. Process isolation added. | ✅ Done |
| 56 | 2026-03-10 | Bug Fix | **BUG (High):** `date`/`datetime` validation uses `strtotime()`, accepting natural language strings like `"next friday"` and impossible dates like `"2024-02-30"` | ✅ Done |
| 57 | 2026-03-10 | Bug Fix | FIX: Replaced `strtotime()` with `DateTimeImmutable::createFromFormat()` + roundtrip check for `date`; strict format check for `datetime`. PHPStan ✅ | ✅ Done |
| 58 | 2026-03-10 | Improvement | Added Scenario 8 (Natural Language Date Rejection) to `EventSchemaValidationTest`. Verified RED before fix, GREEN after. | ✅ Done |
| 59 | 2026-03-10 | Improvement | Added Rule 5 to `docs/ai-rules/300-security.md` §Schema-First Validation: never use `strtotime()`/`Carbon::parse()` for format validation in canonical pipelines. | ✅ Done |
| 60 | 2026-03-10 | Bug Fix | **BUG (High):** `null` custom field values always rejected by semantic validator — `is_bool(null)` etc. always return `false`, routing valid clear-field operations to DLQ | ✅ Done |
| 61 | 2026-03-10 | Bug Fix | FIX: Added `null` guard (`if ($value === null) { continue; }`) before `validateFieldType()` call. Added Scenario 9 regression test. Added Null-Sentinel Rule to `000-core-standards.md`. PHPStan ✅ Tests 9/9 ✅ | ✅ Done |
| 62 | 2026-03-10 | Bug Fix | **BUG (High):** `datetime` validator accepted impossible dates like `2024-02-30T10:00:00Z` and `2024-13-01T10:00:00Z` — `\DateTimeImmutable::createFromFormat()` silently overflows (Feb 30 → Mar 1) so `$dt !== false` passes, allowing invalid datetimes into the pipeline | ✅ Done |
| 63 | 2026-03-10 | Bug Fix | FIX: Added roundtrip check `$dt->format('Y-m-d\TH:i:sP') === $normalized` to `datetime` case in `validateFieldType()`. Comparison targets `$normalized` (Z→+00:00 replaced) not `$value` to avoid false negatives on valid UTC strings. Added `2024-02-30T10:00:00Z` and `2024-13-01T10:00:00Z` overflow cases to Scenario 8 `$rejectedDatetimeValues`. PHPStan ✅ Tests 9/9 ✅ | ✅ Done |
| 64 | 2026-03-10 | Bug Fix | **BUG (Medium) #1:** `datetime` validator silently rejects RFC 3339 values with fractional seconds (e.g. `2024-01-15T10:30:00.123Z`) — Stage 1 (JSON Schema) accepts them but Stage 2 strict format `Y-m-d\TH:i:sP` rejects them, creating a DLQ trap with no user-visible error | ✅ Done |
| 65 | 2026-03-10 | Bug Fix | FIX: Added `preg_replace('/\.\d+(?=[+\-])/', '', $normalized)` after the Z→+00:00 normalization to strip fractional seconds before strict format validation. Comparison target remains `$normalized` (post-strip); roundtrip check preserved; overflow detection unaffected. PHPStan ✅ Tests 9/9 ✅ (70 assertions) | ✅ Done |
| 66 | 2026-03-10 | Improvement | Added Rule 6 (Multi-Stage Pipeline Contract Alignment) to `docs/ai-rules/300-security.md` §Schema-First Validation: downstream stages must accept the same superset as upstream or normalize before validating to prevent silent DLQ traps | ✅ Done |
| 67 | 2026-03-10 | Improvement | Cosmetic: Replaced shallow `(object) $payload` cast in Scenario 5 with deep `json_decode(json_encode($payload))` to precisely mirror what Stage 1 passes to Stage 2 in production — eliminates shallow-cast vs stdClass structural discrepancy | ✅ Done |
| 68 | 2026-03-10 | Performance | Created `Domain\Events\Services\SchemaValidatorService` — holds shared `JsonSchema\Constraints\Factory` instance and resolves schema file URIs; eliminates per-call disk I/O | ✅ Done |
| 69 | 2026-03-10 | Performance | Refactored `ValidateCanonicalEnvelopeAction::execute()` to resolve `SchemaValidatorService` via DI and pass shared `Factory` to both `new Validator()` calls — `UriRetriever` cache now survives across invocations | ✅ Done |
| 70 | 2026-03-10 | Performance | Added `json_encode` null guard in `ValidateCanonicalEnvelopeAction::execute()` — throws `SchemaValidationException` immediately if payload is not JSON-serializable; routed to DLQ via existing `\Throwable` catch | ✅ Done |
| 71 | 2026-03-10 | Improvement | Added Rule 5 (Validator/Parser Class Instantiation in Hot Paths) to `docs/ai-rules/500-performance.md` — prohibits `new ValidatorClass()` in hot-path methods; mandates singleton DI binding for cache-carrying classes | ✅ Done |
| 72 | 2026-03-10 | Performance | Added Redis cache-aside layer to `ValidateSemanticCustomFieldsAction::execute()` — full `account_id+record_type` map cached at `field_metadata:{accountId}:{schemaName}` with 3600s TTL via `Cache::store('redis')->remember()`; in-memory `collect()->only()` filter applied post-cache so key serves any field combination | ✅ Done |
| 73 | 2026-03-10 | Performance | Created `Domain\Metadata\Observers\FieldMetadataCacheObserver` — invalidates `field_metadata:{account_id}:{record_type}` on `saved` and `deleted` events; `$afterCommit = true` prevents race-condition cache poisoning | ✅ Done |
| 74 | 2026-03-10 | Performance | Registered `FieldMetadataCacheObserver` on `FieldMetadata` model in `EventServiceProvider::boot()` | ✅ Done |
| 75 | 2026-03-10 | Improvement | Added Driver Guard Rule to `docs/ai-rules/500-performance.md` §3 Caching Strategy — prohibits `Cache::tags()` without explicit `Cache::store('redis')`; default `file` driver throws `BadMethodCallException` at runtime | ✅ Done |
| 76 | 2026-03-10 | Bug Fix | **BUG (High):** `DeadLetterRouter::dispatch()` uses single-attempt try/catch — a transient DB connection failure silently discards the failure evidence the DLQ exists to preserve | ✅ Done |
| 77 | 2026-03-10 | Bug Fix | FIX: Replaced single-attempt catch with 3-attempt bounded retry loop (100ms/200ms exponential backoff via `usleep`); on exhaustion writes structured JSON to `dead_letter_fallback` daily log channel. PHPStan ✅ Tests 9/9 ✅ | ✅ Done |
| 78 | 2026-03-10 | Improvement | Added `dead_letter_fallback` daily log channel to `config/logging.php` — raw `daily` driver, independent of `TenantLogger`, 30-day retention, `critical` level; serves as durable out-of-band fallback when tenant DB is unavailable | ✅ Done |
| 79 | 2026-03-10 | Improvement | Added Last-Resort Writer Pattern rule to `docs/ai-rules/100-backend-laravel.md` §Background Jobs & Concurrency — any class whose primary function records failure evidence MUST apply bounded-retry + durable-fallback contract regardless of synchronous call context | ✅ Done |
| 80 | 2026-03-10 | Performance | Created migration `2026_03_10_113324_add_indexes_to_dead_letter_events_table` — `index('event_id')`: lookup key for incident response ("find all DLQ entries for this event"), non-unique due to retry scenarios | ✅ Done |
| 81 | 2026-03-10 | Performance | Added composite `index(['source', 'created_at'])` — supports primary forensic query `WHERE source = ? AND created_at > ?`; `source` as equality anchor, `created_at` for range scan within that anchor | ✅ Done |
| 82 | 2026-03-10 | Performance | Added `index('created_at')` — covers dashboard queries "all DLQ entries in the last N hours regardless of source"; not covered by the composite index above | ✅ Done |
| 83 | 2026-03-10 | Bug Fix | **BUG (Low):** `json_encode($payload)` called twice per invocation — guard check result discarded, then re-encoded inside `json_decode(json_encode($payload))`; redundant computation on every canonical event | ✅ Done |
| 84 | 2026-03-10 | Bug Fix | FIX: Introduced `$encoded = json_encode($payload)` variable; guard checks `$encoded === false`, `json_decode($encoded)` reuses cached result. Added Scenario 10 (`json_encode` guard coverage — previously untested path). PHPStan ✅ Tests 10/10 ✅ (72 assertions) | ✅ Done |
| 85 | 2026-03-10 | Bug Fix | **BUG (Medium):** `DeadLetterRouter::dispatch()` fallback log uses bare `json_encode($payload)` — returns `false` for non-serializable payloads (PHP resources, closures), recording the string `"false"` instead of payload data and silently defeating the log's purpose | ✅ Done |
| 86 | 2026-03-10 | Bug Fix | FIX: Replaced `json_encode($payload)` with `json_encode($payload, JSON_PARTIAL_OUTPUT_ON_ERROR) ?: '[non-serializable payload]'` in fallback log context. PHPStan ✅ Tests 11/11 ✅ (78 assertions) | ✅ Done |
| 87 | 2026-03-10 | Improvement | Added **Fallback Logger Serialization Safety** heuristic to `docs/ai-rules/100-backend-laravel.md` §Background Jobs & Concurrency — mandates fault-tolerant `JSON_PARTIAL_OUTPUT_ON_ERROR` encoding in all durable fallback log calls that capture external payloads | ✅ Done |
| 88 | 2026-03-10 | Improvement | Added Scenario 11 to `EventSchemaValidationTest` — verifies `dead_letter_fallback` log `payload` context is a non-empty string even when payload contains a PHP resource; uses `Log::spy()` + Mockery channel mock to capture critical context | ✅ Done |
| 89 | 2026-03-11 | Enhancement | Added `actorId` field to `schemas/envelope/v1.schema.json` — required `string` property; added property definition in `properties`; bumped `schemaVersion` enum from `["v1"]` to `["v1.1"]` to signal contract evolution to Epic 4 & 6 producers | ✅ Done |
| 90 | 2026-03-11 | Test Update | Updated all 10 payload fixtures in `EventSchemaValidationTest` to include `actorId` and `schemaVersion: 'v1.1'` (Scenarios 1–6, 8–11); Scenario 7 untouched — calls Stage 2 directly, envelope schema never evaluated | ✅ Done |
| 91 | 2026-03-11 | Test Update | Added Scenario 12: payload without `actorId` → Stage 1 `SchemaValidationException` (Fail-Closed audit logging contract; covers Acceptance Criteria 1) | ✅ Done |
| 92 | 2026-03-11 | Test Update | Added Scenario 13: `actorId: 'system'` → Stage 1 acceptance, `$validated->actorId === 'system'` asserted (automated process identity convention; covers Acceptance Criteria 2) | ✅ Done |
| 93 | 2026-03-11 | Doc Update | Updated canonical envelope example in `docs/designs/data-sync/V10.md` to include `schemaVersion: "v1.1"` and `actorId: "8832"` fields; aligns contract doc with schema for Epic 4 & 6 emitter developers | ✅ Done |

---

## Deliverables Summary

| File | Type | Status |
|------|------|--------|
| `schemas/envelope/v1.schema.json` | JSON Schema | ✅ |
| `schemas/payloads/project/v1.schema.json` | JSON Schema | ✅ |
| `schemas/payloads/project-task/v1.schema.json` | JSON Schema | ✅ |
| `database/migrations/…_create_field_metadata_table.php` | Migration | ✅ |
| `database/migrations/…_create_dead_letter_events_table.php` | Migration | ✅ |
| `src/Domain/Events/Exceptions/SchemaValidationException.php` | Exception | ✅ |
| `src/Domain/Events/Models/DeadLetterEvent.php` | Model (Tenant) | ✅ |
| `src/Domain/Metadata/Models/FieldMetadata.php` | Model (Tenant) | ✅ |
| `src/Domain/Events/Actions/ValidateCanonicalEnvelopeAction.php` | Domain Action | ✅ |
| `src/Domain/Events/Actions/ValidateSemanticCustomFieldsAction.php` | Domain Action | ✅ |
| `src/Domain/Events/Support/RecordTypeRegistry.php` | Domain Support | ✅ |
| `src/Domain/Events/Services/SchemaValidatorService.php` | Domain Service | ✅ |
| `src/Domain/Metadata/Observers/FieldMetadataCacheObserver.php` | Domain Observer | ✅ |
| `tests/Unit/Domain/Events/Actions/EventSchemaValidationTest.php` | Pest Test | ✅ |

---

## Definition of Done Checklist
- [x] PHPStan Level 5 — 0 real errors (1 suppressed with justification)
- [x] All tests pass: `php artisan test tests/Unit/Domain/Events/`
- [x] No generic variable names (`$data`, `$item`, `$result`)
- [x] No `dd()`, `dump()` or unguarded `console.log()`
- [x] No TODOs without Jira reference
- [x] DDD compliance: Actions return Models, never HTTP responses
- [x] Tenancy: Tenant models use `UsesTenantConnection`
- [x] Testing: `beforeEach` uses SQLite `:memory:` with explicit connection
- [x] PR Review completed — see `pr_review_epic1.md`

## Next Steps
→ Epic 2: Gateway & Token Registry (`epic2_implementation_plan.md`)