Major Life Sciences Tech — Marketing Data Governance & Consent

Summary

Built an API-led consent and identity governance layer on AWS to synchronize opt-in/out and global unsubscribe between Eloqua and SAP CRM, enforce policy at the edge, and keep one source of truth for outreach decisions. Result: lower compliance risk and cleaner campaign targeting through global unsubscribe hygiene.

---

Problem

• Consent lived in multiple systems (Eloqua forms, SAP CRM), often out of sync across shared email identities.
• Global unsubscribe rules weren’t uniformly enforced, risking non-compliant sends.
• No central audit of who changed what, when, and why; support teams lacked replayable evidence.

---

Solution Mechanics

Primary pattern: API-led orchestration (Java + Spring Boot on Amazon EKS).

Secondary: event-driven updates (EventBridge/SQS) for reliability and fan-out.

• Edge & Security

  • AWS API Gateway as the governed front door; OAuth2 client credentials, idempotency keys, request/response validation (OpenAPI/JSON Schema).
  • AWS WAF + rate limiting and header enforcement; AWS Secrets Manager for credentials; KMS encryption at rest; TLS in transit.

• API Orchestration Layer (Amazon EKS / Spring Boot)

  • Consent API — receives opt-in/out & preference updates from channels; validates payloads; persists audit; returns immediate ACK.
  • Eloqua Sync Worker — polls/webhooks Eloqua endpoints for consent deltas; normalizes fields; pushes to SAP via adapter.
  • SAP Adapter — calls SAP CRM (RFC/HTTP) to upsert contact email-opt status and source/date; handles retries and timeouts.
  • Identity Resolution (Preprocessor) — maintains ContactMaster (unique email) and ContactPlus (email↔multiple SAP IDs) with Shared-Winner / Not-Winner flags; ensures global unsubscribe updates propagate to all linked contacts.
  • Policy Engine (OPA/Rego) — blocks outbound outreach if global unsubscribe is active; enforces region-specific consent rules.

• Data, Events & Audit

  • Amazon RDS for PostgreSQL — consent ledger (immutable append + current state), correlation keys, replay pointers.
  • Amazon S3 — immutable audit envelopes (original requests, normalized records, responses); partitioned by date/region.
  • Amazon SQS — durable work queues (ingest→normalize→SAP); DLQs and replay utilities.
  • Amazon EventBridge — emits ConsentChanged events to downstream analytics & suppression services.
  • Observability — CloudWatch metrics/logs (ingest→apply p95, SAP call success, DLQ depth), alarms to SNS.

• Governance & DX

  • Versioned OpenAPI for Consent API; postman collections; red/black rollout of mapping changes.
  • PII minimization (email, contact IDs, timestamps only); data retention policies per region.

Diagram 1 - Context Diagram — AWS consent & identity governance

Diagram 2 - Sequence — Opt-in/out sync between Eloqua and SAP

Diagram 3 - Identity Resolution — Shared email (ContactPlus ↔ ContactMaster)

---

Process Flow

1. Channel/Eloqua triggers an opt-in/out or global unsubscribe event; API Gateway receives it, validates schema, and ACKs with a correlation ID.
2. Consent API writes an immutable envelope to S3 and an append-only consent ledger row to Postgres; emits a work item to SQS.
3. Identity Resolution resolves the email to one-or-many SAP contact IDs (Shared-Winner/Not-Winner) and computes the outbound set.
4. SAP Adapter updates SAP CRM for all impacted contacts (idempotent upserts with retries and exponential backoff).
5. Eloqua Sync Worker reconciles any Eloqua-side field/state gaps (opt status, source, date, global subscription).
6. Policy Engine (OPA) evaluates effective consent; if global unsubscribe is active, downstream EventBridge notifies suppression lists and campaign tools.
7. CloudWatch tracks ingest→apply latency, SAP success, DLQ depth; operators can replay from S3/SQS by time window or correlation ID.
8. Analytics query S3 via Athena/Glue to audit changes, measure hygiene coverage, and drive CX improvements.

---

Outcomes

• Single source of truth for consent and global unsubscribe across Eloqua ↔ SAP. (Verified: consistent reads via Consent API.)
• Near-real-time sync with ingest→apply p95 under 15 minutes during peak. (Modeled/verified in pre-prod load tests.)
• Defensible audit with immutable envelopes and correlation IDs; replayable incident analysis. (Verified capability.)
• Cleaner targeting and reduced false-sends through policy enforcement and identity resolution. (Proxy via hygiene coverage.)

---

Strategic Business Impact

• Reduced compliance risk (Proxy): uniform enforcement of global unsubscribe and region-specific policies.
• Improved campaign performance (Proxy): cleaner lists and fresher consent raise deliverability and conversion.
• Lower support load (Modeled): fewer “unsubscribe didn’t work” tickets via end-to-end reconciliation.

Method tags: Verified (capabilities observed in env), Modeled (latency targets under load), Proxy (coverage/adoption indicators).

---

Role & Scope

Owned architecture and delivery for API Gateway policies, EKS services (Consent API, SAP Adapter, Eloqua Sync, Identity Resolution), Postgres schema, S3 audit model, SQS/EventBridge wiring, OPA policies, and observability.

---

Key Decisions & Trade-offs

• API-led front door with idempotent writes → strict contracts and predictable SLAs; adds an extra hop vs direct integration.
• OPA + JSON Schema over heavy rules engines → simpler runtime, faster iteration; requires good versioning and tests.
• Immutable S3 envelopes for audit → cheap retention and replay; ad-hoc querying offloaded to Athena.
• Shared-email resolution centralized → consistent global unsubscribe; additional storage and compute for merges and flags.

---

Risks & Mitigations

• Out-of-sync systems (partial failures) → exactly-once semantics via idempotency keys + replay from S3/SQS.
• Identity ambiguity (shared emails) → deterministic Shared-Winner selection + propagation to all linked contacts; periodic reconciliations.
• Regional policy drift → OPA policies versioned, peer-reviewed, and canary-released.
• Throughput spikes → autoscale EKS; pre-provision SQS; circuit breakers on SAP Adapter; backpressure metrics/alerts.

---

Suggested Metrics (run-time SLOs)

• Ingest→apply p50/p95/p99 and error rate by source.
• Reconciliation lag (Eloqua↔SAP) and coverage % (contacts in sync).
• DLQ depth/age and replay success rate.
• Policy blocks count (global unsubscribe) and suppression-list propagation latency.
• Ledger correctness: % idempotent upserts; duplicate-correlation rate.

---

Closing principle

Consent is a first-class domain, not a side effect. Lock contracts at the edge, resolve identities centrally, and make every change auditable and replayable.