Chapter 12: Commodity From a Provider¶

Control Where the Value Is¶

1. What to Buy, and What Never to Buy¶

The three structures spend money on providers for what they do well, and keep control of what they compete on. The line between those is not obvious, and it is almost never written down — so it drifts toward the provider, one integration at a time.

The line, stated plainly:

Buy the commodity	Keep the control
delivery, transport, storage	routing, policy, approval
scaling and availability of the primitive	what is allowed, and who decides
the protocol's mechanics	the audit trail of who did what

A provider that delivers a message should not decide whether the message may be sent. A provider that stores a record should not decide who may read it. The moment a provider's console becomes the place a policy is set, the control has moved.

💰 Cost Check: the provider's bill is visible and small; the leak is not on it. Policy set in a console costs a rule nobody can test. An approval in a vendor UI costs an audit you cannot assemble. Untrusted content handed to a tool costs an incident with a plausible explanation. The seam contract is the cheapest artefact in this chapter and the one that stops all three.

🎯 What You'll Learn¶

Where the provider's responsibility ends and yours begins
The seam contract that makes the boundary checkable
The failure modes that appear at the seam, not the provider

Where the Control Leaks¶

Leak	What it looks like
policy in the vendor's console	a rule nobody can diff or test
approval in the provider's UI	a decision with no record in your log
a seam with no contract	a behaviour you discover in an incident
credentials in the wrong layer	a provider key where a user's identity belongs

flowchart LR
  S["sender"] --> P["your control<br/>policy · approval · audit"]
  P --> V["provider<br/>transport · scaling"]
  V --> D["destination"]
  P -.->|"no control"| V2["provider decides"]

What this shows: the provider sits behind your control layer; when the arrow reverses, the provider is deciding.

✅ What to Check¶

Name one policy that is set in a provider's console. That is the control that leaked.
For each provider, can you state what it guarantees, and what you must never assume?

2. The Seam Contract¶

A seam contract answers four questions, and it belongs in your repository:

who owns what — which side is responsible for a failure;
what the provider guarantees — delivery? ordering? idempotency? at what limits?
what you must never assume — the guarantees it does not make;
what you will do when it breaks — retry, drop, escalate, or fail closed.

The seam contract, as a shape:

# seams/<provider>.yaml
provider: <name>
owns:
  provider: [<delivery, scaling, ...>]
  you: [<policy, approval, audit, retries, ...>]
guarantees:
  delivery: <at-least-once | exactly-once | unspecified>
  ordering: <guaranteed | not>
  limits: { rate: <n>, quota: <n> }
never_assume: [<what a test would catch if you did>]
on_failure: <retry with backoff | drop and alert | fail closed>

The critical field is never_assume. Most integrations fail not because the provider broke a promise, but because they assumed a promise the provider never made — ordering, exactly-once delivery, or a quota that was always enough in testing.

✅ What to Check¶

Does each provider have a seam contract, or an assumption?
Which field would have caught your last integration bug?

3. Idempotency, Retries and Ordering¶

Mechanic	The design decision	The default
idempotency	can the same input be processed twice safely?	every handler carries a key
retries	on what errors, with what backoff, up to what count?	retry transient, never blind
ordering	is order guaranteed, or must you sequence?	assume unordered unless stated
duplicates	does the system tolerate them, or dedupe?	dedupe at the boundary

Fail closed where the consequence is a trust or money decision: a payment webhook that cannot be verified is not processed with a warning. Retry with backoff where the consequence is delay. The choice is per seam, and it belongs in the contract, not in the handler.

Rate limits and quotas are part of the contract. A provider's limit is not a surprise; it is an unread field. The failure mode — a burst that hits a ceiling and silently drops — is a capacity assumption, not an outage.

✅ What to Check¶

Can each handler process the same message twice without a second effect?
Does the contract name the rate limit and what happens at it?

4. Human Gates Inside the Pipeline¶

Some flows must pause for a human: a send, a payment, a deletion. The gate belongs inside the pipeline, in your control layer — not in the provider's UI, where it leaves no record in your log.

The rule: an approval is a record with an actor, a timestamp and a decision. A console click is a click; a logged decision is evidence.

The audit trail is part of the product, not a log bolted on afterwards. The reason is commercial as much as technical: a client who asks "who approved this send?" is buying the trail as much as the delivery.

✅ What to Check¶

Where does the last human approval in your flow happen — your log, or the provider's UI?
If a client asked who approved a message, could you answer with a record?

5. The New Risk Class: Untrusted Content and Agents¶

Wherever untrusted content — a message body, a web page, a ticket — reaches an agent, treat it as input to validate, in the same class as SQL injection or a malformed payload.

The problem	The mitigation
content that instructs the agent	strip or quarantine instruction-shaped content
a provider payload that carries a directive	parse into a schema; never pass raw text to a tool
an agent with a credential it can be persuaded to use	least privilege, and a hook that refuses the action

The rule is not "trust the model less". It is that an agent reading untrusted content is a parser reading untrusted input, and parsers validate before they act.

✅ What to Check¶

Which of your agents read content a stranger wrote?
Is there a schema between that content and any tool the agent can call?

6. Build the Workbench¶

Artefacts¶

Artefact	What it is for	Where it lives
seam contract	owns · guarantees · never-assume · on-failure	`seams/<provider>.yaml`
worker	the handler, idempotent, with an approval gate	your control layer
round-trip test	send → receive → verify, including a replay	the test suite
audit record	approvals and outcomes, in your log	the execution log

Decisions¶

Choice	Default	What changes if you choose otherwise
where policy lives	your control layer	the provider's console, where it cannot be tested
delivery assumption	at-least-once, dedupe at the boundary	a duplicate side effect
on failure	fail closed for trust decisions, retry for delay	a warning where a stop was needed
approvals	a logged record	a click that leaves no evidence

✅ Checks¶

The seam contract names never_assume and an on_failure policy.
A replay test proves a duplicate does not double-process.
Every approval is in the log, with an actor and a timestamp.

Sequence¶

Write the seam contract for one provider. Checkpoint: never_assume is filled.
Make the handler idempotent and add the replay test. Checkpoint: the replay does not double-process.
Put the approval in your layer, logged. Checkpoint: the decision is in the log, not the console.
Add the untrusted-content schema between the payload and any tool. Checkpoint: a directive-shaped payload is refused.

🎯 Done When¶

Policy and approval live in your layer, testable and logged.
A duplicate message does not double-process.
Untrusted content reaches no tool without a schema in between.

7. 🔑 Chapter Summary¶

Buy the commodity; keep the control where the value is.

The line — providers own transport and scaling; you own routing, policy, approval and the audit.
The seam contract — owns · guarantees · never-assume · on-failure, in your repository.
Idempotency, retries, ordering and quotas are contract fields, not handler details.
Human gates live inside your pipeline, as records, not console clicks.
Prompt injection is input validation — a schema between untrusted content and any tool.