> ## Documentation Index
> Fetch the complete documentation index at: https://developer.fitprotracker.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Delivery semantics
> At-least-once delivery, idempotency on EventId, retry & backoff, and ordering guarantees.
This page is the contract. Read it before you write your handler — internalizing these four rules will save you a support ticket.
The four rules:
1. **At-least-once.** We may deliver the same event more than once.
2. **Dedupe by `eventId`.** It's a globally unique idempotency key.
3. **Return 2xx within 10 seconds** or we treat it as a failure and retry.
4. **Don't rely on ordering** between events. Use `eventTimestamp` to sort.
## At-least-once delivery
We commit to **at-least-once** delivery. That means:
* If your endpoint responds 2xx, we mark the delivery successful and never re-send that event ID.
* If your endpoint times out, returns non-2xx, or we hit a transient infrastructure issue mid-delivery, **we retry** — and you may end up seeing the same event ID twice.
We do **not** offer exactly-once. That's a deliberate trade-off: building exactly-once across an internet boundary requires partner-side cooperation (a transactional dedupe store), and most partners need to build that anyway for other reasons. We chose to make the dedup point explicit rather than pretend.
## Idempotency on `eventId`
Every event carries a unique `eventId` in the envelope:
```json theme={null}
{ "eventId": "9f1c7e2a8c4d4b1b9e3f5a6d7c8b9a0e", "eventType": "contact.status_changed", ... }
```
When your endpoint receives an event:
[HMAC-SHA256 check](/webhooks/signing) against the raw body. Reject any request that fails verification.
Look it up in your dedupe store (Postgres table, Redis SET, DynamoDB item — anything atomic with a unique constraint on the ID).
* **Seen** → return 200 immediately, skip processing.
* **New** → continue.
Do an atomic insert-if-not-exists. If the insert fails because of a unique-constraint violation, treat as a duplicate (another worker is processing it, or it's a retry that's racing with the first attempt) and return 200.
Apply the change, update your records, fire side effects.
A 2xx response signals "I have it; don't send it again." We mark the delivery durable at that point.
### Minimal Postgres example
```sql theme={null}
CREATE TABLE webhook_dedupe (
event_id CHAR(32) PRIMARY KEY,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
processed_at TIMESTAMPTZ
);
-- At delivery time:
INSERT INTO webhook_dedupe (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- If this returns a row, you're the first to see this event — process it.
-- If it returns no rows, someone else already inserted it — return 200.
```
**How long to retain dedupe entries?** A month is plenty. Our retry window is well under 24 hours (see backoff below), so anything older than that won't recur. Periodically purge old rows so the table doesn't grow unbounded.
## Retries and backoff
If your endpoint fails (non-2xx response, timeout > 10s, connection refused), the failed delivery is **redelivered by Azure Service Bus** under its default policy: exponential backoff in the seconds-to-tens-of-seconds range, up to a small handful of attempts (typically 5). After Service Bus exhausts retries, the message lands in a dead-letter queue and we stop trying for that specific event.
**Note this is per-event retry, not endpoint-level retry.** A handful of retries over \~minutes — not a long-tail schedule that spans hours.
### Auto-pause on consecutive failures
What protects you from a sustained outage is a separate mechanism: **three consecutive event-delivery failures in a row pauses the subscription**. The counter increments on each failed event and resets to zero on any 2xx response.
| State | Trigger | Effect |
| -------------------- | -------------------------- | ------------------------------------------------------------------- |
| Active, healthy | 2xx response | `consecutiveFailures` resets to 0 |
| Active, one failure | first non-2xx | `consecutiveFailures = 1`; next event still attempted |
| Active, two failures | second non-2xx in a row | `consecutiveFailures = 2`; one more chance |
| **Paused** | **third non-2xx in a row** | `Status` → `paused`; no new events delivered until an admin resumes |
When a subscription auto-pauses:
* The admin grid renders a Paused badge on the row (amber dot + "Paused" pill).
* FPT logs a Warning to elmah so internal monitoring catches it.
* **In a near-term release**, the gym owner will get an email notification with a link to investigate. (v1.1 ships the admin-visible state + log signal; SendGrid email wiring lands in v1.1.x.)
* An admin clicks the resume button (`▶`) in the actions column to flip back to active.
A brief partner blip (a few seconds) triggers Service Bus retries and you recover transparently — your `consecutiveFailures` counter never gets to 1 if any of the SB redeliveries succeed. A multi-minute outage probably trips one event's full SB retry budget; if the very next event also fails, you're at 2; the one after that puts you in paused state. The auto-pause is intentionally conservative so a healthy long-running partner isn't paused by a single bad deploy window.
**Why not the long-tail retry schedule** (1min, 5min, 30min, ...)? In practice, partners who are down for an hour usually stay down for hours, and rapid auto-pause + visible admin signal beats invisible queueing. The 3-failure threshold is also the contract Close.com uses for the same reason.
## What counts as success vs failure
| Response | Meaning |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| 2xx (200–299) | **Success.** We won't send this event again. |
| 4xx (400–499) | **Permanent failure.** We don't retry — your endpoint indicated the request itself is unprocessable. Logs only. |
| 5xx (500–599) | **Transient failure.** Retry on the schedule above. |
| Timeout (>30s) | **Transient failure.** Retry on the schedule above. |
| Connection refused / DNS failure | **Transient failure.** Retry. |
| TLS handshake failure | **Permanent failure.** We don't retry — typically misconfigured certificates. We email the subscription owner. |
**Don't return 4xx for transient issues** (database locked, downstream API slow). 4xx tells us "this event is permanently bad" — we'll skip retries and drop it. For "try again later" semantics, return 5xx or just time out.
## Ordering
We **don't guarantee ordering** across events. If a contact's status flips Lead → Member → VIP within 200ms, you might receive the two `contact.status_changed` events in either order. Plan for this:
* **Sort by `eventTimestamp`** when order matters
* **Coalesce** at your end if you only care about the latest state (look up the contact's current state at receipt time via your own data, not the event)
* **Don't make decisions from a single event's `previousX → newX` chain** if multiple events of that type can fire close together — the previous values you see may be stale
## Why these rules exist
Each one prevents a specific support ticket we've seen with other webhook integrations:
| Rule | Ticket it prevents |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| At-least-once + dedupe by eventId | "Our customer was charged twice / had two contacts created" |
| 10-second timeout, return 2xx fast | "All our retries are firing because our endpoint is slow" |
| 4xx vs 5xx distinction | "Our endpoint returned 400 for a transient DB error and we never got the event again" |
| Auto-pause threshold of 3 (not 1) consecutive failures | "A single transient blip auto-disabled our subscription overnight" |
| No ordering guarantee | "Events arrived out of order and corrupted our state" |
| HMAC signature verification | "How do we know it's really Fit Pro Tracker?" |