Delivery semantics

This page is the contract. Read it before you write your handler — internalizing these four rules will save you a support ticket.

The four rules:

At-least-once. We may deliver the same event more than once.
Dedupe by eventId. It’s a globally unique idempotency key.
Return 2xx within 10 seconds or we treat it as a failure and retry.
Don’t rely on ordering between events. Use eventTimestamp to sort.

At-least-once delivery

We commit to at-least-once delivery. That means:

If your endpoint responds 2xx, we mark the delivery successful and never re-send that event ID.
If your endpoint times out, returns non-2xx, or we hit a transient infrastructure issue mid-delivery, we retry — and you may end up seeing the same event ID twice.

We do not offer exactly-once. That’s a deliberate trade-off: building exactly-once across an internet boundary requires partner-side cooperation (a transactional dedupe store), and most partners need to build that anyway for other reasons. We chose to make the dedup point explicit rather than pretend.

Idempotency on `eventId`

Every event carries a unique eventId in the envelope:

{ "eventId": "9f1c7e2a8c4d4b1b9e3f5a6d7c8b9a0e", "eventType": "contact.status_changed", ... }

When your endpoint receives an event:

Verify the signature

HMAC-SHA256 check against the raw body. Reject any request that fails verification.

Check whether you've seen this eventId before

Look it up in your dedupe store (Postgres table, Redis SET, DynamoDB item — anything atomic with a unique constraint on the ID).

Seen → return 200 immediately, skip processing.
New → continue.

Insert the eventId BEFORE processing

Do an atomic insert-if-not-exists. If the insert fails because of a unique-constraint violation, treat as a duplicate (another worker is processing it, or it’s a retry that’s racing with the first attempt) and return 200.

Process the event in your business logic

Apply the change, update your records, fire side effects.

Return 200

A 2xx response signals “I have it; don’t send it again.” We mark the delivery durable at that point.

Minimal Postgres example

CREATE TABLE webhook_dedupe (
    event_id      CHAR(32) PRIMARY KEY,
    received_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at  TIMESTAMPTZ
);

-- At delivery time:
INSERT INTO webhook_dedupe (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- If this returns a row, you're the first to see this event — process it.
-- If it returns no rows, someone else already inserted it — return 200.

How long to retain dedupe entries? A month is plenty. Our retry window is well under 24 hours (see backoff below), so anything older than that won’t recur. Periodically purge old rows so the table doesn’t grow unbounded.

Retries and backoff

If your endpoint fails (non-2xx response, timeout > 10s, connection refused), the failed delivery is redelivered by Azure Service Bus under its default policy: exponential backoff in the seconds-to-tens-of-seconds range, up to a small handful of attempts (typically 5). After Service Bus exhausts retries, the message lands in a dead-letter queue and we stop trying for that specific event. Note this is per-event retry, not endpoint-level retry. A handful of retries over ~minutes — not a long-tail schedule that spans hours.

Auto-pause on consecutive failures

What protects you from a sustained outage is a separate mechanism: three consecutive event-delivery failures in a row pauses the subscription. The counter increments on each failed event and resets to zero on any 2xx response.

State	Trigger	Effect
Active, healthy	2xx response	`consecutiveFailures` resets to 0
Active, one failure	first non-2xx	`consecutiveFailures = 1`; next event still attempted
Active, two failures	second non-2xx in a row	`consecutiveFailures = 2`; one more chance
Paused	third non-2xx in a row	`Status` → `paused`; no new events delivered until an admin resumes

When a subscription auto-pauses:

The admin grid renders a Paused badge on the row (amber dot + “Paused” pill).
FPT logs a Warning to elmah so internal monitoring catches it.
In a near-term release, the gym owner will get an email notification with a link to investigate. (v1.1 ships the admin-visible state + log signal; SendGrid email wiring lands in v1.1.x.)
An admin clicks the resume button (▶) in the actions column to flip back to active.

A brief partner blip (a few seconds) triggers Service Bus retries and you recover transparently — your consecutiveFailures counter never gets to 1 if any of the SB redeliveries succeed. A multi-minute outage probably trips one event’s full SB retry budget; if the very next event also fails, you’re at 2; the one after that puts you in paused state. The auto-pause is intentionally conservative so a healthy long-running partner isn’t paused by a single bad deploy window.

Why not the long-tail retry schedule (1min, 5min, 30min, …)? In practice, partners who are down for an hour usually stay down for hours, and rapid auto-pause + visible admin signal beats invisible queueing. The 3-failure threshold is also the contract Close.com uses for the same reason.

What counts as success vs failure

Response	Meaning
2xx (200–299)	Success. We won’t send this event again.
4xx (400–499)	Permanent failure. We don’t retry — your endpoint indicated the request itself is unprocessable. Logs only.
5xx (500–599)	Transient failure. Retry on the schedule above.
Timeout (>30s)	Transient failure. Retry on the schedule above.
Connection refused / DNS failure	Transient failure. Retry.
TLS handshake failure	Permanent failure. We don’t retry — typically misconfigured certificates. We email the subscription owner.

Don’t return 4xx for transient issues (database locked, downstream API slow). 4xx tells us “this event is permanently bad” — we’ll skip retries and drop it. For “try again later” semantics, return 5xx or just time out.

Ordering

We don’t guarantee ordering across events. If a contact’s status flips Lead → Member → VIP within 200ms, you might receive the two contact.status_changed events in either order. Plan for this:

Sort by eventTimestamp when order matters
Coalesce at your end if you only care about the latest state (look up the contact’s current state at receipt time via your own data, not the event)
Don’t make decisions from a single event’s previousX → newX chain if multiple events of that type can fire close together — the previous values you see may be stale

Why these rules exist

Each one prevents a specific support ticket we’ve seen with other webhook integrations:

Rule	Ticket it prevents
At-least-once + dedupe by eventId	”Our customer was charged twice / had two contacts created”
10-second timeout, return 2xx fast	”All our retries are firing because our endpoint is slow”
4xx vs 5xx distinction	”Our endpoint returned 400 for a transient DB error and we never got the event again”
Auto-pause threshold of 3 (not 1) consecutive failures	”A single transient blip auto-disabled our subscription overnight”
No ordering guarantee	”Events arrived out of order and corrupted our state”
HMAC signature verification	”How do we know it’s really Fit Pro Tracker?”

Getting started

Webhooks

Event catalog

At-least-once delivery

Idempotency on `eventId`

Minimal Postgres example

Retries and backoff

Auto-pause on consecutive failures

What counts as success vs failure

Ordering

Why these rules exist

​At-least-once delivery

​Idempotency on eventId

​Minimal Postgres example

​Retries and backoff

​Auto-pause on consecutive failures

​What counts as success vs failure

​Ordering

​Why these rules exist

At-least-once delivery

Idempotency on `eventId`

Minimal Postgres example

Retries and backoff

Auto-pause on consecutive failures

What counts as success vs failure

Ordering

Why these rules exist