RVM Provider Failover System

The Three Provider Modes

Admins set the mode (and the failover threshold) on the RVM Provider page, reachable from the SuperAdmin sidebar under "Telecom & Numbers".

The Source of Truth: RVMProviderConfig

There is exactly one config record, keyed config_key: "master". Every send path and the failover guard read it. Fields:

All reads/writes go through functions/manageRVMProvider (admin-only). A manual change always sets failover_triggered: false — the admin is back in control.

How a Drop Gets Routed

1. Frontend calls sendOneOffDrop as it always has (no UI change).
2. sendOneOffDrop reads RVMProviderConfig. If active_provider === "slybroadcast", it forwards the request (with the user'sAuthorization header) to sendSlybroadcastDrop and returns. Drop Cowboy code never runs.
3. Otherwise ("dropcowboy" or "both"), the existing Drop Cowboy path runs unchanged — DC is always primary.
4. If the config can't be read for any reason, it falls through to Drop Cowboy (safe default).

The Slybroadcast Path (Isolated)

functions/sendSlybroadcastDrop mirrors the Drop Cowboy one-off structure but delivers via Slybroadcast's hosted-MP3 gateway (vmb.json.php). It independently:

• Enforces the same subscription / free-tier / monthly drops-limit rules as the DC path.
• Normalizes audio (CloudConvert → MP3) when the file isn't already mp3/wav/m4a.
• Generates ElevenLabs TTS from a voice clone + script, or uses an uploaded audio file.
• Rotates the customer's assigned sender numbers (excluding BYON) for caller ID.
• Logs an RVMDelivery record tagged provider: "slybroadcast".
• Registers a disposition callback (c_dispo_url) carrying the record's foreign_id.

functions/slybroadcastWebhook receives that disposition, matches the record byforeign_id (falling back to session id, then a pending phone match), flips its status to success/failure, and increments the customer's drops_used on a fresh success — exactly mirroring the DC "you only pay for what lands" billing model.

Automatic Failover (DC → Sly)

The failover guard lives inside functions/dropCowboyWebhook as a fully additive function (checkProviderFailover). It runs on every Drop Cowboy failure webhook that represents a fresh pending→terminal transition (duplicate/late webhooks are ignored, so they can't artificially pile up the streak).

1. Short-circuit immediately unless active_provider === "both" and failover_triggered is false.
2. Pull the most recent account-wide Drop Cowboy terminal deliveries; take the latest failover_threshold of them.
3. If every one of those is a failure → flip active_provider to "slybroadcast", set failover_triggered: true, stamp the time, and set updated_by: "system".
4. Email the super-admin alert list (see below). All new drops now route to Slybroadcast.
5. It stays on Slybroadcast until an admin opens the RVM Provider page and switches back — which clears the flag.

The Failover Alert

When an auto-failover fires, the guard reuses the existing critical-alert plumbing:

• Reads CriticalAlertSettings (master record) for the recipient list and the global kill switch.
• Respects the per-alert toggle: the alert key rvm_provider_failover can be turned off in the Critical Alerts UI.
• Sends through functions/sendSecOpsEmail (Resend), which logs each send to AlertLog.
• The email links straight to the RVM Provider settings page so an admin can review and switch back.

Code Map

• entities/RVMProviderConfig — the single master toggle record
• functions/manageRVMProvider — admin get/set the toggle & threshold (clears failover flag on manual change)
• functions/sendOneOffDrop — reads the toggle, routes to Sly when active, else runs DC unchanged
• functions/sendSlybroadcastDrop — isolated Slybroadcast send (audio + TTS + rotation + logging)
• functions/slybroadcastWebhook — Slybroadcast disposition postback handler
• functions/dropCowboyWebhook → checkProviderFailover() — additive auto-failover guard
• components/admin/criticalAlertsCatalog — defines the rvm_provider_failover alert
• pages/AdminRVMProvider — the toggle UI (SuperAdmin → Telecom & Numbers)
• functions/slybroadcastTest — admin diagnostic to fire one real Sly voicemail

Scope: The Toggle Governs Every Send Path

The provider toggle now controls all outbound RVM sending across the app — not just one-off drops. Each send path reads RVMProviderConfig at send time and, when active_provider === "slybroadcast", routes that drop through Slybroadcast instead of Drop Cowboy. All the shared prep (audio normalize, ElevenLabs TTS, sender rotation, throttling, delivery logging) is provider-agnostic and runs identically either way.