When the payment bug is in your vendor's product, not your code.

The problem

eback is a production platform where nonprofits across Latin America and the US register, pass a multi-step verification pipeline, get a public profile, and receive online donations. Two codebases, a Next.js web app on Firebase and a 2nd-gen Cloud Functions backend, with GoHighLevel as the CRM behind all of it. Around 72 live organizations and a live donation pipeline. I inherited it from a previous developer.

An end-to-end walkthrough of register → publish → invite → donate surfaced roughly 25 real defects. Three were quietly serious:

• Paid donations were silently dropped. The CRM's order webhook payload diverged from what the code expected (items vs lineItems, amount vs totalAmount, completed vs paid). Valid donations failed to parse and never became records. Donors got charged. The org saw nothing.
• CRM sync failed silently. No flag, no alert, nobody knew. The CRM returned a 400 on duplicate contacts that the code didn't handle, so sync aborted mid-way and opportunities could duplicate on retry.
• The CRM was data-poor. Dozens of custom fields existed and were never populated: legal, governance, financial, compliance, board members, key executives.

Then, once the plumbing was trustworthy, the real problem surfaced. Donations were still failing, and not because of our code. The CRM vendor's own donation iframe had a race condition: their client script populated the tracking ID asynchronously, so a donor who submitted quickly sent a null tracking ID and the vendor's backend rejected the order with a 422 before Stripe was ever contacted. It affected every organization's embed. It was not fixable from our side. And each of the ~30 organizations had that form hand-pasted into their page individually, so they were inconsistent, unstyled, and impossible to change centrally.

What I did

First, made the existing pipeline trustworthy. One shared webhook normalizer across order-created and order-updated, so every paid donation becomes a record, updates the org's totals, and notifies staff. Duplicate-contact 400s handled by find-and-adopt. Opportunity creation made idempotent. Silent failures flagged on the org record, with an audit callable that reconciles any org's Firestore-to-CRM state on demand. Field coverage went from a handful to 57 contact and 59 opportunity fields, then a MERGE-safe backfill pushed corrected mappings into 65 live orgs without wiping anything.

The one architectural change that mattered most in that phase: decoupling the pipeline stage change from the custom-field writes. The CRM rejects an entire field payload if any single value is invalid, so the stage now always advances and the customer-facing org ID always lands, even when a non-critical field write fails. A bad field can never stall a nonprofit's progress.

Then replaced the payment path rather than waiting on the vendor. A single native donation form for every organization: pick an amount, choose one-time or monthly, pay through Stripe Checkout, land on a branded thank-you page. Same look everywhere, fully under our control, every donation recorded. Because Stripe is reached directly, the flow is structurally immune to the vendor's race condition.

Shipping a payment rewrite onto a live donation platform is the risky part, so it went out behind controls:

• A per-organization boolean read at page render. Reverting one org is a field flip, no deploy and no code change.
• A preview route that forces the native form on for a single request without touching any organization's live page, so the whole flow could be QA'd in production conditions at zero risk.
• Phased rollout, organization by organization, rather than a flag day.

Built out what the new payment path made possible. A disaster-relief campaign line (urgent acts) with image submissions, an approve/reject flow that syncs to the CRM, and a per-campaign monthly-giving toggle, so a long-running crisis response can accept recurring support while ordinary campaigns stay one-time by default. Donation notifications now email the receiving organization rather than only the donor, with a separate three-way sequence for relief campaigns that keeps the donor, eback, and the beneficiary org all informed. Stripe receipts wired in.

Hardened and modernized the platform underneath. Closed a Firestore security-rules bypass where a stray true short-circuited a rule. Wrote real rules for pending donations. Moved rules deployment into CI, then onto keyless authentication so no long-lived service-account key sits in the pipeline. Upgraded the Cloud Functions runtime to Node 22 and firebase-admin two major versions. Fixed an impersonation flow that could trap an admin behind an email-verification wall, and a set of donor-visibility rules that returned permission errors on the donor's own donation history.

The detail that mattered: recognizing that the bug was in the vendor's product and acting on it. The instinct is to keep debugging your own integration, or to file a ticket and wait. We did file it. But the honest read was that a broken payment path on a donation platform is an existential bug, the vendor's fix timeline was not ours to control, and routing around their payment layer entirely was both faster and left us permanently less exposed. Knowing when to stop fixing someone else's software is an engineering decision, not an escalation.

Outcome

• One native donation flow replaced ~30 hand-pasted per-org forms, consistent and centrally controllable, no longer exposed to the vendor's race condition.
• Recurring monthly giving live, including per-campaign control for disaster relief.
• Receiving organizations are notified the moment a donation lands, instead of finding out by checking.
• CRM sync is trustworthy: failures visible and recoverable, 65 live orgs backfilled, a field glitch can no longer block a nonprofit's journey.
• A security-rules bypass closed, rules deployment automated and keyless, runtime and admin SDK modernized.
• 127 merged PRs across the two repos between May and August 2026, every data operation dry-run before apply, with a documented rollback lever throughout.