LeadRefinery — How the Platform Works

The machines that run it

The platform runs on rented hosts. It used to run on one host, shared with fifteen containers from other projects, and that host ran out of memory twice in three days — killing the workflow tracker both times. A separate, purpose-built set of hosts is now live. This is what changed, what each part does, and what it costs.

Two panels. Before: one host running every part of the platform, which also
                 built its own software and shared the machine with fifteen containers from other
                 projects, at about 38 dollars a month. It ran out of memory on 31 August and 2
                 September, and the kernel killed the workflow tracker both times. Now: four hosts
                 with one job each — an app host at 20.49 dollars, a calling host at 20.49, a
                 record host with dedicated processing power at 50.99, and a build-and-rehearse
                 host in Europe at 41.99 — plus storage volumes, shared file storage, and network
                 addresses at 26.22. Total 177.18 dollars a month, across two regions, over the
                 150-dollar-a-month ceiling approved for this build.
The build-and-rehearse host is in Europe, where the same computing power costs far less, and it never places a real call. The three hosts that touch a real client stay in the United States. The app host was resized larger and three storage volumes were added after the first build, which is why the total now passes the original budget.

What each host does now

Host What runs on it Region $/month
The app hostThe screens your team uses, and the event reports the voice provider and the carrier send after each callUnited States37.49
The calling hostPlaces every call — the safety checks, the dial itself, and writing down what happenedUnited States20.49
The record hostThe permanent record, and the tracker for every call and lead journey in progress. Dedicated, not shared, processing powerUnited States50.99
The build-and-rehearse hostBuilds every release and rehearses it fully. Also runs the practice environmentEurope41.99
Storage and addressesThree storage volumes, recordings and files in a shared bucket, plus the fixed network addresses each host answers on—26.22
Total177.18

The two regions are a deliberate choice, not an accident. The same computing power costs three to five times more in the United States than it does in Europe. Placing a real call needs a host physically close to the client's own market. Building software and rehearsing it needs no such thing, so that work moved to the cheaper region, and no real call is ever placed from it.

The build approved a ceiling of 150 dollars a month. The first built hosts cost 142.35 dollars, leaving 7.65 dollars of spare room. A later resize of the app host, plus three storage volumes added for growing data, used up that spare room and more. The running total is now 177.18 dollars a month, above the original ceiling. A person has not yet re-approved a higher one.

Why the old host fell over, and what replaced it

Date Cause Result
31 AugBuilding new software and the database competed for memory on one hostThe kernel killed the workflow tracker
2 SepThe same, during a deployThree deploys cancelled in one morning

The fix was not a bigger single host. It was to stop building software on a host that also serves clients at all. A separate host now builds and rehearses every release, and production only ever receives a finished, already-tested copy — so a deploy can never again compete with the database for memory.

What it costs as volume grows

Calls / month What changes $/month
Everything on one host, as before—38
Today's built hostsThree production hosts, one job each, plus the storage volumes added since177.18
500,000 calls a month (estimated)A second app host and calling host for redundancy, a bigger record hostabout 175, before adding a storage volume

Retell and Telnyx costs for the calls themselves are orders of magnitude above every figure on this page. The computers decide whether calling can happen safely at all; they are not what makes calling expensive.

Two environments

Stage Production
Hosts1, everything co-located3, one job each
Shard count4 — fine, nobody measures stage512 — confirmed live, by counting the record host's own rows
RedundancyNoneNone yet; a second app and calling host are the planned next step at higher volume
RetentionWorkflow history: 3 days. Rehearsal rows: 7 days by default, now adjustableWorkflow history: 7 days. The compliance record is kept separately, for 10 years
Builds software?Yes — every release, plus the practice environmentNo — only ever receives an already-built, rehearsed copy
$/month41.99100.36

The shard count, which cannot be changed later

Four rows showing how each component scales. The app host and the calling
                 host scale horizontally with no practical limit, because the workflow tracker
                 hands work to whichever host is listening. The record host scales vertically by
                 resizing to a bigger size, a few minutes and reversible, then a read replica. The
                 counters store and the file bucket are not a concern. The fourth row, highlighted,
                 is the workflow tracker: its shard count is fixed when the cluster is first
                 created and cannot be changed afterwards, so raising it means a new cluster.
                 Production runs 512 shards; development and staging still use 4.
The instinct is that the record host is the scaling problem. Measured, it is not.
Shards Workflow start, p99
4 — used in development and staging524 msThrew an error under a 100,000-lead rehearsal
512 — running in production216 ms~293 open journeys per shard at 150,000 leads
2048211 msNo gain over 512, real idle cost

The new production cluster was built with 512 shards from its first day, verified three ways: the count each shard reports, the server's own version, and a direct count of the database's shard rows. Nothing had to be migrated, because the correction happened before any real client call ran through it. Correcting it later, after real calls exist, would mean migrating them.

How long a record is kept

The workflow tracker holds a call's or a lead's progress only while it is useful for running the system — seven days after it finishes, then the platform deletes it. This is not the permanent record. The ten-year compliance record that an auditor can ask for lives in the database, not in the workflow tracker, so bounding this short-lived copy costs nothing that must be kept. Before this bound was set, the old host's tracker had grown larger than the compliance database itself, purely from its own operating exhaust.

What is not solved yet

  • Recordings sit in European storage, because no United States region was available at the time. Whether a recording of a call to a person in the United States must stay in the United States is a question that needs an answer before recordings are kept at real volume.
  • The health and performance dashboards run on the build-and-rehearse host, in Europe — a deliberate saving, but it means those dashboards depend on a host in a different region from the one placing the calls they describe.
  • The 500,000-calls-a-month tier does not fit inside today's spending limit; reaching it needs a further, separately approved increase.
  • The running total already passes the ceiling approved for this build, before that higher tier is even reached. A bigger app host and three new storage volumes used up the spare room a growing platform was going to need anyway.