What Is Cadence Workflow, and Why Should You Care?
Cadence Workflow is a fault-oblivious, stateful code platform developed at Uber and now a CNCF sandbox project. Think of it as a supervisor for your distributed processes — you write workflow logic as regular code, and Cadence makes sure it completes even when servers crash, networks partition, or processes get killed mid-execution.
Imagine you are building an order processing system. An order comes in, you charge a card, reserve inventory, send a confirmation email, and update the warehouse. If any step fails halfway, you need retries, timeouts, compensation logic, and a way to track where things are. Cadence handles all of that for you. You write the happy path as a simple sequence of function calls, and Cadence takes care of durability, retries, and state tracking behind the scenes.
Teams at companies like Uber, Snap, Stripe, and DoorDash use Cadence to run business-critical workflows — order processing, payment reconciliation, data pipeline orchestration, and user onboarding — at massive scale. If you have ever wondered how big platforms handle multi-step processes without everything falling apart, this is one of the answers.
v1.4.1 is a meaty release. The headline feature is Cadence Schedules reaching general availability — a first-class scheduling primitive that finally replaces hand-rolled cron workflows. But that is not all: you also get Active-Active domain support for MySQL, domain-level multi-tenancy isolation primitives, and a metrics migration from expensive timer-based metrics to OpenTelemetry-aligned histograms. Let us get into it.
What Is New in v1.4.1
Cadence Schedules Is Generally Available
For years, Cadence users who needed to run workflows on a schedule had two options: use an external cron job that kicks off a workflow execution, or build a self-perpetuating workflow with ContinueAsNew and a timer. Both approaches work, but both are exactly the kind of thing Cadence is supposed to abstract away.
With Cadence Schedules reaching GA, you now get a first-class scheduling primitive built directly into the platform. The full API — Create, Describe, Update, Delete, Pause, Unpause, List, and Backfill — is implemented across the type system, Thrift transport, frontend handlers, and CLI.
The scheduler runs as a per-domain durable workflow with ContinueAsNew, which means schedules survive server restarts and cluster failures just like any other workflow. You get configurable overlap policies (SKIP, CONCURRENT, BUFFER, CATCH_UP_ALL) so you can control what happens when a scheduled fire overlaps with a still-running execution. Missed fire times? Catch-up logic handles that. Need to replay a specific window? Backfill with per-fire override policies has you covered.
// Create a schedule that fires every 5 minutes
ScheduleClient.create("nightly-reconciliation", {
spec: {
cron: "*/5 * * * *"
},
action: {
startWorkflow: {
workflowType: "ReconciliationWorkflow",
input: { region: "us-east-1" }
}
},
overlapPolicy: "SKIP",
catchupWindow: "5m"
})The scheduler worker is disabled by default and can be enabled per-domain via dynamic config. Schedule search attributes are included in the default visibility key set automatically — no manual index changes needed. See the announcement blog post and the concepts documentation for the full rundown.
Active-Active Domains Get MySQL Support
If you are running Cadence in a multi-region setup, Active-Active is how you keep workflows running in two data centers simultaneously — not active-passive with failover, but genuinely active in both. Previously, Active-Active only worked with Cassandra. With v1.4.1, MySQL is now a supported backend for Active-Active domains.
The release also improves the failover workflow to support passing the specific set of cluster attributes that should be failed over, giving operators finer-grained control during regional incidents. And a security-minded fix makes it impossible to convert a domain to active-active via the FailoverDomain RPC — a change that prevents an operational footgun.
Domain Multi-Tenancy: Isolation Below the Domain Level
Cadence has always enforced rate limiting, task prioritization, and fair scheduling at the domain level to prevent noisy-neighbor problems between customers. But what happens when a platform team consolidates multiple use cases into a single domain? Historically, nothing good — those use cases had no isolation between them.
Domain Multi-Tenancy (still in active development) introduces task-list-level isolation primitives that give you the same guarantees within a domain that domain-level isolation provides between domains. The key capabilities in v1.4.1:
- Hierarchical weighted round-robin (IWRR) scheduler — fair task scheduling across task lists within a domain, so no single task list can monopolize worker capacity.
- Task list nice values — priority control between task lists, letting you express that the payments task list matters more than the analytics one.
- Per-task-list rate limiting — applied to both regular and sticky polls, preventing one task list from starving others sharing the same domain.
- Task list name and kind in history — stored in history events and transfer tasks to enable routing decisions.
- Per-task-list latency metrics — history task latency tracked at the task list level for granular observability.
Note: this feature requires a Cassandra schema upgrade. Transfer task and timer task tables gain new columns for task list name and kind. Apply migrations before rolling out.
Timer to Histogram Metric Migration
Metrics infrastructure is expensive, and Cadence has been paying a tax it no longer wants to pay. The project is migrating from Tally’s costly Timer metrics to ExponentialHistogram — aligning with Prometheus and OpenTelemetry standards while significantly reducing infrastructure costs.
v1.4.1 enables dual-emission for all timer metrics, meaning you can emit both legacy timers and new histograms simultaneously during the transition. New GaugeMigration and CounterMigration frameworks round out the migration toolset. The behavior is controlled via the histograms block in your service config YAML:
histograms:
default: timer # global default: "timer", "histogram", or "both"
names:
task_latency: true # true = emit histogram, false = suppress
persistence_latency: trueIf the histograms block is not configured, behavior is identical to previous releases — only timers are emitted. The recommended rollout path: start with default: timer, opt individual metrics into both to validate, switch to default: both when confident, and eventually set default: histogram to drop legacy timers entirely.
Legacy timers will eventually be removed. The histograms config block is transitional and will become validation-only once the migration is complete.
Performance and Stability Improvements
Beyond the headline features, v1.4.1 ships several performance wins that operators will appreciate:
- O(1) task list manager lookup by name (#7733) — eliminates a scalability bottleneck in high-task-list-count deployments.
- Rate-limiter token waste fix (#7977) — significantly reduces idle CPU usage under low task rates. If you have seen Cadence workers spinning CPU on empty queues, this is your fix.
- Reduced task processing latency (#8130) — history queue processors are now notified before shard lock release.
- Automatic mutable state corruption detection and repair (#7850) — Cadence can now self-heal corrupted mutable state without operator intervention.
- Runtime-tunable batcher RPS (#7824) — batcher RPS and concurrency can be adjusted at runtime via signals, no restart needed.
Bug Fixes Worth Noting
Two separate getTasksPump deadlocks in the matching engine are resolved (#7855, #7930). The replicator no longer crashes the process on panic — it converts panics to errors instead (#8063), improving multi-region cluster stability. And the Kafka default protocol version is fixed from 0.10.2.0 to 2.1.0 (#7890), which resolves a startup crash with Sarama v1.45+ against modern Kafka clusters.
CLI and Observability Updates
The CLI gains concurrency_limit support (#8028), operational dynamic config commands — get, update, delete (#8101), and --latest_time support for the DecisionCompletedTime reset type (#8151). On the observability front, Prometheus label normalization for authorization latency and cache metrics makes dashboards cleaner (#8215, #8120).
Infrastructure Improvements
- Domain audit log support for MySQL and SQLite.
- Region-specific S3 access points in archival (#8015) — critical for compliance-sensitive multi-region deployments.
- Custom Docker config file support (#8155).
That is v1.4.1 in a nutshell — a release that proves Cadence is not just maintaining, it is building the primitives that platform teams need to run workflows like infrastructure.
