Durable reminders with Temporal

A proactive assistant has to remember things for you: tell me when the match starts, remind me after this song, nudge me tomorrow morning. The obvious build is a table of pending watches, a cron job that polls it, a lock so two pollers don't fire the same reminder, and retry bookkeeping for when the push service is down. Every piece is simple. Together they are where bugs live.

When I built a watcher service for an assistant platform, we ran it on a workflow service built on Temporal instead. This post explains how that works and why it fits.

Workflows, activities and workers

Temporal splits your code into two kinds:

• Workflows hold the orchestration logic: wait, check, then notify. Workflow code must be deterministic, so no random numbers, no direct clock reads and no network calls.
• Activities do the side effects: call an API, send a push notification, read a database. They can fail, and Temporal retries them with the policy you set.

Your own services run as workers. They poll a task queue on the Temporal server, run your code and report back. The server never runs your code; it stores what happened.

The trick: event history and replay

Every step a workflow takes is recorded as an event: activity scheduled, activity completed, timer started, timer fired, signal received. If a worker crashes halfway, another worker picks the workflow up and runs the code again from the top. Steps that already happened are answered from the history instead of being repeated: a finished activity returns its recorded result, and a timer that already fired returns at once.

The code ends up exactly where it stopped, local variables and all. That is why workflow code must be deterministic: the replay has to make the same decisions as the first run.

One workflow per watch

A watch maps neatly onto one workflow, started with a stable ID such as watch-{user}-{topic}:

@workflow.defn
class WatchWorkflow:
    def __init__(self):
        self.event = None

    @workflow.signal
    def on_event(self, event):
        self.event = event

    @workflow.run
    async def run(self, watch):
        while True:
            await workflow.wait_condition(
                lambda: self.event is not None, timeout=watch.check_every)
            if self.event and watch.matches(self.event):
                await workflow.execute_activity(
                    send_reminder, watch,
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=RetryPolicy(maximum_attempts=5))
                return
            self.event = None

That loop gets a lot for free:

• Durable timers. Waiting an hour or three days costs nothing. It is a row in Temporal's database, not a sleeping thread, and it fires even if every worker restarted in between.
• Signals. Other systems push events into a running workflow by its ID. Queries read its state without changing it.
• Retries. If the push service is down, the activity retries with backoff. The workflow just sees success or a final failure.
• No duplicates. Starting a second workflow with the same ID is rejected, so a double click never means a double reminder.

What it costs

Temporal is not free of trade-offs. You run a server and its database, or pay for Temporal Cloud. Changing workflow code needs care, because running workflows replay the new code; you mark changes with versioning so old and new runs each take the right path. Long-lived loops should continue as new now and then so their history stays small. And it is not an event bus: Kafka still moves the firehose of events, while Temporal runs the durable, stateful process that waits for the few that matter.

For reminders, the trade is worth it. The polling, locking, retry and recovery code I would otherwise write and debug simply isn't there.