chore(zero-cache): introduce ZeroEvents for surfacing real-time events for ops/observability #4736

darkgnotic · 2025-08-14T10:24:24Z

Introduce ZeroEvents, a channel for surfacing realtime "events" for the purposes of production ops and observability.

As a monitoring signal, events differ from (1) metrics, which are periodic and limited to numeric values, and (2) logs, which are textual and generally have a multi-minute processing delay. Events, on the other hand, are designed for low latency delivery / processing with structured data payloads.

ZeroEvents are published using the Cloud Events specification, with configuration designed to facilitate integration with knative sink bindings. However, any CloudEvent routing framework can be used to route and handle the Cloud Events.

The first specific ZeroEvent type is used for publishing the status of replication. ReplicationStatusEvents are emitted:

periodically during initial sync
when replication begins (i.e. indicating that initial sync has finished)
whenever a schema change is processed

The events themselves contain the replicated schema data, which facilitates verification by the operator that the expected tables and columns are being replicated.

…s for ops

vercel · 2025-08-14T10:24:30Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Project	Deployment	Preview	Comments	Updated (UTC)
replicache-docs	✅ Ready	Preview	Comment	Aug 14, 2025 10:35am
zbugs	✅ Ready	Preview	Comment	Aug 14, 2025 10:35am

github-actions · 2025-08-14T10:25:31Z

Bencher Report

Branch	darkgnotic/zero-events
Testbed	Linux

🚨 2 Alerts

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Lower Boundary (Limit %)
src/client/zero.bench.ts > pk compare > pk = N	Throughput operations / second (ops/s) x 1e3	📈 plot 🚷 threshold 🚨 alert (🔔)	21.51 ops/s x 1e3 (-24.64%) Baseline: 28.54 ops/s x 1e3	24.50 ops/s x 1e3 (113.90%)
src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers)	Throughput operations / second (ops/s) x 1e3	📈 plot 🚷 threshold 🚨 alert (🔔)	2.35 ops/s x 1e3 (-8.12%) Baseline: 2.56 ops/s x 1e3	2.47 ops/s x 1e3 (105.16%)

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%)	Lower Boundary operations / second (ops/s) x 1e3 (Limit %)
src/client/custom.bench.ts > big schema	📈 view plot 🚷 view threshold	366.99 ops/s x 1e3 (-4.54%) Baseline: 384.45 ops/s x 1e3	353.76 ops/s x 1e3 (96.40%)
src/client/zero.bench.ts > basics > All 1000 rows x 10 columns (numbers)	📈 view plot 🚷 view threshold	1.61 ops/s x 1e3 (-1.04%) Baseline: 1.63 ops/s x 1e3	1.59 ops/s x 1e3 (98.58%)
src/client/zero.bench.ts > pk compare > pk = N	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	21.51 ops/s x 1e3 (-24.64%) Baseline: 28.54 ops/s x 1e3	24.50 ops/s x 1e3 (113.90%)
src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers)	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	2.35 ops/s x 1e3 (-8.12%) Baseline: 2.56 ops/s x 1e3	2.47 ops/s x 1e3 (105.16%)

🐰 View full continuous benchmarking report in Bencher

github-actions · 2025-08-14T10:26:38Z

Bencher Report

Branch	darkgnotic/zero-events
Testbed	Linux

Click to view all benchmark results

Benchmark	File Size	Benchmark Result kilobytes (KB) (Result Δ%)	Upper Boundary kilobytes (KB) (Limit %)
zero-package.tgz	📈 view plot 🚷 view threshold	1,280.67 KB (+0.37%) Baseline: 1,276.00 KB	1,301.52 KB (98.40%)
zero.js	📈 view plot 🚷 view threshold	207.51 KB (0.00%) Baseline: 207.51 KB	211.66 KB (98.04%)
zero.js.br	📈 view plot 🚷 view threshold	58.11 KB (0.00%) Baseline: 58.11 KB	59.28 KB (98.04%)

🐰 View full continuous benchmarking report in Bencher

arv

LGTM

arv · 2025-08-15T08:45:38Z

packages/zero-events/package.json

+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "module": "true",


Not sure what this is?

sorry for the lack of context. i which we could have comments in package.json

eventually i intend to publish this package alone as something like rocicorp/zero-events, so that production / operational code (i.e. cloudzero) can import just the ZeroEvent interfaces and not full zero package.

but "module": "true", is not part of package.json spec as far as I can tell.

Oh, then it looks like I just cargo-culted it from another (mistaken) package.json 😅

arv · 2025-08-15T08:46:12Z

packages/zero-events/package.json

+  },
+  "devDependencies": {
+    "@rocicorp/eslint-config": "^0.7.0",
+    "@rocicorp/prettier-config": "^0.3.0",


vitest?

It does look like this package has no test so you should remove the test scripts

arv · 2025-08-15T08:52:11Z

packages/zero-events/src/status.ts

+export const replicationStatusEventSchema = statusEventSchema.extend({
+  type: v
+    .literal('zero/events/status/replication/v1')
+    .assert(v => v.startsWith(STATUS_EVENT_PREFIX)),


I'm not sure what this assert does here. Seems like it is more of a "safety check". You can probably achieve the same thing using typescript satisfies

Okay, I got something to work using template literals, so we can avoid the somewhat cryptic use of assert.

arv · 2025-08-15T09:23:10Z

packages/zero-cache/src/observability/events.ts

+  void publishFn(lc, event);
+}
+
+export async function publishCriticalEvent<E extends ZeroEvent>(


is this just the same as publishEvent? What is the usecase?

Most events are published fire-and-forget in the steady state.

But for errors, particularly when the server is going to be shut down, I expose this method that is intended to be awaited to make sure that the error event gets published before the server shuts down. This is to guarantee that the error makes it out of the server before it dies.

arv · 2025-08-15T09:26:07Z

packages/zero-cache/src/observability/events.ts

+  return errorDetails;
+}
+
+const pathUnused = {push: () => {}, pop: () => {}};


Path unused can also be an array.

Hmmm ... can you clarify? I'm looking at this method:

mono/packages/shared/src/json.ts

Line 184 in 842c807

export function isJSONValue(v: unknown, path: Path): v is JSONValue {

The usage of isJSONValue usually takes an array. Array implements Path :-)

Aah, I see. Roger.

arv · 2025-08-15T09:31:04Z

packages/zero-cache/src/services/change-source/pg/initial-sync.pg-test.ts

+            component: 'replication',
+            stage: 'Initializing',
+            status: 'OK',
+            description: /Copying \d+ upstream tables at version \w+/,


TIL, you can use RegExps in these

arv · 2025-08-15T09:32:04Z

packages/zero-cache/src/services/change-source/pg/initial-sync.pg-test.ts

+            type: 'zero/events/status/replication/v1',
+            component: 'replication',
+            stage: 'Initializing',
+            status: 'OK',


Why doesn't this have the description?

This is the first status we publish when we start initial sync, before we've touched upstream at all. I couldn't think of any useful information to embellish here ... seems like "Initializing" is enough? It will be followed up with the "Copying ... " event pretty quickly. 🤷

arv · 2025-08-15T09:33:08Z

packages/zero-cache/src/services/change-source/pg/initial-sync.ts

+        lc,
+        'Indexing',
+        `Creating ${indexes.length} indexes`,
+        5000,


What is 5000? and how is it exposed/tested?

It's the interval between continuous updates. So during initial sync, events will be published every 5 seconds to surface the latest state of initial sync. In the PG case, it's mainly the replica size that will change.

It isn't tested. 🤫

arv · 2025-08-15T09:39:01Z

packages/zero-cache/src/services/replicator/replication-status.ts

+  publishEvent,
+} from '../../observability/events.ts';
+
+const byKeys = ([a]: [string, unknown], [b]: [string, unknown]) =>


I'm not sure if this is hot but for hot code do not destructure arrays

Not super hot ... events should be infrequent. But I'll remember this. Thank you! 🙏

arv · 2025-08-15T09:40:36Z

packages/zero-cache/src/services/replicator/replication-status.ts

+    interval = 0,
+  ): this {
+    this.stop();
+    publishEvent(


would it make more sense to make publishEvent a property that is passed in instead of overriding it for tests?

that would be cleaner ... bit it would be a lot of plumbing to get to this object from the top level of the test. i chose the lazy / less invasive route. i think it's "okay" because tests are run serially per process ... but please correct me if i'm mistaken.

darkgnotic

Thank you @arv!

darkgnotic · 2025-08-17T04:11:55Z

packages/zero-cache/src/observability/events.ts

+  void publishFn(lc, event);
+}
+
+export async function publishCriticalEvent<E extends ZeroEvent>(


Most events are published fire-and-forget in the steady state.

But for errors, particularly when the server is going to be shut down, I expose this method that is intended to be awaited to make sure that the error event gets published before the server shuts down. This is to guarantee that the error makes it out of the server before it dies.

darkgnotic · 2025-08-17T04:13:17Z

packages/zero-cache/src/observability/events.ts

+  return errorDetails;
+}
+
+const pathUnused = {push: () => {}, pop: () => {}};


Hmmm ... can you clarify? I'm looking at this method:

mono/packages/shared/src/json.ts

Line 184 in 842c807

export function isJSONValue(v: unknown, path: Path): v is JSONValue {

darkgnotic · 2025-08-17T04:14:35Z

packages/zero-cache/src/services/change-source/pg/initial-sync.pg-test.ts

+            type: 'zero/events/status/replication/v1',
+            component: 'replication',
+            stage: 'Initializing',
+            status: 'OK',


This is the first status we publish when we start initial sync, before we've touched upstream at all. I couldn't think of any useful information to embellish here ... seems like "Initializing" is enough? It will be followed up with the "Copying ... " event pretty quickly. 🤷

darkgnotic · 2025-08-17T04:16:02Z

packages/zero-cache/src/services/change-source/pg/initial-sync.ts

+        lc,
+        'Indexing',
+        `Creating ${indexes.length} indexes`,
+        5000,


It's the interval between continuous updates. So during initial sync, events will be published every 5 seconds to surface the latest state of initial sync. In the PG case, it's mainly the replica size that will change.

It isn't tested. 🤫

darkgnotic · 2025-08-17T04:17:04Z

packages/zero-cache/src/services/replicator/replication-status.ts

+  publishEvent,
+} from '../../observability/events.ts';
+
+const byKeys = ([a]: [string, unknown], [b]: [string, unknown]) =>


Not super hot ... events should be infrequent. But I'll remember this. Thank you! 🙏

darkgnotic · 2025-08-17T04:19:04Z

packages/zero-cache/src/services/replicator/replication-status.ts

+    interval = 0,
+  ): this {
+    this.stop();
+    publishEvent(


that would be cleaner ... bit it would be a lot of plumbing to get to this object from the top level of the test. i chose the lazy / less invasive route. i think it's "okay" because tests are run serially per process ... but please correct me if i'm mistaken.

darkgnotic · 2025-08-17T04:21:00Z

packages/zero-events/package.json

+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "module": "true",


sorry for the lack of context. i which we could have comments in package.json

eventually i intend to publish this package alone as something like rocicorp/zero-events, so that production / operational code (i.e. cloudzero) can import just the ZeroEvent interfaces and not full zero package.

darkgnotic added 2 commits August 14, 2025 18:23

chore(zero-cache): introduce ZeroEvents for surfacing real-time event…

038ce2c

…s for ops

publish status events for single-node mode too

1bef4a4

darkgnotic requested a review from cesara August 14, 2025 10:24

darkgnotic requested a review from arv August 14, 2025 10:24

vercel bot deployed to Preview – replicache-docs August 14, 2025 10:24 View deployment

vercel bot deployed to Preview – zbugs August 14, 2025 10:25 View deployment

update test

0b2e744

darkgnotic enabled auto-merge (squash) August 14, 2025 10:34

vercel bot deployed to Preview – replicache-docs August 14, 2025 10:35 View deployment

vercel bot deployed to Preview – zbugs August 14, 2025 10:35 View deployment

darkgnotic merged commit b0f6be9 into main Aug 14, 2025
13 of 15 checks passed

darkgnotic deleted the darkgnotic/zero-events branch August 14, 2025 10:55

arv reviewed Aug 15, 2025

View reviewed changes

darkgnotic commented Aug 17, 2025

View reviewed changes

darkgnotic mentioned this pull request Aug 18, 2025

chore(zero-cache): readability fixes #4759

Merged

chore(zero-cache): introduce ZeroEvents for surfacing real-time events for ops/observability #4736

chore(zero-cache): introduce ZeroEvents for surfacing real-time events for ops/observability #4736

Uh oh!

Conversation

darkgnotic commented Aug 14, 2025

Uh oh!

vercel bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚨 2 Alerts

Uh oh!

github-actions bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

arv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darkgnotic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

vercel bot commented Aug 14, 2025 •

edited

Loading

github-actions bot commented Aug 14, 2025 •

edited

Loading

github-actions bot commented Aug 14, 2025 •

edited

Loading