feat(replicache): add `zeroOption` to allow Zero to hook into Replicache events #3933

tantaman · 2025-03-06T03:28:26Z

zeroOption is a minimal interface that Replicache will call when:

pullEnd
persist
refresh
mutate

happen. Replicache will invoke

advance whenever it's main head moves forward.
getTxData to fork IVM sources whenever it needs to rebase
init on construction to populate the IVM sources

Invoking the above three methods in persist.ts, refresh.ts, replicache-impl.ts::mutate, replicache-impl.ts::maybeEndPull will happen in the next PR.

vercel · 2025-03-06T03:28:30Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
replicache-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 6, 2025 3:03pm
zbugs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 6, 2025 3:03pm

github-actions · 2025-03-06T03:30:06Z

Bencher Report

Branch	mlaw/rep-tx-data
Testbed	localhost

Click to view all benchmark results

Benchmark	File Size	Benchmark Result kilobytes (KB) (Result Δ%)	Upper Boundary kilobytes (KB) (Limit %)
zero-package.tgz	📈 view plot 🚷 view threshold	1,002.65 KB (+0.07%) Baseline: 1,001.92 KB	1,021.96 KB (98.11%)
zero.js	📈 view plot 🚷 view threshold	180.87 KB (+0.02%) Baseline: 180.83 KB	184.45 KB (98.06%)
zero.js.br	📈 view plot 🚷 view threshold	50.44 KB (-0.01%) Baseline: 50.44 KB	51.45 KB (98.03%)

🐰 View full continuous benchmarking report in Bencher

github-actions · 2025-03-06T03:30:54Z

Bencher Report

Branch	mlaw/rep-tx-data
Testbed	localhost

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) (Result Δ%)	Lower Boundary operations / second (ops/s) (Limit %)
src/client/zero.bench.ts > basics > All 1000 rows x 10 columns (numbers)	📈 view plot 🚷 view threshold	85.90 ops/s (+12.06%) Baseline: 76.66 ops/s	70.19 ops/s (81.71%)
src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers)	📈 view plot 🚷 view threshold	108.80 ops/s (+12.94%) Baseline: 96.34 ops/s	88.44 ops/s (81.29%)

🐰 View full continuous benchmarking report in Bencher

arv · 2025-03-06T09:22:21Z

packages/replicache/src/replicache-options.ts

-export interface ReplicacheOptions<MD extends MutatorDefs> {
+export interface ReplicacheOptions<
+  MD extends MutatorDefs,
+  TZeroData extends ZeroTxData = ZeroTxData,


I think it would be better to create a new interface that extends ReplicacheOptions. ReplicacheImpl can use that type. That way we do not have to change the public API of replicache.

I think you can either add one more option here:

mono/packages/replicache/src/replicache-impl.ts

Lines 388 to 391 in de356bd

constructor(

options: ReplicacheOptions<MD>,

implOptions: ReplicacheImplOptions = {},

) {

or you can extend ReplicacheImplOptions which is all internal stuff the Zero and Reflect use.

arv · 2025-03-06T13:47:01Z

packages/replicache/src/replicache-options.ts

-export interface ReplicacheOptions<MD extends MutatorDefs> {
+export interface ReplicacheOptions<
+  MD extends MutatorDefs,
+  TZeroData extends ZeroTxData = ZeroTxData,


I think you can either add one more option here:

mono/packages/replicache/src/replicache-impl.ts

Lines 388 to 391 in de356bd

constructor(

options: ReplicacheOptions<MD>,

implOptions: ReplicacheImplOptions = {},

) {

or you can extend ReplicacheImplOptions which is all internal stuff the Zero and Reflect use.

arv · 2025-03-06T13:48:55Z

packages/replicache/src/replicache-options.ts

+ */
+export type ZeroOption<T extends ZeroTxData> = {
+  /**
+   * Allow Zero to initialize its IVM state from the given hash and store.


store -> dag store for clarity

arv · 2025-03-06T13:56:40Z

packages/replicache/src/transactions.ts

 * database.
 */
-export interface WriteTransaction extends ReadTransaction {
+export interface WriteTransaction<TZeroData extends ZeroTxData = ZeroTxData>


I would prefer if we do not change Replicache public types.

For Reflect we have:

https://github.com/rocicorp/reflect-archive/blob/209cecc2c899bfbe16e57f32dcad1be8f75cf024/packages/reflect-shared/src/types.ts#L21

We have the same for Zero:

mono/packages/zero-client/src/client/custom.ts

Lines 70 to 80 in 743c954

export class TransactionImpl implements ClientTransaction<Schema> {

constructor(repTx: WriteTransaction, schema: Schema) {

must(repTx.reason === 'initial' || repTx.reason === 'rebase');

this.clientID = repTx.clientID;

this.mutationID = repTx.mutationID;

this.reason = repTx.reason === 'initial' ? 'optimistic' : 'rebase';

// ~ Note: we will likely need to leverage proxies one day to create

// ~ crud mutators and queries on demand for users with very large schemas.

this.mutate = makeSchemaCRUD(schema, repTx);

this.query = {};

}

mono/packages/zero-client/src/client/custom.ts

Lines 90 to 98 in 743c954

export function makeReplicacheMutator(

mutator: CustomMutatorImpl<Schema>,

schema: Schema,

) {

return (repTx: WriteTransaction, args: ReadonlyJSONValue): Promise<void> => {

const tx = new TransactionImpl(repTx, schema);

return mutator(tx, args);

};

}

but somehow we need to plumb through the correct IVM fork. Only Replicache has the information required to create that fork (base head hash, new head hash) and when to fork (when rebase begins).

arv · 2025-03-06T13:58:49Z

packages/zero-client/src/client/replicache-types.ts

 }

-export interface WriteTransaction extends ReplicacheWriteTransaction {
+export interface WriteTransaction<T extends ZeroTxData = ZeroTxData>


I thought we did this... I don't yet understand why we have to change the Replicache interface.

When Replicache starts a rebase it'll ask Zero (via zeroOption) to create a fork of the IVM sources at the given head. This fork will be added to WriteTransaction so Zero can use it in it's mutators.

Previously I assumed there were a finite (or at least small) set of heads and that Zero could look them up by name rather than having to pass the fork along in the Replicache transaction. It turns out there isn't a small set of heads being rebased onto. Looking up by name also opens us up to concurrency issues.

Let me put up the PR that actually uses this argument.

I think I see the big picture better now. I still wish we could keep the Replicache public interfaces as they are.

I have a few tricks up my sleeves that would allow this to work without changing the interface. I think it is OK to add public properties (using symbols) to the implementation without changing the interface for example. Other options include using weakmaps as sidetables.

…cache zeroOption is a minimal interface that Replicache will call when: - pullEnd - persist - refresh - mutate happen. - replicache will invoke `advance` whenever it's main head moves forward. - replicache will invoke `getTxData` to fork IVM sources whenever it needs to rebase - replicache will invoke `init` on construction to populate the IVM sources

When `zero` is running atop `replicache`, `zeroData` will be available. `zeroData` is the `IVMSourceBranch` that mutations should be using in the current transaction. It is templated here `TZeroData = unknown` so Replicache does not need to know about `IVMSourceBranch`.

tantaman · 2025-03-06T14:47:14Z

@arv - I removed all changes to the Replicache public interface.

tantaman · 2025-03-06T14:53:36Z

@arv - I removed all changes to the Replicache public interface.

well not quite. readonly zeroData: ZeroTxData | undefined; is still on interface WriteTransaction

Let me try swapping that to a symbol.

arv

LGTM

Thanks

vercel bot deployed to Preview – zbugs March 6, 2025 03:28 View deployment

tantaman requested review from arv and grgbkr March 6, 2025 03:28

arv reviewed Mar 6, 2025

View reviewed changes

arv requested changes Mar 6, 2025

View reviewed changes

tantaman added 4 commits March 6, 2025 09:26

chore(replicache): thread args to get zeroData to WriteTransaction

55958d8

feat(zero): pull ivm branch from WriteTransaction

ae03ca2

tantaman force-pushed the mlaw/rep-tx-data branch from 4b6392e to f5231b4 Compare March 6, 2025 14:45

vercel bot deployed to Preview – replicache-docs March 6, 2025 14:45 View deployment

vercel bot deployed to Preview – zbugs March 6, 2025 14:46 View deployment

chore(replicache): remove public interface type impact of zeroOption

8f5799a

tantaman force-pushed the mlaw/rep-tx-data branch from f5231b4 to 8f5799a Compare March 6, 2025 15:02

vercel bot deployed to Preview – replicache-docs March 6, 2025 15:03 View deployment

vercel bot deployed to Preview – zbugs March 6, 2025 15:03 View deployment

arv approved these changes Mar 6, 2025

View reviewed changes

tantaman enabled auto-merge (rebase) March 6, 2025 18:42

tantaman merged commit ab3459d into main Mar 6, 2025
11 checks passed

tantaman deleted the mlaw/rep-tx-data branch March 6, 2025 18:49

	constructor(
	options: ReplicacheOptions<MD>,
	implOptions: ReplicacheImplOptions = {},
	) {

	export class TransactionImpl implements ClientTransaction<Schema> {
	constructor(repTx: WriteTransaction, schema: Schema) {
	must(repTx.reason === 'initial' \|\| repTx.reason === 'rebase');
	this.clientID = repTx.clientID;
	this.mutationID = repTx.mutationID;
	this.reason = repTx.reason === 'initial' ? 'optimistic' : 'rebase';
	// ~ Note: we will likely need to leverage proxies one day to create
	// ~ crud mutators and queries on demand for users with very large schemas.
	this.mutate = makeSchemaCRUD(schema, repTx);
	this.query = {};
	}

	export function makeReplicacheMutator(
	mutator: CustomMutatorImpl<Schema>,
	schema: Schema,
	) {
	return (repTx: WriteTransaction, args: ReadonlyJSONValue): Promise<void> => {
	const tx = new TransactionImpl(repTx, schema);
	return mutator(tx, args);
	};
	}

feat(replicache): add zeroOption to allow Zero to hook into Replicache events #3933

feat(replicache): add zeroOption to allow Zero to hook into Replicache events #3933

Uh oh!

Conversation

tantaman commented Mar 6, 2025

Uh oh!

vercel bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tantaman commented Mar 6, 2025

Uh oh!

tantaman commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(replicache): add `zeroOption` to allow Zero to hook into Replicache events #3933

feat(replicache): add `zeroOption` to allow Zero to hook into Replicache events #3933

vercel bot commented Mar 6, 2025 •

edited

Loading

github-actions bot commented Mar 6, 2025 •

edited

Loading

github-actions bot commented Mar 6, 2025 •

edited

Loading

tantaman commented Mar 6, 2025 •

edited

Loading