[rfc] design exploration: implementations for operators against NestedIterator #2103
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
and maybe this obviates `genCached` too?
8177b75 to
0e379a9
Compare
f2c2781 to
ad56a6c
Compare
| const event = Symbol(); | ||
| const node = Symbol(); | ||
| type Event = Add | Remove | NoOp; | ||
| const ADD = 1; |
There was a problem hiding this comment.
What if we just give them string values so we can catch/find if there is any reason to do arithmetic. I don't think there is, but then we'll know for sure.
There was a problem hiding this comment.
One reason to use SMIs over string is efficiency. SMIs are stack allocated and strings heap allocated.
it all depends on how hot things are and with global warming it is going to get pretty hot
| import type {JSONObject} from '../../shared/src/json.js'; | ||
|
|
||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| type TODO = any; |
There was a problem hiding this comment.
a silly way to put cast things as any without needing an eslint comment each time 😅
| type Remove = typeof REMOVE; | ||
| type NoOp = typeof NO_OP; | ||
| type Entry<Type = JSONObject> = { | ||
| [node]: Type; |
There was a problem hiding this comment.
Why does the parent row get referred to by this symbol, but child iterators get actual names based on their table?
There was a problem hiding this comment.
mergeDistinct needed a way to find and id for a row. Giving the parent row a stable name made that easier to do. Otherwise we'd need to iterate all the keys in the row and find the thing that isn't iterable.
| type Remove = typeof REMOVE; | ||
| type NoOp = typeof NO_OP; | ||
| type Entry<Type = JSONObject> = { | ||
| [node]: Type; |
There was a problem hiding this comment.
If we're going to use a symbol here let's try to pick a unique name. How about Entity that is the name we used before. Or Row?
There was a problem hiding this comment.
Also is it enforced that every value flowing through the pipeline has a unique ID as it does today?
There was a problem hiding this comment.
every row will have a primary key and we can't mix rows from different tables in the same iterable level so I think this is true.
There was a problem hiding this comment.
Sorry I pressed publish on this review too quickly and my comments were kind of cryptic. Here's what I should have said:
- It's better if we pick names for things in the system that are unique and not already used by other concepts. The name "node" is already often used to refer to pipeline nodes. So what about using a different name like "row" or "entity"? The reason this is important is so that when talking about the system we can just use a short name for a concept rather than having to qualify it. Like we can say "entity" rather than "pipeline entry node".
- In the existing ivm, we require all things flowing through the pipeline to have a unique ID. Are we going to in the new system too? If we are then how come there's no id field here? is it just because this is a sketch and the id wasn't required, or is its absence important somehow? If there's going to be an ID for each of the thingies flowing through the pipeline, then the name "entity" or "PipelineEntity" makes even more sense.
Sorry for obsessing so much about names, but I think a big part of system design is just choosing good names.
There was a problem hiding this comment.
I've renamed things in the PR that is meant to be merged into main:
- https://github.com/rocicorp/mono/pull/2109/files : packages/zql/src/zql/ivm-2/iterable-tree.ts
I've also gone back to requiring all Entity types to have an id field. We'll need to revisit this when we add compound primary keys which we've discussed before.
This should be the final form (in terms of types) of
NestedIterable. The operators (join/map/filter) are sketches to confirm the shape ofNestedIterableis what we need.NestedIterableis anIterableofEntry.I've explored:
As well as:
The latter set uses
restartableso many iterators can be gathered from the baseIterable, allowing all forks to get the same data.Related document: https://www.notion.so/replicache/NestedIterable-5123f11b877e41b7bc9f00486d491d8b#adb39380a6f74402ace02cb85fe0c405
There are some possible future explorations:
joinwhen dealing with deltas and many:1 & many:many relationshipssortinformation totopkalong with theiterableBut it seems like diminishing returns at this point.