Have you thought about exposing a simple high-level function, such as wait_for_next_blocik, instead? That would avoid having to pick a specific communication mechanism for this feature.

the blocks could be announced even before their insertion into the ledger - they could be streamed optimistically, with a tiny confirmation message being sent to the IPC once they are fully accepted

Somewhat of a tangent: For validators, snarkOS sync does not insert blocks into the ledger until they are confirmed by the next block. The goal is for clients to eventually move to the same, safer, mechanism.
It would make sense to think about whether we can propagate unconfirmed blocks to reduce latency in the future, and if snarkVM should have a notion of unconfirmed blocks.

Have you thought about exposing a simple high-level function, such as wait_for_next_blocik, instead? That would avoid having to pick a specific communication mechanism for this feature.

I'm not sure if this is desirable, as other potential mechanisms (e.g. the network) could be too slow to introduce at this stage, as it could slow down the validator. This way we can ensure that the announcements are near-instantaneous.

I was thinking of something very simple, like adding a condition variable (or tokio::sync::Notify) to Ledger. There is already a lock around current_block. Whenever current_block is updated, you could call notify` and then have a function like this.

async fn wait_for_block_height(&self, next_height: u32) {
       while self.current_block.read().height() < next_height {
                self.block_notify.notified().await;
       }
}

One could then spawn a task in snarkOS, or whatever uses snarkVM, that waits on this function and executes some custom logic (like writing to a UNIX socket).

Part of the design is to not have to have a dependency on snarkVM at all, it's a large library.

Rebased, now that the former base is in. It was clean, and there were no other changes.

the indexer is not connected to the validator but the feature is enabled

Every time a block is baked, a WARN log is displayed indicating that the feature is active, but the IPC channel is not available.

similarly the indexer goes down mid-stream

The same thing happens, there's just an additional initial ERROR log indicating a socket write error. If the indexer is restarted, the publishing resumes on its own.

would a large block impact consensus at all (serialisation cost)?

I've calculated the average serialization speed in a test run, and got ~857MiB/s (and this was in a backfilling run with parallel serialization, and small blocks, both of which were certain to decrease the calculated speed). This means that a 1MiB block would be serialized in ~1.1ms, and that's with a machine below validator specs.

I think we should document what functionality the new feature enables somewhere. You could use this README change from the tracing PR as a basis.

You might want to add a note that this is an "unstable" feature and that the wire format can change in the future, just to be safe.

I filed a dedicated issue to later document this feature, so as to not cause any conflict with the changes linked by @kaimast.