Skip to content

Conversation

@TamHuynhTee
Copy link
Collaborator

No description provided.

…thin 1 minute, sends visitor id to authorizer
@linear
Copy link

linear bot commented Feb 27, 2025

P-835 SDK: Refactor with event queue & batching

Currently we send and retry events individually. We should instead queue and batch the sending of events.

Allow batching of events.

This is needed to scale because Tinybird's event API has a 1000 RPS limit https://www.tinybird.co/docs/get-data-in/ingest-apis/events-api#limits

References

P-950 Some users with only 1 Session are labelled as 'Returning'

If you go to https://app.formo.so/teams/94f54b87-0209-4411-90dc-0e24a186c729/projects/formo.so/users

Screenshot 2025-02-11 at 17.46.25.png

And click on some of the 'Returning' users (which should have num_sessions > 1

https://app.formo.so/teams/94f54b87-0209-4411-90dc-0e24a186c729/projects/formo.so/profiles/0xcc2a855946a3c20683858fe6ee15acf8b836f0b3

Screenshot 2025-02-11 at 17.47.53.png

This user only has 1 session (Feb 7 2025)
Why is this user being labelled as 'Returning'?

P-898 Debug why a high number of duplicate events are emitted

https://app.formo.so/teams/82557c97-0996-4873-a876-7456f56692e9/projects/s5x8Pj_0OndkY0Uyt_WaX/profiles/0x91ebce363cfb97222d73939569be9c5a11cc33c6

Screenshot 2025-02-17 at 17.52.03.png

Could this be an issue in the accountsChanged or disconnect listeners:

  private registerAddressChangedListener(): void {
    const listener = (...args: unknown[]) =>
      this.onAddressChanged(args[0] as string[]);

    this._provider?.on("accountsChanged", listener);
    this._providerListeners["accountsChanged"] = listener;

    const onAddressDisconnected = this.onAddressDisconnected.bind(this);
    this._provider?.on("disconnect", onAddressDisconnected);
    this._providerListeners["disconnect"] = onAddressDisconnected;
  }

"access": "public"
},
"license": "MIT",
"dependencies": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the bundle size now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

98kb now

}
this.eventQueue.enqueue(requestData, (err, _, data) => {
if (err) {
console.error(err);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw later on there should be a config for 'log levels'
Unless the SDK enables logs, we shouldn't print logs by default since it's only for our debugging

Tracked here https://linear.app/getformo/issue/P-1022/add-log-levels-to-sdk-config

@yosriady
Copy link
Contributor

The beforeunload handler is async, but the browser might not wait for it to complete
Consider using navigator.sendBeacon() as a fallback

// Add beacon support for more reliable delivery on page unload
window.addEventListener("beforeunload", async (e) => {
  e.stopImmediatePropagation();
  if (navigator.sendBeacon) {
    const data = this.queue.map(item => item.message);
    navigator.sendBeacon(this.url, JSON.stringify(data));
  } else {
    await this.flush();
  }
});

@yosriady
Copy link
Contributor

@TamHuynhTee

Improving duplicate detection suggestions

  1. Hashing Payloads: Instead of stringifying the entire payload, compute a hash of the payload. This can be done using a hashing algorithm like SHA-256. Hashing is generally faster and results in a fixed-size output, making comparisons more efficient.
   import { createHash } from 'crypto';

   private hashPayload(payload: any): string {
     return createHash('sha256').update(JSON.stringify(payload)).digest('hex');
   }
  1. Use a Set for Hashes: Maintain a Set of hashes for quick lookup. This reduces the need to iterate over the entire queue to check for duplicates.
 private payloadHashes: Set<string> = new Set();

   private checkDuplicate(newMessage: RequestEvent): boolean {
     const hash = this.hashPayload(newMessage);
     if (this.payloadHashes.has(hash)) {
       return true;
     }
     this.payloadHashes.add(hash);
     return false;
   }

@yosriady
Copy link
Contributor

@TamHuynhTee Can you also rename the FormoAnalyticsProvider apiKey to writeKey?
I'm updating the install instructions here https://github.com/getformo/formono/pull/335/files

@TamHuynhTee TamHuynhTee requested a review from yosriady February 28, 2025 10:54
@yosriady
Copy link
Contributor

@TamHuynhTee LGTM, feel free to publish

Also please review https://github.com/getformo/formono/pull/335

@TamHuynhTee TamHuynhTee merged commit 39c461f into main Mar 3, 2025
2 checks passed
@TamHuynhTee TamHuynhTee deleted the feature/event-queue branch March 3, 2025 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants