[exporter/tinybird] limit request size to 10mb #41989
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
62d8c4b to
80357cd
Compare
dmitryax
reviewed
Aug 21, 2025
| type option func(*tinybirdExporter) error | ||
|
|
||
| // withMaxRequestBodySize sets the maximum size of the request body in bytes. | ||
| func withMaxRequestBodySize(size int) option { |
There was a problem hiding this comment.
This seems like isn't applied?
There was a problem hiding this comment.
This is only used in tests to verify that the request size limit works as expected, but it is not exposed to the user. The main reason for this is that the EventsAPI has a hard limit that has not changed for a long time, and it is not expected to change in the future.
MoreraAlejandro
approved these changes
Aug 25, 2025
6 tasks
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When a request to the EventsAPI exceeds 10 MB, split it into N smaller requests, each no larger than 10 MB.
If any of these requests fail, data processing stops and an at-most-once delivery approach is followed:
This change ensures that large requests (≥ 10 MB) do not result in complete data loss. Some data may still be lost because processing halts after the first failure, but this can be improved later by returning the missing data via
consumererror.Traceand derivatives, allowing the collector to retry (out of the scope for this PR).Link to tracking issue
Fixes #41782
Testing
Tested were added to ensure that the at-most-once delivery approach is followed
Documentation