Skip to content

Conversation

@AlexisSouquiere
Copy link
Collaborator

@AlexisSouquiere AlexisSouquiere commented Dec 8, 2023

Fix #1487

  • Follow Micronaut guidelines about sending large amount of data (StreamedFile + PipedInputStream/PipedOutputStream for custom datasource)
  • Add button to download all the messages next to the filters
    image
  • Add a warning message to warn people about the amount of data
    image
  • Take into account the filters set by the users

Tested locally with a topic of 600K records, downloaded a 220Mb file in less than 1min

Downloaded file format is the following:
image
With Avro messages
image
With Tombstone
image

Feedback is appreciated 😄

@AlexisSouquiere AlexisSouquiere marked this pull request as draft December 8, 2023 14:22
@AlexisSouquiere AlexisSouquiere marked this pull request as ready for review December 11, 2023 09:46
@AlexisSouquiere
Copy link
Collaborator Author

AlexisSouquiere commented Dec 11, 2023

I will have to take into account the endTimestamp filter in the new download endpoint if #1629 is merged before

@tchiotludo
Copy link
Owner

tchiotludo commented Dec 17, 2023

@AlexisSouquiere your PR #1629 is merged before, sorry 😅

@tchiotludo
Copy link
Owner

@AlexisSouquiere you want to add extra commit on that one to handle endTimestamp?

@AlexisSouquiere
Copy link
Collaborator Author

It's fine for me I took into account the new filter on the download endpoint :)
PR is ready

@tchiotludo tchiotludo merged commit c89db95 into tchiotludo:dev Jan 16, 2024
@arindampatra33
Copy link
Contributor

@tchiotludo @AlexisSouquiere This commit added a @JsonIgnore to Topic field of Record , this is breaking the tail functionality

@AlexisSouquiere
Copy link
Collaborator Author

@arindampatra33 thanks for the alert. I will check and fix it

@arindampatra33
Copy link
Contributor

arindampatra33 commented Feb 15, 2024

@AlexisSouquiere its failing with below error in the UI when its trying to read topic.name
I guess you have to revert the JsonIgnore topic is an important field

Tail.jsx:398 Uncaught TypeError: Cannot read properties of undefined (reading 'name')
at rowId (Tail.jsx:398:33)
at Table.jsx:519:17
at Array.forEach ()
at o.value (Table.jsx:517:10)
at Wa (react-dom.production.min.js:182:192)
at Ba (react-dom.production.min.js:181:224)
at Ss (react-dom.production.min.js:263:490)
at vc (react-dom.production.min.js:246:265)
at mc (react-dom.production.min.js:246:194)

@arindampatra33
Copy link
Contributor

@AlexisSouquiere If you added it just to have lesser number of columns in the downloaded data , you can create a new model and populate. from Record data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Ability to download all of a topic's messages

3 participants