-
Notifications
You must be signed in to change notification settings - Fork 1.1k
NIO CountBytes example #975
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NIO CountBytes example #975
Conversation
Let me start with general comments:
This is not true. This error comes from the way you loop and compute long total = 0;
int readCalls = 0;
for (int read = 1; read > 0; ) {
readCalls ++;
read = chan.read(buf);
buf.flip();
total += read; // On the last call you read -1 and sum it to total size, thus the mismatch
} In general the for loop seems a bit strange to me, I would have used something like (not tested): while (chan.read(buf) > 0) {
readCalls++;
total += buf.position();
buf.flip();
}
readCalls++; // We must count the last call
What does this mean? Is it slower at reading the size metadata or at downloading the file? Can you share some numbers? |
@@ -0,0 +1,82 @@ | |||
package com.google.cloud.examples.nio; |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
Thank you! Indeed it's obvious now.
Downloading the file. Here's what I see:
We can see that gsutil takes 1/4th of the time. The way it prints it suggests that it did 4 parallel downloads (each of 1/4th the size). |
Added licence text, plus some esthetic changes.
/** | ||
* CountBytes will read through the whole file given as input. | ||
* | ||
* <p>It's meant for testing that NIO doesn't read too slowly. |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
Are we OK to merge? |
@jean-philippe-martin just merged. As usual: thanks! |
You are welcome! |
A CountBytes examples that reads through the provided file (using NIO), counting the bytes.
It also reports elapsed time, allowing for a simple form of benchmarking.
Amusingly, it currently says that the file sizes are off by one: it sees one fewer bytes than Files.size() reports. This is also noticeably slower than gsutil (perhaps because it's single-threaded?).
Sample output: