Skip to content

Enhancement: Save/reuse clone chunk dictionaries to speed up cloning & support transmission resume #63

@max-sistemich-kisters

Description

@max-sistemich-kisters

Hi Olle,
I have beed testing bita on a project I'm working on and it is a very promising tool.
Unfortunately, there are two features/improvements that are missing for my use case:

  1. The dictionary creation is by far the slowest part of the cloning process for me. That part alone takes almost 3 minutes. The actual transmission is much faster then (5-10 seconds). If we could avoid this duplicate work, it would translate to multiple days of extra battery life on my project (assuming normal operation).
    While looking into this, I saw that you had already planned for a mechanism to tackle this at some point (see Implement chunk index cache #3).
    Could you tell me whether there was a specific reason to not implement this?

  2. If the download fails due to a bad connection, I think it would be very helpful to be able to resume transmissions. My suggestion would be to save the dictionary of the downloaded chunks during cloning. So something like:

    $ bita clone \
      --seed /dev/mmcblk0p1 \
      --seed-dictionary /path/to/archive/of/dev/mmcblk0p1 \
      http://file.to.clone \
      <output path or stream> \
      --save-dictionary /path/for/cloned/dictionary
    

    Now if the download succeds, we can use the seed dictionary for a future clone. However, if it fails, we could continue the transmission by either having a clone resume option or the option to clone with multiple seeds. So something like:

    $ bita clone \
      --seed /dev/mmcblk0p1 \
      --seed-dictionary /path/to/archive/of/dev/mmcblk0p1 \
      --seed /path/to/faild/update/or/other/file/that/is/expected/to/share/data/with/server/file \
      --seed-dictionary /path/to/failed/dictionary \
      http://file.to.clone <output path or stream> \
      --save-dictionary /path/for/cloned/dictionary
    

    Of course both of these features would require the user to make sure that the seeds are not modified after dictionary creation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions