Skip to content

Iterator-consuming parallel_map #580

@leogao2

Description

@leogao2

It seems like parallel_map currently consumes the entire input before starting, which eats up a ton of memory; this is undesirable in pipelines where the input is a very large iterator.

Expected behavior: something similar to pool.imap with multiprocessing.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions