Use multiprocessing.Pool for parallel I/O (!431) · Merge requests · Duncan Macleod / gwpy

Duncan Macleod requested to merge github/fork/duncanmmacleod/table-read-nproc into master Apr 12, 2017

This PR modifies gwpy.io.mp.read_multi to use multiprocessing.Pool (instead of ProcessQueue) for parallel file reading, enabling trivial parallel I/O for astropy registered readers (e.g. ASCII). Files are now passed through the pool one-by-one, so readers in gwpy.table.io only need to be built to handle a single file at a time.

Use multiprocessing.Pool for parallel I/O

Merge request reports