|
Supercomputing 2007 Storage Challenge Finalist |
|
|
|
 The Synergy Lab, led by Prof. Wu Feng, and Argonne National Laboratory have collaborated to land a finalist slot in the SC|07 Storage Challenge. The collaborators leveraged mpiBLAST at Virginia Tech (VT) and MPICH2 at Argonne National Laboratory (ANL) to create a "Parallel Metadata Environment for Distributed I/O and Computing" (ParaMEDIC) framework, which accelerated mpiBLAST by 25-fold over the distributed I/O and computing environment between ANL and VT.
mpiBLAST is an open-source parallelization of the BLAST genome sequence-search library. It uses database segmentation to allow different worker processors to search unique segments of the database and write the output to a shared filesystem. For distributed systems sharing a filesystem through a low-bandwidth or high-latency network, writing this output to the filesystem can be a challenging task, eventually forming a performance bottleneck.
ParaMEDIC provides an environment that decouples computation and I/O in applications and drastically reduces I/O overhead through metadata processing. For mpiBLAST, ParaMEDIC partitions worker processes into compute and I/O workers. Compute workers convert their output to metadata and send it to I/O workers. I/O workers process this metadata to re-create the actual output and write it to the filesystem. This results in a dramatic reduction in I/Otime, thus accelerating mpiBLAST by as much as 25-fold.
ParaMEDIC to the rescue!
Team Members: Pavan Balaji (ANL), Wu Feng (VT), Jeremy Archuleta (VT)
For more information, please see http://sc07.supercomputing.org/schedule/event_detail.php?evid=11237 |
|
