ADIOS is an open source I/O framework for scientific computing. It addresses the challenges of exascale data management by delivering a high-performance, scalable, and flexible I/O. ADIOS is available in bindings for C++, C, Fortran, and Python, and it can be used on a variety of platforms, including supercomputers, personal computers, and cloud systems. It has out-of-the-box support for MPI and serial environments.
ADIOS provides a unified application programming interface (API) that abstracts the details of data transport. This allows scientific applications to focus on what they produce and consume in terms of n-dimensional variables, attributes, and steps. ADIOS can be used in a variety of applications, such as file storage for checkpoint-restart and analysis, data streaming for code-coupling, and in situ analysis and visualization workflows. It also provides high-level APIs that make it easy to integrate with rich data analysis ecosystems, such as Python and C++. In addition, runtime configuration files are provided so users can fine-tune available parameters to enable efficient data movements without recompiling their codes to make it more grammatically correct. ADIOS also supports data compression via third-party libraries, including ZFP, SZ, MGARD, BLOSC, bzip2, and png.
Features
- Manage massive data sets, from generation to analysis to movement, as well as short-term and long-term storage.
- Self-describing data in binary-packed (.bp) format for rapid metadata extraction.
- Separate and extract relevant information from large data sets.
- Make real-time decisions based on in-transit or in-situ analytics.
- Expand to other transport mechanisms, such as wide area networks, remote direct memory access, and shared memory, with minimal overhead.
- Utilize the full capabilities of emergent hardware technologies, such as high-bandwidth memory and burst buffers.
- Develop engines and operators and load them as plugins at runtime.
History
ADIOS (Adaptable I/O System) started as a GT student project and was further developed with funding from the HECURA program in a grant jointly awarded to Georgia Tech, UNM, ORNL and Sandia. As originally conceived, ADIOS was a thin wrapper layer between existing external I/O systems and the application, and allowed I/O details to be changed via an external XML specification. For example, ADIOS allowed output to be changed from MPI-IO synchronous, to MPI-IO collective, to POSIX I/O, to asynchronous I/O using datatap (another HECURA project) or no I/O (for baseline testing). ADIOS was designed based on the data requirements of 7 major scientific codes (GTC, Chimera, GTS, XGC1, XGC0, FLASH, and S3D). The use of ADIOS allowed these codes to run unchanged on a variety of different machine architectures and configurations where different I/O methods might confer different performance advantages. Early results from ADIOS were reported in the CLADE workshop associated with the 2008 HPDC conference as well as in posters at SC07.
Over time, ADIOS evolved to have its own file format (BP, or Binary Packed) and to support “data staging” (communication between different running MPI applications via the ADIOS API). Data staging was supported via direct communication with FlexPath, or indirectly using staging servers (DataSpaces).
In 2016, DOE’s Exascale Computing Project supported an effort to create a sustainable version of ADIOS, giving a sub-award to a team from ORNL, LBNL, GT, Rutgers, and Kitware. As part of this effort, ADIOS was rewritten from the ground up using C++11 and using more modern design methods. The existing BP3 file format was re-implemented, and newer versions (BP4 and BP5) were developed. The Sustainable Staging Transport (SST) was developed to support direct data staging via RDMA, a new interface to DataSpaces was produced, and an HDF5 interface was supported. A more rigorous test environment was also developed for Continuous Integration testing. This new version was called ADIOS2 to separate it from the original, but support for ADIOS1 has been discontinued, so the term ADIOS now only refers to the ADIOS2 version.
License
Apache-2.0
Citation
https://www.softxjournal.com/article/S2352-7110(19)30256-0/fulltext