In the Autumn of 1992, the HPSS Collaboration was established to produce a highly scalable high performance storage system. The HPSS Collaboration is based on the premise that no single organization has the experience and resources to meet all the challenges represented by the growing imbalance between computing power and data collection capabilities, and storage system I/O, capacity, and functionality.
HPSS Collaboration
HPSS Collaboration Team
Over 20 organizations including industry, Department of Energy (DOE), other federal laboratories, universities, National Science Foundation (NSF) supercomputer centers, French Commissariat a l’Energie Atomique (CEA) and Gleicher Enterprises have contributed to various aspects of this effort. Today, the primary HPSS development team consists of:
- IBM Consulting (Houston, TX)
- Lawrence Berkeley National Energy Research Supercomputer Center (Berkeley, CA)
- Lawrence Livermore National Laboratory (Livermore, CA)
- Los Alamos National Laboratory (Los Alamos, NM)
- Sandia National Laboratories (Albuquerque, NM)
HPSS Collaboration Organization
Careful thought went into the basic organization of the HPSS Collaboration to bring together the independent entities to successfully produce world-class storage software. Its basic governing document is a Collaboration Agreement spelling out intellectual property rights of the development partners and their management and organization. While the DoE national laboratories and IBM are development partners, IBM is solely responsible for commercializing and deploying HPSS outside of the HPSS Collaboration. There is an Executive Committee (EC) co-chaired by IBM and DOE lab representatives that sets major development and other policies. The EC meets several times a year, primarily by teleconference. HPSS development is overseen by a Technical Committee (TC) coordinated by an IBM project manager. The TC members are generally organized around the major architectural modules of the system and lead small teams of developers within their organization. Both the TC and developers meet at least once per week by teleconference, and once or twice a year in-person. Development of the system follows industry standard software engineering practices, which has been a major factor in its success in producing a stable maintainable product. Today, the HPSS Collaboration follows the Agile Software Development Methods.
Early Requirements and Future Storage Demands
The HPSS Collaboration’s high performance storage system needed to provide scalable hierarchical storage management (HSM), archive, and file system services. No product meeting the requirements existed. When the HPSS Collaboration began, scientific computing power and storage capabilities at a site, such as a DOE national laboratory, was measured in a few tens of gigaflops, data archived in HSMs in a few tens of terabytes at most, data throughput rates to an HSM in a few megabytes/sec, and daily throughput with the HSM in a few gigabytes/day. At that time, the DOE national laboratory and the HPSS Collaboration design team recognized that we were headed for a data storage explosion driven by computing power rising to teraflops/petaflops requiring data stored in HSMs to rise to petabytes and beyond, data transfer rates with the HSM to rise to gigabytes/sec and higher, and daily throughput with an HSM in tens of terabytes/day. Therefore, we set out to design and deploy a system that would scale by a factor of 1,000 or more and evolve from the base above toward these expected targets and beyond.
The IEEE Mass Storage Reference Model v5 was established, and the High Performance Storage System (HPSS) software was developed.
Because of the highly scalable HPSS architecture, these targets have been successfully met. We now recognize that computing power will rise to exaflops with a corresponding rise in the need to scale storage in its various dimensions by another factor of 1,000. Further, other major application domains, such as real-time data collection, also require such extreme-scale storage. We believe the HPSS architecture and basic implementation, built around a scalable relational database management system (IBM’s Db2) make it well suited to this challenge.
HPSS is typically delivered with a FUSE file system interface and an S3/Glacier interface, which are ideal for traditional-HPC, AI, and hybrid-cloud workloads. The annual fee for HPSS is not tied to any attribute of scale from petabytes to exabytes for disk or tape, and from millions to billions of files or objects stored in a single instance of HPSS.
In 2025, IBM deployed a 3-exabyte HPSS solution capable of sustaining a client-ingest-rate of 5.5 petabytes-per-day to HPSS tape, while simultaneously recalling 3.9 petabytes-per-day from HPSS tape back to the client, through a 10 PB HPSS disk cache deployed on economical block storage devices. With these milestones in our rearview mirror, the HPSS Collaboration is now focusing on another factor of 1,000 (zettabytes and trillions), with an emphasis on “ease of use”.