IBM scans ten billion files during your lunch hour
IBM has unveiled a single system that can scan 10 billion files in only 43 minutes.
If that doesn’t sound impressive, keep in mind IBM’s last record was one billion files scanned in three hours in 2007.
IBM made the achievement with its General Parallel File System (GPFS) – commonly used by top supercomputers – running on a cluster of ten eight-core systems paired with 6.8 terabytes of solid-state storage.
Today’s demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing, multi-zettabyte world
“GPFS’s advanced algorithm makes possible the full use of all processor cores on all of these machines in all phases of the task (data read, sorting and rules evaluation),” IBM explained.
“The appliances sustainably perform hundreds of millions of data input-output operations, while GPFS continuously identifies, selects and sorts the right set of files among the 10 billion on the system,” IBM said.
The breakthrough makes it possible to hold huge amounts of data on a single platform, while keeping it manageable.
“Today’s demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing, multi-zettabyte world,” said Doug Balog, vice president of storage platforms at IBM.
“This has the potential to enable much larger data environments to be unified on a single platform and dramatically reduce and simplify data management tasks such as data placement, ageing, backup and migration of individual files.”
Disclaimer: Some pages on this site may include an affiliate link. This does not effect our editorial in any way.