The Importance of the MLPerf Benchmark
And a look at Inspur’s involvement and benchmark results.
As technological innovation accelerates and AI applications become more widely adopted for everyday operations, businesses are increasingly looking to invest in the proper equipment to run these applications and procedures. With this high capital investment comes the need for a way to gauge the performance of different available systems on the market.
One of the most reliable tools for this is currently the MLPerf, a full system benchmark evaluating the performance of machine learning services, software, and hardware. Founded by researchers and engineers from Baidu, Google, Harvard University, Stanford University, and the University of California Berkeley, the MLPerf plays a critical role in the industry by establishing industry-standard metrics and gaining the support of over 70 organizations worldwide. Inspur was one of the earliest supporters for MLPerf benchmark and was a founding member of MLCommons, an open engineering consortium now administering the MLPerf.
How Does It Work?
The MLPerf benchmarks work like this. A set of representative tasks, like Image Classification and Object Detection, are agreed upon with one or more models for each task. Then volunteer(s) create reference implementations that set the example for requirements on the models for companies to replicate.
There are a number of MLPerf Inference rules that apply to all benchmark implementations. These rules include requiring open source codes, prohibiting benchmark detection in the system, requiring results to be replicable, and many more. All MLPerf submissions also get reviewed by other submitters, including potential direct competitors. These processes enable the MLPerf Benchmark to provide unbiased evaluations of training and inference performance.
Why It’s Important for the Market
With the increasing supply and demand for AI products on the market, it is important to have an authoritative, representative panel of benchmarks. For instance, SPEC integer and floating point tests are useful for determining a platform’s CPU performance qualifications for the CPU market. The TPC suite stress tests a whole system’s ability to process large volumes of data. Other more traditional HPC tests include the HPL (High Performance Linpack), STREAM memory bandwidth and HPCG (High Performance Conjugate Gradients) benchmarks. Combined multiple results from a well-rounded slew of tests help organizations make informed decisions based on their computing and business needs.
In a chat with The Next Platform, Inspur senior AI product manager Gavin Wang said, “We think that MLPerf today is based on the most popular AI workloads and scenarios, such as computer vision, natural language processing, and recommendation systems,” He added, “When you look at the eight tasks in the MLPerf training benchmark, they represent the catalog of neural network models, and these scenarios are very representative for customers.”
The MLPerf benchmark will also ultimately drive better products. It compels engineers to design systems optimized for various AI algorithms in common applications, and AI system manufacturers have valuable feedback to build platforms based on customer requirements. With improved performance and architecture, those AI systems then climb the rankings of the MLPerf tests, driving positive revenue and feedback, leading to even further optimization, and the cycle goes on.
Inspur’s MLPerf Benchmark Results
The Next Platform released a publication discussing the MLPerf v0.7 training benchmark results, in which Inspur’s NF5488A5 system achieved first place in ResNet50 performance by successfully training the model against the ImageNet library in 33.37 minutes. This result was 16.1% faster than Nvidia’s DGX A100 system, which used the exact same CPU and GPU components, two AMD EPYC 7724 processors and eight Nvidia A100 SMX4 40GB GPU accelerators.
Inspur’s continuously excellent performance in MLPerf’s benchmarks demonstrate its dedication to deliver systems with industry-leading performance that effectively tackles the world’s most complex AI challenges.