Individual submitters may then – in this case Nvidia and Graphcore – hold more a directly competitive briefing, touting their wares relative to competitors. It’s a friendly affair with polite comment. MLCommons conducts a general briefing with representatives of most of the participants present. The Open division is intended to foster faster models and optimizers and allows any ML approach that can reach the target quality.” – MLCommons.Īn interesting pattern has emerged in releasing MLPerf results – at least for training and inferencing. MLPerf has two divisions: “The Closed division is intended to compare hardware platforms or software frameworks ‘apples-to-apples’ and requires using the same model and optimizer as the reference implementation. He pointedly noted Google (TPU) did not submit in the closed division and that Habana (Intel) only submitted on two workloads (BERT and ResNet 50). Also, as pointed out by Nvidia’s Paresh Kharya, senior director of product management, datacenter computing, Nvidia was again the only submitter to run all eight workloads in the closed – apples to- apples – division. Nvidia GPU performance has been unarguably impressive in all of the MLPerf exercises (click on charts shown below to enlarge them). Relying mostly on large systems – Nvidia supercomputer Selene, sixth on the Top500, and large deployments of Microsoft Azure ND A100 v4 series instances – Nvidia took top honors. For the moment, and depending how one slices the numbers, Nvidia remains king.

Additionally, MLPerf submissions go through a month-long peer review process, which further validates the reported results.”Īlso important is MLPerf’s slow expansion beyond being been mostly a showcase for Nvidia accelerators – the jousting between Nvidia and Graphcore is a case in point. At Habana we find that MLPerf benchmark is the only reliable benchmark for the AI industry due to its explicit set of rules, which enables fair comparison on end-to-end tasks. As machine learning evolves, MLPerf evolves and thus continually expands and updates its benchmark scope, as well as sharpens submission rules. To that end AI leaders from academia, research labs, and industry decided on a set of benchmarks and a defined set of strict rules that ensure fair comparisons among all vendors. “The MLPerf community aims to design fair and useful benchmarks that provide “consistent measurements of accuracy, speed, and efficiency” for machine learning solutions. Consider this statement of support from Habana Labs (owned by Intel) in today’s blog: Submitters for this round included: Azure, Baidu, Dell, Fujitsu, GIGABYTE, Google, Graphcore, Habana Labs, HPE, Inspur, Lenovo, Nvidia, Samsung and Supermicro.īroadly, MLPerf seems to be steadying its position as a looked-for benchmark for machine learning (training, inferencing, and HPC workloads).

That’s roughly unchanged from the 13 organizations that participated in June (training v1.0) and a considerable drop from the 650 results submitted at that time. The latest benchmark round received submissions from 14 organizations and released over 185 peer-reviewed results for machine learning systems spanning from edge devices to data center servers, according to MLCommons. The time-to-train to a predetermined quality is one of the critical answers that the benchmarks can help provide. The idea is to review specific results and configurations over the eight training workloads to find answers based on your needs. Both setups may be excellent for the required tasks. Comparing the performance of an Azure deployment using up to 2,000 A100s to a Dell server running two CPUs and four GPUs, for example, isn’t especially informative by itself. Making sense of the MLPerf results has never been easy because of the many varying system configurations of the submissions. Also, noteworthy, Microsoft Azure made its debut in the MLPerf training exercise with Nvidia-accelerated instances and impressive performances. One of the other AI accelerator companies that participated, Graphcore, stood its ground by holding a separate media and analyst briefing to tout its own MLPerf performance, contending that its IPU-based systems were faster and offer a better bang-for-the-buck than similarly-sized Nvidia A100-based systems. MLCommons today released its fifth round of MLPerf training benchmark results and Nvidia GPUs again dominated the rankings, but competitors were not taking those results sitting down.

Graphcore benchmarks