Machine learning (ML) implementations in developer tools have been on the rise, prompting a consortium of some of the largest companies including Google, Facebook, Microsoft as well as a handful of universities to launch ML inference benchmarks. The MLPerf Inference v0.5 will be used to measure system performance and power efficiency of AI apps.
The benchmarks will show how quickly a trained neural network can process new data to provide useful insights when the inferences are measured, according to the creators of MLPerf.
“The MLPerf inference benchmark working group follows an ‘agile’ benchmarking methodology: launching early, involving a broad and open community, and iterating rapidly,” MLPerf’s website states.
This can be applied to measuring the performance of various applications that include capabilities such as autonomous driving and natural language processing running on smartphones, PCs, edge servers and cloud computing platforms.
“By creating common and relevant metrics to assess new machine learning software frameworks, hardware accelerators, and cloud and edge computing platforms in real-life situations, these benchmarks will establish a level playing field that even the smallest companies can use,” general chair Peter Mattson and the inference working group co-chairs explained in a statement.
Currently, the benchmarks will focus on three common ML tasks including:
- Image classification: predicting a “label” for a given image from the ImageNet dataset, such as identifying items in a photo.
- Object detection: picking out an object using a bounding box within an image from the MS-COCO dataset, commonly used in robotics, automation, and automotive.
- Machine translation – translating sentences between English and German using the WMT English-German benchmark, similar to auto-translate features in widely used chat and email applications.
