Summary

As an important strategic partner of Intel, after Intel launched the 2nd Generation Intel Xeon Scalable processor with stronger computing performance and better user experience support, Inspur and Intel jointly developed artificial intelligence inference based on this processor Solutions that accelerate AI inference and are committed to helping industry and enterprise customers reduce deployment time and costs while ensuring their performance to meet customers’ growing needs. Intel Select Solutions for AI Inferencing provides customers with an efficient artificial intelligence inference algorithm solution based on the proven Intel architecture, which can be quickly launched and deployed, enabling you to innovate and get to market faster. In order to accelerate AI Inferencing and time to market, Intel Select Solutions for AI Inferencing also combines several Intel and third-party software and hardware technologies.

Background

In the context of artificial intelligence and machine learning, training is the stage where neural networks try to learn from data. Reasoning puts learning into practice, and trained models are used to infer and predict outcomes—classify, identify, and process new input data based on what you have learned. The solution is a deep learning inference solution for some of the fastest growing areas of artificial intelligence, such as video, natural language processing, and image processing. As a “turnkey platform” solution, with proven IA building blocks, partners can innovate and bring this integrated tool’s building block solution to market, making AI simple and efficient. Given that neural network models have been trained, model inference will be a challenge. This solution provides a starting point for customers to deploy efficient artificial intelligence inference algorithms. Use OpenVINO to accelerate inference, which shortens the time from enterprise data to strategic decisions, provides low latency and high-end to end throughput, and reduces enterprise costs.

Customer Benefits

Test Results

In the test comparison between Intel Xeon Gold 6240 and Intel Xeon Gold 6130, the performance can be accelerated by 1.35-1.43 times.

• ResNet50 training performance reaches 1.35 times acceleration
• Inception_v3 training performance reaches 1.43 times acceleration

Inference performance (FP32) can achieve 1.3-3 times acceleration on Intel Xeon Gold 6240 and Intel Xeon Gold 6130, with the best performance of multiple streams (BS = 1)

• ReNnet50 multi-steam inference for best performance up to 3x acceleration
• Inception_v3 multi-steam best inference performance up to 2.6 times faster
• A3C multi-steam optimal performance can reach 1.32 times acceleration

Compared with the FP32 model, using VNNI to upgrade and optimize TensorFlow, the multi-steam inference performance of INT8 on the Intel Xeon Gold 6240 can be accelerated by 1.9-3.2 times.

• Under 40 parallel streams, RN50 int8 performance can reach 3.5 times acceleration
• Under 40 parallel streams, the performance of Inception int8 can be 3.2 times faster
• W & D Int8 performance can be accelerated up to 1.9 times under 40 parallel streams
• Under 20 parallel streams, SSD Mobilent int8 performance can be doubled

Compared with TensorFlow, OpenVINO can deeply improve INT8 performance and achieve multiple speedups

• Based on Intel Xeon Gold 6240, compared to TensorFlow, ResNet50 INT8 performance can achieve 1.3 times acceleration
• Based on Intel Xeon Gold 6240, compared to TensorFlow, Inception_v3 INT8 performance can achieve 1.4 times speedup

Intel, the Intel logo and Xeon are trademarks of Intel Corporation in the United States and/or other countries.