MLU-X1000 MLU extender Product Manual V0.2.0 Preliminary 2020.07.10 Cambricon Director 1. Preface ..........................................................................................................................................................1 1.1. Copyright Declaration.....................................................................................................................1 1.2. Versioning ...........................................................................................................................................2 1.3. Update history ...................................................................................................................................2 2. Overview ......................................................................................................................................................3 3. Product Specification Overview ...........................................................................................................4 3.1 Overview of Product Specification Parameters ......................................................................4 3.2 Overview of structural specifications .........................................................................................4 3.3 Overview of electrical specifications ..........................................................................................5 3.4 Summary of heat dissipation specifications ............................................................................5 4. Component Profile ...................................................................................................................................7 4.1 CMX-BB1 ............................................................................................................................................7 4.2 CMX-PA4 ............................................................................................................................................9 4.3 CMX-PDB ............................................................................................................................................9 4.4 CMX-LINKB ..................................................................................................................................... 10 4.5 CMX-IBB ........................................................................................................................................... 10 4.6 Front panel ...................................................................................................................................... 11 4.7 Back panel ....................................................................................................................................... 12 5. Electrical specifications ......................................................................................................................... 13 5.1 PCIE topology description .......................................................................................................... 13 5.2 CCLINK interface description..................................................................................................... 14 5.3 Power Interface Description....................................................................................................... 16 6. BMC management system .................................................................................................................. 18 6.1 BMC functional description........................................................................................................ 18 Copyright 2020 Cambricon Corporation i Cambricon 7. Heat dissipation specifications........................................................................................................... 19 7.1 Description of the heat dissipation environment ............................................................... 19 7.2 Wind-resistance curve of MLU-X1000 system .................................................................... 20 7.3 MLU-X1000 Thermal Simulation Model ............................................... !
8. Optional components ........................................................................................................................... 21 8.1 PCIE High Speed Cable ............................................................................................................... 21 8.2 CCLINK High Speed Cable ......................................................................................................... 21 8.3 Network ............................................................................................................................................ 21 8.4 Hard hard disk ................................................................................................................................ 22 9. Cambricon NeuWare development environment .................................................................... 23 Copyright 2020 Cambricon Corporation ii C ambricon 1. Preface 1.1. Copyright Declaration Disclaimer Cambricon Technologies Corporation Limited(hereinafter referred to as "Cambricon ") does not represent, guarantee (express, implied or statutory) or guarantee the information contained in this document and expressly waives any and all implied guarantees of saleability, ownership, non-aggression of intellectual property or applicability for a specific purpose, and cambricon does not assume any liability arising from the application or use of any product or service. cambricon shall not be liable for any breach of contract, damages, costs or problems arising from :(1) any way of using cambricon products contrary to this Guide; or (2) customer product design. Limitation of liability In no case shall Cambricon be liable for any damage caused by the use or inability to use this Guide
(including but not limited to damage such as loss of profits, business disruption and loss of information), even if Cambricon has been advised that such damage may be suffered. Although the customer may suffer any damage for any reason, according to the terms and conditions of sale of the products of the Cambricon, the total and cumulative liability of Cambricon to the customer for the products described in this Guide shall be limited. Accuracy of information The information provided in this document is owned by Cambricon and Cambricon reserves the right to make any changes to this document information or to any products and services without notice. The information contained in this guide and all other information of the Cambricon documents cited in this guide are provided
"as is ". Cambricon does not guarantee the accuracy or completeness of information, texts, patterns, links or other items contained in this guide. Cambricon may make changes to this Guide or to the products described in this Guide without notice, but does not undertake to update this Guide. The performance tests and grades listed in this guide are to be measured using a specific chip or computer system or component. After such tests, the results shown in this guide reflect the general performance of Cambricon products. Any difference in system hardware or software design or configuration will affect actual performance. As mentioned above, Cambricon does not represent, warrant or guarantee that the products described in this Guide will apply for any particular purpose.Cambricon does not represent or guarantee testing all parameters of each product.The customer is solely responsible for ensuring that the product is suitable and applicable to the application of the customer plan and for performing the necessary tests on the application, Copyright 2020 Cambricon Corporation 1 Cambricon with a view to avoiding the default of the application or product. The fragility of customer product design can affect the quality and reliability of Cambricon products and lead to additional or different circumstances and/or requirements beyond the scope of this guide. Notice of Intellectual Property The Cambricon and Cambricon symbols are trademarks and/or registered trademarks of Cambricon Technologies Corporation Limited in the United States and other countries. Other companies and product names shall be trademarks of the respective companies associated with them. This guide is copyrighted and protected by the provisions of copyright laws and treaties worldwide.This guide can not be reproduced, reworked, modified, published, uploaded, published, transmitted or distributed in any way without the prior written permission of Cambricon. Except for the customer's right to use this guide information and products, according to this guide, Cambricon does not grant any other express or implied rights or permits. It is doubtful that the Cambricon does not grant any (express or implied) rights or permits to the customer based on any patent, copyright, trademark, trade secret or any other Cambricon intellectual property or ownership. Copyright Declaration 1.2. Versioning Cambricon Technologies Corporation Limited reserves all rights. 1Table 1.1 Version Record MLU-X1000 MLU extender Product Manual Document name Version number Author Date created V0.2.0 Cambricon 2020.07.10 1.3. Update history V0.2.0 Update time:
Update:
Initial version Copyright 2020 Cambricon Corporation 2 Cambricon 2. Overview MLU-X1000 MLU extender is a construction unit of artificial intelligence supercomputing. The extender inherits 4 MLU290-M5 intelligent processing cards, and provides up to 2 POPs of adaptive precision computing power. The supercomputing system from 4 cards to 16 cards is constructed by using the Cambrian CCLink inter chip direct connection technology, which provides a highly agile, highly reliable and high-performance computing foundation for the Artificial Intelligence Computing Center. Copyright 2020 Cambricon Corporation 3 3. Product Specification Overview Cambricon Specification indicators Model Core architecture Core frequency Video decoding Memory capacity Memory width Memory bandwidth ECC protection System interface CCLINK external interface CCLINK interface bandwidth TDP power consumption 3.1 Overview of Product Specification Parameters MLU-X1000 MLU extender Specification Parameters are as follows :
1Table 3.1 MLU-X1000 Specification Parameters Integer speed (INT8) 2048 TOPS (Dense) Calculation accuracy support INT16,INT8,INT4,FP32,FP16 Note MLU-X1000 Cambricon MLUv02 1GHz Support 128GB 16384 bits 4096GB/s Yes 8Ports 8*100 GB /S 2300W 2* PCI Express 4.0x16 Heat dissipation scheme Air-cooled, compatible with liquid-cooled 3.2 Overview of structural specifications The structure specifications of the MLU-X1000 MLU extender are as follows:
Copyright 2020 Cambricon Corporation 4 Cambricon Specification indicators Shape Weight Package Shape Package Weight 2Table 3.2 Structural Specification for MLU-X1000 437mm*87mm*735mm 1000mm*635mm*230mm Note 29Kg 39Kg 3.3 Overview of electrical specifications MLU-X1000 MLU extender electrical specifications as follows:
3Table 3.3 Electrical Specification for MLU-X1000 Specification indicators System interface Number of PCIE ports PCIE bandwidth Number of CCLINK ports CCLINK bandwidth BMC management interface Host management interface Input voltage Note PCIE Gen 4X 16 2Ports 128GB /s 8Ports 800GB /s IPMI V2.0 SMBUS AC100-127V,60/50Hz,Max. Output Power 1250W, AC200-210V,60/50Hz,Max. Output Power 2700W, AC210-240V,60/50Hz,Max. Output Power 3000W DC240V,Max. Output Power 3000W(China mainland only) When voltage is below AC200Vboth PSUs must be supplied at the same time. 3.4 Summary of heat dissipation specifications The heat dissipation specification of MLU-X1000 MLU extender is as follows:
4Table 3.4 Heat dissipation specifications of MLU-X1000 Specification indicators Note Working temperature 0-35, altitude of 900m below Working humidity 20%RH-85%RH Copyright 2020 Cambricon Corporation 5 Cambricon Storage temperature Storage humidity
-4075 5%RH-95%RH Noise SDP @23, sound power 7.2 bels Working altitude 3000 m (900-3000m, for each increase of 300 m supported working temperature drop 1) Copyright 2020 Cambricon Corporation 6 Cambricon 4.1 CMX -BB 1 4. Component Profile CMX -BB 1 is the baseboard which carries MLU290-M5 Intelligent processing card. Each CMX -BB 1 can carry 4 MLU290-M5 Intelligent processing cards. The details are shown in the following figure:
Serial 1Figure 4.1 CMX -BB1 graphic 1Table 4.1 CMX -BB1 Description Note Serial Note Copyright 2020 Cambricon Corporation 7 Cambricon number number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 CCLINK-0A &0B CCLINK-2A &2B CCLINK-1A &1B CCLINK-1A &1B PCIE 0 PCIE 1 IPMI UID COM HUB0 COM HUB1 COM HUB2 AC INDICATOR FRONT PANEL CONN. PDB MGT.CONN. OAM MODULE 0 16 17 18 19 20 21 22 23 24 25 26 27 28 29 OAM MODULE 2 OAM MODULE 1 OAM MODULE 3 F AN 4 F AN 3 F AN 2 F AN 1 F AN 0 PCIE SWITCH 0 PCIE SWITCH 1 54V POWER BUSBAR HANDLE 0 HANDLE 1 FRONT PCIE CONN. Copyright 2020 Cambricon Corporation 8 Cambricon 4.2 CMX -PA4 CMX -PA4 is a PCIE board, which is placed on the host server and provides Mini SAS HD interface for connection with MLU-X1000.The details are shown in the following figure:
2Figure 4.2 CMX -PA4 graphic 2Table 4.2 CMX -PA4 Description Serial number Note mini SAS HD CONN. PCIE GOLDEN FINGER PCIE RETIMER Serial number 3 Note 1 2 4.3 CMX -PDB CMX -PDB is the power distribution board.The details are shown in the following figure:
3Figure 4.3 CMX -PDB graphic 3Table 4.3 CMX -PDB Description Serial number Note Note Serial number Copyright 2020 Cambricon Corporation 9 Cambricon 1 2 3 4 4.4 CMX -LINKB 54V POWER BUSBAR PSU CONN.0 PSU CONN.1 INTRUTION 5 6 7 8 SSD POWER CONN.0 SSD POWER CONN.1 PDB MGT.CONN. 12V POWER BUSBAR CMX -LINKB is passive connection board.The details are shown in the following figure:
4Figure 4.4 CMX -LINK B graphic 4Table 4.4 CMX -PDB Description Note SSD MGT.CONN.0 SSD MGT.CONN.1 OCULINK 0 OCULINK 1 Serial number 5 6 7 8 Note OCULINK 2 OCULINK 3 IBB CONN. FRONT PCIE CONN. Serial number 1 2 3 4 4.5 CMX -IBB CMX-IBB is the backplane of Infiniband card. Each CMX-IBB can place two Infiniband cards.The details are shown in the following figure:
Copyright 2020 Cambricon Corporation 10 Cambricon Serial number 1 2 Serial number 1 2 3 4 5 5Figure 4.5 CMX -IBB graphic 5Table 4.5 CMX-IBB Description Note IBB CONN. IB SLOT 0 Serial number 3 Note IB SLOT 1 4.6 Front panel The front panel of the chassis is shown as follows:
6Figure 4.6 Front panel of chassis 6Table 4.6 Description of front panel of chassis Note Switching keys UID keys Reset button PSU 0 PSU 1 Serial number 7 8 9 10 11 Note SSD 1 SSD 2 SSD 3 NIC 0 NIC 1 Copyright 2020 Cambricon Corporation 11 Cambricon SSD 0 4.7 Back panel The rear panel of the chassis is shown as follows:
7Figure 4.7 Rear panel of chassis 7Table 4.7 Description of rear panel of chassis Serial number Serial number Note PCIE 0 PCIE 1 CCLINK-0A CCLINK-0B CCLINK-2A CCLINK-2B CCLINK-1A CCLINK-1B CCLINK-3A CCLINK-3B Note IPMI UID COM HUB 0 COM HUB 1 COM HUB 2 AC INDICATOR POWER CORD 0 POWER CORD 1 10 11 12 13 14 15 16 17 18 6 1 2 3 4 5 6 7 8 9 Copyright 2020 Cambricon Corporation 12 Cambricon 5. Electrical specifications 5.1 PCIE topology description MLU-X1000 MLU extender uses 2 miniSAS HD interfaces to connect with the host server, and there are 2 PCIE switching chips to connect the PCIE devices inside.PCIE interconnection topology is shown as follows:
NVMe NIC PCIe Switch PCIe Switch reserved NVMe NIC Host Server 1Figure 5.1 PCIE Connection topology PCIE signal rate is 16 Gbps, and the cable loss is controlled within 15 dB @8GHz. It is recommended to use 1 meter cable with a diameter of 30 AWG. The pins of the miniSAS HD connectors used by PCIE interfaces are defined as follows:
Copyright 2020 Cambricon Corporation 13 Cambricon miniSAS HD pin Note Pin internal processing 1Table 5.1 PCIE Interface pin definition RX [15:0]P/N PCIE input signal TX [15:0]P/N PCIE output signal External AC coupling capacitance External AC coupling capacitance SMBUS interface clock signal 4.7 K pull-up to 3.3 V SMBUS interface data signal 4.7 K pull-up to 3.3 V Reset signal REFCL K P/N PCIE clock signal PRESENT Opposite side in position detection signal 4.7 K pull-up to 3.3 V 5.2 CCLINK interface description MLU-X1000 MLU extender is equipped with 4 MLU290-M5 Intelligent processing cards, each card has 6 CCLINK ports. Among them, 4 ports are used for internal interconnection and 2 ports are used for external interconnection. The CCLink interconnection topology between the internal cards is as follows:
SMCLK SMDAT PERST#
Copyright 2020 Cambricon Corporation 14 Cambricon QSFP-DD 3 MLU290-1 MLU290-0 3 QSFP-DD Q S F P
D D 0 0 Q S F P
D D 1 2 4 5 4 2 5 1 Q S F P
D D 0 4 0 Q S F P
D D 1 2 5 4 2 5 1 QSFP-DD 3 MLU290-2 MLU290-3 3 QSFP-DD 2Figure 5.2 CCLINK internal connection topology CCLINK interconnection between extenders refer to the following figure:
Server QSFP-DD Rack Rack MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 MLU290-0 SerDes 0 MLU290-0 SerDes 3 MLU290-2 SerDes 0 MLU290-2 SerDes 3 MLU290-1 SerDes 0 MLU290-1 SerDes 3 MLU290-3 SerDes 0 MLU290-3 SerDes 3 3Figure 5.3 External CCLINK connection reference The signal rate of CCLINK is 50 Gbps, and the cable loss is controlled within 10 dB @12.5GHz. It is recommended to use 1 meter cable with a diameter of 30 AWG or 2 meter cable with a diameter of 28 AWG . CCLINK interface uses QSFP DD connectors whose pins are defined as follows:
Copyright 2020 Cambricon Corporation 15 Cambricon 2Table 5.2 CCLINK Interface pin definition QSFP-DD pins Note RX [8:1]P/N capacitance inside SERDES signal input with AC coupling External AC coupling TX [8:1]P/N capacitance inside capacitance is not SERDES signal output with AC coupling External AC coupling Internal processing of pins capacitance is not required required I2C interface clock signal of optical module 4.7 K pull-up to 3.3 V I2C interface data signal of optical module 4.7 K pull-up to 3.3 V ModPrsL Optical module in position signal output 4.7 K pull-up to 3.3 V Selection signal of optical module, default pull-
1K pull-down to up inside GND Reset signal, low level effective 4.7 K pull-up to 3.3 V Interrupt signal of optical module, OC gate, low 4.7 K pull-up to 3.3 V level indicates an interrupt signal Initialization mode 1K pull-down to GND VccRx,VccRx1,Vcc1,Vcc2 Power signal VccTx ,VccTx1 5.3 Power Interface Description MLU-X1000 MLU extender Input Power Requirements:
3Table 5.3 MLU-X1000 Input Power Supply Specifications Input voltage Max. Input Current AC 100-127V,60/50Hz AC 200-210V,60/50Hz AC 210-240V,60/50Hz DC 240V (China mainland only) 9.85 A-12.5A 12.8-13.5A 12.5-14.5A 12.5A SCL SDA ModSelL ResetL IntL InitMode MLU-X1000 MLU extender is able to reduce power consumption adjustment for instantaneous power changes above the s level. The power regulator can support power fluctuations within the ms level
(e.g .1.2 x TDP). Copyright 2020 Cambricon Corporation 16 Cambricon 4Table 5.4 EDPp specifications of MLU-X1000 EDP TBD Duration TBD Copyright 2020 Cambricon Corporation 17 Cambricon 6. BMC management system The BMC management system of MLU-X1000 is compatible with server management standards IPMI 2.0, with high reliability of hardware monitoring and management functions. 6.1 BMC function description MLU-X1000 MLU extender BMC management system main functions and features as follows:
1Table 6.1 BMC Functional description Function Note Remote control Management through SOL functions Information management Management of equipment model, asset information and version information Status monitoring Real-time monitoring of power supply, temperature, working status and other operating states information Heat dissipation control Modulate fan speed according to environment temperature, equipment working load and abnormal conditions Alarm management Report the alarm information in real time and deal with it accordingly WEB interface management Provides visual WEB interface for query and management IPMITool tool management Support IPMITool Copyright 2020 Cambricon Corporation 18 Cambricon 7. Heat dissipation specifications 7.1 Description of the heat dissipation environment The working environment of MLU-X1000 is as follows:
1Table 7.1 Working environment of MLU-X1000 Items Specification parameters Working environment temperature 0~35 Relative humidity 20%~85% no condensation Noise 62~88 dBA Note: There will be 62~88dbA noise during normal operation. Please take adequate sound insulation measures in advance. MLU-X1000 air volume description:
MLU-X1000 can provide up to 360 CFM of air volume Do not block the front and rear ventilation areas of the chassis during operation of MLU-X1000 When installing MLU-X1000, please reduce the air resistance around the inlet and outlet of the Please follow the instructions to arrange the cable to minimize the air resistance of the air flue Please install the chassis cover before using MLU-X1000. If CXM1000 is used without the chassis cover, the components may be damaged. If you need to replace the fan, please make sure to complete within 25s to avoid overheating of chassis the system. Copyright 2020 Cambricon Corporation 19 Cambricon 7.2 Wind resistance curve of MLU-X1000 The system wind resistance curve of MLU-X1000 is shown below:
1Figure 7.1 Wind resistance curve of MLU-X1000 2Table 7.2 Air Volume VS Pressure Drop of MLU-X1000 Air volume (CFM) Air pressure (Pa) 1737 1408 1044 735 0 400 360 310 260 0 Copyright 2020 Cambricon Corporation 20 Cambricon 8. Optional components 8.1 PCIE High Speed Cable MLU-X1000 uses miniSAS HD high-speed cable for PCIE Gen4 interconnection.Compatible cable models are as follows:
1Table 8.1 MLU-X1000 PCIE Compatible Cable Manufacturers Model Specifications Molex 2040431030 1 m ,30 AWG 8.2 CCLINK High Speed Cable MLU-X1000 uses QSFP-DD high-speed cable for CCLINK interconnection.Compatible cable models are as follows:
2Table 8.2 MLU-X1000 CCLINK Compatible Cable Manufacturers Model Specifications Molex Molex TE TE 2015911012 1 m 30 AWG 2015913020 2 m 28 AWG 2366016-4 1 m 30 AWG 2366101-3 2 m 28 AWG 8.3 Network MLU-X1000 can use InfiniBand network card or ROCE network card for cluster interconnection. Compatible network card models are as follows:
3Table 8.3 Network Card Compatibility Manufacturers Model Specifications Copyright 2020 Cambricon Corporation 21 Mellanox MCX653105A-HDAT Half high and half long single PCIE 4.0 Cambricon 8.4 Hard disk Compatible NVMe hard disk models for MLU-X1000 are as follows:
4Table 8.4 NVMe Hard Disk Compatibility Manufacturers Model Specifications HGST HUSMR7619BHP301 NVMe 1.92Tb Copyright 2020 Cambricon Corporation 22 Cambricon 9. Cambricon NeuWare development environment NeuWare fully supports various mainstream programming frameworks (e.g. TensorFlow Caffe PyTorch and MXNet). For the above programming framework, users can easily develop and deploy deep learning applications on Cambrian MLU290-M5. At the same time, the NeuWare provides a complete runtime system and driver software to facilitate the rapid integration of the system. NeuWare also provides a range of tools including application development, function debugging, performance tuning, etc. Among them, application development tools include machine learning library, runtime library, compiler, model retraining tool and specific field (such as video analysis field) SDK; function debugging tools can meet different levels of debugging requirements such as programming framework and function library; performance tuning tools include performance profiling tools and system monitoring tools. 1Figure 9.1 Cambricon NeuWare Copyright 2020 Cambricon Corporation 23 Cambricon 10. Compliance MLU-X1000 MLU extender is compliant with the regulations listed in this chapter. The compliance marks can be found on the labels of each devices. FCC statement This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation. This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense. Caution: Any changes or modifications not expressly approved by the party responsible for compliance could void the user's authority to operate this equipment. CE statement
- This product must not be used in residential areas.
- This product may cause interference if used in residential areas. Such use must be avoided unless the user takes special measures to reduce electromagnetic emissions to prevent interference to the reception of radio and television broadcasts. Copyright 2020 Cambricon Corporation 24