The world first hyper-converged and 100% ‘hot water’ liquid cooled supercomputer at JINR
Technical DataSheet Суперкомпьютер имени Н.Н. Говоруна
Technical Info Суперкомпьютер имени Н.Н. Говоруна
- Performance:138 TFLOPS + 73 TFLOPS
- Nodes / Сores (CPU + CoCPU) / Threads (CPU + CoCPU):Intel Xeon (40 / 1440 / 2880) + Intel Xeon Phi (21 / 1512 / 6048)
- CPU:Intel® Xeon® Gold 6154 (18 cores, 3.0 GHz) / Intel® Xeon Phi™ 7290 (72 cores, 1.5 GHz, 16 GB GDDR5)
- Motherboard:Intel® Server Board S2600BP / Intel® Server Board S7200AP
- Total RAM of the nodes:7.7 TB / 2 TB
- Total disk storage of the nodes:80 ТБ NVMe
- Interconnect:Intel Omni-Path 100 Gb/s
- Service network:2x 1/10 GigE independent sensor and control networks
- Occupied Area:1.92 м.кв. (20.67 sq. ft.)
RSC Group, a leading Russian and well-known in the world supercomputing industry developer and integrator of innovative ultrahigh-dense, scalable and energy efficient solutions for high-performance computing (HPC) and data centers, deployed in March 2018 the world first hyper-converged and 100% ‘hot water’ liquid cooled supercomputer at Joint Institute for Nuclear Research (JINR) located in Dubna, Moscow Region (Russia). This new system was named after Nikolay Govorun, former Director of JINR’s Computing and Automation Lab in 1988-1989, who worked on development of information technologies in this international scientific organization since 1966.
New supercomputer of the Joint Institute for Nuclear Research has been developed and deployed for the Theoretical Physics Laboratory (TPL) named after N.N. Bogolyubov and the Information Technology Laboratory (ITL) with support of JINR management. The project was completed by RSC Group with Intel’s support.
“It's great to note that we launched the new heterogeneous hyper-converged supercomputer named after professor Govorun at JINR’s Information Technology Laboratory in the year of 60th anniversary of commissioning of the first Ural-1 supercomputer at our institute in 1958. Our scientists and research groups now have a powerful and modern tool that will greatly accelerate theoretical and experimental research of nuclear physics and condensed matter physics,” – said Vladimir Korenkov, Director of the Information Technology Laboratory of the Joint Institute for Nuclear Research.
“We are glad that RSC’s innovative HPC solutions will help to advance the Russian science and to improve the efficiency of international scientific research cooperation that has a long history in JINR,” - said Alexander Moskovsky, CEO of RSC Group.
The project involves plans of radical acceleration of comprehensive theoretical and experimental research in the fields of particle physics, nuclear physics and condensed matter physics. This includes deployment of NICA accelerator complex based at JINR to recreate quark-gluon plasma – a special state of matter of the Universe in first moments after the Big Bang. NICA mega-project started in 2013 and it is planned to commission this accelerator facility in 2020.
New supercomputer of the Joint Institute for Nuclear Research will enable resource-intensive massive parallel calculations to solve problems of lattice quantum chromodynamics for studying the properties of hadronic matter with high energy density and baryon charge and in presence of strong electromagnetic fields. It will greatly accelerate simulation of collision dynamics of relativist heavy ions and provide new resources to study the properties of highly correlated systems in the physics of new materials. It will also be used to develop and adapt software for NICA mega-project for new computing architectures, create an innovative hardware and software environment for high performance computing and simulating complex scientific problems and to train IT specialists in all relevant areas.
New JINR's supercomputer is a natural development of HybriLIT heterogeneous platform with significant increase of performance. Theoretical peak performance of the new supercomputer is about 500 Teraflops (in double precision).
The new supercomputer includes hyper-converged, high dense and energy efficient RSC Tornado solution based on Intel server products with 100% ‘hot water’ direct liquid cooling. The Joint Institute for Nuclear Research deployed new universal RSC Tornado computing cabinets with record-breaking power density and precision liquid cooling balanced for continuous usage of high temperature cooling agent (up to +63°C at cabinet entry point). JINR’s specifications required optimal cabinet temperature mode with +45°C constant cooling agent temperature at node entries (+57°C peak temperature).
‘Hot water’ mode enabled deployment of free cooling solution (24x7x365) using only dry coolers running at up to +50°C ambient temperature and complete elimination of freon circuit and chillers. As the result, average power efficiency factor (PUE) is less than 1.027. The system uses less than 3% of consumed power for cooling, which is an outstanding result for HPC industry.
Computing nodes of JINR system are based on Intel server products: the most powerful 72-core Intel® Xeon Phi™ 7290 processors, Intel® Xeon® Scalable family processors (Intel® Xeon® Gold 6154), Intel® Server Board S7200AP and Intel® Server Board S2600BP, Intel® SSD DC S3520 with SATA interface in M.2 form-factor and the newest high-speed Intel® SSD DC P45xx drives NVMe interface and 1TB capacity.
High-speed data transfer between JINR’s supercomputer nodes uses innovative Intel® Omni-Path Architecture communication technology providing up to 100 Gbps non-blocking switching rate. It is based on 48-port Intel® Omni-Path Edge Switch 100 Series with 100% liquid cooling to provide high efficiency of the cooling system in ‘hot water’ mode and the lowest TCO of the system. Intel® Omni-Path Architecture meets current requirements of resource-intensive user applications and provides sufficient throughput reserve for the future.
Innovative monitoring and management system based on RSC BasIS software stack guarantees high availability, redundancy and simplicity of usage of computing systems based on RSC solutions. It can be used to manage single nodes and the entire solution, including infrastructure components. All elements of the system (computing nodes, power supplies, hydraulic regulation modules, etc.) have an integrated management module providing broad capabilities for detailed telemetry and flexible management. Cabinet design supports replacement of computing nodes, power supplies and hydraulic regulation modules (with redundancy) in hot-swap mode without interruption of system operation. Most components of the system (such as computing nodes, power supplies, network and infrastructure components, etc.) are software-defined, and this significantly simplifies and speeds up initial deployment, maintenance and future upgrades of the system. 100% liquid cooling of all components ensures their longevity.
Joint Institute for Nuclear Research (JINR) is an international inter-governmental scientific research organization established by eleven member countries in 1956. Now 18 countries are members of JINR: Azerbaijan, Armenia, Belarus, Bulgaria, Vietnam, Georgia, Kazakhstan, Democratic People's Republic of Korea, Cuba, Moldova, Mongolia, Poland, Russia, Romania, Slovakia, Uzbekistan, Ukraine, Czech Republic. The institute has government-level Cooperation Agreements with Hungary, Germany, Egypt, Italy, Serbia and South African Republic.
Main areas of scientific and experimental research In JINR: particle physics, nuclear physics, condensed matter physics. International Scientific Council is responsible for development of JINR’s science policy.
JINR includes seven laboratories: Theoretical Physical Laboratory named after N. N. Bogolyubov, Nuclear Problems Laboratory named after V.P. Jelepov, Nuclear Reactions Laboratory named after G.N. Flerov, High Energy Physics Laboratory named after V.I. Vexler and A.M. Baldin, Neutron Physics Laboratory named after I.M. Frank, Information Technology Laboratory, Radiation Biology Laboratory. Each laboratory conducts research on a scale of a large academic research institute. JINR has about 4500 staff employees including over 1200 scientists with members and correspondent members of national academies of sciences, over 260 doctors and 570 candidates of sciences, about 2000 engineering and technical employees.
JINR has an impressive collection of experimental physical appliances: the first superconducting accelerator of heavy cores and heavy ions in Europe and Asia (nuclotron), U-400, U-400M cyclotrons, new DC-280 cyclotron, unique neutron pulse reactor IBR-2 and proton accelerator (FM cyclotron) used for radiation treatment. JINR successfully works on construction of a mega science grade experimental facility — NICA (Nuclotron based Ion Collider facility). JINR actively works on creating a modern heavy ion accelerator complex DRIBs (Dubna Radioactive Ion Beams) and a super heavy element factory, which is a key part of this project.
JINR has powerful HPC facilities integrated with global computer networks through high-speed communication channels (100 Gbps). The project of JINR’s Multi-Functional Information and Computing Facility is intended to create a technological base for scientific research in unified computing environment combining numerous technological solutions, concepts and techniques. JINR successfully operates the only Tier-1 grid center in JINR countries for СМS experiment on LHC, which became one of the best CMS centers in WLCG ‒ global grid infrastructure for data processing, analysis and storage.