ESSS Rocky’s 200 million particle simulation on Oracle Cloud pushes DEM analysis to a new height
Published on: March 4, 2022
The largest particle model ever reported using a CFD-DEM simulation applies the most powerful NVIDIA GPU shapes of Oracle Cloud to optimize a gas cyclone separator design.
Oracle Cloud Infrastructure (OCI) and Engineering Simulation and Scientific Software (ESSS) are elevating Discrete Element Modeling (DEM) technology with NVIDIA through the successful demonstration of a first-of-its-kind simulation of 200 million particle analysis using ESSS’ Rocky DEM software on OCI’s bare-metal GPU shape.
The NVIDIA A100 Tensor Core GPU was a key component in this simulation. It allows CUDA-based GPU-accelerated software such as Rocky DEM to utilize its full capacity, handling tens of millions of particles deployed in a DEM simulation model. Enabled by Rocky’s multi-GPU parallel processing capabilities and the power of OCI’s bare-metal GPU systems, the demo dramatically expands the range of applications where DEM simulations can analyze problems that include hundreds of millions of particles. In addition, the most recent release of Rocky, a powerful parallel load-balancing algorithm, enables more efficient GPU resource utilization than previous versions.
Recently, OCI and ESSS designed a study for a gas cyclone separator, a device commonly used to separate and remove particles from an airflow stream. Such a device can be found in home vacuum cleaners and has many industrial uses including separating abrasives, pollutants, dust particles, and various production plant emissions. In this study, particle-laden air was injected into a centimeter-sized cyclone. While the air swirled inside the cyclone, dust particles of different diameters bumped into each other, bouncing off the interior walls of the cyclone. The rapid swirling motion allowed the heavier particles to move towards the cyclone walls making their way out through the bottom outlet while somefiner particles and a cleaner air stream are released from the top outlet.
Looking under the hood
Solving a problem as big as 200 million particles demands powerful computing hardware, which OCI provides in the form of a tailored cloud computing infrastructure, as illustrated in Figure 1. The muscle of this solution is the Bare Metal BM.GPU4.8 server instance, loaded with eight NVIDIA A100 Tensor Core GPUs with 40GB of GPU memory in each, for a server total of 320GB in GPU memory. Rocky solver was installed on top of Oracle Linux, an Oracle-optimized general-purpose Linux distribution that carries all required dependencies to enable NVIDIA GPUs and Rocky DEM software.
However, the challenging computational work doesn’t stop when the simulation completes. Post-processing a large model is a compute-intensive task also. For that purpose, hardware-accelerated rendering with GUI-enabled instances is a must, and we selected OCI’s VM.GPU3.1 virtual machine, featuring the NVIDIA V100 Tensor Core GPU.
With the computing hardware taken care of, there was still one concern: storage of the results. Huge simulation cases tend to generate a massive amount of data, which uses a lot of disk space. To avoid moving such a large amount of data between the processing server, BM.GPU4.8 and the post-processing server VM.GPU3.1, a more efficient storage solution was developed. A third virtual machine was launched to serve as an NFS-server (Network File System) with terabyte-level block storage mounted to it. This storage solution made intermediate output data available for post-processing on the fly, and also allowed the bare-metal BM.GPU4.8 server to be released when the simulation work was finished, creating a more cost-effective solution. Similarly, the post-processing virtual machine could be released while waiting for new intermediate output data.
Particle simulation on Oracle Cloud: results
The solution was obtained using a CFD-DEM coupling approach. The computational fluid dynamics (CFD) part, meaning the airflow field solution, was obtained from Ansys Fluent and used in Rocky through its one-way coupling approach. In the other end, Rocky handled the entire DEM portion as well as particle-fluid interaction computations.
The geometry of the cyclone and CFD mesh are shown in Figure 2. Cyclone has 0.3m of height and 0.15m of diameter, air is injected at 30m/s and simulation was computed for 0.12s. Particles start leaving the domain at 0.025s, meaning the flow was solved for approximately 5 flow-through times. The dust particles carried by the airflow, sized 5 to 50μm (10-6 meters), were injected into the cyclone at ~2 billion particles/second, bringing cyclone mass load to ~50% level.
Simulation numbers are summarized in Table 1 below:
The output of the simulation can be seen in the animation video below. The particle count grows quickly from zero to tens of millions, creating the expected spiral airflow pattern. The injection rate was achieved and, at the end of the video, a total of 184 million particles were simultaneously swirling in the cyclone. At this point approximately 200GB GPU memory was used, leaving more than one-third of total GPU memory available. This headroom will be used in expanded Rocky models capturing more detail and simulating more particles in the future on OCI’s BM machines.
Vinicius Daroz earned his BS in Mechanical Engineering and an MSc in Engineering Turbulence from the Post-Graduate Program of Mechanical and Materials Engineering (PPGEM) at Federal University of Technology, Paraná (UTFPR). He is currently working at ESSS as a CFD-DEM Applications Engineer on the Rocky DEM technical team. Before joining ESSS, Vinicius was a researcher at the Center of Research on Rheology and Non-Newtonian Fluids (CERNN), working with the Oil & Gas industry to study the transport of solids in turbulent flows.