Instinct - AMD MI210¶
Acknowledgments¶
We greatly appreciate the donation of this test platform from AMD to Dr. Spencer Bryngelson, a CRNCH faculty member working on computational fluid dynamics (CFD) codes including MFC.
System Specifications¶
| Queues | CPU | Memory (GB) | Network | Cards | Notes |
|---|---|---|---|---|---|
| rg-gpu | 2x AMD EPYC 7713 (Milan) | 512 GB DDR4, 3200 MHz, 32 GB DIMMs | Connect-X 6 (MT28908), 10 GE | 2x MI210 |
Instinct is a single node server with two AMD 210 GPUs, which are very similar to the MI250x GPUs deployed in ORNL’s Frontier. This server has two Milan CPUs, 512 GB of DDR4 memory, and a Connect-X 6 networking card.
Software and Tools¶
| Distro | Kernel | Standard Compilers | Other Compilers | MPI | Miscellaneous |
|---|---|---|---|---|---|
| RHEL8 | 4.18.0 | GCC 8.5 | ROCm 5.2.0, AOCC 3.2.0 | AOCL 3.2.0, ROCm libraries 5.2.0, uProf 3.6 |
How do I get to Instinct?¶
As with most CRNCH resources, you need to either log in via the gateway node, rg-login, or access the system from the campus network via VPN or an on-campus connection.
To request an allocation on Instinct using slurm:
//Request an allocation of 1 hr, partition rg-gpu, and specify the node name for the server with -w
salloc -t 1:00:00 -p rg-gpu -w instinct
To request an entire node (and all the memory) you can run:
//Request an allocation of with 2 sockets, all cores in each socket, and 2 threads per core, partition rg-gpu,
//specify the node name for the server with -w, and request unlimited memory (otherwise cgroups limits to 1 GB per core)
salloc --sockets-per-node=2 --cores-per-socket=64 --threads-per-core=2 -p rg-gpu -w instinct --mem=0
Compiling for the MI210 GPUs¶
Currently we suggest using the ROCm compiler stack to compile or GCC 12+.
Modules should be available but you may need to update your module path:
[]$ module avail
-------------------------------------------- /projects/tools/x86_64/rhel-8/modulefiles -------------------------------------
aocc/3.2.0 aocl/aocc-3.2.0 aocl/gcc-3.2.0 (D) rocm/5.2.0
[]$ module load rocm/5.2.0
[]$ which hipcc
/opt/rocm-5.2.0/bin/hipcc
Useful ROCm tools include hipcc, amdclang, aompcc, amdflang, rocgdb, and rocprof.
Running OpenACC with GCC13¶
Alex Woods has compiled GCC13 for testing OpenACC on the Instinct GPU, and we have put his script in the internal docs here When compiling with the new gcc install, add the flags -fopenacc -foffload-options=-march=gfx90a for the Instinct MI2XX GPUs. Note that performance will likely be slow.
Running jobs¶
Once you have compiled your code, you can request a longer job to do testing.