Instinct - AMD MI210

Acknowledgments

We greatly appreciate the donation of this test platform from AMD to Dr. Spencer Bryngelson, a CRNCH faculty member working on computational fluid dynamics (CFD) codes including MFC.

Current Status

BUGS / Feature Requests

  • N/A

System Specifications

Queues CPU Memory (GB) Network Cards Notes
rg-gpu 2x AMD EPYC 7713 (Milan) 512 GB DDR4, 3200 MHz, 32 GB DIMMs Connect-X 6 (MT28908), 10 GE 2x MI210  

Instinct is a single node server with two AMD 210 GPUs, which are very similar to the MI250x GPUs deployed in ORNL’s Frontier. This server has two Milan CPUs, 512 GB of DDR4 memory, and a Connect-X 6 networking card.

Software and Tools

Distro Kernel Standard Compilers Other Compilers MPI Miscellaneous
RHEL8 4.18.0 GCC 8.5 ROCm 5.2.0, AOCC 3.2.0   AOCL 3.2.0, ROCm libraries 5.2.0, uProf 3.6

How do I get to Instinct?

As with most CRNCH resources, you need to either log in via the gateway node, rg-login, or access the system from the campus network via VPN or an on-campus connection.

To request an allocation on Instinct using slurm:

//Request an allocation of 1 hr, partition rg-gpu, and specify the node name for the server with -w
salloc -t 1:00:00 -p rg-gpu -w instinct

To request an entire node (and all the memory) you can run:

//Request an allocation of with 2 sockets, all cores in each socket, and 2 threads per core, partition rg-gpu,
//specify the node name for the server with -w, and request unlimited memory (otherwise cgroups limits to 1 GB per core)
salloc --sockets-per-node=2 --cores-per-socket=64 --threads-per-core=2 -p rg-gpu -w instinct --mem=0

Compiling for the MI210 GPUs

Currently we suggest using the ROCm compiler stack to compile or GCC 12+.

Modules should be available but you may need to update your module path:

 []$ module avail

 -------------------------------------------- /projects/tools/x86_64/rhel-8/modulefiles -------------------------------------
aocc/3.2.0    aocl/aocc-3.2.0    aocl/gcc-3.2.0 (D)    rocm/5.2.0

 []$ module load rocm/5.2.0
 []$ which hipcc
 /opt/rocm-5.2.0/bin/hipcc

Useful ROCm tools include hipcc, amdclang, aompcc, amdflang, rocgdb, and rocprof.

Running OpenACC with GCC13

Alex Woods has compiled GCC13 for testing OpenACC on the Instinct GPU, and we have put his script in the internal docs here When compiling with the new gcc install, add the flags -fopenacc -foffload-options=-march=gfx90a for the Instinct MI2XX GPUs. Note that performance will likely be slow.

Running jobs

Once you have compiled your code, you can request a longer job to do testing.

Useful training material

  • OmniTrace - AMD’s newest tracing and profiling tool.

Vendor-provided Documents and Resources