ARC: A Root Cluster for Research into Scalable Computer Systems

Official Annoucement of the ARC Cluster (local copy)
NCSU write-up on the ARC Cluster (local copy)
TechNewsDailyStory (local copy)

main system funded in part by NSF through CRI grant #0958311

Cooling door equipment and installation funded by NCSU CSC; GPUs funded in part by a grant from NCSU ETF funds, and by NVIDIA and HP donations


1728 cores on 108 compute nodes integrated by Advanced HPC. All machines are 2-way SMPs with AMD Opteron 6128 (Magny Core) processors with 8 cores per socket (16 cores per node).

6128 Opteron Processor (single core):
  • 128KB split (64KB+64KB) I+D L1 caches, 2-way associative, 64B/line (private)
  • 512KB L2 cache, 16-way associative, 64B/line (private)
  • 12MB L3 cache, 96-way associative, 64B/line (shared)
  • 800MHz-2GHz Core speed
  • 3200MHz bus speed (HT)
  • 1800MHz memory controller
  • 80 Watts peak power (1.3V)
  • 45nm SOI CMOS fab
  • G34 socket


Networking, Power and Cooling:


System Status


All software is 64 bit unless marked otherwise.

Obtaining an Account

Accessing the Cluster

Using OpenMP (via Gcc)

Running CUDA Programs (Version 7.0)

Running MPI Programs with Open MPI and Gcc (Default)

Running MPI Programs with MVAPICH and Gcc (Alternative)

MPI Job Submission with Torque (Default)

Batch submission is realized via Torque via OpenPBS over the Maui Cluster Scheduler.

Using the PGI compilers V13.9 (Alternative)

(includes OpenMP and CUDA support via pragmas, even for Fortran)

Dynamic Voltage and Frequency Scaling (DVFS)

Power monitoring

Sets of three compute nodes share a power meter; in such a set, the lowest numbered node has the meter attached (either on the serial port or via USB). In addition, two individual compute nodes have power meters (with different GPUs). See this power wiring diagram to identify which nodes belong to a set. The diagram also indicates if a meter uses serial or USB for a given node. We recommend to explicitly request a reservation for all nodes in a monitored set (see qsub commands with host name option). Monitoring at 1Hz is accomplished with the following software tools (on the respective nodes where meters are attached):

Virtualization with KVM

Virtualization support is realized via KVM.

Follow instructions for VM creation and see the MAC guidelines for network connectivity.





Hadoop Map-Reduce and Spark

Simple setup of multi-node Hadoop map-reduce with HDFS, see also free AWS setup as an alternative and the original single node and cluster setup. But follow the instructions below for ARC. Other components, e.g., YARN, can be added to the setup below as well (not covered). To get rid of ssh errors, you need to add a secondary node server and other optional services. This is not required, it's an option.

You can also run Spark on top of Hadoop as follows, which will also default to the HDFS file system:

export SPARK_DIST_CLASSPATH=$(hadoop classpath)
export SPARK_HOME=/usr/local/spark
$SPARK_HOME/bin/run-example SparkPi 10

Advanced topics (pending)

For all other topics, access is restricted. Request a root password. Also, read this documentation, which is only accessible from selected NCSU labs.

This applies to:

Known Problems

Consult the FAQ. If this does not help, then please report your problem.


Additional references: