ARC: A Root Cluster for Research into Scalable Computer Systems

main system funded in part by NSF through CRI grant #0958311

GPUs funded in part by a grant from NCSU ETF funds

GPUs funded in part by nVidia donations

Hardware

1728 cores on 108 compute nodes integrated by Advanced HPC. All machines are 2-way SMPs with AMD Opteron 6128 (Magny Core) processors with 8 cores per socket (16 cores per node).

6128 Opteron Processor (single core): 128KB split (64KB+64KB) I+D L1 caches, 2-way associative, 64B/line (private) 512KB L2 cache, 16-way associative, 64B/line (private) 12MB L3 cache, 16-way associative, 64B/line (shared) 800MHz-2GHz Core speed 3200MHz bus speed (HT) 1800MHz memory controller 80 Watts peak power (1.3V) 54nm SOI CMOS fab G34 socket Nodes: Supermicro H8DGG-QF Motherboard with IPMI 2.0 Transport 1022GG-TF with 2 GPU slots + 1 half-size PCI-E slot 32GB DRAM Mellanox ConnectX-2 VPI InfiniBand Adapter Card - Part ID: MHQH19B-XTR login node: login-0-0 (1TB HDD+nVidia C2050+ OCZ RevoDrive 80GB SSD under /mnt) nodes: compute-0-XXX, XXX=0..107 (1TB HDD, OCZ RevoDrive 120GB SSD under /mnt) 12 nodes with nVidia C2070 36 nodes with nVidia C2050 60 nodes with nVidia GTX480 5 PFS (Lustre) nodes: mds-0-114 and oss-0-YYY, YYY=108-111 (each OSS has 10TB HDDs as RAID5) 1 experimental node: compute-0-115 head node: arcs (has 12TB HDDs as RAID5) with Supermicro H8DGU-F Motherboard and Transport 2022G-URF backup node: arcm (same configuration as arcs)

• 8 Mellanox InfiniScale IV IS5025 QDR 36-Port InfiniBand Switches (fat tree)
• 1 Mellanox InfiniScale IV IS5030 QDR 36-Port InfiniBand Switch (managed)
• 3 48-Port SMC SMC8848M 10/100/1000Mbps Managed Layer 2 Switches (stacked, eth0)
• 3 48-Port Cisco Linksys SRW2048 WebView Managed Gigabit Switches (eth1)
• 1 16-Port Cisco Linksys SRW2016 WebView Managed Gigabit Switch (eth1)
• 1 APC Smart-UPS 1500VA
• 40 Cyberpower PDU15BV14F PDUs
• 41 Watts Up? Pro Energy Monitor 99333 (USB) / older serial version
• 4 APC AR3100 NetShelter SX 42U racks
• 4 Coolcentric RDWTS-04 passive cooling doors
• 2 Web Power Switch devices

Pictures

System Status

• Monitor the System Status with Ganglia (only some subnets in NCSU domain)
  • Logged Problems
• Monitor the Server Room Temperature (should be < 80F)
  • Monitor Temperature over Time (should be < 80F)

Software

• Rocks 5.3 / CentOS 5.7 Linux x86_64
  • Roll Documentation
• OpenFabrics 1.5.4
• gcc/gfortran (C compiler)
• MVAPICH
• Open MPI
• Torque/Maui (PBS)
• Sun/Oracle Grid Engine
• OpenMP (via Gcc or PGI compilers)
• NVIDIA CUDA
• PGI compilers 10.5
• ATLAS (BLAS)
  
• Lustre
• PVFS2

Obtaining an Account

• for NCSU students/faculty/staff in Computer Science:
  • Send an email to your advisor asking for ARC access and indicate your unity ID.
  • Have your advisor endorse and forward the email to David Fiala.
  • If approved, you will be sent a secure link to upload your public DSA key for SSH access.
• for NCSU students/faculty/staff outside of Computer Science:
  • Send a 1-paragraph project description with estimated compute requirements (number of processors and compute hours per job per week) in an email to your advisor asking for ARC access and indicate your unity ID.
  • Have your advisor endorse and forward the email to David Fiala.
  • If approved, you will be sent a secure link to upload your public DSA key for SSH access.
• for non-NCSU users:
  • Send a 1-paragraph project description with estimated compute requirements (number of processors and compute hours per job per week) in an email to your advisor asking for ARC access. Indicate the domain name that you will login from (e.g., csc.ncsu.edu). In an attachment, provide a public DSA key (a file named id_dsa.pub) with a signature of the machine that you will use to access the cluster.
  • Have your advisor endorse and forward the email to David Fiala.
  • If approved, you will be sent a secure link to upload your public DSA key for SSH access.

Accessing the Cluster

• Login for NCSU users:
  • Login to a machine in the .ncsu.edu domain.
  • Then issue:
    • ssh arc.csc.ncsu.edu
  • Or use your favorite ssh client under Windows from an .ncsu.edu machine.
• Login for users outside of NCSU:
  • Login to the machine that your public key was generated on.
    Non-NCSU access will only work for IP numbers that have been added as firewall exceptions, so please use only the computer (IP) you indicated to us any other computer will not work.
  • Then issue:
    • ssh arc.csc.ncsu.edu
  • Or use your favorite ssh client under Windows.

Using OpenMP (via Gcc)

• The "#pragma omp" directive in C programs works.

  gcc -fopenmp -o fn fn.c
  

Running CUDA Programs

• Append to your ~/.bashrc:

  export PATH=".:~/bin:/usr/local/bin:/usr/bin:$PATH"
  export PATH="/usr/local/cuda/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib:$LD_LIBRARY_PATH"
  export MANPATH="/usr/share/man:$MANPATH"
  

  Log out and back in to activate the new settings.

• Install the SDK in your directory:

  sh /home/rocks/install/contrib/5.3/x86_64/nvidia/4.1rc2/gpucomputingsdk_*_linux.run
  

• Test the SDK:

  cd NVIDIA_GPU_Computing_SDK/C
  #edit common/common.mk, before "# This line invokes..." add
    #ADD THIS LINE HERE
    LIB += -lpthread
  make
  ./bin/linux/release/bandwidthTest
  ./bin/linux/release/matrixMul
  

• Running on the cluster: see Torque below for details
  • qsub -q cuda ... # job submitted to GPU/CUDA queue (GTX480/C2050/C2070) -- compromise between single/double precision performance
  • qsub -q gtx480 ... # job submitted to GPU/CUDA queue (GTX480) -- best for single precision arithmetic
  • qsub -q c2050 ... # job submitted to GPU/CUDA queue (C2050/C2070) -- best for double precision arithmetic
  • qsub -l nodes=2:ppn=1 -q cuda ... # job for two tasks on two nodes with GPU/CUDA support

• Developing/Debugging CUDA Programs under Windows with Parallel Nsight (free download, you need a CUDA-capable NVIDIA card under Windows for this)

Running MPI Programs with Open MPI and Gcc on the Login Node (Default)

• Compile MPI programs:

  mpicc -O3 -o pi pi.c
  

• Execute the program on 2 processors (using Open MPI over TCP on eth0 interface):

  mpirun -np 2 pi
  

• Notice: You will automatically use TCP on the login node but Infiniband (32X faster) when running on compute nodes under Open MPI.

Running MPI Programs with MVAPICH and Gcc (Alternative)

• Create the file hostfile.
• Append to your file ~/.bashrc:

  export PATH="/usr/mpi/gcc/mvapich-1.2.0/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/mpi/gcc/mvapich-1.2.0/lib/shared:$LD_LIBRARY_PATH"
  

  Log out and back in to activate the new settings.
• Compile MPI programs:

  mpicc -O3 -o pi pi.c
  

• Execute the program on 2 processors (using MVAPICH):

  qsub -I -l nodes=2
  mpirun -np 2 -hostfile $PBS_NODEFILE pi
  

• Notice: You need to be on a compute node to run with MVAPICH as it uses Infiniband (no support for any other protocols provided).

• MVAPICH2: Append to your file ~/.bashrc:

  export PATH="/usr/mpi/gcc/mvapich2-1.7/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/mpi/gcc/mvapich2-1.7/lib/shared:$LD_LIBRARY_PATH"
  

  Log out and back in to activate the new settings.

MPI Job Submission with Torque (Default)

• On login-0-0, issue:
  • to submit: qsub ...
    • qsub -q cuda ... # job submitted to GPU/CUDA queue (GTX480 or C2050) -- compromise between single/double precision performance
    • qsub -q gtx480 ... # job submitted to GPU/CUDA queue (GTX480) -- best for single precision arithmetic
    • qsub -q c2050 ... # job submitted to GPU/CUDA queue (C2050) -- best for double precision arithmetic
    • qsub -l ncpus=4 ... # ask for four tasks (processors) -- packed as up to 16 tasks per node
    • qsub -l nodes=4:ppn=16 ... # job for four nodes with 16 processors on each node (64 tasks)
    • qsub -l nodes=2:ppn=1 -q cuda ... # job for two tasks on two nodes with GPU/CUDA support
    • qsub -l nodes=2,cput=00:5:00 ... # job for two tasks + 5 minutes CPU time
    • qsub -l nodes=2,walltime=01:00:00 ... # job for two tasks + 1 hour wall time
    • qsub -W depend=afterany:1234 ... # job starts after job 1234 has finished (successfully or not)
  • to submit interactive: qsub -I # one node, shell will open up
  • to submit interactive: qsub -I -nodes=20 #two nodes w/ 20 tasks
  • to submit interactive: qsub -I -l host=compute-0-54.local #specifically on node 54
  • to submit interactive: qsub -I -l host=compute-0-54.local+compute-0-55.local #on 54+55
  • to submit interactive with X11: qsub -I -X ...
  • to check job status (PBS): qstat
  • to check job queues (Maui): showq ...
  • to remove: qdel ...
  • to check node status: pbsnodes ...
  • see documentation: "man pbs", qsub and Torque link above
• Sample job script for Open MPI
• Sample job script for MVAPICH
• to run a command on all nodes of a job: pbsdsh ...

Using the PGI compilers V10.5 (Alternative)

(includes OpenMP and CUDA support via pragmas, even for Fortran)

• Append to your ~/.bashrc:

  PGI=/usr/local/pgi; export PGI
  PATH=/usr/local/pgi/linux86-64/10.5/bin:$PATH
  MANPATH=$MANPATH:$PGI/linux86/10.5/man
  LM_LICENSE_FILE=/usr/local/pgi/license.dat
  export LM_LICENSE_FILE PATH
  

  Log out and back in to activate the new settings.
  • For Fortran 77, use: pgf77 -V x.f
  • For Fortran 95, use: pgf95 -V x.f
  • For HPF, use: pghpf -V x.f
  • For C++, use: pgCC -V x.c
  • For ANSI C, use: pgcc -V x.c
  • For debugging, use: pgdbg

  • For AMD 64-bit, add option: -tp=barcelona-64
  • For OpenMP, add option: -mp
  • For CUDA, add options: -Mcuda -ta=nvidia,cc20
    • with filename.cuf: supports Fortran CUDA pragmas, e.g., !$cuf kernel do(1) <<< *, 128 >>>
    • with filename.f: supports Fortran ACC pragmas (for CUDA), e.g., !$acc do parallel
    • with filename.c: supports C ACC pragmas (for CUDA), e.g., #pragma acc for parallel
• PGI Documentation
  • PGI Accelerator Quick Reference
• OpenAcc Documentation
  • OpenAcc Quick Reference

• PGI+Open MPI, append to your ~/.bashrc:

  export PATH="/usr/mpi/pgi/openmpi-1.5.1/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/mpi/pgi/openmpi-1.5.1/lib64:$LD_LIBRARY_PATH"
  

• PGI+MVAPICH: Append to your file ~/.bashrc:

  export PATH="/usr/mpi/pgi/mvapich-1.2.0/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/mpi/pgi/mvapich-1.2.0/lib/shared:$LD_LIBRARY_PATH"
  

  Log out and back in to activate the new settings.
• PGI+MVAPICH2: Append to your file ~/.bashrc:

  export PATH="/usr/mpi/pgi/mvapich2-1.7/bin:$PATH"
  export LD_LIBRARY_PATH="/usr/mpi/pgi/mvapich2-1.7/lib/shared:$LD_LIBRARY_PATH"
  

  Log out and back in to activate the new settings.

Power monitoring

• export PATH=".:~/bin:/usr/local/bin:/usr/bin:$PATH"
• export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
• for serial meters, use:

  mlogger -p 0 -o

• for USB meters, use:

  wattsup ttyUSB0 watts 

• Linux support for Watts Up serial version
  • man page / Power Aware Tools
  • Watts Up Pro Manual (serial)
  • Watts Up Pro Data Protocol (serial)
• Linux support for Watts Up USB version
  • Watts Up Pro Manual (USB) (local copy)

Virtualization with KVM

Lustre

• TBA

PVFS2

• Mounted as /pvfs2
• about 30TB of storage over 4 servers (7.3TB each) under software RAID5
• Currently limited by 1Gpbs switch connection (eth1)

Advanced topics (pending)

This applies to:

• booting your own kernel
• installing your own OS

Known Problems

References:

• A User's Guide to MPI by Peter Pacheco
• Debugging: Gdb only works on one task with MPI, you need to "attach" to other tasks on the respective nodes. We don't have totalview (an MPI-aware debugger). You can also use printf debugging, of course.

Additional references:

• Intro to UNIX at NCSU
• Emacs reference card
• Gdb reference card
• Gnu manuals (make, gdb and many more)