Opt⁶⁴ Cluster ⇔ 16+1 nodes, 32 dies, 64 cores, 64 bit (x86_64)

Hardware

All machines are 2-way SMPs with dual-core AMD Opteron 265 processors, per core:

64KB split I+D L1 caches, 2-way associative, 64B/line
1MB shared L2 cache, 16-way associative, 64B/line

Tyan Thunder K8SD Pro Motherboard (2882-D)
Transport GX28 (B2882)
Tyan M3289 IPMI
server: opt (has 750GB RAID5 array NFS mounted to nodes, UPS)
nodes: optXX, XX=00..15
experimental node: opt16

Software

All software is 64 bit unless marked otherwise.

Monitor the System Status with Ganglia
htop to monitor per-processor activity on a node
Fedora Core 5 Linux x86_64
gcc (C compiler)
gcc32/g77 (32 bit C and Fortran compilers) -- interoperable w/ 64 bit??? untested
MPICH2 (also MPI API)
locally: man MPI
OpenMP v2.5 (via gcc)
ATLAS (BLAS)
locally: /usr/share/doc/atlas-devel-3.6.0/doc
PAPI3 (using perfctr)
locally: man PAPI

Reservation

Before you use any node, check the calendar if someone needs exclusive access to some nodes. Do NOT use such nodes. Reserve your own nodes if you need access. (Password obtained through Vincent Freeh.)

Access

Request a user id from Vince Freeh (server "long") or Frank Mueller (server "opt"). Please indicate your unity ID and student ID.
Accounts on long will differ in name from your unity ID, accounts on opt use your unity ID but do not share any unity file space.
Install ~/.ssh/config on your local Linux machine. If you're not using Linux, skip this step.
On Linux, ssh into any optXX machine (XX=00..15).
```
ssh <your-username›@optXX
```
Your initial password is your student ID.
On windows, use your favorite ssh client to log in to opt.csc.ncsu.edu.
Then log in to one of the nodes (same as above).
Please change your password. If you got your account through Vincent Free, ssh to opt; otherwise ssh to one of the optXX nodes. Then change you password:
```
yppasswd
```
The new password will be effective no later than at the full hour.

Using OpenMP

The "#pragma omp" directive in C programs works.
```
gcc -fopenmp -o fn fn.c
```
Gcc 4.1 for FC5 is back-patched with the 4.2 OpenMP support. It may not be the fastest but it works.

Using PAPI

Compile:
```
gcc -o fn fn.c -lpapi
```
Make sure your LD_LIBRARY_PATH is set. See below.

Interactively running MPI programs

Create the file ~/.rhosts and set the access rights:
```
chmod 600 .rhosts
```
Create a file ~/.mpd.conf and set the access rights:
```
chmod 600 .mpd.conf
```

Append to your file ~/.bashrc:

export PATH=".:~/bin:/usr/bin:/usr/local/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/lib64:/usr/local/lib:$LD_LIBRARY_PATH"

Log out and back in to optXX to activate the new settings.

Check if MPD is running:
```
mpdtrace
```
This should return a list of available nodes to run MPI jobs on. If not, see below: user-level MPD

Compile MPI programs:

mpicc -O3 -o pi pi.c

If you're using BLAS/ATLAS:

mpicc -O3 -o pi pi.c -L/usr/lib64/atlas -lcblas -latlas

Execute the program on 2 processors (using MPICH2):
```
mpirun -np 2 pi
```
Try again with a different number of processors.
Use the new MPICH2 run/execute interface (more options, see man page):
```
mpiexec -n 2 pi
```
Try again with a different number of processors.

User-level MPD

If mpdtrace is not working, you can either wait for 5 minutes and hope for the best as it tries to restart every 5 minutes to find dead and new live nodes. This may or may not work. Alternatively, run your own MPD:

Figure out which nodes are alive. You can either Monitor the System Status with Ganglia or do this: ping optXX for XX=00..15. Still, they may not work properly. Try: rsh optXX pwd -- maybe working?
Create a file like ~/mpd.hosts but only include the nodes that are working!
Edit your ~/.mpd.conf file to remove/comment out the line
```
MPD_USE_ROOT_MPD=1
```
If you have 2 working nodes and you are logged in to optYY, issue:
```
mpdboot --ifhn=optmpiYY -n 2 -r rsh --ncpus=4
```
You can now run MPI jobs.
Inquire about running jobs:
```
mpdlistjobs
```
Stop MPD daemons:
```
mpdallexit
```
Clean up MPD daemons and files:
```
mpdcleanup
```
Vary you mpdboot parameters. In the above case, you'll run 4 jobs per node (since we have 2-way SMPs, each with a dual core, i.e., 4 processor cores per node). Check out the man pages.

MPI batch job submission

To reduce the per-node activity and avoid competition of jobs for resources, larger MPI jobs should be submitted as batch jobs. They can also be scheduled at certain times of the day (reserved in the calendar).

Batch submission is realized via Torque over OpenPBS (coming later: Maui Cluster Scheduler).

On opt, issue:

submit: qsub ...
check: qstat
remove: qdel ...
documentation: "man pbs" and Torque link above

Advanced topics

For all other topics, access is restricted. Request a root password. Also, read this documentation, which is only accessible from selected NCSU labs.

This applies to:

IPMI
diskless booting
disk imaging
booting your own kernel
booting xen (non-root access in the future)
installing your own OS

Known Problems

Please let us know if you have solutions to these problems.

suspected low bandwidth between server (opts) and clients (optXX), noticeable for NFS bandwidth (but NFS is not the problem, as tested on opt)
- related to iptables (stopping iptables restores performance)
- suspected modules: ip_conntrack_netbios_ns ip_conntrack (kmem error on module reload)
- or is NAT known to reduce bandwidth by 50% ???
Xen kernel incompatible with perfctr and our powernow patch

References:

A User's Guide to MPI by Peter Pacheco
Debugging: Gdb does not work with MPI, and we don't have totalview (an MPI-aware debugger). Hence, you are down to good old printf debugging, unfortunately.
Linux Cluster Info (add cluster tools to the wish list)

Opt64 Cluster ⇔ 16+1 nodes, 32 dies, 64 cores, 64 bit (x86_64)

Hardware