Installation instructions for multiple scientific applications
including Bioinformatics, Computational Chemistry, and Climate
Modeling on the cluster. Corresponding folders contain example scripts
to execute these applications on the cluster after successful
installation.

Contents:

• AlphaFold: Instructions to install and run AlphaFold on the cluster.
• CryoSPARC: Instructions to install and run CryoSPARC on the cluster.
• Dynamite: Instructions to set up and execute Dynamite on the cluster.
• Knitro: Examples to use Knitro with Matlab, Python, and C on the cluster.
• LANDIS: Instructions to either install LANDIS or use its singularity image and run it on the cluster.
• PariGP: Instructions to load PariGP’s module on the cluster and example scripts to execute it.
• Schrodinger: Instructions to run Schrodinger on the cluster along with an example script to start it in batch mode.
• TotalView: Instructions to load the TotalView module and use that to run the application on the cluster in MPI mode.
• WRF_Model: Instructions to configure and compile WRF on the cluster.

TAU (Tuning and Analysis Utilities) is a comprehensive profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, Java, and Python. It is capable of gathering performance information through instrumentation of functions, methods, basic blocks, and statements. All C++ language features are supported including templates and namespaces. The instrumentation consists of calls to TAU library routines, which can be incorporated into a program in several ways:

• Automatic instrumentation using the compiler
• Automatic instrumentation using the Program Database Toolkit (PDT)
• Manual instrumentation using the instrumentation API
• At runtime using library call interception through the tau_exec command
• Dynamically using DyninstAPI
• At runtime in the Java virtual machine

Data Analysis and Visualization:

• Profile data: TAU’s profile visualization tool, ParaProf, provides a variety of graphical displays for profile data to help users quickly identify sources of performance bottlenecks. The text based pprof tool is also available for analyzing profile data.
• Trace data: TAU provides the JumpShot trace visualization tool for graphical viewing of trace data. TAU also provide utilities to convert trace data into formats for viewing with Vampir, Paraver and other performance analysis tools.

Programming models and platforms: TAU supports most commonly used parallel hardware and programming models, including Intel, Cray, IBM, Sun, Apple, SGI, GPUs/Accelerators, HP, NEC, Fujitsu, MS Windows, using MPI, OpenMP, Pthreads, OpenCL, CUDA and Hybrid.

Examples

To get started with TAU on the FAS cluster you can try the below examples:

• Example 1: Profiling mpi4py

References:

• TAU website
• Official Documentation

PARI/GP is a cross platform and open-source computer algebra system designed for fast computations in number theory: factorizations, algebraic number theory, elliptic curves, modular forms, L functions… It also contains a wealth of functions to compute with mathematical entities such as matrices, polynomials, power series, algebraic numbers, etc., and a lot of transcendental functions as well as numerical summation and integration routines. PARI is also available as a C library to allow for faster computations.

Usage

PARGI/GP is available through FASRC Lmod. There is one release to run PARI/GP sequentially and another release to run in parallel through MPI. You can distinguish which module is which by their module dependencies. For example, as of August 2023, two pari modules are available:

[jharvard@boslogin04 ~]$ module spider pari

-------------------------------------------------------------------------------
pari:
-------------------------------------------------------------------------------
Description:
PARI/GP is a cross platform and open-source computer algebra system
designed for fast computations in number theory: factorizations, algebraic
number theory, elliptic curves, modular forms, L functions.

Versions:
pari/2.15.4-fasrc01
pari/2.15.4-fasrc02

-------------------------------------------------------------------------------
For detailed information about a specific "pari" package (including how to
load the modules) use the module's full name. Note that names that have a
trailing (E) are extensions provided by other modules.
For example:

$ module spider pari/2.15.4-fasrc02
-------------------------------------------------------------------------------

The pari module compiled with MPI can only be loaded when the compilers gcc 12 and openmpi 4.1.4 have been loaded as explained when you execute module spider pari/2.15.4-fasrc02:

[jharvard@boslogin04 ~]$ module spider pari/2.15.4-fasrc02

-------------------------------------------------------------------------------
pari: pari/2.15.4-fasrc02
-------------------------------------------------------------------------------
Description:
PARI/GP is a cross platform and open-source computer algebra system
designed for fast computations in number theory: factorizations, algebraic
number theory, elliptic curves, modular forms, L functions.

You will need to load all module(s) on any one of the lines below before the
"pari/2.15.4-fasrc02" module is available to load.

gcc/12.2.0-fasrc01 openmpi/4.1.4-fasrc01

Help:
pari-2.15.4-fasrc02
PARI/GP is a cross platform and open-source computer algebra system
designed for fast computations in number theory: factorizations, algebraic
number theory, elliptic curves, modular forms, L functions.

Examples

For examples, see our User Codes repo for PARI|GP.

References

• PARI/GP documentation
• Intro to GP programming tutorial
• Parallel GP tutorial

The Weather Research and Forecasting (WRF) model is a widely used, state-of-the-art atmospheric simulation system designed for both meteorological research and operational forecasting.

On the FASRC cluster, WRF is intended to be run in high performance computing (HPC) mode, leveraging multiple compute nodes and large-scale parallelism for efficient, large-domain simulations.

Usage

For usage information on WRF, see SPACK package manager: WRF

Examples

We have some examples for WRF Module in User Codes.

CryoSPARC is a closed source, commercially-developed piece software for analyzing single-particle cryoelectron microscopy data. It supports CUDA based GPU-accelerated analysis through the PyCUDA library. It consists of several applications which are bundled in two separate binary packages, termed

• CryoSPARC Master (cryosparcm)
• CryoSPARC Worker

The Master package is meant to use relative little compute resources, and at least some sysadmins seem to have decided to allow users to run this directly on login nodes. CryoSPARC Worker can be run on a separate node or the same node, but typically should have access to GPU compute resources. The worker nodes must have password-less SSH access to the master node as well as unfettered TCP on a number of ports. The authoritative list of requirements for installation can be found in the CryoSPARC guide.

In addition to instantiating worker nodes and connecting them to the Master node, CryoSPARC can also be configured with a “Cluster Lane” which submits jobs via the SLURM job scheduler. This is the install strategy described in this document.

CryoSPARC Master Program

The master program is called with the cryosparcm command documented here. The major mechanism for customizing the behavior of cryosparcm is the config file located in cryosparc_master/config.sh. It has the license, path to the MongoDB database, master hostname, and the base tcp port. Ensure these settings in your config.sh file are correct or you will experience errors.

See the User Code repo for examples.

CryoSPARC Operation

At the top level, cryosparcm is really a Supervisor based shell script which manages at least six different applications. For instance, running cryosparcm start will bring up the following applications

• app (cli)
• command_core
• command_rtp
• command_vis
• database (MongoDB)
• webapp

These mostly communicate with one another over TCP. The TCP ports used by each of the component programs are not individually configurable, but the base port to which the user connects is configurable in the cryosparc_master/config.sh. Of note, the hostname of the node running the master application is also typically hardcoded in this config.sh file. However, if this is left unset, it will take the hostname of the machine on which cryosparcm start is called.

Obtaining a Free Academic CryoSPARC License

CryoSPARC is free for academic use. However, it does require a license. You can request a license on the CryoSPARC webpage. The process of requesting a license is described in detailed here.

Installing CryoSPARC on Cannon

We provide a configure script to get up and running with CryoSPARC on Cannon, found in our FASRC/User_Codes git repo. When running CryoSPARC, use a GPU computing node so that the correct CUDA modules are loaded and functioning.

Examples

There are example config files for your environment available in FASRC/User_Codes on git to help you get up and running with CryoSPARC, and some additions for your .bashrc file.

Resources

• CryoSPARC documentation
• Downloading and Installing CryoSPARC
• GPU Computing on the FASRC cluster

See Alphafold2 and Alphafold3.

AlphaFold in the FASRC Cannon cluster

Alphafold typically runs within a Docker container. On the FASRC cluster, Podman can be used to run such containers. However, Singularity is recommended for ease of use in an HPC cluster environment.

Singularity images

The AlphaFold singularity images are stored in a cluster-wide location, meaning that individual users do not have to copy the singularity images to use them. Singularity images are located in:

/n/singularity_images/FAS/alphafold/

Each singularity image is tagged with the Alphafold version

[jharvard@holylogin03 ~]$ ls -l /n/singularity_images/FAS/alphafold/
total 23G
-rwxr-xr-x. 1 root root 4.8G May 25  2023 alphafold_2.3.1.sif
-rwxr-xr-x. 1 root root 2.9G May 25  2023 alphafold_2.3.2.sif
-rwxr-xr-x. 1 root root 4.9G Dec 18  2024 alphafold_3.0.0.sif
-rwxr-xr-x. 1 root root 2.8G Oct 16 16:32 alphafold_3.0.1.sif
-rwxr-xr-x. 1 root root 4.5G Nov  2  2022 alphafold_v2.2.4.sif
-rw-r--r--. 1 root root  817 Dec  5  2024 readme.txt

• Version 3.0.1: Created by FASRC with a Singularity definition file, which was based on the Dockerfile by Google Deepmind
• Version 3.0.0: Created by FASRC with a Singularity definition file, which was based on the Dockerfile by Google Deepmind
• Version 2.3.2: Downloaded from Catguma DockerHub
• Version 2.3.1: Downloaded from TACC DockerHub
• Version 2.2.4: FASRC built a Singularity container based on the Singularity definition file from https://github.com/prehensilecode/alphafold_singularity.

Databases

The Alphafold database is stored in a cluster-wide location, meaning that individual users do not have to download the AlphaFold database to run their simulations. The database is stored in SSD as recommended by the developers. Database locations:

Alphafold2

/n/holylabs/rc_admin/Everyone/alphafold_databases/v2

Alphafold3

/n/holylabs/rc_admin/Everyone/alphafold_databases/v3

Model parameters

Alphafold3

To run Alphafold3, you must request the model parameters from Google. See Obtaining model parameters.

Google will provide a file file_name.bin.zst. Extract with unzstd file.bin.zst. Place file.bin in a lab share (do not put in netscratch) — this will be the location of you --model_dir.

Running Alphafold

You will find example scripts in the FARC User_Codes repo.

Resources

• Get Started with AlphaFold code examples on FASRC Github
• Alphafold3 Github
• Alphafold2 GitHub
• University of Virginia AlphaFold docs
• AlphaFold discussion about Singularity implementation
• Job Efficiency and Optimization Best Practices