Examples of data that can be stored on FASRC storage are:

• Datasets
• Code
• Scientific software
• Research results

Examples of data that should not be stored on FASRC storage include:

• Clerical or lab administrative data
• Data related to personnel, grant proposals, business operations, or general lab management
• Data with personally identifiable or financial information 

FASRC storage filesystems are only approved for Data Security Level 1 (DSL1) and  DSL2 research data on the Cannon cluster. DSL3 data must be stored in the approved FASSE cluster project. Research data containing information classified as DSL 4 must be stored on an appropriate storage solution that is approved for DSL4 sensitive data.*

*A limited number of DSL4 projects exist in their own isolated environments

If it comes to the attention of the FASRC Staff that non research related data is being stored on the FASRC systems, we will alert the lab’s PI.

To view alternative storage options for administrative data, please refer to the FASRC website.  Additional information is also provided on the Harvard Security website regarding Data Security levels.

Google Drive (Administrative file storage)

• Store, access, and share administrative files from any device, as data is stored in the cloud
• My Drive is designed for personal documents to be shared individually. 
• Shared drive is for collaborative files and folders owned at the team level.
• Approved for Medium Risk Confidential (L3) data. 
• Central Administration, GSE, HBS, HDS, HKS, and HMS (Quad), and HSPH require local approval for account requests.
• Google Drive storage request form: This form can be used to request a new personal or shared Google Drive, or request additional storage space for an existing Google Drive. The default Google Shared Drive storage limit is 5 GB. Eligible users may request additional storage using the form. The requests will need to be approved by FAS, as they are currently responsible for the costs.

Dropbox (Administrative file storage)

• Secure data storage for faculty and research staff
• Store, sync, and share data files in the cloud
• Collaborate on documents with added version history

Microsoft 365

• OneDrive (Administrative file storage)
  • Store work-related administrative files
  • Share with colleagues within and outside of Harvard
  • Users have 2TB of file storage
  • Approved for Medium Risk Confidential (L3) data.
• Sharepoint (Document management tool)
  • Secure location to store, organize, share, and access information 
  • Default of 25 TBs of expandable storage
  • Approved for Medium Risk Confidential (L3) data.
  • Can be shared externally with non-Harvard faculty and staff. 
  • Storage for Level 4 data is available upon request with restrictions. 

Atlassian Confluence (Wiki web environment)

• Website allowing users to create, edit, and publish content collaboratively through a web browser. 
• Access can be given to individuals, groups, to the Harvard community, or to the public. 
• Each page has its own URL, page history, access restrictions, file attachments, and comments.

Access Control Lists (ACLs or FACLs) are used to manage granular permissions on individual files or directories (folder). Primarily they are used to give one or more persons access to a file or directory that is not dependent upon the owner or group attached to the file/folder. For instance, if one user owns a file and wants to allow another user to also have write access to it without giving the group write access.

While FASRC manages a large number of groups for access to the many storage shares we host, we generally do not micromanage access at the individual level such as ‘this person should have access but not this person’ situations. In some instances with multiple users that might be best handled with an additional group, but that is added support overhead. For access issues involving these individual access scenarios, an ACL may be the best option when you need to grant someone else more granular permissions on files you own.

Usage

From a login or other node on the cluster, type man getfacl and man setfacl

Example 1: getfacl

This example shows how to see what FACLs are set.

[harvard_lab]# ls -l
total 12         (the '+' sign indicates that ACLs have been applied)
drwxrwsr-x+ 28 jharvard harvard_lab 4096 Feb 19 20:06 Everyone
drwxrwsr-x+ 7 jharvard harvard_lab 4096 May 9 20:03 Lab
drwxrwsr-x+ 74 jharvard harvard_lab 4096 Oct 10 2023 Users
[harvard_lab]# getfacl .
# file: .      (shows the FACL settings, in this case a group, harvard_lab_admins, has special permissions)
# owner: root
# group: harvard_lab
# flags: -s-
user::rwx
group::r-x
other::r-x
default:user::rwx
default:group::r-x
default:group:harvard_lab_admins:rwx
default:mask::rwx
default:other::r-x

Example 2: setfacl

This example shows how to allow another user read/write/execute access to a file you own.

[jharvard]$ ls -l test
-rw-r--r--. 1 jharvard harvard_lab 30 May 17 16:41 test
[jharvard]$ setfacl -m u:testuser:rwx test
[jharvard]$ ls -l test
-rw-rwxr--+ 1 jharvard harvard_lab 30 May 17 16:41 test
[jharvard]$ getfacl test
# file: test
# owner: jharvard
# group: harvard_lab
user::rw-
user:testuser:rwx
group::r--
mask::rwx
other::r--

Additional documentation about the use of ACLS can be found at:

• An introduction to Linux Access Control Lists (ACLs) (RedHat)
• How to create and use file access control lists in Linux (RedHat)
• Access Control Lists (TACC)

If you have received a monthly storage bill from the FASRC Storage Service Center, this document will help you to understand its contents. Please bear in mind that the bill may show a lot more information than you expected, however other labs may need these details to help them correctly route charges to the correct projects or billing codes.

Storage allocations are billed monthly and are based on the total size of the allocation(s). A lab with a single 16TB allocation on Tier 0, for example, has a total allocation cost of $800 per year [at the time of this writing]. That lab will pay 1/12 of that total amount each month, so approximately $66.66 each month*.

Additionally, the billing system provides a breakdown of usage by users. This includes any user with data in your storage and may include disabled users (see note below). Please note that while this breakdown shows a dollar amount per user and a 33-digit string attached, those values are solely for your information. These values are useful for labs who need to charge back or split costs for projects/grants and/or determine who is using what percentage of their allocation.

Please Note:  FASRC does not police lab storage and, as such, we do not remove data from your storage unless asked to by a PI or lab manager. Each lab is responsible for the management and cleanup of their stored data. If you have difficulty removing items from your storage, please contract FASRC for assistance.

* – We recognize that there will be rounding issues and FASRC will endeavor to round down so that the cost of allocations do not exceed the quoted yearly cost.

EXAMPLE BILL

The example below shows an allocation of 16TB of Tier 0 storage at $50 per TB per year
Total cost per year = $800

$4.16 per TB/month unit cost
Total billed per month= ~ $66.56

PLEASE NOTE: Billing is NOT  based on usage, but the total allocation. A breakdown by user is provided for those who need to charge back to grants, but the total amount is the same whether the storage is empty or full.

However, a situation may arise where you are not at quota but over quota and when trying to remove files using rm, you receive the error

rm: cannot remove ‘{somefilename}’: No space left on device

NOTE: Any time you are deleting files, it is important that you check to ensure you enter the correct filename. A good rule of thumb is to use the full path to a file (instead of relative path) or cd to the directory containing the file first. Also, be extra cautious when using wildcards like * .

WORKAROUND

A workaround is to identify some larger file(s) to remove and to reduce that file(s) size to zero bytes. Once enough space is recovered to get you under quota, you should be able to use rm again. To do this on files you’ve identified for removal, use the truncate command:

truncate -s 0 FILENAME
To truncate a single file down to zero bytes.

or

truncate -s 0 FILE1 FILE2 FILE3
To truncate multiple files down to zero bytes

Example:

truncate -s 0 ~/Jobfiles/August/job12653287.out
It’s always safer to use the full path to a file.
~ as used here is a Unix shortcut for the path to your home directory.

Alternately, you could also use the cat command to insert zero bytes of data into a large file:

cat /dev/null > ~/mybig.file

/dev/null is a special Unix device that is always zero bytes in size

CHECKING USED SPACE

You can check to see your current total home directory (shortcut “~”) usage using the du command (plus the summary, total, and human-readable options) like so:

[jharvard@holylogin01 ~]$ du -sch ~
80G .
80G total

If you are on a login node, you can also view your computed quota directly like so:

[jharvard@holylogin01 ~]$ df -h .
Filesystem                      Size Used Avail Use% Mounted on
rcstorenfs:/ifs/rc_homes/home13 95G  80G   15G  9%   /n/home13

This shows that the user jharvard has used 80GB out of 100GB. The 95GB shown as the Size is called a soft quota. That is the threshold at which the system will notify you that you are going over.

Please bear in mind that due to the size of our home directories neither is instantaneous. The notification and re-calculation of quotas happens some time during a 24 hour period. So if you manage to go over quota before the next calculation is done, you won’t receive a soft notice. Similarly, it may take some time for your actual usage and computed quota to match again.

To find which files or directories are using the most space:

[jharvard@holylogin01 ~]$ cd ~
[jharvard@holylogin01 ~]$ du -h --max-depth=1 .
384K ./.config
232K ./Test
2.0G ./spack
...

This will show a listing of files and sizes and you can repeat the command down the directory tree to find files to delete.

What To Do If du and df Are Different

If you find that df says you are at quota while du show a lower number, you may have sparse files which are not being accounted for properly in your du(but are accounted for by the filesystem’s quota check).

To check for this, you will need to use the --apparent-size flag in du to show the logical size and find the culprit(s)

cd ~

du -ch --apparent-size --max-depth=1 .

This will show the logical size of directories and should point you to the cause.

Clearing Disk Space

.local

This hidden folder is located in your $HOME . It typically grows in size when pip install is executed outside of a conda/mamba or python virtual environments to install packages, for example while in a Jupyter or interactive session. See the warning on pip installs. This is because such installations get placed in your ~/.local , resulting in $HOME getting full.

In order to manage ~/.local, do the following:

1. Make sure that there are no jobs currently running under your profile by executing: squeue -u <username
2. Rename/turn-off .local folder: mv ~/.local ~/.local.off

.conda

Conda/Mamba environments can be quite bulky based on the number and types of packages installed inside them and should be stored in your PI’s $LAB directory. See Mamba environments in a desired location. However, if such environments are created using the default location, $HOME/.conda, then the storage size of ~/.conda can be managed as follows:

1. Remove unused packages and clear caches of Conda/Mamba:
   module load python
source activate <your-environment>
conda clean --all 
   This deletes only the unused packages in your ~/.conda/pkgs directory.
2. Remove unused conda/mamba environments:
   module load python
conda info -e
conda env remove --name <your-environment>

.cache

This directory, located in ~/.cache, can grow in size with the general use of the cluster, Open OnDemand, or VSCode. In order to manage this space, do the following:

1. Make sure that there are no jobs currently running under your profile by executing: squeue -u <username
2. Remove the folder: rm -r ~/.cache

.singularity

This folder, located in ~/.singularity, typically grows in size when a container is pulled to the cluster using Singularity. You can manage the size of this folder by cleaning its corresponding cache: singularity cache clean all

In order to avoid ~/.singularity folder from filling up, you can set a temporary directory while pulling a container and redirect the location for storing its cache. For example: export SINGULARITY_TMPDIR=/tmp/

Then, pull the container using Singularity as usual.

Note: It is best to run the above commands from a compute node, in an interactive session, as login nodes are not performant and are meant for lightweight activities only. One can ignore the interactive session being used for managing $HOME size and move forward with turning off ~/.local or deleting ~/.cache.

Users and labs seeking cloud computing resources can and are encouraged to make use of NERC. While FASRC does not directly provide NERC services, FASRC users and labs are free to make use of their services for cloud computing needs.

NERC Links and Documentation

• NERC User Guide / Onboarding Process
• NERC Documentation
• NERC Account Request Guide
• NERC Coldfront Interface (requires NERC account)
• NERC Coldfront Guide

Need help or have questions?

Current NERC users or those with questions about the service should contact NERC via their online help system or by emailing help@nerc.mghpcc.org

To get more information about the storage options and their features please visit our Data Storage page. Please feel free to reach out to us at  rchelp@rc.fas.harvard.edu with any questions.

Storage Options and Cost

* Disaster Recovery (aka DR) means that the entire share can be restored in the event of hardware failure or other ‘disaster’. This DRE copy is not accessible to the end user and not suitable for recovering individual files that have been accidentally deleted.

** Snapshot means that a snapshot of the filesystem is periodically taken (for up to 7 days) and can be accessed by the end user to recover individual files that are accidentally deleted.

Important notes:

• If you require more than approximately 100TB in a single allocation, please contact us first to discuss or drop by Office Hours
• Billing is through the FIINE system. See below to get access to FIINE
• Billing is done monthly
• The cutoff day for billing is the 15th of each month. Any changes done to the allocation will be reflected in next month’s bill
• The service center needs 33 digit billing code to provide the service. It’s an internal service so we can’t create POs for billing
• To read your current bill read here
• For billing questions and queries email billing@rc.fas.harvard.edu

Request or Manage Storage Allocations

• To request an allocation or manage an existing one, the PI (or a previously designated storage manager) should log into Coldfront
• If you cannot access Coldfront, or you are a PI who would like to designate a storage manager for your lab, please contact FASRC

Lead Time for New Tape (Tier 3) Allocations

There is a minimum setup time of roughly 2 weeks for new tape allocations. This timeframe assumes we receive the completed tape setup from our service partner NESE without delay. Delays there are beyond our control and could increase lead time. Please note that any storage changes made after the 15th of the month will be reflected in the following month’s billing.

Billing for Allocations

Charges for storage allocations are billed monthly. Expense code(s) can be applied to each allocation and can be sub-divided among multiple billing codes.
See our Service Center FAQ for answers to common questions.

See also How to read your Storage Service Center bill

To manage billing for an existing allocation, you will need:

• A FASRC account. You very likely already have this unless you are a new PI.
  If you do not have an account, please view: How Do I Get a FAS Research Computing Account?
• Access to the FIINE billing system.
  • PLEASE NOTE: If you have an account that was created before ~ 2021 and cannot log into FIINE, you likely need to  go through the Informatics onboarding tool to get your Harvard Key information updated in our system. https://onboard.rc.fas.harvard.edu/onboard/

Instructions for expense code management and billing record review in FIINE are available at:
https://ifx.rc.fas.harvard.edu/docs/user/fiine.html

Starfish – Data Management

Starfish – Scans the different storage servers to provide a view, usage details, metadata, and tagging based on the projects. View our Starfish documentation for more details about starfish and examples to query the data.

Coldfront – Lab and Allocation management:

Coldfront – Provides a view on PI projects and allocations. New allocation and updates to existing allocation can be requested using Coldfront. View the Coldfront article for more details about Coldfront and its use.

Fiine -FAS Instrument Invoicing Environment

Fiine – For lab/finance administrator to manage the expense codes per project/user and view invoices.

FAQ – Storage Service Center

Please note that most, file systems, including lab directories on holylabs, cannot be mounted on your desktop. 

• Scratch – Scratch space (/n/netscratch) cannot be mounted in this manner. It is only available on the cluster. If you need to transfer data to/from scratch, you can use an SFTP or SCP client to connect to the cluster and then change to /n/netscratch/[your lab’s space] . You can also use Globus for large external transfers.
• Active Lab Storage (Tier 0) and shares whose name begin with holy generally cannot be mounted. If you need to transfer data to/from such shares, you can use an SFTP or SCP client to connect to the cluster and then change to the path of your lab share. You can also use Globus for large external transfers.
• Active Lab Storage (Tier 1) and (Tier 2) shares can be exported via Samba if a valid need exists. Otherwise, if you need to transfer data to/from such shares, you can use an SFTP or SCP client to connect to the cluster and then change to the path of your lab share . You can also use Globus for large external transfers.
• Long-term storage (tape) cannot be mounted and is only accessible via Globus.

Connect to the VPN

If using wireless connections, cluster storage must be routed through a VPN connection. If on wired connections inside Harvard, the VPN client is not required. If you don’t already have one setup, follow the VPN setup instructions.
NOTE: If you have set up custom DNS on your computer, this may cause issues connecting to shares.

Find the filesystem path (if not known)

If you already know the path, skip to instructions for your operating system below.

Mounting your HOME DIRECTORY

If you have cluster access, you can mount your home directory as a drive. You can figure out the path to your home directory by using ssh to login to the cluster. Use cd ~ to go to your home directory (on a Unix-like system, the ~ character is a shortcut to ‘my home directory’). Then type pwd to show where your home directory resides. [jharvard@boslogin02 ~]$ cd ~ [jharvard@boslogin02 ~]$ pwd /n/home08/jharvard  The home08 is the part we need in this example in order to construct the full path to your home directory.  Since all home directories are mounted from the same server, we don’t need to figure that part out. The path, therefore, that you will need for connecting is the combination of the server name, rcstore.rc.fas.harvard.edu. followed by the word homes to signify that it’s a home directory, the sub-folder your home directory resides in (home08 in this example), and your RC username.

For this example, this would result in:
For Windows \\rcstore.rc.fas.harvard.edu\homes\home08\jharvard
For Mac OSX smb://rcstore.rc.fas.harvard.edu/homes/home08/jharvard

Mounting a LAB SHARE

First, it’s important to note that most lab shares are not mountable. Also, cluster-only filesystems such as scratch (netscratch, holyXXXX, or local scratch) are never mountable. If you need to transfer data to/from such shares, you can use an SFTP or SCP/Rsync, or use Globus for large external transfers.

If you don’t already know the path to your lab’s share and believe it should be mountable, asking a lab-mate or your PI for the path is the quickest option. If your lab-mates do not know and you believe your share is mountable, please contact FASRC.

You can also try and see if your lab’s share is mounted on our Samba cluster using the instructions further down the page, but with one of the following paths:
For Windows \\smbip.rc.fas.harvard.edu\ (browse for your lab’s name, see below)
For Mac OSX smb://rcstore.rc.fas.harvard.edu/ (browse for your lab’s name, see below)

If found, you can use the path shown there to mount your lab’s share.

Operating System-Specific Instructions

Macs use Connect to Server

If you’re using a Mac, go to a Finder window (or click on the desktop) and choose Go > Connect to Server from the menu. In the server address box, enter the server and path combination as described above prepended with the smb:// protocol specifier (please note that Macs use “/” where Windows uses of “\”). Using the example information above, the value might be smb://rcstore.rc.fas.harvard.edu/homes/home08/jharvard to mount the home directory of user jharvard. If you are mounting a lab share path, enter that instead (example: smb://smbip.rc.fas.harvard.edu/jharvard_lab). If you’ve selected the proper volume, you should get a login prompt. Use your FASRC credentials here. Note that you must include the rc\ domain specifier at the beginning of your user name.

PCs use Map Network Drive

You can connect to shared storage on a Windows PC by using the Map Network Drive button in a file explorer window (click the yellow folder icon in the taskbar).

Select This PC in the left-hand pane.

Click Computer, which will present a drop-down menu, and then from that menu click Map network drive.

In the Map Network Drive utility, select a free Drive letter.
Then enter the combination of fileserver address and path in the Folder field.
For the example, in the home directory described above the path would be \\rcstore.rc.fas.harvard.edu\homes\home08\jharvard.
If you are mounting a lab share path, enter that instead (example: \\smbip.rc.fas.harvard.edu\jharvard_lab).
Make sure Connect using different credentials is checked.
Click Finish to continue.Optional: If you want this drive to reconnect every time you log on to your computer, check Reconnect at sign-in. Just bear in mind that it will not re-connect if you are not on the VPN or your normal campus wired jack.


The reason you must check Connect using different credentials in the Map Network Drive box is that your PC has a local account (and a local ‘domain’) and it will default to that if you do not specify another username and domain. If you don’t select this checkbox and attempt to connect, it will try to authenticate with your local PC information and after three failed attempts will result in a lockout (FYI: Don’t worry, lockouts expire automatically in about 5 minutes).

When you are prompted to Enter Network Credentials, prepend your FASRC username withRC\ to specify you are connecting to the RC domain with your username.
Example: RC\jharvard means ‘Connect to the server and path I entered above as RC domain user jharvard’.

This will prompt you for your password. If instead you get an error message about a read-only filesystem, it could be because mount.cifs is not installed on your system. Using this method, you will need to reissue the command every time you boot your computer.Some users prefer using smbclient to connect to Samba/SMB/CIFS shares. This is an optional package you will need to install on your own.

SFTP/Filezilla 

If you are unable to mount your lab storage using one of the above methods, or your lab’s share is simply not available via Samba, you always have the option of using SFTP. This is especially useful if you need to maintain a different VPN connection and cannot connect to our VPN. SFTP to a login node does not require a FASRC/FAS VPN connection as login nodes use two-factor authentication.

We recommend FileZilla as a reliable, cross-platform SFTP client. Note that SFTP uses SSH and our two-factor authentication, so you will need to ensure you have OpenAuth set up, and that you have have cluster access and a home directory. If you are unsure, SSH to a login node first. If you need to request cluster access, see our doc on adding groups/access.

Scratch Policy

Each lab is allotted 50TB of scratch space for its use in their jobs. This is temporary high-performance space and files older than 90 days will be deleted through a periodic purge process. This purge can run at any time, especially if scratch is getting full and is also often run at the start of the month during our monthly maintenance period.

There is no charge to labs for netscratch, but please note that it intended as volatile, temporary scratch space for transient data and is not backed up. If your lab has concerns or needs regarding scratch space or usage, please contact FASRC to discuss.

Modifying file times (via touch or other process) when initially placing data in scratch is allowed, however doing so subsequently to avoid deletion is an abuse of the filesystem and will result in administrative action from FASRC. To reiterate, you may initially modify the file date(s) on new data so that it is not in the past, but should not modify it further.  If you have longer-term needs, please contact us to discuss options.

Networked, shared netscratch

The cluster has storage built specifically for high-performance temporary use. You can create your own folder inside the folder of your lab group. If that doesn’t exist or you do not have write access, contact us.

IMPORANT: netscratch is temporary scratch space and has a strict retention policy. 

/n/netscratch is short-term, volatile, shared scratch space for large data analysis projects.

The /n/netscratch filesystem is managed by the VAST parallel file system and provides excellent performance for HPC environments. This file system can be used for data intensive computation, but must be considered a temporary store. Files are not backed up and will be removed after 90 days. There is a 50TB total usage limit per group.

Large data analysis jobs that would fill your 100 Gb of home space can be run from this volume. Once analysis has been completed, however, data you wish to retain must be moved elsewhere (lab storage, etc.). The retention policy will remove data from scratch storage after 90 days.

Local (per node), shared scratch storage

Each node contains a disk partition, /scratch, also known as the local scratch that is useful for storing large temporary files created while an application is running.

IMPORTANT: Local scratch is highly volatile and should not be expected to persist beyond job duration.

The /scratch volumes are a directly connected (and therefore, fast) to temporary storage location that is local to the compute node. Many high performance computing applications use temporary files that go to /tmp by default. On the cluster we have pointed /tmp to /scratch. Network-attached storage, like home directories, is slow compared to disks directly connected to the compute node. If you can direct your application to use /scratch for temporary files, you can gain significant performance improvements and ensure that large files can be supported.

Though there are /scratch directories available to each compute node, they are not the same volume. The storage is specific to the host and is not shared. For details on the /scratch size available on the host belonging to a given partition, see the last column of the table on Slurm Partitions. Files written to /scratch from holy2a18206, for example, are only visible on that host. /scratch should only be used for temporary files written and removed during the running of a process. Although a ‘scratch cleaner’ does run hourly, we ask that at the end of your job you delete the files that you’ve created.

$SCRATCH VARIABLE

A global variable called $SCRATCH exists on the FASRC Cannon and FASSE clusters which allows scripts and jobs to point to a specific directory in scratch regardless of any changes to the name or path of the top-level scratch filesystem. This variable currently points to /n/netscratch so, for example, one could use the path $SCRATCH/jharvard_lab/Lab/jsmith in a job script. This will have the added benefit of allowing us to change scratch systems at any time without your having to modify your jobs/scripts.

This page describes the resources which are available to each user account and lab, and is a guide for day-to-day usage.

See also our Introduction to FASRC Cluster Storage video

Home Directories

Every user whose account has cluster access receives a 100 GB home directory. Your initial working directory upon login is your home directory. This location is for your use in storing everyday data for analysis, scripts, documentation, etc. This is also where files such as  you .bashrc reside. Home directories paths look like /n/homeNN/XXXX where homeNN is home01–home15 and XXXX is your login. For example, user jharvard’s home directory might be /n/home12/jharvard. You can also reach your home directory using the Unix shortcut ~, as in: cd ~

• Size Limit: 100GB (hard limit)
• Availability: All cluster nodes. Can be mounted on desktops and laptops
• Backup: Daily snapshots. Retained for 2 weeks
• Retention policy: indefinite
• Performance: Moderate. Not appropriate for I/O intensive or large numbers of jobs
• Cost: Provided with each user account

Your home volume has good performance for most simple tasks. However, I/O intensive or large numbers of jobs should not be processed in home directories. Widespread computation against home directories would result in poor performance for all users. For these types of tasks, the scratch filesystem is better suited.

Home directories are private to your account and will follow you no matter should you change labs, but are not suitable for storing HRCI/level 3 or above data. This is a violation of Harvard security policies.

Your home directory is exported from the disk arrays using CIFS/SMB file protocols and so can be mounted as a ‘shared drive’ on your desktop or laptop. Please see this help document for step-by-step instructions.

Home directories are backed up into a directory called .snapshot in your home. This directory will not appear in directory listings. You can cd or ls this directory specifically to make it visible. Contained herein are copies of your home directory in date specific subdirectories. Hourly, daily, weekly snapshots can be found. To restore older files, simply copy them from the correct .snapshot subdirectory. NOTE: If you delete your entire home directory, you will also delete the snapshots. This is not recoverable.

The 100 GB quota is enforced with a combination of a soft quota warning at 95GB and a hard quota stop at 100 GB. Hitting quota during processing of large data sets can result in file write/read failures or segmentation faults. You can check your usage using the df command: df -h ~ (where ~ is the unix shortcut for ‘home’)

TIP: If you are trying to determine usage, you might try using du -h -d 1 ~ to see the usage by sub-directory, or du -ax . | sort -n -r | head -n 20 to get a sorted list of the top 20 largest.

When attempting to log in when your home directory is over quota, you will often see an error in the .Xauthority file:
/usr/bin/xauth: error in locking authority file .Xauthority Logging into an NX or other virtual service will fail as the service cannot write to your home directory.

When at or over quota, you will need to remove unneeded files. Home directory quotas are global and cannot be increased for individuals. You may be able to use lab or scratch space to assist with copying or moving files from your home directory to free up space.

Lab Directories

Each lab which uses the cluster receives a 4 TB lab directory (as of 2022 – these will reside in /n/holylabs/LABS). This location is for each lab group’s use in storing everyday data for analysis, scripts, documentation, etc. Each such lab will have a directory on our high-performance scratch filesystem (see below).

• Size Limit: 4TB (hard limit), 1 million files
• Availability: All cluster nodes. Cannot be mounted on desktops and laptops
• Backup: Highly redundant, no backups
• Retention policy: Duration of the lab group
• Performance: Moderate. Not appropriate for I/O intensive or large numbers of jobs
• Cost: Provided with each lab group

Lab directories have good performance for most simple tasks. However, I/O intensive or large numbers of jobs should not be processed in lab directories. Widespread computation against lab directories would result in poor performance for all users. For these types of tasks, the scratch filesystem is better suited.

This lab directory is owned by the lab’s PI and is intended only to be used for research data on the cluster. research storage should not be used for administrative files and data.

Lab directories are not suitable for storing HRCI/level 3 or above data. This is a violation of Harvard security policies.

The 4 TB quota is enforced with a combination of a soft quota warning and a hard quota stop at 4 TB. Hitting quota during processing of large data sets can result in file write/read failures or segmentation faults. If your lab requires additional storage, see our Data Storage page for a list of available storage options.