XStream Storage Resources

Home directories

User home

Each user on XStream has an home directory referenced by $HOME. It is a small and low performance NFS storage space used to keep scripts, binaries, source files, small log files, etc.

On XStream, the home directory is not intended to be used for computation. The Lustre parallel file system $WORK is much larger and faster, thus much more suited for computation, and should be excellent for large job data files (see below).

Group home

Each group on XStream has a shared home directory referenced by $GROUP_HOME. Like $HOME, it is an NFS storage space used to store small files shared by all members of your primary POSIX group.

  • Note: in $GROUP_HOME, only the owner of the files can delete them.

Backups

User and group home directories are backed up every night, but there are no snapshots. Contact Research Computing Support in order to recover any lost files.

In general, we recommended that you regularly backup your files outside of XStream.

Local scratch

A local SSD-based scratch space is available on each compute node (NOT on login nodes). It is made of 3 x Intel SSD (MLC) aggregated using Linux dm-raid for a total of 480 GB per node (447 GB usable). To access this local scratch space, please use the $LSTOR or $TMPDIR environment variables. This space may be purged when the compute node reboot or when this space becomes full.

Parallel file system

A Lustre file system is available on XStream and is mounted on /cstor on login and compute nodes. Its specifications are detailed in tech specs. On XStream, this single parallel file system has multiple purposes:

  • perform fast I/O
  • store large computational data files
  • allow multi-node jobs to write coherent files

Important note

  • The parallel file system is not backed up.

User work directory

Each user on XStream has a large work directory referenced by $WORK. It is a Lustre storage space used to work on large input or result files, checkpoint/restart files, etc.

Group work directory

Each group on XStream has a shared work directory referenced by $GROUP_WORK. Like $WORK, it is a Lustre storage space and shared by all members of your primary POSIX group.

  • Note: in $GROUP_WORK, only the owner of the files can delete them.

Filesystem Quotas

Limits

Default filesystem hard-limit quotas are described in the tables below. User and group quota values are not cumulative (the first limit reached takes precedence).

Filesystem Quotas per user (Stanford) Quotas per user (XSEDE)
$HOME Volume: 10 GB
Inodes: 1,000,000
Volume: 5 GB
Inodes: 500,000
$WORK Volume: 10 TB
Inodes: 10,000,000
Volume: 1 TB
Inodes: 5,000,000
Filesystem Quotas per group
$GROUP_HOME Volume: 100 GB
Inodes: 10,000,000
$GROUP_WORK Volume: 50 TB
Inodes: 50,000,000

Quota info

NFS home quota info

To get your current user quota info ($HOME and $GROUP_HOME):

$ quota -s

To get your current group quota info ($HOME and $GROUP_HOME):

$ quota -gs

Lustre work quota info

To get your current user and group quota info on lustre ($WORK and $GROUP_WORK):

$ lfs quota -h $WORK