Questions Answered in this Article
Answers
Question 1: Are you a candidate for High-Performance Computing (HPC)?
All faculty, staff and students have the ability to use the High Performance Computing (HPC) but only faculty and staff can request access.
Students should discuss with their professor which project they should be added to.
Professors/researchers should include the following information for students in their class or project. :
LastName, FirstName, Department, EmailAddress@uncw.edu, "Project/Professor"
When you can't perform your research in a reasonable amount of time with the resources you have on hand, it is time to discover HPC. HPC is an ideal solution for:
-
Research that requires significant computational power to run long running jobs that iterate over large sets, and are notoriously slow?
-
Leveraging advanced software modules to take advantage of HPC's clustered CPU/GPUs?
Question 2: How do I get started?
High-Performance Computing Services, available through UNCW Information Technology Services (ITS), provides entry and intermediate-level computing resources and consulting support for research and instruction.
Any faculty member can request a project on the HPC cluster at no charge. Faculty may also request a temporary project for a course or research.
Graduate students, post docs, and other collaborators must be added to a project by a faculty member and/or PI. Undergraduates or non-thesis master's students who are not working in a research group must work with a faculty advisor to request access.
Submit a Research Computing (HPC) Access Request.
Question 3: Is my application available on HPC?
HPC has a sizable application catalog that includes common titles supporting diverse fields of expertise. A comprehensive list of HPC staff maintained software can be found here.
If an application is not listed in the software catalog, it can be requested for consideration or the researcher can install the package in their project space as per the software installation guidelines.
UNCW/ITS staff are not domain experts and do not know every software package, but they can give you advice on:
Submit a HPC consultation request
Question 4: How do I transfer files to HPC?
Transferring files using HPC's "OnDemand" Dashboard
From the Files> drop down navigate to your home directory. From here you can select files using the options provided.
NOTE: There is a 10 GB file size limitation for file transfers using HPC's "OnDemand" dashboard.
To move files larger than 10GB or multiple files you will need to use sftp/ssh to transfer the data.
Transferring files using sftp/ssh.
*Please note: In order to transfer files using sftp/ssh one must be a member of the sftponlygroup. Researchers from UNCW are added to the sftponly group by default.
To verify your account settings, open a terminal on the HPC login device. At the prompt type "id".
The "id" command will display your user id (uid), any groups and their permissions.
 NOTE: HPC ids are currently limited to 8 characters and thus may differ from your email address.
MAC and Linux users
Open a terminal window and type: ssh-keygen
When prompted, hit enter to accept the default values. A file will be created in the hidden directory (.ssh/) with the key that was generated.
You will need to copy the ssh-rsa key to the clipboard.
Log into the OnDemand dashboard, select Files/Home Directory and then open in terminal.
In your terminal navigate to your .ssh directory, locate the authorized_keys file and append to it the ssh-rsa blob from the local host.
Save and close the file
Return to your local host and initiate a file transfer.
sftp servxfer.hpc.ncsu.edu
Question 5: How much data can I store on HPC?
The main types of storage available to the general user are the home directory and the scratch directory. Many users will only use these spaces.
Home directory
The home directory is intended for scripts, small application source code, and executables. There is not enough space in the home directory for large data files.
Each account has 1 GB of quota in /home/user_name.
This space is backed up daily. One copy of each file is retained in the backup. Deleted files are retained for about 7 days.
To check how much space is available in the home directory, type quota -s.
Scratch directory
Scratch space is intended for the storage requirements for running jobs. Applications should use scratch space during job execution, i.e., jobs should be submitted from /share.
Scratch space is not backed up and files that have not been accessed for 30 days are automatically deleted.
Each project has 10 TB of quota in /share/group_name. Users should create a sub directory under this location for their use:
cd /share/group_name
mkdir user_name
-
To find the group_name, see the first listed output from the groups command. This is the default group.
To check how much space is being used in a directory, type du -h -d 0. Depending on the amount of files, this may take a long time.
To check overall /share quota usage use command mmlsquota -g group_name
This space is not backed up. Files that have not been accessed in 30 days are automatically deleted.
Supplemental group storage
Supplemental storage is available for data and applications.
Directory for user maintained software
HPC provides space for user installed software. The space is backed up daily. If a /usr/local/usrapps/group_name directory does not exist, see the requirements for requesting the space on the HPC software page.
Directories in /usr/local/usrapps may not be used for data or as a working space from which to execute jobs. A compute node cannot write to /usr/local/usrapps.
Instructional use mass storage directory
Since a class lasts longer than 30 days, it may be necessary to store large inputs and examples in case of purge; in that case, the instructor may request a mass storage space in /gpfs_archive.
For more details and an example use case on using mass storage for a class, see Hazel for Instructional use: access, storage, software, and training.
There is a 1 TB group quota on this space. Mass storage is accessible only from the login nodes. It is not available from compute nodes and cannot be used as an alternative to scratch space for running jobs.
The mass storage is a hierarchically managed file system consisting of a disk cache and a tape library. In order to speed archival and retrieval, a tar archive of a directory, including all files and subdirectories, should be used to store data. Storing many individual files may be distributed across numerous tapes and take a long time to retrieve in the event that the recovery of an entire directory is required.
Submit a HPC consultation request
For further assistance, please request technical support.