ICE 2.0: Restructuring and Growing an Instructional HPC Cluster

The Instructional Cluster Environment (ICE) resource at Georgia Tech's (GT) Partnership for an Advanced Computing Environment (PACE) had its roots in a request and funding from the GT College of Computing (COC) [4]. The PACE team initially constructed two ICEs. The first was funded by COC, which would support COC courses. Building on its success, PACE funded and built a second cluster shortly after, available for use by any course at GT. These were dubbed COC-ICE and PACE-ICE, respectively. After five years, these resources continue to offer high-quality learning environments for a variety of courses. In the 2022-2023 academic year, 55 courses employed ICE, providing access to over 5000 students. In addition, ICE serves regular workshops providing training in scientific computing, taught by PACE research faculty or other GT faculty, and it has also hosted two Open Hackathons in partnership with Nvidia [20].

An intervening datacenter migration [15] allowed for an opportunity to refresh the hardware and software stack of the two clusters, while keeping the architecture constant. However, the separation of the resources began to lead to unnecessarily increased workload on the PACE team, when the purpose and design of each cluster was essentially the same. During a group-wide migration to the Slurm scheduler, we saw an opportunity to simplify the team-wide workload and generally improve the ICE resources, by combining them into a single cluster through our ICE 2.0 project.

The ICE cluster is distinctive in that it uses the same basic hardware types as our larger research-oriented clusters (although it has more variation in order to support student exploration of hardware architectures), and we hope to highlight here the ways in which the design for a campus-wide resource aimed at instructional use differs from those, which are more commonly documented in the cyberinfrastructure (CI) community. While there are a handful of systems purposefully built for educational purposes in the RCD community, for which publications exist, they often take the form of so-called "Microclusters" [1, 2, 6] using clusters of low-performance boards, cloud resources [19], or are discussed largely from a pedagogical perspective [3].

One of the key challenges in widely-available HPC systems is that of managing authentication - whether due to the size of the system itself [5], multiple layers of web software on top of HPC-system level software [16], or simply the struggles inherent in interfacing with separately controlled Identity and Access Management (IAM) systems [17].

The availability of HPC resources for use in non-CS and non-HPC-specific courses is described as a key method of propagating HPC knowledge at the university level by a recent working group on HPC education [24]. In addition to use in traditional courses, the ICE system supports a variety of workshops, similar in spirit to those at other institutions [7], which help grow and strengthen our HPC user community. Having a purpose-built resource and tightly integrated account management workflow for this greatly reduces the strain of providing new users access to a more research-focused cluster [10] with minimal human intervention in the account provisioning process [17].

Our migration from two separate clusters to one instructional resource provides an excellent opportunity to share design lessons learned over the last several years, which we have implemented in the final ICE 2.0 design and policies. Initially, we discuss our goals for the ICE project, including how we hope to improve resource utilization and wait times, goals of the merge, and migration from the Adaptive Computing Torque/Moab [8, 26] scheduler and resource manager to Slurm [27]. We describe the storage layout and data merge in detail, including an autofs configuration that allows us to interface cleanly with central IAM for account provisioning. Finally, we describe a floating partition and Quality-of-Service setup which allows us to satisfy a variety of stakeholder requirements while providing a straightforward set of QOS levels for new users.

The ICE 2.0 project aimed to restructure our instructional services and improve student experiences by increasing resource access, updating our scheduler and scientific software, and introducing faster storage. We also sought to streamline systems administration and report usage metrics more effectively.

The division between two ICE clusters, one for the College of Computing and one for other colleges, limited student access to resources, increasing wait times in periods of peak utilization, and increased the administrative effort to maintain the systems. In collaboration with COC, PACE merged the two clusters to form the new ICE. Overflow between nodes increases availability of resources for all courses and decreases wait times while preserving priority for COC courses on former COC-ICE nodes and for other colleges’ courses on former PACE-ICE nodes. All students also gained the opportunity to experiment with the additional GPU architectures added by COC in recent years, including Nvidia A40 and A100 and AMD MI210 architectures. Merging the clusters also eliminated confusion, as students and instructors frequently attempted to access the wrong login node via ssh or browsed to the wrong Open OnDemand [11] interface between the similarly-named clusters.

ICE 2.0 introduced the Slurm scheduler to ICE, replacing the Moab and Torque schedulers previously in place. PACE's research clusters were in the progress of migrating schedulers in the same year, while also receiving a brand-new software stack with updated compilers, libraries, and scientific software applications [14]. ICE received the same software, aligning the research and instructional environments, which eases the transition between clusters for those engaged in advanced computing for both research and education while requiring the PACE team to maintain only one software stack for both systems. The combined software stack greatly eases the administrative burden of maintaining a separate educational cluster.

The third goal of ICE 2.0 was to improve storage speed and capacity, in response to feedback from faculty teaching courses on the cluster. COC and PACE collaborated to purchase a Lustre parallel filesystem, much like the one on the flagship Phoenix research cluster [13], to provide new faster storage for ICE. This device provides faster performance than the home directories on ICE and is set up as scratch space for temporary storage. We also aimed to improve provisioning of shared directories, increasingly popular as space for course materials or collaborative work, and to introduce fixed methods and schedules for the removal of old data from ICE storage systems.

Beyond student-facing aspects, we also aimed to improve the administration of the cluster. Merging the two instructional clusters simplified management of the systems, while new data collection and organization offers better metrics for evaluating the impact of ICE on students.

Originally, the COC-ICE and PACE-ICE clusters were managed as isolated resources, each with their own dedicated scheduler and resource manager. Per Table 1, COC-ICE totaled 51 servers with a collection of 2^nd Generation Intel Scalable Xeon (Cascade Lake) and 3^rd Generation AMD EPYC (Milan) processor CPU nodes and GPU nodes with an array of AMD MI210 Instinct and Nvidia Quadro RTX6000, Tesla V100, and Ampere A40 and A100 accelerators; meanwhile, PACE-ICE offered Cascade Lake CPU and Nvidia Quadro RTX6000 or Tesla V100 GPU nodes. While PACE-ICE was fairly simple in implementation, having only two Torque queues for CPU and GPU jobs respectively, COC-ICE provided a more robust queuing structure, with nine queues configured with different resource limits according to policy, such as job size, length, or purpose (grading versus coursework), and architecture (as Intel and AMD hardware had separate queues due to differing software stacks). With the merger of the two clusters and the migration to Slurm, we opted to simplify scheduling by:

1. collapsing the COC-ICE queues by leveraging two-dimensional resource limits such as CPU-time,
   
2. maintaining common priority policies for users coming from different colleges, and
   
3. creating a modular infrastructure that could be augmented by additional resource additions.
   

Priority and resource access are managed using a combination of partitions and QOS designations, which are applied passively via a Lua job submit filter to enforce policy. To preserve priority based on original hardware purchases while enabling access to a larger pool of resources for spillover capability, the partitions in Slurm were arranged as a two-level system using the "PriorityTier" attribute [25]: the higher priority partition defined as a floating resource with CPUs or GPUs based on the previous cluster using a partition QOS, and the lower priority partition mapping to all hardware.

Floating partitions in Slurm are virtual divisions of the hardware that avoid limiting each partition to specific physical hardware, as a traditional partition would. The floating partition prescribes a maximum fraction of the total cluster, or of a subset of that cluster (such as GPU nodes) that can be occupied by jobs in that partition at any given time. Within each priority tier, there are two partitions, one for CPU-only nodes and the other for GPU nodes. As summarized in Table 2, three levels of QOS designations manage scheduling priorities and resources limits for scheduling jobs within a partition: Instructor/TA for quick turnaround on grading, Enrolled Students to ensure assignments can be completed on time, and, in the future, General Students to allow independent work on the cluster's idle cycles. Described further in Section 5, user allowed and default QOS are determined based on entitlements in LDAP, meaning that this whole process is fully automated based on user management at the college and registrar level.

Following the finalization of the ICE 2.0 merger and Slurm configuration details, the College of Engineering (COE) proceeded with a hardware purchase for 2023, represented in Table 1, to augment ICE and provide priority access to students from the college. As intended, the additional college was able to be implemented as a third pane in the configurations, validating the templated approach almost immediately after its inception.

Lastly, the Lua job submit plugin is leveraged to route jobs correctly to the partitions based on user QOS and resource requests, as shown in Figure 2, which shows the general situation with abstracted college names. Colleges which have made a financial investment in ICE (COC and COE) have their own priority partitions reflecting their purchased hardware fraction, while students enrolled in other colleges’ courses have priority through a partition reflecting PACE's own investment in ICE as a college-equivalent. Each user's job is submitted to the specific high priority floating CPU or GPU partition and the low priority“ice” CPU or GPU partition; additionally, unless a specific GPU architecture is requested, the job submit plugin appends the feature list to request an Nvidia GPU. The effect of this configuration is to automatically prioritize jobs based on purchased hardware, but also leverage opportunistic cycles from the cluster as a whole, improving cluster utilization and efficiency while reducing queue wait times. Example configuration files for this are included in the artifacts linked to in Appendix  A.

Account management for ICE is handled through GT's LDAP server. The details of enabling this setup have been described previously for the original ICE implementation [4] and largely remain in place. For the merged ICE, several new features were added to incorporate priority on different portions of the cluster, to improve the availability of usage metrics, and to ensure old data can be properly removed.

Collecting usage metrics is essential to understanding the use of valuable cyberinfrastructure resources and justifying investment in them. For an instructional cluster, this includes courses supported, numbers of students running compute jobs, utilization fraction of CPU and GPU resources, and more. We analyze much of this information through our instance of Open XDMoD [21].

Course enrollments are translated within the identity and access management system into school-based POSIX groups, which populate on the cluster. From here, we run scripts every 30 minutes (via cron) to identify new users added and create home directories and Slurm accounts for them. Slurm accounts, new in ICE 2.0, are based on the specific POSIX group, as described in Section 4, allowing us to tie an individual student's compute job to the school offering the course in which they are enrolled. For privacy reasons, individual course enrollments are not tracked. The full updated process of account provisioning is shown in Figure 3, including a top-level POSIX group for enabling ssh to the login nodes via access.conf and access to the Open OnDemand portal.

With the addition of Slurm accounts, we can now determine metrics in XDMoD by school offering the course, providing much more fine-grained detail beyond the prior information, which was limited to only two buckets: COC and all other colleges.

To further increase our understanding of the students on ICE, we began recording student majors from the identity management database, which reflects the diversity of student population learning scientific computing skills through ICE beyond only the schools offering ICE courses.

The POSIX groups for instructors and TAs are also recognized by the updated account creation script, populating them into the higher- priority “grade” QOS on the scheduler. This priority was previously introduced for TAs in COC to grade student coding assignments and is now expanded to support any instructor efforts campuswide to prepare or grade course materials.

The account management scripts have also been updated to track membership in the relevant access POSIX groups each day. This allows us to implement the storage policies in Section  3.3 and remove data for former students after sufficient time has passed. A generic version of the account sync script is described in  A.4.

Because ICE has a distinct purpose from PACE's research clusters, facilitation is handled differently. Most students in courses using ICE have no prior experience with high-performance computing, and PACE facilitators cannot support thousands of students directly. Instead, a tiered support structure is used, where students are directed to their instructors and teaching assistants (TAs) for questions, while instructors and TAs are encouraged contact PACE facilitators directly. PACE provides detailed documentation for all students, instructors, and TAs on use of ICE, its scheduler, software, and hardware. At the start of each semester, PACE offers customized training to instructors and TAs to ensure their comfort and readiness on ICE, focused on specific aspects required for their courses, and additional support is provided throughout the semester whenever instructors and TAs have questions. Courses uses ICE in highly variable ways, requiring different approaches to facilitation and resource utilization. Vertically Integrated Projects courses [9] have a research focus, where students spend multiple semesters in team projects. These courses tend to more closely resemble utilization on the research clusters, with longer jobs and more complex workflows. Many other courses work exclusively in Jupyter notebooks through Open OnDemand [11], often designed such that each student requires only a single CPU or a single GPU for a short period of time. Instructors teaching students with limited computing experience value the standardized environment offered by ICE. The wide variety of approaches, from research workflows to those that do not require true high-performance computing, require a broad set of resources and approaches to facilitation.

Building on the five-year-old Instructional Cluster Environments (ICE), Georgia Tech's PACE team completed the ICE 2.0 project to update and improve our instructional environment, merging two clusters, adopting the Slurm scheduler, updating scientific software, compilers, and libraries, adding a parallel filesystem, and improving account management and metrics. The upgraded cluster is designed to provide an increasing student population with a better educational experience, with more resource availability, more compute architectures, faster storage, and current software.