If you had to guess what organization runs the world’s fastest, most powerful computer network, which would you choose? If your mind jumped to an organization like NASA or a financial or research institution, you’re close. The title, however, goes to SCinet, the temporary network built annually for the SC Conference, an event showcasing high-performance computing (HPC) research and technology.
The SC Conference hosts exhibits from the world’s most advanced research organizations and companies, so the bandwidth and speed requirements are tremendous.
I've gained insights through years of volunteering with SCinet. Leading a team of nearly 200 networking experts for SC23, I've seen firsthand how both volunteers and equipment contributors learn from the process.
One of the more interesting takeaways from recent efforts is the potential for maximizing performance or even reducing costs by relying on networking equipment from multiple vendors. Interoperability in networking technology has come a long way in recent years, and SCinet is a case study in the potential of bespoke networks for specialized use cases and more.
A Closer Look at SCinet
The 2023 version of SCinet, which will serve the attendees of SC23 in Denver from November 12-17, will deliver 6.71 terabits per second (Tbps) of WAN capacity and feature twelve 400 gigabits per second (Gbps) WAN connections. It includes state-of-the-art hardware, software, and services donated by 30 contributors, with a combined value exceeding $40 million.
The nuts-and-bolts installation of SCinet for SC23 is a roughly month-long sprint of volunteers representing more than 80 research and education institutions and commercial contributors. The bulk of the planning happens in the year leading up to the conference, starting with gathering the network requirements of our research participants.
With requirements in hand, the SC architecture team works with our partners and the SCinet technical teams to build a network architecture that meets participant needs.
VXLAN and Alien Wave Insights From SCinet
In recent years, there’s been a noticeable shift in the parts of the networking industry toward interoperability and open standards. For example, the openROADM group was established as a multi-source agreement (MSA) “comprised of carriers and vendors working together to define, characterize, and enable open disaggregated optical transport for a range of carrier deployment and interoperation criteria.”
What distinguishes SCinet from many other networks is our volunteers and contributors working together towards the creation of a very diverse architecture of hardware and software. Just consider that while a traditional network may only have two or three vendors, SCinet has 30.
This diversity doesn't just provide our volunteers a unique sandbox to experiment with a wide range of equipment, enriching their knowledge for their home institutions, it also offers our contributors a valuable lens to gauge how well their products mesh and interoperate with those of their peers.
This year, one of the more interesting things we pursued with respect to interoperability was our wide area network (WAN) team’s use of “Alien Waves” and our routing team's use of EVPN/VXLAN.
Chris Skaar, the WAN team co-chair, describes “alien waves” as “the transportation of optical spectrum sourced from a third party across an existing line system.” For instance, this could entail the use of transponders from one vendor to inject wavelengths into, and across, a different vendor's line system.
SCinet employs alien waves for a variety of reasons. Most importantly, it fosters a diverse multi-contributor environment. This approach not only showcases various contributors' products but also ensures SCinet isn't overly dependent on one contributor, thereby avoiding potential supply strains. Additionally, this method produces practical interoperability data beneficial for vendors, researchers, and SCinet volunteers.
Corey Eichelberger, the routing team co-chair explains VXLAN and its importance: “VXLAN enables connecting Layer 2 networks between any combination of wide-area network connections and different SC23 exhibitor facing network ports without having to use VLAN tagging through the middle of the network.”
He added: “This allows us to avoid both complex tagging configurations and potential spanning tree loops that can cause major network disruptions. This is not typically done across different vendor platforms. SCinet will use VXLAN across four different contributors’ equipment at SC23.”
Through my experience with SCinet I’ve gained a clear picture of the advantages of greater diversity in network architectures. For example, during the pandemic, when everyone was struggling with supply chain challenges, the SCinet approach allowed us to build a diverse network with many contributors thus alleviating the supply chain challenges faced by sourcing from a single contributor.
More diversity may also help organizations reduce the overall costs of their network installations. More importantly, however, interoperability means more options for configuring bespoke solutions for specialized workloads and requirements, such as artificial intelligence (AI), weather modeling, or high-energy physics.
For example, several network research exhibition experiments for SC23 required a large amount of bandwidth between Los Angeles, California, and the Colorado Convention Center in Denver. To support these experiments, the exhibitors asked the SCinet WAN team to provide 1.2 Tbps of transport between the sites.
One research and education contributor volunteered the optical spectrum on their line system, and two other contributors offered optical transponders for utilizing the spectrum. In the end, through close collaboration, we were able to capitalize on interoperability across the contributors' solutions to meet the exhibitors’ bandwidth requirements.
Of course, in many cases, economies of scale or other factors may dictate that a single-vendor solution still makes the most sense, but generally speaking, the possibilities are changing for the better.
Three Interoperability Lessons for IT Leaders
Volunteering for the SCinet team has exposed me to a host of leadership and networking lessons that would be hard to get elsewhere. On a personal note, as a network engineer, I have a newfound appreciation for the value of careful project management and risk management, and for having good backup plans in place.
While there is a lot of technical ground I could cover regarding the lessons I’ve learned through my work on SCinet, from a high level, these are my three most important takeaways:
- Social networking is important for engineering growth and education: SCinet is made up of volunteers who are expert network and systems engineers from the R&E, federal, and commercial realms. Making connections and being able to call on these experts for consultation, expertise, or advice is invaluable to our volunteers and their home organizations.
- Foster an environment of continuous learning and innovation: IT leaders should see the technological landscape not just as a set of tools, but as a dynamic field rife with opportunities for learning and innovation. By encouraging their teams to interact with diverse systems and stay updated with cutting-edge technologies, they foster an environment where innovation thrives and the organization stays future-ready
- Champion user and researcher involvement when designing a network: It is important to involve your users and researchers when you first design or upgrade your network architecture. Understanding their end-to-end workflows will allow you to use technology and hardware (possibly from various vendors) that best fit their needs and accelerate their work. For example, researchers may have partners all over the world, and the faster they can share their data, the faster they can achieve results. Building part of your network to meet the needs of long-distance file transfers may be very different than meeting the general networking needs of your wireless network users.
How to get involved with SCinet
SCinet only happens thanks to individual volunteers giving their time, and HPC-related companies or institutions contributing materials and services.
Volunteers join more than 180 professionals from around the globe working together to design, deliver, install, and operate SCinet at SC.
Our team members come from educational institutions, government agencies, high-performance computing sites, research and education networks, equipment vendors, and telecommunications carriers. To learn more about how to get involved, please visit our Participate in SCinet webpage.
Hans Addleman, the International Networks Technical Director at Indiana University, is the SC23 SCinet Chair.
