The high-throughput computing community — including scientists, researchers, developers, administrators and facilitators — gathered at UW–Madison on June 3 to celebrate 40 years of “The Condor Project,” now known as HTCondor.
The brainchild of Miron Livny, lead investigator of research computing at the Morgridge Institute and director of the Center for High-Throughput Computing, HTCondor uses a vast network of computing resources to power a wide range of research projects across the globe.
HTCondor distributes computing power by matching and prioritizing jobs with available machines depending on the needs of the user and the properties of their dataset. This workload management system streamlines the influx of large datasets and computational inquiries that are born from curiosity-driven research.
“We are powerful enablers of scientific discovery,” Livny says in his opening remarks to kick off Throughput Computing Week HTC25, an annual conference which took place the week of June 2-6 this year.
In addition to 40 years of HTCondor, this year’s conference recognized several other milestones for high-throughput computing resources developed at UW–Madison: 20 years of OSG, 5 years of PATh, and 2 years of Pelican.
This suite of technologies support hundreds of registered users and institutions studying astrophysics, genomics, climate systems, artificial intelligence, wildlife populations, and more. While the research areas are varied, they all share the same problem: massive amounts of scientific data requiring massive amounts of computing power.
Erik Wright, assistant professor of biomedical informatics at the University of Pittsburgh, leads research in antibiotic resistance and is the number one user of these high-throughput computing resources in the field of biology.
“All of us across all these different domains are connected by the unfathomable scale of the things that we study,” he says in his Keynote Presentation at HTC25, leaning into the overarching theme of community during the conference. “This is a game changer for research.”
At the HTCondor Celebration event, several speakers offered their reflections from the past 40 years. Each one of them emphasized that the impact of the Condor Project has never come just from building the technologies, but from building a community.
“We are powerful enablers of scientific discovery.”
Miron Livny
The concept of community in this space didn’t grow out of necessity but rather was planted at the very beginning in Livny’s PhD dissertation as a guiding principle for what was to come: a shared resource built from communities of computers and communities of people.
“It’s really the underpinning element of what I’m trying to do with my involvement at the Morgridge Institute, it’s the community and it’s the people,” Livny says.
As an independent research institute, Morgridge has a distinct approach to pursue “fearless science” — giving scientists the freedom to drive innovation and pursue high-risk, high-reward endeavors.
“The flexibility that Morgridge provides to Miron and his team is a fantastic example of how we can support scientific collaborations with UW–Madison and other partners,” says Morgridge CEO Brad Schwartz. “It’s a remarkable advance in their field. They have created a resource that they continue to improve upon, and in return, that facilitates the achievements others. This is the hallmark of a highly impactful breakthrough.”
Livny says the success of the past 40 years was built on relationships and trust — and this is just the beginning.
“We created an ecosystem of people who depend on us, their science depends on us. This is a non-trivial responsibility that we must continue and carry on as we celebrate 50, 60, 70 years, because we are not done yet.”