High-throughput computing: Fostering data science without limits
December 6, 2022
Biology and big data are now completely inseparable.
Most modern biology produces data sets too massive to manage by conventional standards, and the challenge will increase exponentially as the sophistication of the science grows.
The Center for High-Throughput Computing (CHTC), a joint partnership of UW-Madison School of Computer, Data & Information Sciences and the Morgridge Institute, sees this onslaught of data and says: Bring it on.
“We have established a goal of never letting the amount of data limit the experimental approach of the scientists,” says Miron Livny, the founder of high-throughput computing (HTC). Livny has been championing HTC for more than three decades as a UW-Madison computer scientist, and more recently as the Morgridge Institute’s lead investigator of research computing.
HTCondor is a task-scheduling software approach that essentially breaks a larger computational task into smaller pieces, allowing researchers to analyze more data (hence the term “high throughput”). The team now handles 250-300 projects a year, double that of five years ago, and uses hundreds of millions of hours of computing time.
And that’s just at UW-Madison. The global Open Science Grid provides HTC resources to the world, where it is the backbone system for Nobel Prize-winning projects such as detecting gravitational waves and discovering new subatomic particles. Just this year, it made a splash for its contribution to the discovery of a massive black hole in the center of our galaxy.
This service is gaining adherents on campus because scientists are learning that it is more than someone asking, “What technology do you need?” Research computing is a collaboration, and the people HTC brings to the equation are more important than the technology.
Livny says the HTC Facilitation Team is a great example. The emphasis on facilitators was way ahead of its time, almost unheard of in computer science circles. These are the translators who can work their magic between the technology and the bench experiments — finding the best way to maximize the data for the scientists.
Read the Full Story on the Morgridge Website.