![]() Ideally, the ideas get taken up by existing systems, though. There are only so many cycles the world is willing to spend on producing (or learning) production-grade HPC tools, and lots of ideas never really make it to launch, or never really catch on once they are out in the world. That’s partly because of something you’ve doubtless already noticed: high-performance programming is an enormous amount of work, and getting the best performance puts a lot of pressure on compiler writers. Third, parallel scientific computing has been a concern for as long as I’ve been alive – literally – and yet I’ve said very little about the many other parallel systems that have been explored. ![]() But this course is deliberately oriented toward the HPC side of the world. This is mostly a matter of scope: if I was doing a course on systems for ML, I might talk about all these things (and maybe Chris De Sa does talk about these things in his ML systems course!). No Spark, no Hadoop, no TensorFlow or PyTorch or Keras or anything like that. Second, I’ve said nothing about any of the big parallel systems enjoying wide use for data analytics and machine learning. First, we haven’t said anything about accelerator programming, even though that’s a pretty giant part of the HPC landscape right now the dominant programming systems there are CUDA and OpenCL, though there are lots of competitors (including OpenMP, which I think we mentioned in passing). There are a couple things that you might think are obvious holes. ML systems (TensorFlow, PyTorch, Keras, …)?.Data analytics systems (Spark, Hadoop, …)?.With that said, what else should we be talking about when it comes to the landscape of parallel programming environments? Beyond MPI and OpenMP We’ve also spent a lot of time talking about how to think about performance and parallelism more generally, which I think is useful independent of the specific programming environment in question. I don’t think we’ve done them full credit – there are books worth of stuff on both systems. Both of these systems have been around since the 1990s, so over 20 years. We’ve spent some time on the two biggest modern programming environments for high-performance parallel computing: MPI and OpenMP. ![]() Model, tune, and measure! Parallel programmingīefore we talk about the big picture, let’s talk about where we are. The key, though, should you choose to follow this path, is that you should do your playing with an eye to performance. Indeed, there are a lot of parallel programming languages and frameworks out there! And I want to point to a few of them, because playing with one of these can make the basis for a fine class project. But before that, I want to take a step back and talk about the broader parallel programming picture. I’m going to spend the bulk of this lecture talking about parallel global address space (PGAS) languages generally, and UPC and UPC++ in particular. And second, I originally planned to finish recording and posting by Thursday, not by the end of the following weekend! What can I say, it has been an eventful week. First, I originally planned to talk about transform methods in this slot I only thought better of it as I started thinking about what to say about the final project. This slide deck is not exactly when and where I planned things at the start of the semester, in two ways. CS 5220 Applications of Parallel Computers Beyond MPI and OpenMP
0 Comments
Leave a Reply. |