Multilevel Parallel Patterns

Parallel patterns represent an important high-level abstraction for programming complex parallel hardware systems and they have been endorsed by several top IT companies that offer their own pattern libraries (e.g. Intel Thread Building Blocks, Microsoft PPL). However, most of these pattern libraries are tailored to shared- memory multicore processors with the “flat” architecture of cores. On the other hand, most of the high-performance systems and supercomputers are built as distributed systems of computational nodes, where each node comprises a multicore processor (with a possible addition of accelerators). Adapting parallel patterns to such architectures would allow easy exploitation of massive computing power that these systems offer, while not requiring programmers to be hardware experts.

The aim of this project is to develop a prototype pattern library that implements basic parallel patterns (e.g. farm and pipeline) on distributed systems where each node comprises a multi- or many-core processor. The key idea is to implement multilevel patterns that will, at the system level, distribute parallel work between different nodes in a distributed system, and then at the node level, distribute work to the cores of a single node. The library can be written in Erlang or in C++ using a combination of MPI (for distribution of work at the system level) and some threading or pattern mechanism (e.g. C++17 threads, pthreads, OpenMP, Intel TBB) at the node level. Possible extensions include developing benchmark applications to test the library and considering more than two levels of work distribution (e.g. in systems with non- uniform memory architecture).

Skills the student will develop

Designing and implementing new programming models and abstractions

Distributed and shared-memory parallel programming (in Erlang or C++)

Understanding architecture of modern high-performance and supercomputing systems