Programmers of this writer’s generation were privileged to see first concurrency, then true parallelism emerge on microcomputer platforms. So we had time to absorb the disciplines and motifs of parallelism gradually, as we encountered them in projects.
In my own case, exposure to these ideas began through writing interrupt-driven assembly language routines for eight-bit CPUs, proceeded to more formal work with process concurrency in XINU (is not Unix … a Unix-like OS developed by Doug Comer at Purdue for use in teaching OS architecture), HP-UX and other second- and third-gen Unix flavors, and thread programming in Windows and Linux – all before multi-core fully arrived.
A student’s challenge
By contrast, today’s students have a very different challenge. All this technology is now readily available, commonplace, inescapable, ubiquitous. So students can feel as if there’s no way into the topic that doesn’t require learning a great deal of fundamentally tricky, mind-bending stuff all at once.
And this can be overwhelming, particularly when the “multi-core parallel” part gets bound up with what may be unfamiliar (albeit fascinating) technical and mathematical content pertinent to the character of a particular hardware platform or class of applications, such as the mechanics of shader cores or complex algorithms for graphics, fluid dynamics, etc. It can feel like too much – especially for those looking to master multi-thread/multi-core and/or many-core parallel programming to broaden professional horizons, rather than pursue immediate and specific career goals.
Where should I start?
So where do you start with this stuff? As with any complicated topic, there are various ways into parallel development, which might be summarized as “from the bottom up,” and “from the top down.”
These are best ballasted with some practical instruction, which can and will typically come from many sources. Currently working coders would, however, be hard-pressed to find a better, more accessible resource than Clay Breshears’ book, The Art of Concurrency, subtitled “A Thread Monkey’s Guide to Writing Parallel Applications.” Breshears currently works as a Courseware Architect for Intel’s Scale Programs Organization, designing training courses in multi-core/multi-threaded development, but was for some years a Senior Parallel Application Engineer, and several years in academic research. He has a PhD in computer science from the University of Tennessee.
Breshears’ book manages to strike exactly the right tone for people who already code for a living and know their way around a comma-delimited imperative language (e.g. C++). He covers ground quickly but completely, jumping right into classifying parallel applications, talking about real tools like OpenMP and Intel Threading Building Blocks (TBB), and dispensing liberal nuggets of programming wisdom. At the same time, however, Breshears introduces dramatically new ideas at a measured pace that eliminates head-scratching.
Helping everyone understand
That’s important: Breshears is mirroring an approach now being pioneered at the upper end of undergraduate CS teaching of parallel computing, which seeks to help everyone taking a course to understand the material. Among other places, the approach is well articulated by University of Pennsylvania professor Patrick Cozzi in a long, detailed blog post on his recent delivery of a new course on GPU programming for undergraduates (http://blog.virtualglobebook.com/2011/05/reflections-on-teaching-gpu-programming.html).
One of the important points Breshears makes, again and again, is that tools never solve the whole problem – and indeed, they’re not supposed to. Instead, he maps how good programmers use the tools to solve the problem at hand – for example, by using TBB to create a task queue to serve worker threads on demand. As he introduces these design patterns, he generally maps them to the parallel development patterns enumerated in another important resource: Tim Mattson’s Patterns for Parallel Programming – another recommended acquisition for the self-learner.
Start with C and C++
In working through the ideas presented by Breshears, it makes sense to start with C and C++ as fundamental tools for bottom-up exploration: first using the Cilk Plus array notation to explore the high-payoff/low-complexity topic of SIMD vectorization, then beginning threadwise experiments with Intel TBB. Unfortunately, as has been widely acknowledged since the 1980s, imperative languages like C can’t easily and essentially convey “parallel thinking,” the innate acceptance and understanding of the asynchronous/non-sequential nature of parallel program behavior.
Trying different languages
Those who find themselves challenged by these ideas may profit by playing with another programming language whose structure, syntax, and assumptions support asynchronicity and parallelism more innately. Recently, I’ve begun fooling with Erlang and Haskell, both intrinsically functional languages with HPC, parallelism, and threadwise concurrency baked in. As helpful as working with the languages can be, moreover, it’s equally mind-expanding to read through some of the documentation and commentary illuminating how these languages and their associated packages manage parallel development idioms. For example, Bryan O’Sullivan’s online book, Real World Haskell, contains a very useful discussion of modes of parallelism that may help shine light in the corners for anyone struggling with TBB in a C++ environment.
John Jainschigg is a Geeknet contributing editor, and is CEO of World2Worlds, Inc., a digital agency focused on immersive technology and gaming. John’s initial intro to concurrency was via interrupt and re-entrancy programming at the assembler level on Z80 and 68000-based systems. He wrote concurrent, time-critical packet-switching applications on HP-UX RISC machines in the late 1980s, and since then has worked up and down the client-server stack in Java, C++, PHP, and other conventional and scripting languages, and more recently, in task-specific, state-based, radically concurrent languages like LSL.