Create generic asynchronous parallel runtime

[maxfierrog]

Create a wrapper for a generic runtime backed by an opaque implementation of a parallel computation model, like MPI (multi-process), Tokio (async), or Rayon (multi-thread).

The internals of the Runtime can be implemented with Tokio. That is, a Tokio runtime will be encapsulated in a custom Runtime type, and the Tokio runtime will be the coordinator of the parallel implementation's workers. All Runtime calls should be synchronous and blocking, but internally, the Tokio runtime might make asynchronous / non-blocking calls to the underlying model.

In this sense, this is the first part of four things:

1. Synchronous runtime wrapper
2. Asynchronous generic coordinator
3. Bindings to parallel impls. (OpenMPI, rayon, more Tokio, etc.)
4. Thread-safe database implementation

Example

/// Available types of runtime. All of these assume a shared file system.
enum Runtime {
  ThreadPool { threads: u32 },
  Async { tasks: u32 },
  MPI { nodes: u32 },
}

/// Arguments to job functions.
struct Args<G, D> {
  game: &G,
  db: &mut D,
}

/// Solve the game.
fn solve<G: ..., D: ...>(game: &G, db: &mut D) -> Result<()> {
  let rt = Runtime::initialize(Runtime::MPI(30))?;
  rt.dashboard(Interface::Web)?;
  rt.route_logs(LOG_FILE)?;
  
  let mut graph = Graph::<JobID>::new(); // Partition solving jobs (JobID = Partition #)
  let mut set = Vec::<JobID>::new(); // Exploration jobs (JobID = State)
  
  starts = stochastic_exploration(game, db);
  graph = rt.independent(
    ExploreArgs { game, db },
    explore_job::<G, D>,
    jobs: set,
  )?;
  
  rt.dependent(
    SolveArgs { game, db },
    solve_job::<G, D>,
    jobs: graph,
  )?;
}

/// Explore the game tree.
fn explore_job<G: ..., D: ...>(args: Args<G, D>, target: JobID) -> Result<()> { ... }

/// Solve a single partition.
fn solve_job<G: ..., D: ...>(args: Args<G, D>, target: JobID) -> Result<()> { ... }

/// Sample random states in the game tree.
fn stochastic_exploration<G: ..., D: ...>(args: Args<G, D>) -> Result<()> { ... }

Recommended Courses (UC Berkeley)

If you can't tell whether this is out of your scope, we recommend the following experience:

• CS 61B (minimum)
• CS 61C (minimum)
• CS 162
• CS 267