DaskDistributedDispatcher.jl

Submit and execute distributed computations. A dask.distributed scheduler and Dispatcher.jl integration.
Author invenia
Popularity
14 Stars
Updated Last
6 Months Ago
Started In
June 2017

DaskDistributedDispatcher

Linux/macOS Build Status Windows Build status codecov

DaskDistributedDispatcher integrates Dispatcher.jl with the python dask.distributed scheduler service.

Documentation:

Overview

Dispatcher.jl builds the graph of julia computations and submits jobs via the julia client to the dask.distributed scheduler, which is in charge of determining when and where to schedule jobs on the julia workers. Thus, the computations can be scheduled and executed efficiently.

Quick Start

At the command line:

dask-scheduler

At the Julia REPL, given some Dispatcher nodes nodes:

addprocs(3)
@everywhere using DaskDistributedDispatcher

for i in workers()
	@spawnat i Worker()
end

dask_executor = DaskExecutor()

node_results = run!(dask_executor, nodes)

For a more detailed explanation, see the documentation linked above.

Frequently Asked Questions

How can the python dask.distributed scheduler be used for julia computations?

The dask.distributed scheduler can be used in a julia workflow environment since it is language agnostic (no information that passes in or out of it is Python-specific). Instead the scheduler communicates with the workers/clients entirely using msgpack and long bytestrings. More information on the protocol used is here.

License

DaskDistributedDispatcher.jl is provided under the Mozilla Public License 2.0.