Dispatcher.jl with the python
dask.distributed scheduler service.
Dispatcher.jl builds the graph of julia computations and submits jobs via the julia client to the
dask.distributed scheduler, which is in charge of determining when and where to schedule jobs on the julia workers. Thus, the computations can be scheduled and executed efficiently.
At the command line:
At the Julia REPL, given some Dispatcher nodes
addprocs(3) @everywhere using DaskDistributedDispatcher for i in workers() @spawnat i Worker() end dask_executor = DaskExecutor() node_results = run!(dask_executor, nodes)
For a more detailed explanation, see the documentation linked above.
Frequently Asked Questions
How can the python
dask.distributedscheduler be used for julia computations?
dask.distributed scheduler can be used in a julia workflow environment since it is language agnostic (no information that passes in or out of it is Python-specific). Instead the scheduler communicates with the workers/clients entirely using msgpack and long bytestrings. More information on the protocol used is
DaskDistributedDispatcher.jl is provided under the Mozilla Public License 2.0.