An extensible, practical and coherent algebra of query combinators
61 Stars
Updated Last
1 Year Ago
Started In
November 2017


DataKnots is a Julia library for querying data with an extensible, practical and coherent algebra of query combinators.

Documentation Build Status Process
Stable Documentation Development Documentation Build Status Code Coverage Status Chat on Zulip Open Issues MIT License

DataKnots is designed to let data analysts and other accidental programmers query and analyze complex structured data.


Let's take some Chicago public data and convert it into a DataKnot.

using DataKnots, CSV

employee_csv_file = """
    """ |> IOBuffer |> CSV.File

chicago = DataKnot(:employee => employee_csv_file)

We could then query this data to return employees with salaries greater than their department's average.

using Statistics: mean

@query chicago begin
    keep(avg_salary => mean(employee.salary))
    filter(salary > avg_salary)
  │ employee                                         │
  │ name       department  position           salary │
1 │ JAMES A    FIRE        FIRE ENGINEER-EMT  103350 │
2 │ JEFFERY A  POLICE      SERGEANT           101442 │

In this example, nouns, such as employee, department and salary, are query primitives. The verbs, such as group, keep, mean and filter are query combinators. Query expressions, such as group(department), are built from existing queries by applying these combinators.

Queries could also be constructed with pure Julia code, without using macros. The query above could be equivalently written:

using Statistics: mean

chicago[It.employee >>
        Group(It.department) >>
        Keep(:avg_salary => mean.(It.employee.salary)) >>
        It.employee >>
        Filter(It.salary .> It.avg_salary)]
  │ employee                                         │
  │ name       department  position           salary │
1 │ JAMES A    FIRE        FIRE ENGINEER-EMT  103350 │
2 │ JEFFERY A  POLICE      SERGEANT           101442 │


DataKnots implements an algebraic query interface of Query Combinators. This algebra’s elements, or queries, represent relationships among class entities and data types. This algebra’s operations, or combinators, are applied to construct query expressions.

We seek to prove that this query algebra has significant advantages over the state of the art:

  • DataKnots is a practical alternative to SQL with a declarative syntax; this makes it suitable for use by domain experts.

  • DataKnots' data model handles nested and recursive structures (unlike DataFrames or SQL); this makes it suitable for working with CSV, JSON, XML, and SQL databases.

  • DataKnots has a formal semantic model based upon monadic composition; this makes it easy to reason about the structure and interpretation of queries.

  • DataKnots is a combinator algebra (like XPath but unlike LINQ or SQL); this makes it easier to assemble queries dynamically.

  • DataKnots is fully extensible with Julia; this makes it possible to specialize it into various domain specific query languages.


At this time, while we welcome feedback and contributions, DataKnots is not yet usable for general audiences.

Our development chat is currently hosted on Zulip:

Current documentation could be found at:

Used By Packages

No packages found.