Reference

DataKnots.It — Constant

It :: AbstractQuery

In a query expression, use It to refer to the query's input.

julia> unitknot[Lift(3) >> (It .+ 1)]
┼───┼
│ 4 │

It is the identity with respect to query composition.

julia> unitknot[Lift('a':'c') >> It]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

It provides a shorthand notation for data navigation using Get, so that It.a.x is equivalent to Get(:a) >> Get(:x).

julia> unitknot[Lift((a=(x=1,y=2),)) >> It.a]
│ a    │
│ x  y │
┼──────┼
│ 1  2 │

julia> unitknot[Lift((a=(x=1,y=2),)) >> It.a.x]
│ x │
┼───┼
│ 1 │

source

DataKnots.unitknot — Constant

unitknot

The unit knot holds an empty tuple.

julia> unitknot
┼──┼
│  │

The unitknot is useful for constructing queries that do not originate from another datasource.

julia> unitknot["Hello"]
┼───────┼
│ Hello │

source

DataKnots.DataKnot — Type

DataKnot(Pair{Symbol}...)

This constructor binds names to datasets, so that they could be used to start a query. The knot created has a single top-level record, each with its own value.

julia> test_knot = DataKnot(:dataset=>'a':'c')
│ dataset │
┼─────────┼
│ a; b; c │

julia> test_knot[It.dataset]
  │ dataset │
──┼─────────┼
1 │ a       │
2 │ b       │
3 │ c       │

Arguments to this constructor are run though convert.

convert(DataKnot, val)

This converter wraps a given value so that it could be used to start a query.

An empty knot can be constructed with missing.

julia> convert(DataKnot, missing)
(empty)

A plural knot is constructed from a vector.

julia> convert(DataKnot, 'a':'c')
──┼───┼
1 │ a │
2 │ b │
3 │ c │

An object that complies with the Table interface, such as a CSV file, can be converted to a DataKnot.

julia> using CSV;

julia> csv_file = "k,v\na,1\nb,\n" |> IOBuffer |> CSV.File;

julia> convert(DataKnot, csv_file)
  │ k  v │
──┼──────┼
1 │ a  1 │
2 │ b    │

get(::DataKnot)

Use get to extract the underlying value held by a knot.

julia> get(convert(DataKnot, "Hello World"))
"Hello World"

getindex(::DataKnot, X; kwargs...)

We can query a knot using array indexing notation.

julia> convert(DataKnot, (dataset='a':'c',))[Count(It.dataset)]
┼───┼
│ 3 │

Query parameters are provided as keyword arguments.

julia> convert(DataKnot, 1:3)[PWR=2, It .^ It.PWR]
──┼───┼
1 │ 1 │
2 │ 4 │
3 │ 9 │

source

DataKnots.Collect — Method

Collect(X₁, X₂ … Xₙ) :: Query

In the combinator form, Collect(X₁, X₂ … Xₙ) adds fields X₁, X₂ … Xₙ to the input record.

julia> unitknot[Record(:x => 1) >> Collect(:y => 2 .* It.x)]
│ x  y │
┼──────┼
│ 1  2 │

If a field already exists, it is replaced.

julia> unitknot[Record(:x => 1) >> Collect(:x => 2 .* It.x)]
│ x │
┼───┼
│ 2 │

To remove a field, assign it the value nothing.

julia> unitknot[Record(:x => 1) >> Collect(:y => 2 .* It.x, :x => nothing)]
│ y │
┼───┼
│ 2 │

Each(X >> Record) :: Query

In the query form, Collect appends a field to the source record.

julia> unitknot[Lift(1) >> Label(:x) >> Collect]
│ x │
┼───┼
│ 1 │

source

DataKnots.Count — Method

Count(X) :: Query

In the combinator form, Count(X) emits the number of elements produced by X.

julia> X = Lift('a':'c');

julia> unitknot[Count(X)]
┼───┼
│ 3 │

Each(X >> Count) :: Query

In the query form, Count emits the number of elements in its input.

julia> X = Lift('a':'c');

julia> unitknot[X >> Count]
┼───┼
│ 3 │

To limit the scope of aggregation, use Each.

julia> X = Lift('a':'c');

julia> unitknot[Lift(1:3) >> Each(X >> Count)]
──┼───┼
1 │ 3 │
2 │ 3 │
3 │ 3 │

source

DataKnots.Drop — Method

Drop(N) :: Query

This query drops the first N elements of its input, preserving the rest.

julia> unitknot[Lift('a':'c') >> Drop(2)]
──┼───┼
1 │ c │

Drop(-N) takes the last N elements.

julia> unitknot[Lift('a':'c') >> Drop(-2)]
──┼───┼
1 │ b │
2 │ c │

source

DataKnots.Each — Method

Each(X) :: Query

This evaluates X elementwise.

julia> X = Lift('a':'c') >> Count;

julia> unitknot[Lift(1:3) >> Each(X)]
──┼───┼
1 │ 3 │
2 │ 3 │
3 │ 3 │

Compare this with the query without Each.

julia> X = Lift('a':'c') >> Count;

julia> unitknot[Lift(1:3) >> X]
┼───┼
│ 9 │

source

DataKnots.Exists — Method

Exists(X) :: Query

In the combinator form, Exists(X) emits a boolean testing if X produces any elements.

julia> X = Lift('a':'c');

julia> unitknot[Exists(X)]
┼──────┼
│ true │

When the query argument X is empty, Exists(X) produces false.

julia> X = Lift([]);

julia> unitknot[Exists(X)]
┼───────┼
│ false │

Each(X >> Exists) :: Query

In the query form, Exists emits a boolean testing if its input has any elements.

julia> X = Lift('a':'c');

julia> unitknot[X >> Exists]
┼──────┼
│ true │

When the query input is empty, Exists produces false.

julia> X = Lift([]);

julia> unitknot[X >> Exists]
┼───────┼
│ false │

source

DataKnots.Filter — Method

Filter(X) :: Query

This query emits the elements from its input that satisfy a given condition.

julia> unitknot[Lift(1:5) >> Filter(isodd.(It))]
──┼───┼
1 │ 1 │
2 │ 3 │
3 │ 5 │

When the predicate query produces an empty output, the condition is presumed to have failed.

julia> unitknot[Lift('a':'c') >> Filter(missing)]
(empty)

When the predicate produces plural output, the condition succeeds if at least one output value is true.

julia> unitknot[Lift('a':'c') >> Filter([true,false])]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

source

DataKnots.First — Method

First(X) :: Query

In the combinator form, First(X) emits the first element produced by its argument X.

julia> X = Lift('a':'c');

julia> unitknot[First(X)]
┼───┼
│ a │

Each(X >> First) :: Query

In the query form, First emits the first element of its input.

julia> X = Lift('a':'c');

julia> unitknot[X >> First]
┼───┼
│ a │

source

DataKnots.Get — Method

Get(lbl::Symbol) :: Query

This query extracts a field value by its label.

julia> unitknot[Lift((x=1, y=2)) >> Get(:x)]
│ x │
┼───┼
│ 1 │

This has a shorthand form using It.

julia> unitknot[Lift((x=1, y=2)) >> It.x]
│ x │
┼───┼
│ 1 │

With unlabeled fields, ordinal labels (A, B, ...) can be used.

julia> unitknot[Lift((1,2)) >> It.B]
┼───┼
│ 2 │

source

DataKnots.Given — Method

Given(X₁, X₂ … Xₙ, Q) :: Query

This evaluates Q in a context augmented with named parameters added by a set of queries.

julia> unitknot[Given(:x => 2, It.x .+ 1)]
┼───┼
│ 3 │

source

DataKnots.Group — Method

Group(X₁, X₂ … Xₙ) :: Query

This query groups the input data by the keys X₁, X₂ … Xₙ.

julia> unitknot[Lift(1:5) >> Group(isodd.(It))]
  │ #A     #B      │
──┼────────────────┼
1 │ false  2; 4    │
2 │  true  1; 3; 5 │

source

DataKnots.Is — Method

Is(T::Type) :: Query

This query asserts that the input has the type T.

source

DataKnots.Is0to1 — Method

Is0to1(X) :: Query

This query asserts that X emits 0 or 1 element.

Each(X >> Is0to1) :: Query

In this form, Is0to1 asserts that its input contains 0 or 1 element.

source

DataKnots.Is0toN — Method

Is0toN(X) :: Query

This query asserts that X may emit any number of elements.

Each(X >> Is0toN) :: Query

In this form, Is0toN asserts that its input contains any number of elements.

source

DataKnots.Is1to1 — Method

Is1to1(X) :: Query

This query asserts that X emits 1 element.

Each(X >> Is1to1) :: Query

In this form, Is1to1 asserts that its input contains 1 element.

source

DataKnots.Is1toN — Method

Is1toN(X) :: Query

This query asserts that X emits 1 or more elements.

Each(X >> Is1toN) :: Query

In this form, Is1toN asserts that its input contains 1 or more elements.

source

DataKnots.Join — Method

Join(X) :: Query

Join(X) evaluates X in the source context and adds it as a field to the input record.

julia> unitknot[Record(:x => 1) >> Each(Record(:y => 2 .* It.x) >> Join(It.x))]
│ y  x │
┼──────┼
│ 2  1 │

source

DataKnots.Keep — Method

Keep(X₁, X₂ … Xₙ) :: Query

Keep evaluates named queries, making their results available for subsequent computation.

julia> unitknot[Keep(:x => 2) >> It.x]
│ x │
┼───┼
│ 2 │

Keep does not otherwise change its input.

julia> unitknot[Lift(1) >> Keep(:x => 2) >> (It .+ It.x)]
┼───┼
│ 3 │

source

DataKnots.Label — Method

Label(lbl::Symbol) :: Query

This assigns a label to the output.

julia> unitknot[Lift("Hello World") >> Label(:greeting)]
│ greeting    │
┼─────────────┼
│ Hello World │

A label could also be assigned using the => operator.

julia> unitknot[:greeting => Lift("Hello World")]
│ greeting    │
┼─────────────┼
│ Hello World │

source

DataKnots.Last — Method

Last(X) :: Query

In the combinator form, Last(X) emits the last element produced by its argument X.

julia> X = Lift('a':'c');

julia> unitknot[Last(X)]
┼───┼
│ c │

Each(X >> Last) :: Query

In the query form, Last emits the last element of its input.

julia> X = Lift('a':'c');

julia> unitknot[X >> Last]
┼───┼
│ c │

source

DataKnots.Lift — Method

Lift(f, (X₁, X₂ … Xₙ)) :: Query

Lift lets you use a function as a query combinator.

julia> unitknot[Lift((x=1, y=2)) >> Lift(+, (It.x, It.y))]
┼───┼
│ 3 │

Lift is implicitly used when a function is broadcast over queries.

julia> unitknot[Lift((x=1, y=2)) >> (It.x .+ It.y)]
┼───┼
│ 3 │

Functions accepting a AbstractVector can be used with plural queries.

julia> unitknot[sum.(Lift(1:3))]
┼───┼
│ 6 │

Functions returning AbstractVector become plural queries.

julia> unitknot[Lift((x='a', y='c')) >> Lift(:, (It.x, It.y))]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

source

DataKnots.Lift — Method

Lift(val) :: Query

This converts any value to a constant query.

julia> unitknot[Lift("Hello")]
┼───────┼
│ Hello │

AbstractVector objects become plural queries.

julia> unitknot[Lift('a':'c')]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

To specify the vector cardinality, add :x0to1, :x0toN, :x1to1, or :x1toN.

julia> unitknot[Lift('a':'c', :x1toN)]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

The missing value makes an query with no output.

julia> unitknot[Lift(missing)]
(empty)

source

DataKnots.Max — Method

 Max(X) :: Query

In the combinator form, Max(X) finds the maximum among the elements produced by X.

julia> X = Lift(1:3);

julia> unitknot[Max(X)]
┼───┼
│ 3 │

The Max of an empty input is empty.

julia> unitknot[Max(Int[])]
(empty)

Each(X >> Max) :: Query

In the query form, Max finds the maximum of its input elements.

julia> X = Lift(1:3);

julia> unitknot[X >> Max]
┼───┼
│ 3 │

source

DataKnots.Min — Method

 Min(X) :: Query

In the combinator form, Min(X) finds the minimum among the elements produced by X.

julia> X = Lift(1:3);

julia> unitknot[Min(X)]
┼───┼
│ 1 │

The Min of an empty input is empty.

julia> unitknot[Min(Int[])]
(empty)

Each(X >> Min) :: Query

In the query form, Min finds the minimum of its input elements.

julia> X = Lift(1:3);

julia> unitknot[X >> Min]
┼───┼
│ 1 │

source

DataKnots.Mix — Method

Mix(X₁, X₂ … Xₙ) :: Query

This query emits records containing every combination of elements generated by X₁, X₂ … Xₙ.

julia> unitknot[Mix(Lift(1:2), Lift('a':'c'))]
  │ #A  #B │
──┼────────┼
1 │  1  a  │
2 │  1  b  │
3 │  1  c  │
4 │  2  a  │
5 │  2  b  │
6 │  2  c  │

source

DataKnots.Nth — Method

Nth(X, N) :: Query

In the combinator form, Nth(X, N) emits the Nth element produced by its argument X.

julia> X = Lift('a':'d');

julia> N = Count(X) .÷ 2;

julia> unitknot[Nth(X, N)]
┼───┼
│ b │

Each(X >> Nth(N)) :: Query

In the query form, Nth(N) emits the Nth element produced by its input.

julia> X = Lift('a':'d');

julia> N = Count(X) .÷ 2;

julia> unitknot[X >> Nth(N)]
┼───┼
│ b │

source

DataKnots.Record — Method

Record(X₁, X₂ … Xₙ) :: Query

This query emits a record with fields generated by X₁, X₂ … Xₙ.

julia> unitknot[Lift(1:3) >> Record(It, It .* It)]
  │ #A  #B │
──┼────────┼
1 │  1   1 │
2 │  2   4 │
3 │  3   9 │

Field labels are inherited from queries.

julia> unitknot[Lift(1:3) >> Record(:x => It,
                                    :x² => It .* It)]
  │ x  x² │
──┼───────┼
1 │ 1   1 │
2 │ 2   4 │
3 │ 3   9 │

source

DataKnots.Sort — Method

Sort(X₁, X₂ … Xₙ) :: Query

This query sorts the input data by the keys X₁, X₂ … Xₙ.

julia> unitknot[Lift(1:5) >> Sort(isodd.(It))]
──┼───┼
1 │ 2 │
2 │ 4 │
3 │ 1 │
4 │ 3 │
5 │ 5 │

source

DataKnots.Sum — Method

Sum(X) :: Query

In the combinator form, Sum(X) emits the sum of elements produced by X.

julia> X = Lift(1:3);

julia> unitknot[Sum(X)]
┼───┼
│ 6 │

The Sum of an empty input is 0.

julia> unitknot[Sum(Int[])]
┼───┼
│ 0 │

Each(X >> Sum) :: Query

In the query form, Sum emits the sum of input elements.

julia> X = Lift(1:3);

julia> unitknot[X >> Sum]
┼───┼
│ 6 │

source

DataKnots.Tag — Method

Tag(name::Symbol, F) :: Query

This provides a substitute name for a query.

julia> IncIt = It .+ 1
It .+ 1

julia> IncIt = Tag(:IncIt, It .+ 1)
IncIt

Tag(name::Symbol, (X₁, X₂ … Xₙ), F) :: Query

This provides a substitute name for a query combinator.

julia> Inc(X) = Lift(+, (X, 1));

julia> Inc(It)
Lift(+, (It, 1))

julia> Inc(X) = Tag(:Inc, (X,), Lift(+, (X, 1)));

julia> Inc(It)
Inc(It)

source

DataKnots.Take — Method

Take(N) :: Query

This query preserves the first N elements of its input, dropping the rest.

julia> unitknot[Lift('a':'c') >> Take(2)]
──┼───┼
1 │ a │
2 │ b │

Take(-N) drops the last N elements.

julia> unitknot[Lift('a':'c') >> Take(-2)]
──┼───┼
1 │ a │

source

DataKnots.Unique — Method

Unique(X) :: Query

This query produces all distinct elements emitted by X.

julia> unitknot[Unique(['a','b','b','c','c','c'])]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

Each(X >> Unique) :: Query

In the query form, Unique produces all distinct elements of its input.

julia> unitknot[Lift(['a','b','b','c','c','c']) >> Unique]
──┼───┼
1 │ a │
2 │ b │
3 │ c │

source

DataKnots.@query — Macro

 @query dataset expr param=...

Applies the query to a dataset with a given set of parameters.

julia> @query unitknot 2x+1 x=1
┼───┼
│ 3 │

source

DataKnots.@query — Macro

 @query expr

Creates a query object from a specialized path-like notation:

bare identifiers are translated to navigation with Get;
query combinators, such as Count(X), use lower-case names;
the period (.) is used for query composition (>>);
aggregate queries, such as Count, require parentheses;
records can be constructed using curly brackets, {}; and
functions and operators are lifted automatically.

julia> @query 2x+1
Lift(+, (Lift(*, (Lift(2), Get(:x))), Lift(1)))

source