First CRAN release. No user-facing behavior changed since 0.2.1; the work here was packaging.
dplyneage now declares R (>= 4.1.0), which the
examples have required since they started using the native
pipe.
The ducklake integration vignette moved to a website-only article. ducklake is not on CRAN, so shipping it as a vignette meant declaring a dependency CRAN cannot resolve. The article still builds on the pkgdown site, and nothing about ducklake support itself changed.
Anything that starts Python — the sqlglot examples, the sqlglot
tests, and the raw-SQL chunks in
vignette("getting-started") — is now skipped when
NOT_CRAN is unset. reticulate provisions its environment
over the network on first use, and CRAN checks run offline.
The copyright and license terms of the bundled React and React
Flow JavaScript are recorded in LICENSE.note, and their
copyright holders are named in DESCRIPTION.
The pure-R engine now handles both ways dbplyr stores select
expressions. Under dbplyr 2.5.x, partial evaluation wraps computed
expressions in quosures and leaves sql() calls unevaluated,
so edge labels picked up a leading ~ and raw-SQL columns
were traced as if they had no sources instead of raising the classed
error that triggers the sqlglot fallback. dbplyr 2.6.0 stores bare
expressions with sql() already evaluated, which is what the
engine was written against. Both layouts now produce identical
lineage.
lineage_flow() widgets that initialize while their
container is hidden (a non-active reveal.js slide, a hidden tabset
panel) now wait for the container to gain nonzero dimensions before
mounting React Flow. Mounting against a zero-size container pinned the
viewport at minZoom, and no later fit — including the Controls fit
button — could recover it, so widgets on non-first Quarto revealjs
slides rendered as a dot in the corner. No host-page JavaScript is
needed anymore to work around this.
The widget’s htmlwidgets resize hook is now
implemented: when the container changes size, the graph re-fits into the
new frame.
lineage_flow(height = ...) is respected when the
widget initializes hidden; previously the binding overrode it with a
600px default because the not-yet-laid-out container measured
zero.
extract_lineage() now gives an actionable error when
passed a plain data frame, pointing to the
dbplyr::memdb_frame() / copy_to() workaround
instead of failing later with a misleading message about Python or SQL
strings. The workaround is also documented in the README and on
?extract_lineage.
New lineage_openlineage() exports lineage as an
OpenLineage RunEvent with ColumnLineage facets
— the interchange format Marquez, DataHub, and OpenMetadata ingest, so
dplyneage-extracted lineage can sit alongside lineage from dbt, Airflow,
or Spark. Edge classifications map to OpenLineage transformation types,
including INDIRECT subtypes for
include_indirect edges.
New lineage_mermaid() exports lineage as a Mermaid
flowchart — paste it into a ```mermaid fence and it renders
natively on GitHub, in Quarto, and in most documentation tools, with no
htmlwidget involved. Tables draw as colored subgraphs, non-identity
edges carry their expression, and indirect edges draw dashed.
The getting-started vignette now covers local data frames:
plain-dplyr pipelines have no lazy query tree to trace, and
dbplyr::memdb_frame() (or any copy_to()) is
the one-line workaround that makes the identical pipeline
traceable.
New include_indirect argument for
extract_lineage(): columns used in
filter()/WHERE, join conditions,
group_by(), and
arrange()/ORDER BY — which shape the result
without appearing in it — draw as dashed edges to each output column,
classified by use ("filter", "join",
"group_by", "sort"). Impact analysis via
lineage_upstream()/lineage_downstream() then
sees them too: dropping a column used only in a filter()
still breaks the pipeline. Both engines support it, and multi-model
pipelines stitch indirect edges across layers.
extract_lineage() now stitches multi-model
pipelines: pass a named list of lazy tables or SQL strings (one element
per model) and any source table matching another element’s name links to
that model’s node, so a bronze/silver/gold flow renders as one multi-hop
DAG. Intermediate models draw as orange transform nodes, terminal models
as green targets, and metadata$models records each model’s
SQL and engine.
Diagrams are laid out by a height-aware layered algorithm: each pipeline hop advances one column, nodes stack with spacing that accounts for their column count (tall tables no longer overlap), and layers are vertically centered.
The ducklake vignette now ends with the stitched whole-lake
diagram and a transitive lineage_upstream() impact
query.
extract_lineage() results are now classed
dplyneage_lineage with a compact print method summarising
engine, tables, output columns, and edge count.
New lineage_edges() and
lineage_tables() flatten a lineage object into plain data
frames — one classified row per column edge, one row per table.
Lineage edges are now classified as identity,
aggregation, or transformation (mirroring
OpenLineage’s transformation types) in both engines. Diagrams label
non-identity edges with the column’s defining expression and animate
aggregations automatically; lineage_json() and
lineage_graphml() carry the classification and expression
on each edge.
New lineage_diff() compares two extractions and
reports added/removed edges and columns — extract lineage on two
branches and fail CI when column provenance changed.
New lineage_upstream() and
lineage_downstream() answer impact questions (“what feeds
this column?” / “what does this column feed?”) by transitive traversal,
without exporting to igraph first.
reticulate has moved from Imports to Suggests: dbplyr pipelines
are analyzed entirely in R, so Python tooling is now only installed by
users who analyze raw SQL. extract_lineage() and
has_sqlglot() explain the requirement when reticulate is
missing.
Schema-qualified tables keep their qualifier:
stg.orders and raw.orders are now distinct
nodes in both engines instead of merging into one orders
node, extract_lineage()’s schema argument
accepts qualified names (list("stg.orders" = ...)), and
automatic schema harvesting looks qualified tables up
correctly.
extract_lineage() no longer lets a real table named
output collide with the synthetic output node, and sources
whose table cannot be determined (NA or empty names) now
connect to the unknown node instead of producing dangling
edges.
The sqlglot engine now records each output column’s actual defining expression (previously it recorded the column name), matching the R engine.
metadata$table_count is now
metadata$node_count, since it counts all diagram nodes
including the output node.
The static SVG fallback in lineage_flow() escapes
table labels before inserting them into HTML.
lineage_flow() now routes each target column’s edges
through its own vertical lane instead of bending every edge at the same
midpoint, so parallel edges no longer draw on top of each other. Edges
fanning into the same target column still merge into one lane on
purpose. Lanes are fractions of the source-to-target span, so they hold
up when nodes are dragged.
New vignette vignette("ducklake-lineage") shows
dplyneage working with ducklake: lineage for
lake pipelines, per-layer diagrams, and time-travel queries
(#1).
New lineage_json() and
lineage_graphml() export extract_lineage()
results (or hand-built node/edge lists) to interchange formats: a clean
JSON schema for scripting, CI diffs, and data catalogs, and column-level
GraphML that loads directly into igraph, Gephi, or yEd for impact
analysis.
extract_lineage() now analyzes dbplyr lazy tables
with a pure-R engine that walks the pipeline’s lazy query tree — no
Python required. Column provenance is read directly from the tree, so
joins (including suffix conflicts and coalesced full-join keys),
aggregates, window expressions, and set operations resolve
exactly.
New engine argument for
extract_lineage(): "auto" (the default) uses
the R engine for lazy tables and falls back to sqlglot for SQL strings
or constructs the R engine cannot trace, such as raw SQL injected with
dbplyr::sql(); "r" and "sqlglot"
force a specific engine. Requires dbplyr >= 2.5.0 for the R
engine.
extract_lineage() results now record which engine
ran in metadata$engine.
First public release.
extract_lineage() extracts column-level lineage from
dplyr/dbplyr pipelines or raw SQL strings, powered by sqlglot’s lineage
engine. Aliases, CTEs, subqueries, set operations (UNION), and
multi-source computed columns (e.g. COALESCE(a.x, b.x)) all
resolve to their true source columns.extract_lineage() automatically reads each referenced
table’s columns from the database connection so unqualified columns are
attributed to the correct table and SELECT * expands. For
raw SQL, pass the new schema argument.lineage_flow() renders interactive React Flow diagrams
with column-level edges, draggable table nodes, hover highlighting, and
zoom/pan controls. Accepts extract_lineage() output
directly in a pipe.create_table_node() and
create_column_edge() for building lineage diagrams
manually, plus lineage_example() as a built-in demo.lineage_flowOutput() and
renderLineageFlow().reticulate::py_require() — no manual setup step.
install_sqlglot() is deprecated and does nothing.