Repository navigation
Description
Background
Substrait types inside expressions (literal types, cast targets, lambda parameter types) carry no struct field names; only relation schemas do. So the consumer names a struct literal's fields c0, c1, .... A column keeps the real names from its table.
DataFusion matches struct fields by name, so combining the two fails, even though they have the same shape:
-- s: List<Struct{a, b}>
SELECT coalesce(s, []) FROM t
-- No function matches 'coalesce(List(Struct("a", "b")), List(Struct("c0", "c1")))'Substrait matches struct fields by position, so the fix is to give the literal the column's field names, position by position. This doesn't change what the plan means.
Why not carry the names in the plan instead? Substrait's struct type is positional by design: names only appear in a relation's schema (NamedStruct), not in the types used inside expressions. Adding them would need a change to the Substrait spec, and every producer to adopt it. Carrying them in a DataFusion-specific extension would only help plans produced by DataFusion, not plans from other producers. So the consumer has to work with positional structs.
Limitation (by design)
Since the literal's names are lost, fields can only be matched by position. If a producer writes a struct literal with the same fields as the column in a different order, e.g. coalesce(s, {b: 20, a: 10}) with s: Struct{a, b}, the values end up in the wrong fields. Producers must write struct literals with fields in the column's order.
Cases
| Where the struct literal is combined with a struct column | Example | Status |
|---|---|---|
| Function arguments that need a common type | coalesce(s, []), nvl(s, NULL) |
Fixed in #27099 |
| Function arguments that are independent (left unchanged) | named_struct('l', s, 'r', <literal>) |
Fixed in #27099 |
| Operators | s = <literal>, s BETWEEN … |
Fixed in #27099 |
| Window function arguments | lag(s, 1, <literal>) |
Fixed in #27099 |
CASE results, including nested CASE |
CASE WHEN … THEN s ELSE NULL END (#14632) |
Fixed in #27099 |
IN lists |
s IN (<literal>, <literal>) |
Fixed in #27099 |
Substrait Nested::List elements |
[s, <literal>] |
Fixed in #27099 |
Set operation inputs, including under LIMIT/ORDER BY/DISTINCT and nested set operations |
SELECT s FROM t UNION ALL SELECT NULL FROM u |
Fixed in #27099 |
Dictionary / run-end encoded columns; List vs LargeList vs FixedSizeList vs ListView |
coalesce(large_list_col, [<literal>]) |
Fixed in #27099 |
| Struct literal combined with a lambda parameter inside the lambda body | array_transform(s, x -> coalesce(x, <literal>)) |
Open |
CAST to a struct type |
CAST(NULL AS STRUCT<…>) in a set operation, as DataFusion's own producer writes it |
Open |
| Set operation input whose top projection only references columns, with the literal below it | SELECT s, random() FROM t UNION ALL SELECT NULL, random() FROM u |
Open |
| Struct literal next to a list of structs | array_has(s, <literal>), array_append(s, <literal>) |
Open |
| Aggregate function arguments | none found yet | Open (no known case) |
In the cases below, the literal and the column only meet through a column reference: the literal is made in one input or subquery, and combined with the column further up. Fixing them needs to follow a column back to the expression that produced it.
| Where the struct literal is combined with a struct column | Example | Status |
|---|---|---|
| Above a join, with the literal in one input | SELECT coalesce(t.s, u.x) FROM t CROSS JOIN (SELECT NULL AS x) u |
Open |
| A join condition | t JOIN (SELECT NULL AS x) u ON t.s = u.x |
Open |
| A scalar subquery | WHERE s = (SELECT named_struct('a', 1, 'b', 'x')) |
Open |
An IN subquery |
WHERE s IN (SELECT named_struct('a', 1, 'b', 'x')) |
Open |
Reproduction tests for the open cases are available; comment here if you'd like to pick one up.
Activity
fix: give Substrait struct literals the field names of the struct the…
fix: give Substrait struct literals the field names of the struct the…