Skip to content

Substrait consumer: struct literals lose field names when combined with struct columns #27517

Open

Description

@LiaCastaneda

Background

Substrait types inside expressions (literal types, cast targets, lambda parameter types) carry no struct field names; only relation schemas do. So the consumer names a struct literal's fields c0, c1, .... A column keeps the real names from its table.

DataFusion matches struct fields by name, so combining the two fails, even though they have the same shape:

-- s: List<Struct{a, b}>
SELECT coalesce(s, []) FROM t
-- No function matches 'coalesce(List(Struct("a", "b")), List(Struct("c0", "c1")))'

Substrait matches struct fields by position, so the fix is to give the literal the column's field names, position by position. This doesn't change what the plan means.

Why not carry the names in the plan instead? Substrait's struct type is positional by design: names only appear in a relation's schema (NamedStruct), not in the types used inside expressions. Adding them would need a change to the Substrait spec, and every producer to adopt it. Carrying them in a DataFusion-specific extension would only help plans produced by DataFusion, not plans from other producers. So the consumer has to work with positional structs.

Limitation (by design)

Since the literal's names are lost, fields can only be matched by position. If a producer writes a struct literal with the same fields as the column in a different order, e.g. coalesce(s, {b: 20, a: 10}) with s: Struct{a, b}, the values end up in the wrong fields. Producers must write struct literals with fields in the column's order.

Cases

Where the struct literal is combined with a struct column Example Status
Function arguments that need a common type coalesce(s, []), nvl(s, NULL) Fixed in #27099
Function arguments that are independent (left unchanged) named_struct('l', s, 'r', <literal>) Fixed in #27099
Operators s = <literal>, s BETWEEN … Fixed in #27099
Window function arguments lag(s, 1, <literal>) Fixed in #27099
CASE results, including nested CASE CASE WHEN … THEN s ELSE NULL END (#14632) Fixed in #27099
IN lists s IN (<literal>, <literal>) Fixed in #27099
Substrait Nested::List elements [s, <literal>] Fixed in #27099
Set operation inputs, including under LIMIT/ORDER BY/DISTINCT and nested set operations SELECT s FROM t UNION ALL SELECT NULL FROM u Fixed in #27099
Dictionary / run-end encoded columns; List vs LargeList vs FixedSizeList vs ListView coalesce(large_list_col, [<literal>]) Fixed in #27099
Struct literal combined with a lambda parameter inside the lambda body array_transform(s, x -> coalesce(x, <literal>)) Open
CAST to a struct type CAST(NULL AS STRUCT<…>) in a set operation, as DataFusion's own producer writes it Open
Set operation input whose top projection only references columns, with the literal below it SELECT s, random() FROM t UNION ALL SELECT NULL, random() FROM u Open
Struct literal next to a list of structs array_has(s, <literal>), array_append(s, <literal>) Open
Aggregate function arguments none found yet Open (no known case)

In the cases below, the literal and the column only meet through a column reference: the literal is made in one input or subquery, and combined with the column further up. Fixing them needs to follow a column back to the expression that produced it.

Where the struct literal is combined with a struct column Example Status
Above a join, with the literal in one input SELECT coalesce(t.s, u.x) FROM t CROSS JOIN (SELECT NULL AS x) u Open
A join condition t JOIN (SELECT NULL AS x) u ON t.s = u.x Open
A scalar subquery WHERE s = (SELECT named_struct('a', 1, 'b', 'x')) Open
An IN subquery WHERE s IN (SELECT named_struct('a', 1, 'b', 'x')) Open

Reproduction tests for the open cases are available; comment here if you'd like to pick one up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      Sponsor
      SponsoredKunjungi sekarang
      Promo