Spark Atlas Connector creates lineage only for direct columns, not upper() or concat_ws() in CTAS

We are testing Spark column-level lineage with Apache Atlas in CDP Private Cloud Base 7.1.9.

A successful Spark CTAS operation creates this projection:

CREATE TABLE tdc_spark_ctas_test.customers_expression_test
USING PARQUET
AS
SELECT
    customer_id,
    first_name AS direct_first_name,
    upper(first_name) AS upper_first_name,
    concat_ws(' ', first_name, last_name) AS full_name_concat_ws
FROM tdc_spark_ctas_test.customers_raw;

The analyzed logical plan contains all four expressions. Atlas creates the target hive_table and all four hive_column entities, but only two spark_column_lineage entities:

- customer_id -> customer_id
- first_name -> direct_first_name

It creates no spark_column_lineage entity for:

- first_name -> upper_first_name
- first_name + last_name -> full_name_concat_ws

The Spark application finishes successfully. Atlas receives and processes the Spark notifications with failedMessageCount: 0, and no relevant Atlas ingestion error is recorded.

The CDP 7.1.9 Atlas reference documents spark_column_lineage entities and their input/output column relationships. Cloudera documentation also lists Spark CTAS as an operation that produces spark_column_lineage. I could not find documentation stating that expression-based projections are excluded or limited to direct column mappings.

Questions:

• Is expression-based column lineage for functions such as upper() and concat_ws() supported in CDP 7.1.9?
• Is this a known Spark Atlas Connector defect?
• Is there a service pack, or later release that adds/fixes this support?
• Is any additional Spark Atlas Connector configuration required?

Environment:

• CDP Private Cloud Base 7.1.9
• Source and target: Hive Metastore tables backed by Parquet