Our goal is to provide a seamless, high-performance environment that bridges the gap between raw data and actionable insights. By integrating AI-driven SQL conversion and strengthening our connector suite, we are empowering teams to work faster, more securely, and with greater visibility across the entire data lifecycle.
To better align with Cloudera’s suite of solutions, the Octopai convention and reference will now be labelled Cloudera Data Lineage.
All materials referencing Cloudera Data Lineage relate to what was formerly Octopai.
New Features
This release focuses on three core pillars: advanced cloud-data integration, enterprise-grade security, and robust pipeline connectivity.
Enterprise-Grade Authentication
The following releases prioritize support for Cloudera authentication protocols.
- Spark Lineage + Kerberos Authentication Support – Secure your Spark lineage leveraging secure integration with industry-standard Kerberos authentication.
- Secure Hive & Impala Kerberos Integration – Supported authentication protocols for secure integration.
Databricks Ecosystem & AI Intelligence
The following releases positions our lineage as a more robust solution over the native Databricks offering!
- Unity Catalog Integration – Lowering the barrier for Databricks complex data catalog analysis. Enhanced Delta Live Tables and analysis with deepened support for Databricks Delta Tables to ensure high-performance ACID transactions and metadata handling.
- Supported Lineage for Databricks Notebooks AI that reside outside Unity Catalog.
- Supporting Lineage of Python jobs operated by Databricks compute engine.
- Databricks/Hive Metastore (HMS) Connector (Multiple Metastores) – Seamlessly bridge your Hive metadata with Databricks environments.
- Streaming, Connectivity & Lineage
- Snowflake Stage Enhancement – Improved support for pipeline lineage, providing end-to-end visibility of data movement into Snowflake.
- Apache Kafka & Kafka Connect – New, robust connectors to support Kafka lineage.
- DataStage Sequencers – Enhanced analysis support for IBM InfoSphere DataStage Sequencer jobs to improve ETL orchestration visibility. This is an industry-first integration!
- Apache NiFi Connector: Full support for NiFi integration with Apache Knox authentication support.
Use Cases
These enhancements prioritize productivity, security, and transparency. By automating manual tasks and hardening our security posture, we enable data teams to focus on innovation rather than infrastructure.
For Data Engineers & Analysts
- Solve the "metadata management" chaos by leveraging Cloudera Data Lineage as a critical risk-mitigation tool that automatically maps the entire lineage—across Cloudera and external systems like Databricks, Oracle, or Snowflake—instantly revealing what data is obsolete versus what is business-critical.
- End-to-End Lineage: With enhanced Databricks and Snowflake Stage support, teams can audit and trace data flow with 100% confidence, simplifying compliance and troubleshooting.
- Seamless Ingestion: Use the new Kafka and NiFi connectors to build real-time pipelines without custom-coding complex integrations.
For Enterprise Security Teams
- Cloudera connector protocols: 5 Cloudera connectors enable unified governance that can be enforced using a single governance model that follows the data, regardless of where it is stored or processed.
- Unified Governance: The Databricks connector allows for a single source of truth for metadata, reducing the risk of "data silos or partial governance and enabling full Databricks lineage (going beyond Cloudera), giving insight into migration success.
To learn more about these features and releases, please visit our [Internal Documentation Portal], which includes technical deep-dives, configuration guides, and demo videos.
To receive more details on how you can benefit from these new integrations and enhancements, please reach out to your Cloudera representative.
