Bridging the Gap: How to Securely Connect Cloudera Data Lineage to Amazon Redshift Serverless

[#1]

Data governance is no longer just a "nice-to-have" feature in any data platform. It is the backbone of modern analytics. For organizations using Cloudera Data Lineage (formerly Octopai), the ability to automatically map data flows across a complex landscape is a superpower. But what happens when your governance tool and your data warehouse live in completely different, isolated networks?

Recently, we tackled a common but tricky architectural challenge: connecting a Windows-based Octopai client running in one AWS VPC to an Amazon Redshift Serverless cluster sitting in a separate, secured VPC. While AWS VPC Peering is the standard solution for this, we discovered that Redshift Serverless introduces some unique networking nuances—specifically around routing tables and DNS—that can trip up even experienced cloud engineers.

This blog post outlines our architecture, the specific "gotchas" we encountered, and the step-by-step path to successfully extracting metadata from the standard tpch dataset using a secure, private connection.

The Objective and Environment

Our goal was simple! Enable the Octopai Windows client to extract metadata from Redshift Serverless without exposing any traffic to the public internet. Cloudera Data Lineage operates on a "Push Model," where the client sits inside the customer firewall, extracts only metadata including databases, tables, columns, stored procedures, functions, views etc., and pushes that lightweight JSON/XML payload to the Cloudera Data Lineage SaaS portal for visualization.

The environment we focused on consisted of two distinct Virtual Private Clouds (VPCs) within the same AWS account:

• The Client VPC: Hosted our Windows EC2 instance running the Octopai extraction client.
• The Data (Customer) VPC: Hosted the Amazon Redshift Serverless workgroup and the tpch sample database.

Both the Cloudera Data Lineage client and Amazon Redshift Serverless with sample TPC-H dataset were set up and ready to go.

Connecting these two isolated worlds required a VPC Peering connection, but as we found out, simply "plugging in the cable" wasn't enough.

Step 1: Laying the Network Bridge

The first phase involved establishing the VPC Peering connection. We initiated a peering request from the Client VPC and accepted it in the Data VPC. This created the potential for network traffic to flow, but potential is not connectivity.

For the connection to work, we had to update the Route Tables. In the Client VPC, we added a route pointing to the Redshift VPC’s CIDR block, targeting the new peering connection. This told our Windows machine exactly how to reach Redshift Serverless. We effectively built a bridge, but we still needed to ensure the destination address was readable.

Step 2: The DNS "Gotcha"

Redshift Serverless provides an endpoint URL (e.g., workgroup.account.region.redshift-serverless.amazonaws.com). By default, this URL resolves to a Public IP address. If our Windows client tried to connect to that Public IP, the traffic would attempt to leave via the Internet Gateway, bypassing our secure peering connection entirely.

The fix was found in the VPC Peering settings. We had to explicitly enable DNS Resolution for both the Requester and Accepter VPCs. Once enabled, AWS magic took over: the exact same Redshift URL began resolving to a Private IP address (e.g., 10.0.130.215) corresponding to the VPC peering link.

Step 3: The Route Table Trap

Even with DNS resolving correctly, our connection attempts timed out. Test-NetConnection showed we could find the IP, but we couldn't complete a TCP handshake.

Test-NetConnection -ComputerName endpoint-wg.x42.us-east-2.redshift-serverless.amazonaws.com -Port 5439

This led to our most critical discovery. We had added the return route (back to the Windows client) to the Main Route Table of the Redshift VPC, assuming it would propagate to all subnets. However, Redshift Serverless subnets often come with explicit associations to custom route tables.

In AWS, an explicit subnet association overrides the Main Route Table. Our Redshift subnet was looking at a custom table that had no idea our Windows client existed. The request was arriving at Redshift, but the reply was being dropped because the return path was missing.

The Solution: We identified the specific subnet used by the Redshift endpoint, found its custom route table (which was not the Main table), and added the route to the Client VPC there. Immediately, the traffic started flowing ( tested using Test-NetConnection)

Step 4: Metadata Extraction

With the network pipe fully established, we configured the Windows client. We installed the Amazon Redshift ODBC driver and set up a System DSN pointing to the Redshift endpoint.

The following parameters were used in the Windows ODBC Data Source Administrator to create the System DSN for Redshift Serverless:

ParameterValue	Description
Data Source Name	RedshiftServerless_Octopai
Driver	Amazon Redshift (x64)
Server	endpoint-wg.x42.us-east-2.redshift-serverless.amazonaws.com (Your Redshift Endpoint URL)
Port	5439
Database	dev (Or your target database name)
User	masteruser
Password	********
SSL Mode	prefer (Or require for stricter security)

We then launched the Octopai client, added a new metadata source using the ODBC connection we created, and did an extraction run. The client successfully queried system tables like SVV_TABLE_INFO and PG_CLASS, extracting the metadata without reading sensitive row-level data.

Adding the Redshift Metadata Source in the Octopai Client

Once the ODBC connection is verified and successful (as tested in Step 3), the final step is to configure the metadata extraction within the Cloudera Data Lineage client application (formerly Octopai).

1. Launch the Octopai Client: Open the Octopai Client application on the Windows EC2 instance.
2. Add a New Source: Navigate to the source management section and select "Add New Source."
3. Choose Redshift: Select Amazon Redshift from the list of available database types.
4. Configure Connection Details:
• Connection Type: Select ODBC.
• DSN: Enter the exact Data Source Name created in Step 4, for example: RedshiftServerless_Octopai.
• Credentials: Re-enter the User (masteruser) and Password defined in the DSN setup, though for DSN connections, the credentials are often pulled directly from the system configuration.
5. Test Connection: Use the built-in test function to ensure the Octopai client can successfully connect to Redshift Serverless using the private VPC peering link. This test validates the entire secure pipeline.
6. Schedule Extraction: Once the connection is successful, save the source configuration and schedule the initial metadata extraction run. The client will now use the secure, privately routed connection to pull metadata structures (tables, columns, views, etc.) from the databases and push the lineage data to the Cloudera Data Lineage SaaS portal.

Summary

Connecting Cloudera Data Lineage to Redshift Serverless is a straightforward process once you understand the underlying network plumbing. The key takeaways for anyone attempting this are:

1. DNS Matters: Always enable DNS resolution in your peering connection to ensure traffic stays on the private backbone.
2. Check Subnet Associations: Never assume a subnet is using the Main Route Table. distinct, explicit route tables are common in secure setups and must be updated individually.
3. Security Groups: Ensure your Redshift Security Group allows inbound traffic specifically from the Client VPC CIDR, not just the peering ID.

With this secure architecture, we now have automated, end-to-end lineage flowing into Cloudera Data Lineage, giving us full visibility into our data lifecycle while keeping our network footprint completely private.