Vertica to Data Lake Connection Failed using WEBHDFS command

ARK86 — Thu, 06 Feb 2025 09:28:31 GMT

Hi,

We have 3 servers VerticaServer, DataLakeNameNode, and DataStageServer. In a DataStage job, we are trying to COPY data into a Vertica table by getting it from external table in DataLake using WEBHDFS. The DataStage job on DR is a copy of production job which is running fine on a daily basis.

I have run following checks to ensure that there are no connectivity issues
1. Ping is working fine between all three servers

2. Telnet from VerticaServer to DataLakeNameNode on port 50070 is working

3. On VerticaServer 'curl -i "http://<IP>:50070/wehdfs/v1/?op=LISTSTATUS"' is returning '200 OK'

4. Webhdfs is up on DataLakeNameNode

In DataStage job, we are getting following error when we run command 'COPY STG.PARTY FROM 'webhdfs://<IP>:50070/warehouse/tablespace/external/hive/STG/PARTY/*' DELIMETER ',' ENCLOSED BY "" NO ESCAPE ABORT ON ERROR'':
"The reported error is: [SQLSTATE 58030] java.sql.SQLException: [Vertica] [VJDBC] (6776) ERROR: Failed to glob [webhdfs://<IP>:50070/warehouse/tablespace/external/hive/STG/PARTY/*] because of error: [http://<IP>:50070/webhdfs/v1/warehouse/tablespace/external/hive/STG/PARTY/?op=LISTSTATUS&user.name=vadmin: Curl Error: Timeout was reached Error Details: Failed to connect to <IP> port 50070: Connection timed out."

I have compared hdfs settings of both Prod and DR on Ambari but there seems to be no difference between them.

Could it be due to environment mismatch? As the same job is working fine on a daily basis on Production.

Would really appreciate the support. Thanks.

Re: Vertica to Data Lake Connection Failed using WEBHDFS command

BlakeSullivan — Thu, 06 Feb 2025 13:35:51 GMT

I also want to know it.

Re: Vertica to Data Lake Connection Failed using WEBHDFS command

VidyaSargur — Fri, 07 Feb 2025 07:07:18 GMT

@ARK86, Welcome to our community! To help you get the best possible answer, I have tagged in our HDFS experts @Asok @rki_ who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Vertica to Data Lake Connection Failed using WEBHDFS command

ARK86 — Sat, 08 Feb 2025 08:56:53 GMT

Thank you Vidya. Looking forward to their response on this.

question Re: Vertica to Data Lake Connection Failed using WEBHDFS command in Support Questions

Vertica to Data Lake Connection Failed using WEBHDFS command

Re: Vertica to Data Lake Connection Failed using WEBHDFS command

Re: Vertica to Data Lake Connection Failed using WEBHDFS command

Re: Vertica to Data Lake Connection Failed using WEBHDFS command