Member since
11-19-2024
2
Posts
0
Kudos Received
0
Solutions
11-19-2024
01:09 PM
@ezerihun I updated this property in the configs in Cloudera Manager for Hive and the Isilon gateway and also updated the query to point to the bucket instead of the endpoint/bucketname as you suggested. I was able to get a create table statement to run successfully (table_type=ICEBERG and stored in the S3 bucket). The problem now is that I cannot insert anything into the table because a specific tez related tar.gz file doesn't exist where it thinks it should. Because we have multiple environments and namespaces on the same Isilon, I've engaged our internal storage team for assistance. However, as far as creating an Iceberg table in an S3 bucket, I think we're good to go. Thank you so much for your suggestion! *I'm hesitant to close this post until we can successfully read from and write to these Iceberg tables in/on S3, but we're further along now then we ever have been.
... View more
11-19-2024
08:19 AM
Background: I'm a new admin to our existing CDP environment. Current prod set up uses Nifi, Ranger, Impala, HMS, Hive on Tez, Isilon/HDFS on Dell Powerscale. New directive is to create databases and Iceberg tables via S3 using buckets on the same Powerscale in a different access zone and redirect Nifi processes writing to hdfs to write to S3 buckets. Steps: I update the core-site.xml and hive-site.xml files with the following properties - "fs.s3a.secret.key", "fs.s3a.access.key" and "fs.s3a.endpoint". Success: I validated connection to the S3 bucket with aws cli and can run Nifi PutS3Object and the file is in the expected S3 bucket directory. Challenge: I'm getting a error in Impala when I attempt to create an Iceberg table in the same bucket to use with the Nifi PutIceberg processor. "ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: CAUSED BY: MetaException: Got exception: org.apache.hadoop.fs.s3a.AWSClientIOException getFileStatus on s3a://ce-dev-bucket-3/s3_test_iceberg: com.amazonaws.SdkClientException: Unable to execute HTTP request: ce-dev-bucket-3.<s3endpoint>.domain.org: Name or service not known: Unable to execute HTTP request: ce-dev-bucket-3.<s3endpoint>.domain.org: Name or service not known" Can anyone suggest any permissions or properties that may be missing?
... View more
Labels:
- Labels:
-
Apache Iceberg
-
Apache Impala
-
Cloudera Hue