question Re: Using Glue Catalog as Metastore for PutIceberg Processor Nifi in Support Questions

Using Glue Catalog as Metastore for PutIceberg Processor Nifi

MaxEcueda — Thu, 25 Jul 2024 12:12:56 GMT

Hi,
Does anyone know if we can use glue catalog as metastore for PutIceberg Processor in NiFI
My use case is we need to stream records in AWS s3 in iceberg format from NiFi and we use glue catalog as metastore.

Would Really appreciate the help

Re: Using Glue Catalog as Metastore for PutIceberg Processor Nifi

DianaTorres — Thu, 25 Jul 2024 15:27:30 GMT

@MaxEcueda Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @joseomjr @SAMSAL who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Using Glue Catalog as Metastore for PutIceberg Processor Nifi

Tanya19 — Fri, 18 Oct 2024 10:53:30 GMT

@DianaTorres Can you point me to the person who can help with solution of above problem statement? @joseomjr @SAMSAL Please help with the solution for same if available. I have usecase where Nifi is being used for putting Iceberg data on S3 using AWS Glue Catalog.

Re: Using Glue Catalog as Metastore for PutIceberg Processor Nifi

DianaTorres — Fri, 18 Oct 2024 23:44:27 GMT

@MattWho @steven-matison Hi! Do you have some insights here? Thanks in advance

Re: Using Glue Catalog as Metastore for PutIceberg Processor Nifi

MattWho — Mon, 21 Oct 2024 12:41:28 GMT

@Tanya19 @MaxEcueda

The PutIceberg and PutIcebergCDC processors only offer Hadoop or Hive Catalog Service provider options currently.

The only mention of Glue Catalog i could find in an Apache NiFi JIra was the following still open Jira: https://issues.apache.org/jira/browse/NIFI-11449

It might be a good idea to create an Apache NiFi jira with as much detail as you can provide around this improvement request for an additional AWS Glue Catalog provider.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt