Skip to main content
MaxEcueda
MaxEcueda
Bronze ⭐️⭐️⭐️
July 25, 2024
Question

Using Glue Catalog as Metastore for PutIceberg Processor Nifi

Hi,
Does anyone know if we can use glue catalog as metastore for PutIceberg Processor in NiFI
My use case is we need to stream records in AWS s3 in iceberg format from NiFi and we use glue catalog as metastore.

Would Really appreciate the help

    4 replies

    DianaTorres
    Community Manager
    DianaTorres
    Community Manager
    July 25, 2024

    @MaxEcueda Welcome to the Cloudera Community!

    To help you get the best possible solution, I have tagged our NiFi experts @joseomjr @SAMSAL  who may be able to assist you further.

    Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

    Translate
    Regards, Diana Torres, Senior Community Moderator
    Tanya19
    Tanya19
    Silver ⭐️
    October 18, 2024

    @DianaTorres Can you point me to the person who can help with solution of above problem statement?  @joseomjr @SAMSAL  Please help with the solution for same if available. I have usecase where Nifi is being used for putting Iceberg data on S3 using AWS Glue Catalog.

    Translate
      MattWho
      Titanium ⭐️⭐️⭐️
      MattWho
      Titanium ⭐️⭐️⭐️
      October 21, 2024

      @Tanya19 @MaxEcueda 

      The PutIceberg and PutIcebergCDC processors only offer Hadoop or Hive Catalog  Service provider options currently.

      The only mention of Glue Catalog i could find in an Apache NiFi JIra was the following still open Jira:  https://issues.apache.org/jira/browse/NIFI-11449

      It might be a good idea to create an Apache NiFi jira  with as much detail as you can provide around this improvement request for an additional AWS Glue Catalog provider.

      Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

      Thank you,
      Matt

      Translate
      DianaTorres
      Community Manager
      DianaTorres
      Community Manager
      October 18, 2024

      @MattWho @steven-matison Hi! Do you have some insights here? Thanks in advance

      Translate
      Regards, Diana Torres, Senior Community Moderator
        Terms & ConditionsAccessibility statement