Member since
07-19-2024
28
Posts
15
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1130 | 08-26-2024 05:24 AM |
08-26-2024
12:15 AM
1 Kudo
We have a streaming requirement where the data is received continuously and we store them in HDFS by merging 10,000 records to avoid multiple IO operations. We generate the batchID using the below calculation, ${now():format('yyyyMMdd')}${now():format("HH"):toNumber():multiply(60):plus(${now():format('mm'):toNumber()}):divide(15)} So this batch will change every 15 minutes and in Nifi sometimes there are 4 flows or 5 flows based on the number of records received during 15-minute time intervals. For example, if we receive 50,000 records in 15 minutes then this flow will be executed 5 times but if there are 40,000 records in 15 minutes then the entire NiFi flow will be executed 4 times. Once the batch execution is completed fully (after 4 flows or 5 flows as per the incoming data volume as explained in the above example) we want this batchID to be stored in the next table for further processing. So, we are working on some logic/condition in NiFi to invoke the "RouteOnAttribute" to check the batchId of the current run and batchId of the previous run. If the current batchID is the same as the previous batchID then no action is taken but if both the values are different then the previous batchID needs to be passed to the next NiFi processor for further processing. So please help us here with how we can store the value of batchId in the previous run and use it in NiFi flow. I have tried using the UpdateAttribute processor with "Store state locally" but it is not helping here to store the batchId it always gives a blank. Please help here with the right approach to achieve this requirement.
... View more
Labels:
- Labels:
-
Apache NiFi
08-06-2024
06:32 AM
Good Day! We are trying to insert the records into HBase tables using the NiFi processor - PutHBaseRecord. But we are facing the below error. Please help us with your expertise. Thanks! Processor Used: PutHBaseRecord Error Message 11:09:31 GST ERROR PutHBaseRecord[id=ee32a0cc-0190-1567-1233-12334fd02c9e] Failed to put records to HBase.: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 5 actions: FailedServerException: 5 times, servers with issues
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache NiFi
08-01-2024
02:44 AM
1 Kudo
Thanks alot @MattWho for your expertise!
... View more
08-01-2024
02:42 AM
1 Kudo
Thanks alot @SAMSAL ! This solution works and thanks once again for the detailed explanation
... View more
08-01-2024
02:23 AM
1 Kudo
Good Day! We are required to convert the CSV data to be stored in parquet format. While converting using the below NiFi flow, we see .member0 & .member1 getting added to the output. Input Data : no,firstname,lastname 1,1234,user 1,test,user 1,test,user 1,test,user 1,test,user 2,babu,user2 c,krish,mohan 4,krishna1234,mohna NiFi Flow : Output Data : no: .member0 = 1 firstname: .member0 = 1234 lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 2 firstname: .member1 = babu lastname = user2 no: .member1 = c firstname: .member1 = krish lastname = mohan no: .member0 = 4 firstname: .member1 = krishna1234 lastname = mohna We believe internally somehow the integer values are assigned to member0 and string values are assigned to member1 if the same column has both string and integer values. Also, we believe this is decided based on the values of 1st row. Please help us with your expertise if we can avoid this member0 and member1 by explicitly mentioning to NiFi to treat all data as a string. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
07-29-2024
06:21 AM
We are facing an error while enabling the HBase_2_ClientService in the NiFi 1.18 version. Hadoop Configuration Files: /etc/hbase/conf/hbase-site.xml,/etc/hadoop/conf/core-site.xml Kerberos Credentials Service: KeytabCredentialsService (this is working fine for other processors) Below are nar file part of the lib folder of NiFi related to Hbase, nifi-hbase-nar-1.18.0.2.1.5.3000-3.nar nifi-hbase_2-client-service-nar-1.18.0.2.1.5.3000-3.nar nifi-hbase_1_1_2-client-service-nar-1.18.0.2.1.5.3000-3.nar nifi-couchbase-services-api-nar-1.18.0.2.1.5.3000-3.nar nifi-couchbase-nar-1.18.0.2.1.5.3000-3.nar Unable to enable this service and getting the below error, Please help us with your expertise to resolve this error. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
07-26-2024
01:08 AM
1 Kudo
Hi @SAMSAL , Thanks a lot for your valuable input! Yes, We are using the GetFile to read the content and PutFile to write the content to the text file. Please let us know if there is any option to update the text file with the content from the flow file instead of replacing or creating a new text file. MergeContent will be used to merge the content of the different flow file items and write them in the file. However, we have a requirement to update or append the existing text file in the folder. Thanks!
... View more
07-26-2024
12:29 AM
1 Kudo
Hi @monica345 , Thanks a lot for your valuable input! We have restarted the NiFi multiple times and find below the list of jar files placed in the NiFi lib folder. Please let us know if there is any additional jar file that needs to be used in NiFi for connecting to Hive. Thanks! slf4j-api-2.0.6.jar nifi-stateless-bootstrap-1.18.0.2.1.5.3000-3.jar nifi-stateless-api-1.18.0.2.1.5.3000-3.jar nifi-server-api-1.18.0.2.1.5.3000-3.jar nifi-runtime-1.18.0.2.1.5.3000-3.jar nifi-property-utils-1.18.0.2.1.5.3000-3.jar nifi-properties-1.18.0.2.1.5.3000-3.jar nifi-nar-utils-1.18.0.2.1.5.3000-3.jar nifi-framework-api-1.18.0.2.1.5.3000-3.jar nifi-api-1.18.0.2.1.5.3000-3.jar logback-core-1.3.5.jar logback-classic-1.3.5.jar log4j-over-slf4j-2.0.6.jar jul-to-slf4j-2.0.6.jar jetty-schemas-5.2.jar jcl-over-slf4j-2.0.6.jar javax.servlet-api-3.1.0.jar aspectjweaver-1.9.6.jar oracle-connector-java.jar mysql-connector-java.jar ojdbc8.jar ojdbc10.jar hive-jdbc-3.1.3000.7.1.7.2000-305.jar libthrift-0.9.3.jar
... View more
07-25-2024
07:14 AM
We are facing the HttpClient error when we try to fetch data from the Hive using the SelectHiveQL NiFi Processor. Please find the error details below and help us resolve this with your expertise. Controller Service - HiveConnectionPool Database Connection URL: jdbc:hive2://localhost:10000/default;principal=hive@test@xyz.com Hive Configuration Resources: /etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml NiFi Processor - SelectHiveQL Error Message : SelectHiveQL[id=23er-1234-9999-4444-17] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$2132/2007351143@6ab4a383 failed to process due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient - Caused by: java.lang.ClassNotFoundException: org.apache.http.client.HttpClient; rolling back session: {} SelectHiveQL[id=sge33-1234-9999-21-33] Processing halted: yielding [1 sec]: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient - Caused by: java.lang.ClassNotFoundException: org.apache.http.client.HttpClient
... View more
Labels:
- Labels:
-
Apache NiFi
07-24-2024
07:02 AM
1 Kudo
Thanks a lot @MattWho for your valuable commands, Please help with configuring the GrokReader for the below input data as I am unable to find the right documentation for configuring the GrokReader with unstructed data. <13>Jul 18 11:39:11 test234104.test.gmail.ae AgentDevice=WindowsLog AgentLogFile=Security PluginVersion=100.3.1.22 Source=Microsoft-Windows-Security-Auditing Computer=mycomputer1 OriginatingComputer=102.123.33.1 User= Domain= EventID=4688 EventIDCode=4688 EventType=8 EventCategory=13312
... View more
- « Previous
-
- 1
- 2
- Next »