Member since
07-19-2024
28
Posts
15
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1145 | 08-26-2024 05:24 AM |
10-08-2024
02:49 AM
Hi Team, We are configuring the job in CML to run every 4 hours as shown below. Currently, the job is getting triggered automatically at 12 PM, 4 PM, 8 PM, and 12 AM. But the jobs are not getting triggered at 4 AM and 8 AM. Also, it's not getting triggered over the weekend. Please help us with your expertise if there is any additional setting or any admin configuration that we need to validate here. Thanks in advance!
... View more
Labels:
- Labels:
-
Cloudera Machine Learning (CML)
10-07-2024
03:46 AM
2 Kudos
Hi @smruti / @caio_contente , Thanks a lot for your response! I am new to NiFi and still exploring various options here. I am unable to find the correct way to pass the query to the PutHiveQL processor. Please help me by providing some steps to configure the PutHiveQL processor to perform the DDL operation. Thanks!
... View more
09-20-2024
07:01 AM
Hi Experts, Please let us know if we can use any NiFi processors for performing MSCK repair on the hive table to reflect the data that are updated by NiFi to HDFS. Thanks! Command : MSCK REPAIR TABLE tblName;
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-16-2024
06:55 AM
Hi Experts, Please let us know if we can use any NiFi processors for truncating hive table data. Thanks!
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-12-2024
06:22 AM
1 Kudo
Thanks a lot, @SAMSAL! The proposed solution worked for us.
... View more
09-02-2024
10:41 PM
1 Kudo
Thanks a lot, @SAMSAL for the technical suggestion! Please find below our comments on both the proposed solutions, Solution 1: Using Sql Calcite function TIMESTAMPADD This method does not perform the conversion. Please refer to the below screenshot for more details and let us know if we are making any mistakes. It returned the same output Solution 2: Using Expression Language This solution works fine when we pass the data directly into the query record. However, we wanted to apply this conversion for the batch of data in the CSV input file. Please refer below for the complete flow Complete NiF Flow : Input File : Id,username,load_date 1,test@test.com,1725000090 Error Message : QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException: Invalid Expression: select '${literal(load_date):multiply(1000):format('yyyy-MM-dd HH:mm:ss')}' mytimestamp from flowfile due to Unexpected token 'load_date' at line 1, column 10. Query: ${literal(load_date):multiply(1000):format(yyyy-MM-dd HH:mm:ss)}: {} Please help us with your expertise if there is any way to pass the column from the flowfile to the Expression language. Thanks in advance!
... View more
08-30-2024
07:15 AM
1 Kudo
We are working on converting the Unix time into date-time using the QueryRecord processor. We understand that QueryRecord is built with the calcite SQL syntax and tried to use the DATE_FROM_UNIX_DATE function. We are receiving the below error. Please help us resolve this issue with your expertise. We can achieve this conversion using the update attribute but we wanted this as part of the Query record as we wanted to apply this conversion on the completed dataset. Query : SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile Error : QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile] - Caused by: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile] - Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 38: No match found for function signature DATE_FROM_UNIX_DATE(<NUMERIC>) - Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature DATE_FROM_UNIX_DATE(<NUMERIC>): {}
... View more
Labels:
- Labels:
-
Apache NiFi
08-27-2024
09:37 AM
@MattWho @SAMSAL @vaishaakb - Good Day! I see most of the NiFi queries are resolved by your expertise. So please help here with your technical suggestion on the NiFi requirement. Thanks!
... View more
08-27-2024
09:30 AM
Thanks a lot, @SAMSAL for your detailed explanations. Appreciate your expertise on the NiFi. I am a beginner to the NiFi world and not much documentation on this topic is available on the internet. Thanks once again for your support!
... View more
08-26-2024
05:24 AM
1 Kudo
Thanks, @SAMSAL for your reply! Below is the POC created for implementing this requirement on my Windows laptop. UpdateAttribute_previousBatchID - The first Update attribute is used to record the batchID from the previous run previousBatchID - ${getStateValue("batchID")} UpdateAttribute_batchID - The second update attribute is used to generate the new batchID . batchID - ${now():format('yyyyMMdd')}${now():format("HH"):toNumber():multiply(60):plus(${now():format('mm'):toNumber()}):divide(15)} RouteOnAttribute - Compare the batchId with PreviousBatchID. UpdateAttribute_processedBatchID - The last update attribute after the RouteOnAttribute is used to get the previous BatchID BatchID_Process - ${previousBatchID} As u can see below, the variables have entry strings in the last step for both the variables highlighted in yellow. Please help to guide me with the right way to use the update attribute to achieve my requirement. Thanks!
... View more