Member since
07-19-2024
29
Posts
15
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1277 | 08-26-2024 05:24 AM |
12-15-2024
09:16 PM
Hi Team, Good Day We have a requirement to start the CDSW AI ML job from the API call. We are currently using Cloudera Machine Learning legacy REST API (Version V1) as per this link: https://docs.cloudera.com/machine-learning/1.5.4/jobs-pipelines/topics/ml-rest-apis.html curl -v -XPOST http://cdsw.example.com/api/v1/<path_to_job> --user "<LEGACY_API_KEY>:" Ex: curl -XPOST https://***/api/v1/projects/bd***/s***/jobs/2/start --user "***" --header "Content-type: application/json" --data "{}" We are working to upgrade the CDSW to CML and as part of this job rewrite exercise we also thought to use Cloudera Machine Learning REST API v2 (Version V2) for the curl command to Start the Job In the below URL, We don't see a method to start the job https://docs.cloudera.com/machine-learning/1.5.4/rest-api-reference/index.html#examples-CMLService-restartApplication-0-python Please share your expertise on the below open queries : 1) When is V1 getting depreciated? Is it recommended to use V1 API now? 2) Which API method is to be used in the Version 2 API call to start the job?
... View more
10-08-2024
02:49 AM
Hi Team, We are configuring the job in CML to run every 4 hours as shown below. Currently, the job is getting triggered automatically at 12 PM, 4 PM, 8 PM, and 12 AM. But the jobs are not getting triggered at 4 AM and 8 AM. Also, it's not getting triggered over the weekend. Please help us with your expertise if there is any additional setting or any admin configuration that we need to validate here. Thanks in advance!
... View more
Labels:
- Labels:
-
Cloudera Machine Learning (CML)
10-07-2024
03:46 AM
2 Kudos
Hi @smruti / @caio_contente , Thanks a lot for your response! I am new to NiFi and still exploring various options here. I am unable to find the correct way to pass the query to the PutHiveQL processor. Please help me by providing some steps to configure the PutHiveQL processor to perform the DDL operation. Thanks!
... View more
09-20-2024
07:01 AM
Hi Experts, Please let us know if we can use any NiFi processors for performing MSCK repair on the hive table to reflect the data that are updated by NiFi to HDFS. Thanks! Command : MSCK REPAIR TABLE tblName;
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-16-2024
06:55 AM
Hi Experts, Please let us know if we can use any NiFi processors for truncating hive table data. Thanks!
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-12-2024
06:22 AM
1 Kudo
Thanks a lot, @SAMSAL! The proposed solution worked for us.
... View more
09-02-2024
10:41 PM
1 Kudo
Thanks a lot, @SAMSAL for the technical suggestion! Please find below our comments on both the proposed solutions, Solution 1: Using Sql Calcite function TIMESTAMPADD This method does not perform the conversion. Please refer to the below screenshot for more details and let us know if we are making any mistakes. It returned the same output Solution 2: Using Expression Language This solution works fine when we pass the data directly into the query record. However, we wanted to apply this conversion for the batch of data in the CSV input file. Please refer below for the complete flow Complete NiF Flow : Input File : Id,username,load_date 1,test@test.com,1725000090 Error Message : QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException: Invalid Expression: select '${literal(load_date):multiply(1000):format('yyyy-MM-dd HH:mm:ss')}' mytimestamp from flowfile due to Unexpected token 'load_date' at line 1, column 10. Query: ${literal(load_date):multiply(1000):format(yyyy-MM-dd HH:mm:ss)}: {} Please help us with your expertise if there is any way to pass the column from the flowfile to the Expression language. Thanks in advance!
... View more
08-30-2024
07:15 AM
1 Kudo
We are working on converting the Unix time into date-time using the QueryRecord processor. We understand that QueryRecord is built with the calcite SQL syntax and tried to use the DATE_FROM_UNIX_DATE function. We are receiving the below error. Please help us resolve this issue with your expertise. We can achieve this conversion using the update attribute but we wanted this as part of the Query record as we wanted to apply this conversion on the completed dataset. Query : SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile Error : QueryRecord[id=2cc603e4-0191-1000-f3c7-ef830564cf72] Unable to query FlowFile[filename=convertfunctionsample.csv] due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile] - Caused by: java.sql.SQLException: Error while preparing statement [SELECT DATE_FROM_UNIX_DATE(1724851471) FROM FLOWFile] - Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 38: No match found for function signature DATE_FROM_UNIX_DATE(<NUMERIC>) - Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature DATE_FROM_UNIX_DATE(<NUMERIC>): {}
... View more
Labels:
- Labels:
-
Apache NiFi
08-27-2024
09:37 AM
@MattWho @SAMSAL @vaishaakb - Good Day! I see most of the NiFi queries are resolved by your expertise. So please help here with your technical suggestion on the NiFi requirement. Thanks!
... View more
08-27-2024
09:30 AM
Thanks a lot, @SAMSAL for your detailed explanations. Appreciate your expertise on the NiFi. I am a beginner to the NiFi world and not much documentation on this topic is available on the internet. Thanks once again for your support!
... View more