Member since
07-19-2024
29
Posts
15
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1332 | 08-26-2024 05:24 AM |
12-15-2024
09:16 PM
Hi Team, Good Day We have a requirement to start the CDSW AI ML job from the API call. We are currently using Cloudera Machine Learning legacy REST API (Version V1) as per this link: https://docs.cloudera.com/machine-learning/1.5.4/jobs-pipelines/topics/ml-rest-apis.html curl -v -XPOST http://cdsw.example.com/api/v1/<path_to_job> --user "<LEGACY_API_KEY>:" Ex: curl -XPOST https://***/api/v1/projects/bd***/s***/jobs/2/start --user "***" --header "Content-type: application/json" --data "{}" We are working to upgrade the CDSW to CML and as part of this job rewrite exercise we also thought to use Cloudera Machine Learning REST API v2 (Version V2) for the curl command to Start the Job In the below URL, We don't see a method to start the job https://docs.cloudera.com/machine-learning/1.5.4/rest-api-reference/index.html#examples-CMLService-restartApplication-0-python Please share your expertise on the below open queries : 1) When is V1 getting depreciated? Is it recommended to use V1 API now? 2) Which API method is to be used in the Version 2 API call to start the job?
... View more
11-15-2024
11:43 PM
1 Kudo
@NagendraKumar Does there any messages found on CML's Cron logs ?
... View more
10-16-2024
08:00 AM
hi @NagendraKumar , You can use puthiveql as described in this article https://community.cloudera.com/t5/Support-Questions/how-to-use-puthiveql-in-NIFI/td-p/204542
... View more
10-02-2024
07:12 AM
@NagendraKumar were you able to truncate the data with the processor that my colleague Smruti mentioned?
... View more
09-12-2024
06:22 AM
1 Kudo
Thanks a lot, @SAMSAL! The proposed solution worked for us.
... View more
09-05-2024
07:45 AM
1 Kudo
@NagendraKumar The image you shared indicates that your PublishKafka processor is producing a bulletin. What is the nature of the exception being reported in that bulletin? I also see what appears to be only one connection exiting your PublishKafka processors. The publishKafka processor has multiple relationships. Are you auto-terminating the failure" relationship? If so, i never recommend doing that. ----------- Now when it comes to monitoring queue sizes/thresholds, you could use the ControllerStatusReportingTask NiFi reporting task to output these stats for all connections to a dedicated logs (see Additional Details... for how to setup dedicated logs via NiFi's logback.xml). You can then create a NiFi dataflow that tails the dedicated connection log and parses the ingested log entries for connections where thresholds exceed your 80% threshold and route those to a putEmail processor for yoru notification needs. (preferred) Another option here is to use the SiteToSiteStatusReportingTask to report specifically on NiFi connections and feed that data in to a dataflow that parses for threshold in excess of 80% and route those to a putEmail processor. This method has less overhead as it does not write to a NiFi log, require tailing logs, can be setup to report on connections, and reports in a more structured format (see Additional Details... ). ---------- From your image I can also see your PublishKafka processor reporting 3 actively running threads. You mentioned the processor becomes hung? Have you analyzed a series of thread dumps to identify where it is getting hung? NiFi also offers a way to monitor for long running tasks: Runtime Monitoring Properties You could use this in conjunction with the SiteToSiteBulletinReportingTask to construct a dataflow that could send an email alert when task are detected on processor that have been running in excess of the configured threshold. This runtime monitoring does have an impact on your overall NiFi performance due to the overhead needed to run it. So if you find it impacts your throughput performance negatively, you'll need to stop using it. __________ Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-27-2024
09:30 AM
Thanks a lot, @SAMSAL for your detailed explanations. Appreciate your expertise on the NiFi. I am a beginner to the NiFi world and not much documentation on this topic is available on the internet. Thanks once again for your support!
... View more
08-06-2024
08:44 AM
Hi @NagendraKumar You must be having some more stack trace which lists the regionserver where it is failing after retries after the below error- org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 5 actions: FailedServerException: 5 times, servers with issues Try to check give the full stack trace and check regionserver logs for those servers. Also, make sure that there are no inconsistencies in hbase.
... View more
08-01-2024
02:44 AM
1 Kudo
Thanks alot @MattWho for your expertise!
... View more
08-01-2024
02:23 AM
1 Kudo
Good Day! We are required to convert the CSV data to be stored in parquet format. While converting using the below NiFi flow, we see .member0 & .member1 getting added to the output. Input Data : no,firstname,lastname 1,1234,user 1,test,user 1,test,user 1,test,user 1,test,user 2,babu,user2 c,krish,mohan 4,krishna1234,mohna NiFi Flow : Output Data : no: .member0 = 1 firstname: .member0 = 1234 lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 1 firstname: .member1 = test lastname = user no: .member0 = 2 firstname: .member1 = babu lastname = user2 no: .member1 = c firstname: .member1 = krish lastname = mohan no: .member0 = 4 firstname: .member1 = krishna1234 lastname = mohna We believe internally somehow the integer values are assigned to member0 and string values are assigned to member1 if the same column has both string and integer values. Also, we believe this is decided based on the values of 1st row. Please help us with your expertise if we can avoid this member0 and member1 by explicitly mentioning to NiFi to treat all data as a string. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi