Member since
02-01-2022
288
Posts
103
Kudos Received
60
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1265 | 05-15-2025 05:45 AM | |
| 5297 | 06-12-2024 06:43 AM | |
| 8296 | 04-12-2024 06:05 AM | |
| 6138 | 12-07-2023 04:50 AM | |
| 3411 | 12-05-2023 06:22 AM |
06-20-2023
06:47 AM
1 Kudo
@wert_1311 Check out this article: https://community.cloudera.com/t5/Community-Articles/Support-Video-Enabling-kubectl-for-CDE/ta-p/314200 Enjoy Breakstone's amazing radio voice! 😉
... View more
06-20-2023
06:42 AM
3 Kudos
@drewski7 This blog is a great place to start: https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/ That said, some recommendations: Recommend 3 nodes. Use 32 or 64gb ram. Set min ram 16, max 32, let nifi/operating system leverage other 32gb. Add more cores and tune Active Thread Count accordingly Be careful which processors are Primary Only and which processors are not. Do not over loadbalance queues, load balance at top of flow, let nifi distribute work load naturally after that. Tune Processor Concurrency and Run Schedule. Be sure to understand how each work. With a good setup tuned as above, have a plan to identify when time is appropriate to scale horizontally (add more nodes). Here are some more docs that get specific into sizing: https://docs.cloudera.com/cfm/2.1.1/nifi-sizing/topics/cfm-sizing-recommendations.html https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/nifi-configuration-best-practices/content/configuration-best-practices.html
... View more
06-15-2023
09:36 AM
@Ray82 Yes, you can achieve this with UpdateRecord. You will need to provide record reader/writer with schema of your upstream and downstream. Then in UpdateRecord you explicitly add properties (+) for each record value you want to update versus using a SQL statement like QueryRecord. Here are some useful community posts on this topic: https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267 https://community.cloudera.com/t5/Support-Questions/NiFi-UpdateRecord-processor-is-not-updating-JSON-path/m-p/186256
... View more
06-15-2023
08:54 AM
@MOUROU I recently built a nifi flow in version 1.21 that uses the NiFI API from within nifi, and it is NOT necessary to get access token. From within nifi i am able to just start using the api calls I needed. It would be worth it to see if 1.16 behaves the same way. That flow is here: https://github.com/cldr-steven-matison/NiFi-Templates/blob/main/NiFi_Template_XML_to_Flow_Definition_JSON.json
... View more
06-14-2023
07:10 AM
@Fredb This is a very difficult one to solve. Does anyone know what would cause the execution of the sample_Import_Load.bat to run correctly from the windows command prompt, but fail when executed via the ExecuteStreamCommand processor with these errors? This is most likely caused by permission issues. Nifi requires specific permissions against files and scripts it touches or executes from within processors. As such, the error is saying the processor does not know where any of the resources exist to run that .bat file. I do not have any experience with nifi on windows, other than to avoid it, but the solution is likely the same as other operating systems. Make sure the nifi user has full ownership of the file(s). Additionally, it is sometimes possible to find deeper errors looking at the nifi-app.log file while testing and/or setting the log level of the processor to be more aggressive.
... View more
06-14-2023
07:05 AM
1 Kudo
@rupeshh Docker container are never fun for this and other reasons. I still think you have missing permissions on the file. I cannot see the ownership of the file listing, and i cannot see the path in the error. At any rate, the error definitely suggests the processor does not see the file. One suggestion would be to use the nifi user and cli on docker to ls the directory and files. If that user cannot see the files, that would indicate the same issue the error states (directory path or file does not exist, or not seen due to permissions).
... View more
06-13-2023
12:22 PM
@rupeshh In order for nifi to be able to see the mounted directory or files within it, it needs to be properly owned to the same user that is running nifi. For example: chown nifi:nifi /some/path Then nifi will be able to see the directories and files.
... View more
06-13-2023
06:55 AM
1 Kudo
@wert_1311 Your error indicates that two of your roles are missing or incomplete. 1. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir/* 2. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-ranger-audit-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-ranger-audit-role:s3:PutObject:arn:aws:s3:::cdp-my-bucket/ranger/audit/* Go back to the quickstart and docs, and make sure completd all the setups. Here is a link with more about the credentials https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-aws/topics/mc-aws-req-credential.html There are steps in the page above describing the 2 roles which have conflicts in your error. The error indicates, in the end of each message, which policies are missing.
... View more
06-09-2023
04:05 AM
@Vasu_ Can you provide screen shots of config for the processor(s) and the SSL Context Service you created? I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic. Additionally, if you did attempt to create a keystore/truststore from the self signed cert, be sure to share the commands you used. Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this
... View more
06-08-2023
06:24 AM
1 Kudo
@Vasu_ I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs. Here is an article i just wrote about Modern NiFi and SSL: https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937 It is important to understand how to make a working SSL Context Service before trying to make custom ones. So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself. The following link has a technical example, and both solutions you can use here. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719
... View more