Member since
02-07-2019
2690
Posts
235
Kudos Received
30
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1088 | 04-15-2025 10:34 PM | |
3277 | 10-28-2024 12:37 AM | |
1414 | 09-04-2024 07:38 AM | |
3241 | 06-10-2024 10:24 PM | |
1381 | 02-01-2024 10:51 PM |
03-21-2025
06:29 AM
To identify which user is writing the files, use HDFS CLI commands such as ls or getfacl
... View more
03-20-2025
09:39 PM
@MarinaM, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.
... View more
03-20-2025
11:59 AM
@pasq Did you know that Cloudera makes managing and deploying your MiNiFi agents easy with Cloudera Edge Management? Our ManagementHub provides a UI to build, deploy and update the dataflow you deploy to one too many MiNiFi agents. You can create parameters for various properties like passwords to provide those at deployment time through Edge Management Check out this Video. NiFi processors with "sensitive" properties are designed to encrypt those password when written to the flow.json.gz. Even in MiNiFi these component classes will expect that the passwords are encrypted. The defined sensitive.props.key property value in the minifi.properties or nifi.propertes is used in the encryption/decryption of these sensitive properties stored in the flow.json.gz file. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
03-17-2025
08:32 AM
@David26 I am not familiar with this specific processor "Plc4xListenRecordProcessor 0.12.0". I could not find it in Apache NiFi 1.x or 2.x versions. I could find it in Maven Central either. Where did you download this nar that you added to your NiFi? What version of Apache NiFi are you using? You may need to reach out to the originator of this custom nar you have added to you NiFi for this processor specific query. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
03-12-2025
10:57 PM
@Zubair123, This article is available exclusively for our customers. If you're a customer, please contact our customer support team for more details. If you’re not, our sales team would happily assist you with any information you need.
... View more
03-10-2025
09:49 PM
2 Kudos
Sorry for my late response. I did try with the JDK version as mentioned in the driver documentation. It didn't work. However, I am now using keytab method for connecting and I am fine with it. @asish thanks a ton for all the support.
... View more
03-06-2025
07:42 AM
Assuming it's a MapReduce job, since you're looking for information related to MapReduce I/O counters. Script to calculate the counter info. [hive@node4 ~]$ cat get_io_counters.sh
#!/bin/bash
# Ensure a job ID is provided
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <job_id>"
exit 1
fi
JOB_ID=$1
# Extract I/O counters from the MapReduce job status
mapred job -status "$JOB_ID" | egrep -A 1 'File Input Format Counters|File Output Format Counters' | awk -F'=' '
/File Input Format Counters/ {getline; bytes_read=$2}
/File Output Format Counters/ {getline; bytes_written=$2}
END {
total_io_mb = (bytes_read + bytes_written) / (1024 * 1024)
printf "BYTES_READ=%d\nBYTES_WRITTEN=%d\nTOTAL_IO_MB=%.2f\n", bytes_read, bytes_written, total_io_mb
}'
[hive@node4 ~]$ Sample Output [hive@node4 ~]$ ./get_io_counters.sh job_1741272271547_0007
25/03/06 15:38:34 INFO client.RMProxy: Connecting to ResourceManager at node3.playground-ggangadharan.coelab.cloudera.com/10.129.117.75:8032
25/03/06 15:38:35 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
BYTES_READ=288894
BYTES_WRITTEN=348894
TOTAL_IO_MB=0.61
[hive@node4 ~]$
... View more
03-04-2025
04:55 AM
@ggangadharan I have tried this and it is working. Thanks for the help.
... View more
03-04-2025
01:59 AM
@nifi-srinikr, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.
... View more
02-18-2025
11:13 AM
1 Kudo
@mridul_tripathi That is not exactly the dataflow I was trying to convey, but good attempt. This is what I was envisioning: It start with fetching the of files from "SFTP1" using the listSFTP and FetchSFTP processors. The ListSFTP processor will create a bunch of FlowFile attributes on the output FlowFile that can be used by the FetchSFTP to fetch the content and add it to the FlowFile. In the FetchSFTP processor you will specify the SFTP1 Hostname, Username, and Password. You will use NiFi Expression language to tell FetchSPT to fetch the specific content based in the FlowFile attributes created by ListSFTP: Next the FlowFile (now with its content from SFTP1) is passed to the CryptographicHashContent processor that will create a new FlowFile Attribute (content_SHA-256) on the flowFile with the content hash. Unfortunately, we have no control over the FlowFile attribute name created by this processor. Next The FlowFile is passed to an UpdateAttribute processor is used to move the (content_SHA-256) FlowFile to a new FlowFile attribute and remove the content_SHA-256 attribute completely so we can calculate it again later after fetch same file from SFTP2. I created a new FlowFile Attribute (SFTP1_hash) where I copied over the hash. Clicking the "+" will allow you to add a dynamic property. Next I pass the FlowFile to ModifyBytes processor to remove the content from the FlowFile. Now it is time to fetch the content for this same Filename from SFTP2 by using another FetchSFTP processor. This FetchSFTP processor will be configured with the hostname for SFTP2, username for SFTP2, and password for SFT2. We still want to use the filename from the FlowFile to make sure we are fetching the same file contents from SFTP2. So you can still use "${path}/${filename}" assuming both SFTP1 and SFTP2 use the same path. If not, you will need to set path manually (<some SFTP2 path>/${filename}). Now you pass the FlowFile to another CryptographicHashContent processor which will have the content fetched from SFPT2 for the same filename. At this point in time your FlowFile has a bunch of FlowFile attributes (including hash of both content from SFTP1 (SFTP1_hash) and SFTP2 (content_SHA256)and only the content from SFTP2. So you'll pass it Now it is time to compare those two hash attribute values to make sure they are identical using an RouteOnAttribute processor. Here will create a NiFi Expression Language (NEL) expression to make this comparison. Clicking the "+" will allow you to add a dynamic property. Each Dynamic property added in this property becomes a new relationship on the processor. ${content_SHA-256:equals(${SFTP1_hash})} This NEL will return the value/string from FlowFile's "content_SH256" attribute and check to see if it is equal to the value/string from the FlowFile's "SFTP1_hash" attribute. If true, the FlowFile will be routed to the new "Content-Match" relationship. If false, it will be routed to the exiting "unmatched" relationship. Here you can decide if just want to auto-terminate the "Content-Match" relationship or do some further processing. The Unmatched relationship will contain any FlowFiles where the content for two files of the same filename have content that did not match. The FlowFile will contain the content from SFTP2. Hope this helps. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more