Adding to what @ChethanYM said, shared traces can also arise from third party tool while connection gets closed abruptly or issue with proxy host/knox if you are using one in JDBC connection settings. If you would like more debugging on this, appending JDBC driver DEBUG logs would be helpful. To do so, add below with JDBC connection string and repro the issue , It will generate driver DEBUG logs and may give some more details about the issue. LogLevel=6;LogPath=/tmp/jdbclog Regards, Krish ... View more

@Rothmny  Based on what you have shared, you should be able to accomplish this with  GenerateFlowFile and ExecuteStreamCommand processors. The GenerateFlowFile processor would be configured to generate a 0 byte FlowFile and scheduled using Cron so that it creates a FlowFile on your required schedule that will then trigger the execution of the ExecuteStreamCommand processor that will execute your python script. For calling rest-api endpoints via NiFi dataflows, you could use the InvokeHTTP processor.  Since the InvokeHTTP processor does not require and inbound connection, you could just have it as first processor in a dataflows scheduled with cron as well.  Or, if the invokeHTTP is called after your script successfully executes you could trigger with the FlowFile that exited the the ExecuteStreamCommandProcessor. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

Thank you so much for helping me out. ... View more

To identify which user is writing the files, use HDFS CLI commands such as ls or getfacl ... View more

@MarinaM, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

@pasq  Did you know that Cloudera makes managing and deploying your MiNiFi agents easy with Cloudera Edge Management?  Our ManagementHub provides a UI to build, deploy and update the dataflow you deploy to one too many MiNiFi agents. You can create parameters for various properties like passwords to provide those at deployment time through Edge Management Check out this Video. NiFi processors with "sensitive" properties are designed to encrypt those password when written to the flow.json.gz.  Even in MiNiFi these component classes will expect that the passwords are encrypted.  The defined sensitive.props.key property value in the minifi.properties or nifi.propertes is used in the encryption/decryption of these sensitive properties stored in the flow.json.gz file. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@David26  I am not familiar with this specific processor "Plc4xListenRecordProcessor 0.12.0".   I could not find it in Apache NiFi 1.x or 2.x versions.  I could find it in Maven Central either.  Where did you download this nar that you added to your NiFi? What version of Apache NiFi are you using? You may need to reach out to the originator of this custom nar you have added to you NiFi for this processor specific query. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Zubair123, This article is available exclusively for our customers. If you're a customer, please contact our customer support team for more details. If you’re not, our sales team would happily assist you with any information you need.  ... View more

Assuming it's a MapReduce job, since you're looking for information related to MapReduce I/O counters. Script to calculate the counter info. [hive@node4 ~]$ cat get_io_counters.sh #!/bin/bash # Ensure a job ID is provided if [ "$#" -ne 1 ]; then echo "Usage: $0 <job_id>" exit 1 fi JOB_ID=$1 # Extract I/O counters from the MapReduce job status mapred job -status "$JOB_ID" | egrep -A 1 'File Input Format Counters|File Output Format Counters' | awk -F'=' ' /File Input Format Counters/ {getline; bytes_read=$2} /File Output Format Counters/ {getline; bytes_written=$2} END { total_io_mb = (bytes_read + bytes_written) / (1024 * 1024) printf "BYTES_READ=%d\nBYTES_WRITTEN=%d\nTOTAL_IO_MB=%.2f\n", bytes_read, bytes_written, total_io_mb }' [hive@node4 ~]$ Sample Output [hive@node4 ~]$ ./get_io_counters.sh job_1741272271547_0007 25/03/06 15:38:34 INFO client.RMProxy: Connecting to ResourceManager at node3.playground-ggangadharan.coelab.cloudera.com/10.129.117.75:8032 25/03/06 15:38:35 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server BYTES_READ=288894 BYTES_WRITTEN=348894 TOTAL_IO_MB=0.61 [hive@node4 ~]$ ... View more