Find answers, ask questions, and share your expertise
Recently active
Sensor data coming from ConsumeMQTT processor looks like this{1,1,1,3465689,B4:E6:2D:34:E1:D9,BE:FB:E4:11:91:5A,-42,192.168.0.21,v0.16,15.88,16.35,46.59,95931.16,0,0,0,3.27,0,0,0,0}Expected Output: { "val 1" : 1, "val 2" : 1,.......}and so on I'm preparing this incoming sensor data to be ingested in influxdb kindly help me out which processor is best for this and required configuration Thanks a lot
Hello!I am trying to read JSON body from a flowfile and write the length of a JSON array of elements to new flowfile attribute. But somehow I am getting aforementioned error all the time.Here is my script. from org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackimport json#Define a subclass of StreamCallback for use in session.write()class PyStreamCallback(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): jsn = IOUtils.toString(inputStream, StandardCharsets.UTF_8) array = json.loads(jsn) # type: dict i = 0 while i <= 1: # ищем вложенность второго уровня, чтобы посчитать кол-во элементов (записей в таблице) root_key = list(array.keys())[0] array = array[root_key] i += 1 self.length = str(len(array)) def get_length_of_array(self): return self.length# end classflowfile = session.get()if(flowfile !=
Hello, Hello,I've been trying to setup NiFi with single user authentication but got stuck in a "Access Unknown: Certificate and Token not found" error, and I'm unable to find a solution. Any help would be greatily apreciate.So, I'm trying to deploy in a remote machine with Docker, with authentication provided by NiFi, using a Nginx as reverse proxy.Maybe I have to pass p12 file to NiFi, but I have no idea how to proced.My docker-compose file is as follows version: '3.9'x-base: &base image: apache/nifi:1.16.2 networks: - nifix-nifi-environment: &nifi-environment NIFI_WEB_HTTPS_PORT: 8443 NIFI_CLUSTER_IS_NODE: "true" NIFI_ZK_CONNECT_STRING: "zookeeper:2181" NIFI_ELECTION_MAX_WAIT: "30 sec" NIFI_ELECTION_MAX_CANDIDATES: 2 NIFI_SENSITIVE_PROPS_KEY: "my-random-string" NIFI_WEB_PROXY_CONTEXT_PATH: "//" NIFI_CLUSTER_NODE_PROTOCOL_PORT: 8082 NIFI_WEB_PROXY_HOST: "nifi0:8443,nifi0,nifi0.mysite.com,n
HiHow can I make Impala read only required partitions not all partitionswe are using the results from a sql query RegardsPhilip
NIFI version 1.16.1 nifi-hive3-nar-1.16.2 While loading data to hive tables using PutHive3Streaming, some of the tables loads are getting failed with errors. I tried to change commit size, it didn't help. out of 50 hive table loads only 5 gets failed with this error and this is repeatable. Record count for these tables are less than 1Million. 2022-06-21 09:53:23,114 ERROR [Timer-Driven Process Thread-21] o.a.n.processors.hive.PutHive3Streaming PutHive3Streaming[id=9c9e3916-2000-1698-a0da-2dc44149819f] Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedorg.apache.nifi.processors.hive.PutHive3Streaming$ShouldRetryException: Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedat org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:512)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:360)at org.apache.hadoop.s
Hello Team, We recently upgraded the CM from version 7.2.X to 7.6.1. Since this is a production cluster we didn't do a restart of Cluster yet. However, we are getting alert with YARN HistoryServer. After checking the logs, we couldn't see any issue from the YARN HistoryServer but we found issues in the Cloudera agent. Please check the logs as shown below. [16/Jun/2022 07:50:44 +0200] 3249 GM JOBHISTORY throttling_logger ERROR (4 skipped) Error fetching metrics at 'https://xxx.xxx.com:19890/jmx'Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 223, in _collect_and_parse_and_return self._adapter.safety_valve)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/url.py", line 305, in urlopen_with_retry_on_authentication_errors return function() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 245, in _op
Hello Team, We are seeing a frequent entropy issue in our customer cluster as shown below. /proc/sys/kernel/random/entropy_avail returns 3754. We also installed the rng-tools in all the nodes and the service rngd is also running. After checking the document https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-data-at-rest-encryption-requirements.html#pnavId1 we can see the ExecStart looks as follows ExecStart=/sbin/rngd -f -r /dev/urandom However, our ExecStart is the default one and looks as follows ExecStart=/sbin/rngd -f can you please share if updating the ExecStart will solve the issue? Best RegardsSayed Anisul Hoque
I am using Nifi 1.6.0. I am trying to copy an S3 file into redshift. The json file on S3 looks like this:[{"a":1,"b":2},{"a":3,"b":4}]However this gives an error because of '[' and ']' in the file (https://stackoverflow.com/a/45348425).I need to convert the json from above format to a format which looks like this:{"a":1,"b":2}{"a":3,"b":4}So basically what I am trying to do is to remove '[' and ']' and replace '},' with '}'.The file has over 14 million rows (30 MB in size)How can I achieve this using Nifi?
I receive a response from InvokeHTTP as a folder->another_folder->file.json.gz and I want to do some operations on the json file using SplitJson processor, so, I need to decompress the file.json.gz I tried to UpdateAttribute to rename the parent folder to be folder.zip then UnpackContent but it's not supporting .gz format Then, I tried instead of UnpackContent, the ExecuteStreamCommand with unzip command with a plan to use PutFile then ListFile to get the file.json.gz and pass to SplitJson but I got this error in ExecuteStreamCommand step Failed to write flow file to stdin due to Broken pipe: java.io.IOException: Broken pipe What's the solution for this? I tried to set Connection Timeout propriety to 120s but still the same error, What is the best way to get file.json.gz, decompress it and pass it to SplitJson processor?
Dear Community, NiFi locally installed on my laptop was working fine until today that I got the following error in the URL. An unexpected error has occurred. Please check the logs for additional details This is the error that I am seeing in the log file. 2022-06-17 09:42:45,569 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@759a678a{nifi-error,/,file:///C:/Users/alshahin/nifi-1.15.3/work/jetty/nifi-web-error-1.15.3.war/webapp/,AVAILABLE}{.\work\nar\extensions\nifi-server-nar-1.15.3.nar-unpacked\NAR-INF\bundled-dependencies\nifi-web-error-1.15.3.war} Any help is appreciated. Thanks Regards Alireza
Hi, I am new to NiFi, I wanted to understand the how can we connect to the oracle database through NiFi?Do we need to download the driver for the same and paste it to NiFi server ? or we have to install the database on the same server and after using that driver then only we can connect to the database. Please help me.
Hi guys,I build a secured NiFi cluster authenticated by LDAP server.I am able to use NiFi Rest Api to access the secured cluster via Bearer Token.I have build a Nifi flow that need to call */nifi-api/* every 20 seconds and I have a problem is the Token is expire after 8 hours (based on my config) and i have to update Bearer Token manually after 8 hours :((I was wondering whether some way that I can get a permanet Bearer Token to Call REST API to access a secured NiFi cluster?Thanks.
Hi, I have done migration from CDH to CDP 7.1.7. We were using Apache Flume previously. But since Flume is removed from CDP, our data flow is broken. I want to know if there is any documents available for setting up standalone Flume that can connect CDP Kafka consumer and save data to HBase tables.Also want to note that I am using Kerberos for my cluster.
Hello Team, We had a situation where one application consumed over 1 TB of disk space which eventually flooded the disk space. We had to kill this application for freeing the space on this disk. Due to this not happening in the future, we want to limit the storage consumption of the YARN application. Could you please share how to configure this? Best Regards
Hi do you have any ide what how i solve the above error
I was trying to setup Nifi cluster.All these hosts pings one another.I have tried everything i could on the web, but no success.This is my configuration for master node:Thanks.MASTERNODE:nifi.state.management.embedded.zookeeper.start truenifi.zookeeper.connect.string =masternode:2181,workernode02:2181,workernode03:2181nifi.cluster.is.node=truenifi.cluster.node.address=masternodenifi.cluster.node.protocol.port=9991nifi.cluster.node.protocol.threads=10nifi.cluster.node.event.history.size=25nifi.cluster.node.connection.timeout=5 secnifi.cluster.node.read.timeout=5 secnifi.cluster.firewall.file=nifi.cluster.protocol.is.secure=falsenifi.cluster.load.balance.host=masternodenifi.cluster.load.balance.port=6342nifi.remote.input.host=masternodenifi.remote.input.secure=falsenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.web.http.host=masternodenifi.web.http.port=9443------------------------------------------------------WORKE
Hi ,Nifi is not able to form a cluster after restart , this was working fine for months without any issues, only after restart i am seeing issues , can anyone please help ? i have below zookeeper configuration and nifi.properties and nifi-app.log information. we are using external zookeeper. zoo.cfg initLimit=100autopurge.purgeInterval=24autopurge.snapRetainCount=30syncLimit=50tickTime=2000dataDir=/ngs/app/xxx/zookeeper-data-356_1admin.enableServer=trueadmin.serverPort=9990standaloneEnabled=falseserver.1=node1:2887:3887;2184server.2=node2:2886:3886;2182server.3=node3:2889:3889;2183quorum.cnxn.threads.size=204lw.commands.whitelist=mntr,stat nifi.properties nifi.state.management.configuration.file=./conf/state-management.xml# The ID of the local state providernifi.state.management.provider.local=local-providernifi.state.management.provider.cluster=zk-providernifi.state.management.embedded.zookeeper.start=falsenifi.state.management.embedded.zookeeper.properties=./conf/
I have a list of json records : [{}, {},{},{}]I splitted them to get : {},{},{},{}i need to load them to postgres as json datatypethe result in postgres should be like that | data ||---------------------------------------|| { "name":"alex", "surname":"rolfi"}|| { "name":"rolf", "surname":"syon"}|| { "name":"med", "surname":"akil"}|i did that with the following processors : but Putdatabaserecord did not solve the problem
Hi. I'm trying to reproduce a typical edge-node submission pattern for PySpark jobs using the CDE Jobs UI. To provide a module with custom Python functions that are declared as UDFs, one can specify:spark_session.sparkContext.addPyFile("python_utils.py")or use the --py-files argument with spark-submit: spark-submit --py-files python_utils.py pyspark_main.py Pure Python functions can then be imported in this way: import pyspark.sql.functions as Fdef add_udf_column(df): from python_utils import python_func python_udf = F.udf(python_func, StringType()) df = df.withColumn("udf_column", python_udf(df["src_column"])) return df Attempting something similar using the CDE Jobs UI, Spark cannot seem to find the custom module. My settings are:Application File: pyspark_main.pyArguments: --py-files python_utils.pyAdvanced Options:Python, Egg, Zip files: Added python_utils.py The error I'm getting is:ModuleNotFoundError: No module named 'python_utils'
While querying postgres database using ExecuteSQL Processor 1.16.1 converts the date into UTC format. However my target system requirement is to store data in local timezone. This happens only if you set the AVRO logical data type to True. If Avro logical types set to false then it returns the localtime. I have few 100 date fields and i cannot convert each of these from the source. How do i force the ExecuteSQL with AVRO type set to return the date as-is from the source. When Avro Type set to false : "install_date" : "2018-07-20 13:31:25.733"When Avro Type set to true: "install_date" : "2018-07-20T05:31:25.733Z",
Hi What does the eye symbol signify - is it just a view? Regards Philip
If i have to send a mail via oozie containing a spreadsheet for some data alert , how can i do it ?
Hi, I need help. I have successfully installed NiFi on Cooperates Linux Machine. But while accessing this NiFi server getting "This site can’t be reached" error. We checked everything but still persisting the issue. Please help me.
Hi,I am getting a csv without a header then adding a header with ReplaceText to it after the ReplaceText the header is added but the rest is crashed.PS: This problem happen with this file only I tried the flow with other csv files and worked fine, can this be a csvReader problem or configuration on the encoding ? Before :The replace : After :
We had an HDP 2.5.3.0 cluster that had an HDFS balancer CLI command that is not found with HDP 3.1.5.0. I am adding new nodes the the 3.1.5 cluster and it appears there is some HDFS balancing going on, albeit slowly. Did the CLI go away with 3.1.5.0 or am I missing something?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.