Q&A

  日志显示为这样，我不知道如何解决，希望有人能给我提提建议

Hello -I have a 3 node nifi cluster, running version 1.19.1. I have noticed that consistently there is an interruption of communication between zookeeper and the nifi nodes and each of the nifi nodes disconnect and then re-connect. Another office, is running a previous version of NiFi and they are experiencing the same behavior i.e., nodes consistently disconnect and then reconnect and they have zeroed in on garbage collection as a cause. Based on review of previous tickets in this system, I believe this interruption may be caused by garbage collection. Can you provide any guidance on how to deal with this issue?  Can garbage collection be tuned/reduced so as to have less of an impact? Thank you

Hi Team, How to invoke api of Content-Type multipart/form-data using invokehttp processor in nifi.It's not working for me.  If anyone has used the same, can you please share.Thank you.

Hi all, I have installed CDP 7.1.8 runtime and CM 7.7.1. When I would like to download the parcel for CFM, I got the error shown belowOn my VM, I can download the parcel using wget, so I don't know why it shows 404 not found.Please kindly help me out, thanks in advance.

[2023-03-20 17:43:00,250] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) ./kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group topicname we are using security.inter.broker.protocol=SSL   Getting below error Error: Executing consumer group command failed due to org.apache.kafka.common.errors.TimeoutException: Call(callName=findCoordinator, deadlineMs=1679341973470, tries=1, nextAllowedTryMs=1679341973571) timed out at 1679341973471 after 1 attempt(s)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=findCoordinator, deadlineMs=1679341973470, tries=1, nextAllowedTryMs=1679341973571) timed out at 1679341973471 after 1 attempt(s)at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)at org.apache.kafka.common

So I'm making a flow that extract data from elasticsearch using SearchElasticsearch processor and dump the data into my table in BigQuery using PutBigQuery processor.The data extracted from elasticsearch is json with new line as delimiter, like this:       {"_index":"twitter","_id":"123", "_source":{"message":"bla bla bla", type:"tweet"}}{"_index":"twitter","_id":"124", "_source":{"message":"blalalala", type:"tweet"}}       And then I'm doing some cleaning to change some column name and make all the hits into one json and write it as pretty json like:     [{"_index":"twitter","_id":"123","_source": {"tweet":"bla bla bla", "type":"tweet"}},{"_index":"twitter","_id":"124","_source": {"tweet":"blalalala", type:"tweet"}}     And then, I'm trying to convert flowfile into JSON with my defined schema, so I used UpdateAtribute so my flowfile has my schema name of atribute.   Then I used ConvertRecord so each record is using the sam

Hi,I am getting the following error on Importing R globalenv module with Python3, other modules are importing fine. Error comes on following modules:from rpy2.robjects import globalenvimport rpy2.robjects as robjectsfrom rpy2.robjects.packages import importrfrom rpy2.robjects import pandas2riimport osfor key in os.environ: if key in ['R_LIBS', 'LD_LIBRARY_PATH', 'R_HOME']: print(key, '=>', os.environ[key])LD_LIBRARY_PATH => /data/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p78.21656418/lib/hadoop/lib/native:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/jpeplite/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pepspark/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pephive/lib/*:/usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/R_LIBS => /usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/:/home/cdsw/R:/usr/local/lib/R/libraryR_HOME => /usr/local/bin/Rimport rpy2from rpy2 import situationfrom rpy2.robjects import globalenvEngine exited with status 2. Co

Hi community,In my CDH5.16 Cluster, unable to start HDFS. Please find the screenshot belowStderr logPlease help me.

Hello Team,I have CDP Setup on premise and I am unable to login in ranger admin console.DetailsCDP: 7.1.7 | CM : 7.6.5 | Non Kerberised | Database : Mysql for all servicesI have tried all the below wayshttps://cloudera.ericlin.me/2020/02/how-to-update-ranger-web-ui-admin-users-password/https://community.cloudera.com/t5/Support-Questions/Ranger-Password-reset/m-p/280992In my case table: x_portal_user under ranger database was empty set.SO I manually inserted values for fieldsinsert into x_portal_user(id,first_name,login_id,password,status) values(1,'Admin','admin','ceb4f32325eda6142bd65215f4c0f371',1);With the above query it got inserted into table. Then did FLUSH PRIVILEGES , commit steps in mysql database.Also tried further insert one more insert statement like above.On Ranger Side configuration:I have disable kerberosAdmin Authentication Method: UnixPlease help/suggest with the solution. Thanks

Hello, I am trying to upgrade nifi from 1.8 to 1.19.1 as per the documentation, but I get the below error, and I am unable to start the nifi service. please help.   Error:at org.apache.nifi.NiFi.main(NiFi.java:331)Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: Sensitive Properties Key [nifi.sensitive.props.key] not found: See Admin Guide section [Updating the Sensitive Properties Key]at org.apache.nifi.properties.NiFiPropertiesLoader.getDefaultProperties(NiFiPropertiesLoader.java:233)at org.apache.nifi.properties.NiFiPropertiesLoader.get(NiFiPropertiesLoader.java:218)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.nifi.NiFi.initializeProperties(NiFi.java:370)... 3 common frames omitted

Hello, I have a 3 node NiFi Cluster (nifi1, nifi2, and nifi3), using v 1.19.1.   The nodes are each disconnecting consistently from the cluster.  They connect, and then disconnect, and then reconnect, and then disconnect over and over... On nifi1, when I log into the GUI, a warning on the right side of the GUI indicates:Event reported for nifi3:9443- Node disconnected from cluster due to Have not received a hearbeat from node in 338 seconds. In the nifi GUI, logging into https://nifi1:9443/nifi, under the cluster section, when i first open this display,  no nodes are indicated as connected.  When I open the cluster window, an error message "Cannot replicate request to Node nifi1:9443 because the node is not connected.  Then, if I wait several minutes and refresh the screen, then all nodes are showing as connected, and then after another refresh nodes disconnect. There is similar behavior when I log into https://nifi2:9443/nifi. I see a

I'm trying to using NIFI dynamically filter flow Files in JSON format and keep only those values that are present is one SQL table. I'm using QueryRecord processor, with query like:SELECT * FROM FLOWFILE WHERE ${key} = ${value}only those values that are present in QueryDatabaseTable variables should be kept. Flow Files to be filtered looks like these:  [ { "CustomerID" : 1, "OrderNbr" : "12", "Amount":100 } ]CustomerID field is the one to check and filter data. My current NIFI flow:   

023-03-15 20:39:23,735 INFO nodemanager.NodeManager: STARTUP_MSG:/************************************************************STARTUP_MSG: Starting NodeManagerSTARTUP_MSG: host = DESKTOP-T8AUQ59/192.168.43.26STARTUP_MSG: args = []STARTUP_MSG: version = 3.3.0STARTUP_MSG: classpath = C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\share\hadoop\common;C:\hadoop-3.3.0\share\hadoop\common\lib\accessors-smart-1.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\animal-sniffer-annotations-1.17.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\asm-5.0.4.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\audience-annotations-0.5.0.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\avro-1.7.7.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\checker-qual-2.5.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-beanutils-1.9.4.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-cli-1.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-codec-1.11.jar;C:\hadoop-3.3.0\share\hado

org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:272)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$2(StandardFlowComparator.java:282)at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:280)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:258)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$0(StandardFlowComparator.java:105)at java.util.HashMap.forEach(HashMap.java:1290)at org.apache.nifi.registry.flow.diff.StandardFlowComp

hi i have issue when i tryin to migrate simple file from s3 to HDFS (onprimes) with AWS DataSync agent. aws side error: Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs cloudwatch: [ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error) \ HDFS name node log:  java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.

Hello,hope someone can point me in the right direction.I need to delete all files starting with some attributes value, so based on Linux way it should be something like this: rm -f /opt/my_path/myfile_* I decided to use Execute Stream Command in this way: The path seem correct, the command generated, pasted into a Linux shell is working fine, however the processor did not delete the files.If i try with a specific file it works fine.Seems that * is not permitted, any clue? Many Thanks to everybody... 

I'm stating with Jolt Transformation Processor in NiFi and I so far to understand the Jolt logic. I want to apply a simple transformation to insert a provided JSON into a list. provided JSON:{   Label1:Value1,   Label2:Value2} expected result:{GeneralLabel:[                   {                      Label1:Value1,                      Label2:Value2                   }                  ]} Any clue about how to do it? In working with shift operation but without results.

Hi Team, I have created an rest endpoint on NIFI as per below Screen.The purpose of this endpoint is to fetch data from https://jsonplaceholder.typicode.com/todos/2 and present as http response of my API.  Below is my template code -<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.3"><description></description><groupId>be506fdd-0184-1000-83e1-c82d0bfe4cb7</groupId><name>SampleTemplatetoCallHttpsEndpoint</name><snippet><processGroups><id>78122aea-0a0d-3713-0000-000000000000</id><parentGroupId>f175ed29-51e9-3358-0000-000000000000</parentGroupId><position><x>0.0</x><y>0.0</y></position><versionedComponentId>78122aea-0a0d-3713-ba11-a970a91b757e</versionedComponentId><comments></comments><contents><connections><id>00ded3d7-2d46-349b-0000-000000000000</id><parentG

Hi , Currently i am using NIFI to consume and publish data using kafka and i have a requirement to perform retry only for topicAuthorizationException thrown by NIFI processor (publishKafka processor). Like similar to RouteonAttribute, is there any processor or any log attribute through which i can parse or read  this topicAuthorizationException error , so that i can perform retry? when kafka topic does not exist, NIFI throws below exception, the same is seen in nifi-app.log, i need to know how can we read this error in NIFI org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test_topic] 

Bonjour, j'ai installé NIFI en suivant la documentation, j'ai bien evidement vérifié que j'avais le bon java, mais au moment d'accéder au portail localhost, je n'y arrive pas, ca me met ça: Bien evidemment je l'ai la,cé avant, voici ce que dit la cmd ou je le lance, aussi je ne trouve pas mes identiifiants comme convenu dans le fichier logs, quelqu'un pourrait m'aider svp 

I am have the next error when try to acces resource manager from the knox web ui errror in gateway.log   INFO knox.gateway (ConfigurableHADispatch.java:failoverRequest(275)) - Failing over request to a different server: https://awsnode01:7183/favicon.ico?doAs=u3212 INFO knox.gateway (ConfigurableHADispatch.java:executeRequest(162)) - Could not connect to server: https://awsnode01:7183/favicon.ico?doAs=u3212 org.eclipse.jetty.io.EofException: Closed INFO knox.gateway (ConfigurableHADispatch.java:failoverRequest(275)) - Failing over request to a different server: https://awsnode01:7183/favicon.ico?doAs=u3212INFO knox.gateway (ConfigurableHADispatch.java:executeRequest(162)) - Could not connect to server: https://awsnode01:7183/favicon.ico?doAs=u3212 org.eclipse.jetty.io.EofException: Closed INFO knox.gateway (ConfigurableHADispatch.java:failoverRequest(275)) - Failing over request to a different server: https://awsnode01:7183/favicon.ico?doAs=u3212INFO knox.gateway (C

Hello, I am following the Apache NiFi Walkthroughs, specifically the NiFi Standalone Using External CA.  I am trying to install a 5 node cluster.  The process I am following is:I created, using TinyCert the following:1.  A signed NiFi server certifcate for each NiFi server in my cluster i.e., nifi1, nifi2, nifi3, nifi4, and nifi5.2.  The matching private keys in PEM format.3.  A signed client certificate that I created on nifi1 (which I then planned to reuse for the other nodes i.e., nifi2, nifi3, nifi4, and nifi5).4.  The matching private key in PEM format.5.  The CA certificate in PEM format. Then, on each server i.e., nifi1, nifi2, nifi3, nifi4, and nifi5, i followed the steps indicated in the walkthrough i.e., 1.  concatenate the server certificate and CA certificate to form the certificate chain.2.  Form the PKCS12 keystore from the certificate chain and private key.3.  Convert the PKCS12 keystore for the NiFi instance in

i have  NIFI mongo db connection and need to fetch only new record , i have a long column (transactiondate) (yyyy-MM-dd HH:mm:ss) and need the new transactions in last 1 sec . 

I have a feed coming in with time spent value set in seconds. Example 900 or 3600Is there way in nifi to convert attribute to 00:15 and 01:00 (HH:mm format)?

There is an error during cdsw model deployment where the package of code is not imported. - CDSW version1.9.2 - Folder Configuration/home/cdswl- ex_package_onel    l- ex_package_twol    l-ex_pkg_code.pyll- predictl    l- predict.py predict.py is the code used in the model deployment.- Package import code within predict.py....from ex_package_one.ex_package_two import ex_pkg_code...- Error logModuleNotFoundError: No module named 'ex_package_one'- Attempt to resolve the errorI printed it out as os.getcwd() from predict.py and it came out as /cdsw/home.sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname("__file__"))))The path was added using the above code, but the same error occurred.No package errors will occur if the code is transferred to /home/cdsw/predict.py.Is there any other way to solve this problem?Thank you for any help.