Find answers, ask questions, and share your expertise
Recently active
error " Cannot find CDH's bigtop-detect-javahome" when add any service in cluster please we need a helpTue Mar 7 14:18:59 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:03 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:08 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:13 EET 2023Cannot find CDH's bigtop-detect-javahome.----------------------and stderr log --------------------------- from cryptography import x509[07/Mar/2023 14:19:12 +0000] 20862 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hdfs/hdfs.sh datanode[07/Mar/2023 14:19:12 +0000] 20862 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 20870[07/Mar/2023 14:19:12 +0000] 20871 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5/opt/cloudera/cm-agent/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer suppor
Failing at 38% NAMENODE : AMBARI server NAMENODE : AMBARI server stderr: errors-440.txt stdout: output-440.txt 2021-07-22 09:41:36,191 - Processing identities...2021-07-22 09:41:36,222 - Creating keytab file for hdpcluster-072221@ on host myserver.com2021-07-22 09:41:36,230 - Processing identities completed. DATANODE : HDP CLUSTER DATANODE : HDP CLUSTER stdout: /var/lib/ambari-agent/data/output-438.txt 2021-07-22 09:41:34,474 - Missing keytabs:Keytab: /etc/security/keytabs/kerberos.service_check.072221.keytab Principal: hdpcluster-072221Command completed successfully! This is the krb5.conf file placed in ambari server (name node) and hdpcluster 3.1.5 (datanode) # Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/# output settings[logging]default = FILE:/tmp/krb5libs.logkdc =
Dear Members, I have accidently created a partition having characters %01 in the name. %01 is start of header `as per the notepad++`.When i am trying to delete the partition using ALTER statement, there is no error but partition also not getting deleted. can anyone suggest how to delete such partitions with special characters. ALTER TABLE <table_name> DROP IF EXISTS PARTITION (code='YATHAH%0188QW');
when we rename partition of a column, would underlying data of the column would also change as per our rename. this is on ORC file format. for ex. I've partition named activeFlag='N', when I rename this partition to activeFlag='Y', would those activeFlag column value change to 'Y' (OR) data would be intact as 'N' only?
Hi,I have two NIFI instance running and i am sending data. Both the nodes receives the data in random order. when i kill one of NIFI instances(first instance), all the data is processed by other NIFI instance(Second instance). And again when i bring back the other NIFI instance which was killed , the data is distributed among them in random order. Here there is no data loss observed, but we are observing some log messages are missing in the nifi-app.log as part of this restart.For example, during this testing if i have sent 100 messages, i see only log messages for 97 messages. Is this expected in NIFI or is there a way we can fix this issue? Is it related to gracefully shutting down the NIFI? Any help would be appreciated. Thanks
I have a custom processor which takes in a flowfile and based on the given attributes it creates new attributes and returns. However, I keep getting this error and I've checked the forums and I can't see where the problem is. Any help is appreciated. o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in Component [LocationParserProcessor[id=630e7e24-0183-1000-905c-8f31dc8a1e22]]org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=82eb8ad5-a9e3-473a-991b-0c4a3ab2bf36,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663856909589-1, container=default, section=1], offset=698, length=188],offset=0,name=file.csv,size=96] transfer relationship not specified. This FlowFile was not created in this session and was not transferred to any Relationship via ProcessSession.transfer() at org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:259) at org.ap
HI i am using cdp 6.3i am trying to connect impala shell it is giving me this errorimpala-shell. i am using 3 nodes , and i tried with all nodes . impala service is working fine and even i can run queries from Hue dashboard but self impala shell is not connecting Running Command : impala-shell Result :Starting Impala Shell without Kerberos authenticationError connecting: TTransportException, Could not connect to mprl508:21000***********************************************************************************Welcome to the Impala shell.(Impala Shell v3.4.0-SNAPSHOT (27b919f) built on Tue Aug 3 21:32:01 UTC 2021)When you set a query option it lasts for the duration of the Impala shell session.***********************************************************************************[Not connected] >
Hello Team, I have a pipeline that starts with an ad-hoc flow file processor as the starting processor. During cluster maintenance, the processors often go to a stopped state. Is there a way, I can get an alert with the note this processor is in a stopped state? Basically, the ask is to send alerts based on the state of the processor. Thanks in advance.
日志显示为这样,我不知道如何解决,希望有人能给我提提建议
Hello -I have a 3 node nifi cluster, running version 1.19.1. I have noticed that consistently there is an interruption of communication between zookeeper and the nifi nodes and each of the nifi nodes disconnect and then re-connect. Another office, is running a previous version of NiFi and they are experiencing the same behavior i.e., nodes consistently disconnect and then reconnect and they have zeroed in on garbage collection as a cause. Based on review of previous tickets in this system, I believe this interruption may be caused by garbage collection. Can you provide any guidance on how to deal with this issue? Can garbage collection be tuned/reduced so as to have less of an impact? Thank you
Hi Team, How to invoke api of Content-Type multipart/form-data using invokehttp processor in nifi.It's not working for me. If anyone has used the same, can you please share.Thank you.
Hi all, I have installed CDP 7.1.8 runtime and CM 7.7.1. When I would like to download the parcel for CFM, I got the error shown belowOn my VM, I can download the parcel using wget, so I don't know why it shows 404 not found.Please kindly help me out, thanks in advance.
[2023-03-20 17:43:00,250] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) ./kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group topicname we are using security.inter.broker.protocol=SSL Getting below error Error: Executing consumer group command failed due to org.apache.kafka.common.errors.TimeoutException: Call(callName=findCoordinator, deadlineMs=1679341973470, tries=1, nextAllowedTryMs=1679341973571) timed out at 1679341973471 after 1 attempt(s)java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=findCoordinator, deadlineMs=1679341973470, tries=1, nextAllowedTryMs=1679341973571) timed out at 1679341973471 after 1 attempt(s)at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)at org.apache.kafka.common
So I'm making a flow that extract data from elasticsearch using SearchElasticsearch processor and dump the data into my table in BigQuery using PutBigQuery processor.The data extracted from elasticsearch is json with new line as delimiter, like this: {"_index":"twitter","_id":"123", "_source":{"message":"bla bla bla", type:"tweet"}}{"_index":"twitter","_id":"124", "_source":{"message":"blalalala", type:"tweet"}} And then I'm doing some cleaning to change some column name and make all the hits into one json and write it as pretty json like: [{"_index":"twitter","_id":"123","_source": {"tweet":"bla bla bla", "type":"tweet"}},{"_index":"twitter","_id":"124","_source": {"tweet":"blalalala", type:"tweet"}} And then, I'm trying to convert flowfile into JSON with my defined schema, so I used UpdateAtribute so my flowfile has my schema name of atribute. Then I used ConvertRecord so each record is using the sam
Hi,I am getting the following error on Importing R globalenv module with Python3, other modules are importing fine. Error comes on following modules:from rpy2.robjects import globalenvimport rpy2.robjects as robjectsfrom rpy2.robjects.packages import importrfrom rpy2.robjects import pandas2riimport osfor key in os.environ: if key in ['R_LIBS', 'LD_LIBRARY_PATH', 'R_HOME']: print(key, '=>', os.environ[key])LD_LIBRARY_PATH => /data/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p78.21656418/lib/hadoop/lib/native:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/jpeplite/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pepspark/lib/*:/data/cloudera/parcels/PTY_BDP-7.2.0.26_CDP7.p0/pephive/lib/*:/usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/R_LIBS => /usr/lib/x86_64-linux-gnu/:/usr/local/lib/R/lib/:/home/cdsw/R:/usr/local/lib/R/libraryR_HOME => /usr/local/bin/Rimport rpy2from rpy2 import situationfrom rpy2.robjects import globalenvEngine exited with status 2. Co
Hi community,In my CDH5.16 Cluster, unable to start HDFS. Please find the screenshot belowStderr logPlease help me.
Hello Team,I have CDP Setup on premise and I am unable to login in ranger admin console.DetailsCDP: 7.1.7 | CM : 7.6.5 | Non Kerberised | Database : Mysql for all servicesI have tried all the below wayshttps://cloudera.ericlin.me/2020/02/how-to-update-ranger-web-ui-admin-users-password/https://community.cloudera.com/t5/Support-Questions/Ranger-Password-reset/m-p/280992In my case table: x_portal_user under ranger database was empty set.SO I manually inserted values for fieldsinsert into x_portal_user(id,first_name,login_id,password,status) values(1,'Admin','admin','ceb4f32325eda6142bd65215f4c0f371',1);With the above query it got inserted into table. Then did FLUSH PRIVILEGES , commit steps in mysql database.Also tried further insert one more insert statement like above.On Ranger Side configuration:I have disable kerberosAdmin Authentication Method: UnixPlease help/suggest with the solution. Thanks
Hello, I am trying to upgrade nifi from 1.8 to 1.19.1 as per the documentation, but I get the below error, and I am unable to start the nifi service. please help. Error:at org.apache.nifi.NiFi.main(NiFi.java:331)Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException: Sensitive Properties Key [nifi.sensitive.props.key] not found: See Admin Guide section [Updating the Sensitive Properties Key]at org.apache.nifi.properties.NiFiPropertiesLoader.getDefaultProperties(NiFiPropertiesLoader.java:233)at org.apache.nifi.properties.NiFiPropertiesLoader.get(NiFiPropertiesLoader.java:218)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.nifi.NiFi.initializeProperties(NiFi.java:370)... 3 common frames omitted
Hello, I have a 3 node NiFi Cluster (nifi1, nifi2, and nifi3), using v 1.19.1. The nodes are each disconnecting consistently from the cluster. They connect, and then disconnect, and then reconnect, and then disconnect over and over... On nifi1, when I log into the GUI, a warning on the right side of the GUI indicates:Event reported for nifi3:9443- Node disconnected from cluster due to Have not received a hearbeat from node in 338 seconds. In the nifi GUI, logging into https://nifi1:9443/nifi, under the cluster section, when i first open this display, no nodes are indicated as connected. When I open the cluster window, an error message "Cannot replicate request to Node nifi1:9443 because the node is not connected. Then, if I wait several minutes and refresh the screen, then all nodes are showing as connected, and then after another refresh nodes disconnect. There is similar behavior when I log into https://nifi2:9443/nifi. I see a
I'm trying to using NIFI dynamically filter flow Files in JSON format and keep only those values that are present is one SQL table. I'm using QueryRecord processor, with query like:SELECT * FROM FLOWFILE WHERE ${key} = ${value}only those values that are present in QueryDatabaseTable variables should be kept. Flow Files to be filtered looks like these: [ { "CustomerID" : 1, "OrderNbr" : "12", "Amount":100 } ]CustomerID field is the one to check and filter data. My current NIFI flow:
023-03-15 20:39:23,735 INFO nodemanager.NodeManager: STARTUP_MSG:/************************************************************STARTUP_MSG: Starting NodeManagerSTARTUP_MSG: host = DESKTOP-T8AUQ59/192.168.43.26STARTUP_MSG: args = []STARTUP_MSG: version = 3.3.0STARTUP_MSG: classpath = C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\etc\hadoop;C:\hadoop-3.3.0\share\hadoop\common;C:\hadoop-3.3.0\share\hadoop\common\lib\accessors-smart-1.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\animal-sniffer-annotations-1.17.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\asm-5.0.4.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\audience-annotations-0.5.0.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\avro-1.7.7.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\checker-qual-2.5.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-beanutils-1.9.4.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-cli-1.2.jar;C:\hadoop-3.3.0\share\hadoop\common\lib\commons-codec-1.11.jar;C:\hadoop-3.3.0\share\hado
org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:272)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$2(StandardFlowComparator.java:282)at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:280)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:258)at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$0(StandardFlowComparator.java:105)at java.util.HashMap.forEach(HashMap.java:1290)at org.apache.nifi.registry.flow.diff.StandardFlowComp
hi i have issue when i tryin to migrate simple file from s3 to HDFS (onprimes) with AWS DataSync agent. aws side error: Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs cloudwatch: [ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error) \ HDFS name node log: java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
Hello,hope someone can point me in the right direction.I need to delete all files starting with some attributes value, so based on Linux way it should be something like this: rm -f /opt/my_path/myfile_* I decided to use Execute Stream Command in this way: The path seem correct, the command generated, pasted into a Linux shell is working fine, however the processor did not delete the files.If i try with a specific file it works fine.Seems that * is not permitted, any clue? Many Thanks to everybody...
I'm stating with Jolt Transformation Processor in NiFi and I so far to understand the Jolt logic. I want to apply a simple transformation to insert a provided JSON into a list. provided JSON:{ Label1:Value1, Label2:Value2} expected result:{GeneralLabel:[ { Label1:Value1, Label2:Value2 } ]} Any clue about how to do it? In working with shift operation but without results.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.