Find answers, ask questions, and share your expertise
Recently active
hello, everyone; I am a freshman as a CDH manager user, My CDH version is CDH 6.0.1, my boss let me configure our CDH cluster to support "docker on yarn", but i found that feature not be supported in CDH 6.x manager, that means i have a only way to support that with configure that by customer configure so I decide to edit yarn-site.xml and container-executor.cfg manually at the node of cluster,but I found that will be covered automatically by cm。 how can i modify the yarn-site.xml and container-executor.cfg and will not be covered by cm. thx
my CDH Cluster has been online more than half year, all the machines and service is normal. but recently , there is one host agent always under bad healthy, and Cloudera Manager will show the red color on this host. then I check the agent log as below: [16/Nov/2019 02:20:11 +0000] 1142 MainThread agent WARNING Long HB processing time: 9.52771306038[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Long HB processing time: 34.8044610023[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Delayed HB: 19s since last[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Long HB processing time: 22.3401939869[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Delayed HB: 7s since last[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Long HB processing time: 36.7459609509[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Delayed HB: 21s since last[16/Nov/2019 02:22:24 +0000] 1142 MainThread agent WARNING Long HB processing t
Installed hadoop 3.3.5. On Single node server. hbase-site.xml <configuration> <property> <name>hbase.master.namespace.init.timeout</name> <value>2400000</value> </property> <property> <name>hbase.regionserver.executor.openregion.threads</name> <value>20</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:54310/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>localhost</value> </property> <property> <name>hbase.master.ipc.address</name> <value>localhost</value> </property> <property> <name>hbase.master.hostname</name&g
I have a json { "REQUESTID" : "379", "REQUESTTYPE" : "V", "REQUESTCONTEXT" : "B2B", "CARDNUMBER" : [ "5537290000000511", "5537290000000522" ], "EXPIRYDATE" : [ "08/20", "09/21" ], "EMAIL" : [ "John.Jones123@abcmail.com", "Jason.456@gmail.com" ], "PHONENUMBER" : "11234565555"}CARDNUMBER, EXPIRYDATE and EMAIL are varchar in sql table when Json is passed to covertToSQL processor, I am seeing empty string in the attribute for these columns in prepared statement (Insert)
What is the default port used by the Apache NiFi processor ListenFTP for the data channel in active FTP mode? For the control channel the default port is 2221. Is the default port for the data channel in active mode 2220? The default ports for FTP are 21 for control channel and 20 for data channel. Because ports below 1024 require root access, the NiFi default port for control channel is 2221.
Hello everyone, we are currently working on automaticlly patching the operating system of our CDP Private Cloud Base cluster. We enabled/configured our cluster to be HA, but for two components (YARN Job History Server and Spark History Server) we couldn't find any solution. We designed our automation to be split over the timespan of three days. Starting at 8 PM each day we update one of the master nodes, one of the edge nodes (e.g. know) and one third of the worker nodes in sequential order. Our problem is how to handle an unexpected failure. Is it possible to wait until the next day or is the availabilty in danger if one server (with the mentioned compontens) isn't booting up etc. How important are the YARN Job History Server and Spark History Server for the successful execution. As far as I understand both are only required for storing/providing logs for executed applications. Do I miss anything here?
Hello, I´m using QueryDatabaseTable with time_stamp max value column to get in real time +2k records per day. Despite I read all docs that I found I can not understand how it works. It supose to do a select every time a new record is inserted in the table but if I take a look in the postgres log I can see that the query is executed every second in search for new records.I though that this processors works like a trigger but it seems like it doesn´t and I´m afraid that this can affect database performance.Can somebody explain how this processor internally works? Thank you in advance
I am using redhat 7.9 OS, and CM 7.7.1.base on the CVE-2022-41946 security issue,https://www.postgresql.org/about/news/postgresql-jdbc-4251-4243-4238-42227jre7-security-update-for-cve-2022-41946-2551/I need to upgrade the cloudera postgresql jdbc driver files I saw the jdbc driver file is the following for the CDH-7.1.8.:/opt/cloudera/cm/common_jars/postgresql-42.2.14.jre7.6ce27816399941030df9ad06f89fa77b.jar/opt/cloudera/cm/common_jars/postgresql-42.2.24.jre7.a7b0f155b668470fb4e212e79724cc7d.jar /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.3.2.jar/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.3.3.jar/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/postgresql-42.2.16.jar I think just do a replacement for those jar files is a risk. How can I upgrade those file to PostgreSQL JDBC Driver version 42.2.27, 42.3.8, 42.4.3, 42.5.1, or later. Thank you!!
hi i have such a problem, i have a flow that downloads a csv file with data and there are columns like:,system name, surname, phone, mail, card name, proffesion. The file looks like this only data is more of course systemnamesurnamephonemailproffesiondepartmentABAJohnWane77777777j.wane@gmail.comaccountantaccountancyBCALillyBlack89898989l.black@gmail.comrecruiterhrI am reading this file record by record and for each record I need to query another system via REST using first name, last name, profession and department, the query looks something like this:https://system address/data/?name=${name}&surname=${surname}&proffesion=${proffesion}&department=${department}I use Invoke HTTP for thisand now this system should return me json with data:{"date": [{"name": "John","surname": "Wane","profession": "ACC","department":"ACC01"}}As you can see in the answer I get a different profession and department. How can I now create a json for each record that will contain data from the c
Hi Team, We are facing Nifi performance issues form a while as we increased no of processors and processor groups. We have increased thread count but still facing the issue. CPU utilization is also not reaching minimum level it is utilizing less than 50%. How to increase the performance in Nifi. We have been using 13.2 version till now and migrated to 18.0 to see if there is performance improvement but still facing the same. Could anyone suggest how to improve performance.
Hello team, I have tried to change the cluster certificates following the indications from the manual, plus internal company documents. I am stucked when starting the cluster, as when I try to start the INFR Solr instance, it returns a "no valid keystore" message. I have tried with the passwords that are configured in the config files for the service, and it works smoothly. I have done manual comparisson to working environments and everything seems to be in order. Any idea where to hit? Thanks in advance 🙂
Hello, The duplicate data from 2023-03-26 to 2023-07-10 must be removed.I'm trying to remove duplicates from the table using this command, but I'm getting an error.Commands:set hive.exec.dynamic.partition.mode=nonstrict;INSERT OVERWRITE TABLE db.table_name PARTITION(dt) select distinct * from db.table_name where dt >= '2023-03-26' AND dt >= '2023-07-10';Error: 23/07/26 16:07:46 [LocalJobRunner Map Task Executor #0]: WARN io.CombineHiveRecordReader: Multiple partitions found; not going to pass a part spec to LLAP IO: {{dt=2023-07-10}} and {{dt=2023-07-11}} 2023-07-26 16:07:47,952 Stage-1 map = 0%, reduce = 0% 23/07/26 16:07:47 [aabca681-0714-44f6-bc8d-9be6d7fca9fc main]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead.Note: The partition of this table is just the dates.Example: show partitions db.table_name;dt=2023-07-04dt=2023-07-05dt=2023-07-06dt=2023-07-07dt=2023-07-08dt=2023-
Hi, I got an error when creating external table with multiple delimiter such as "||".It says "Cannot validate serde: org.apache.hadoop.hive.serde2.MultiDelimitSerDe". I have tried add jar hive-contribute.3.1.3.jar, but still not working and also using regex.My Cloudera Version is CDH 6.1.1.How to solve this problem so i can achieve to set multiple delimiter in hive create external table?Please help. Sincerely, Gideon MaruliBank Mayapada
I am a newbie to Nifi Jolt transform. I've been looking through the document, but still cannot figure out how to do mutiple headings for one array.input json{"dataPackage": {"datatest": {"field": [{"Name": "test1","content": 45},{"Name": "test2","content": 20},{"Name": "test3","content": 23}]}}}Jolt Spec[{"operation": "shift","spec": {"dataPackage": {"datatest": {"field": {"*": {"Name": {"test1": {"#test1": "field[&3].Name","@(2,content)": "field[&3].content"},"test2": {"#test2": "field[&3].Name","@(2,content)": "field[&3].content"},"test3": {"#test3": "field[&3].Name","@(2,content)": "field[&3].content"}}}}}}}}, {"operation": "shift","spec": {"field": {"*": {"@(0,content)": "firstDetails.@(1,Name)"}}}}]current output{"firstDetails" : {"test1" : 45,"test2" : 20,"test3" : 23}}Expected output{"firstDetails" : {"test1" : 45,"test2" : 20},"secondDetails" : {"test3" : 23}}can you please help
hello. I am trying to CopyTable 5 minutes or 1 hour of data from a 15TB HBase Table to another cluster.But then I get hbase.ipc.CallTimeoutException and the transfer keeps failing. #1688313600000 ('2023-07-03 01:00:00') - 1688313900000 ('2023-07-03 01:05:00') : 5 minuteshbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --endtime=1688313900000 'BIG_EMP' #1688313600000 ('2023-07-03 01:00:00') - 1688317200000 ('2023-07-03 02:00:00')hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --
hello. After registering the target cluster as a peer in the replication source clusterEnabling replication on a table results in "ERROR: No meta znode available" error. >list_peers PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false hbase:032:0> enable_table_replication 'emp'ERROR: No meta znode availableFor usage try 'help "enable_table_replication"'Took 73.6962 seconds ....2023-07-20 15:25:44,586 INFO [ReadOnlyZKClient-A-hadoo
Hi, My Nifi Registry is hosted on a server and is configured using client certificates and I have my Nifi instance hosted locally on my machine also using client certificates.I am encountering this issues when I try to access buckets created in Nifi Registry from my Nifi instance: More details of Nifi client certificates:Owner: CN=localhost, OU=NIFIIssuer: CN=localhost, OU=NIFIMore details of Nifi Registry client certificates: Owner: CN=<dns>, OU=NIFIIssuer: CN=localhost, OU=NIFI I read somewhere that the issuer of NiFi Registry cert, should be the owner of the trusted cert in NiFI's truststore and same is the case in my issue but even then, I cant access Nifi registry from my NIfi instance.Can anyone point out any technical issues or suggest where I am going wrong?Thanks
I need to expose data through APIs to 3rd party Apps. Those API endpoints are needed to developed By Nifi itself. is that possible in Nifi? does Nifi act as an API manager?
I've developed a flow that depends on importing an Oracle database dump via data pump (i.e., the impdp command). To run this command, you have to pass in the username and password as part of the parameters of the impdp command to connect to the Oracle database. In early development, I simply put the password in a non-sensitive parameter, and at the appropriate time, I'm using a ReplaceText processor to wipe out the current contents of the flowfile and write in the contents of the shell script used to run the impdp command. Of course, the text I'm writing in to the flowfile includes #{oracle_pw}. I then PutFile to write the script to disk and ExecuteStreamCommand to run the script. It works like a charm. Now that development is over and we're making plans to move this flow into production, I need a way to make this password parameter sensitive and still be able to pass it in to the script. I've given this a lot of thought and done some research, but it seems like I might need to re
command rebalance is not currently available for execution because role balancer must be in state N/A, but it is in state Busy I cant start rebalance on cdh 5.16.2.
Hi Team,I create a dataflow try to insert data into uppercase table in postgesql.But it does not work.Appreciate any help on this issue, I am new to Nifi. Here are my setting for QueryDatabaseTable and PutDatabaseRecord The error message showorg.apache.nifi.processors.standard.db.TableNotFoundException: Table "ACPMA" not found, ensure the Catalog, Schema, and/or Table Names match those in the database exactly Here is a uppercase table ACPMA in postgesql.
Installed Nifi (v1.20) on RHEL7, java version is openjdk version 1.8.0_181. After starting nifi I can see it reports back that is it running and listening for Bootstrap requests, however, a few seconds later it shuts down. Appreciate any help on this, I new to Apache Nifi INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 33666 2023-07-20 10:43:25,365 ERROR [NiFi logging handler] org.apache.nifi.StdErr Defaulting Uptime to NOIMPL due to (java.lang.UnsupportedOperationException) Implementation not available in this environment2023-07-20 10:43:25,504 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: javax/crypto/SecretKey2023-07-20 10:43:25,504 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...2023-07-20 10:43:25,570 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi
Redhat 7 OS.I try to do a in-place upgrade Cloudera Manager from 7.4.4 to 7.7.1 following the docs: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cm_upgrade_after-cdp.html I do anything, and after the upgrade, I found the cloudera-scm-agent is not stable. the cloudera-scm-agent log is: [26/Jul/2023 15:11:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:12:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:13:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:14:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed
I have set up Quickstart VM on my local machine for learning purposes. Was working on Hive and Sqoop when I imported a File into HDFS. The activity was successful. For some reason, I am unable to go inside the folder using Hue. Nothing happens when I click on the folder. Have restarted the Browser. It didn't make any difference. Can you please let me know how to handle this?
Hi,I am a newbiee to nifi and its processors. Want to understand what would be the jolt specification for the following input/output? Or can anyone suggest any other processor . Input JSON:{"cells": {"deviceindicators:1234_32456_789023":"0", "deviceindicators:5678_89213_875943":"110"}} Output JSON:{"cells": {"1234_32456_789023":"0", "5678_89213_875943":"110"}} Want to remove "deviceindicators:" from the key using JoltJSONtransformation.. Please advice
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.