Solved

My current environment is CDH 6.3.2 Impala v3.2.0-cdh6.3.2 kudu 1.10.0-cdh6.3.2 Somehow, creating table with kudu storage gives IllegalArgumentException. It was ok with kudu 1.7.0-cdh5.16.2 CREATE TABLE test_mlee( id BIGINT, name STRING, PRIMARY KEY(id))PARTITION BY HASH PARTITIONS 16STORED AS KUDUERROR: IllegalArgumentException: null  Any comment is appreciated Thank you in advance.

Hi,   I'm upgrading CDH from 5.13.0 to 6.3.1 and can not proceed after "Upgrade HDFS Metadata".   On a 3rd step, after "Starting the JournalNodes." and "Starting metadata upgrade on Active NameNode of nameservice nameservice1.". I cannot proceed with "Waiting for NameNode (master02) to start responding to RPCs.".   It just freezes, but according to log records, it will not continue because the process failed. And it is failed because of a quorum of JournalNodes is not succeeded. master03.ib (10.12.0.3) is not responding.   What can I do? What can cause the issue? Can I run following steps manually?   The logs says the following Spoiler (Highlight to read) 6:43:03.610 PM WARN QuorumJournalManager Waited 55044 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:04.611 PM WARN QuorumJournalManager Waited 56045 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.

Hello ,   Can anyone experienced issue while using ConsumeImap Processor for consuming messages from Secure Email server using Imap port 993.   We are getting below error PKIX path building failed. unable to find valid cerification path   Please write here if anyone faced issue like this.   Thanks and Regards  

We want to store our sensitive information such as passwords, private keys in Vault and retrieve it from Vault on need basis, kindly share your inputs which is much needed.

Hi Everyone,We have enabled hive compaction on transnational table in order to improve fetch query performance.Somehow it's not working at my end, am facing attached error which i observed in "hivemetastore.log". Can somebody please let us know, How this issue can be resolved ? 

Hi,Newbie here. Suddenly one of the nodes lost the heartbeat. Tried to restart ambari-agent and ambari-server. However, the error still persists. Here is the ambari-agent log.   WARNING 2020-02-11 15:24:08,318 base_alert.py:138 - [Alert][ranger_admin_password_check] Unable to execute alert. argument of type 'NoneType' is not iterableINFO 2020-02-11 15:24:14,721 security.py:141 - Encountered communication error. Details: SSLError('The read operation timed out',)ERROR 2020-02-11 15:24:14,721 Controller.py:226 - Unable to connect to: https://xxx1:8441/agent/v1/register/xxx2.comTraceback (most recent call last):File "/usr/lib/ambari-agent/lib/ambari_agent/Controller.py", line 175, in registerWithServerret = self.sendRequest(self.registerUrl, data)File "/usr/lib/ambari-agent/lib/ambari_agent/Controller.py", line 549, in sendRequestraise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))IOError: Request to https://xxx1.com:8441/agent/v1/register/xxx2.com failed due

Hi, folks,   You could help me fill up my hard drive with NiFi. /var/lib/nifi/content_repository   It's taking up 100% of my space, can you help me get it back or can I just delete it all?   Greetings

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 225, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 97, in start self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 139, in configure setup_ranger_db() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 258, in setup_ranger_db user=params.unix_user, File

Hi community I am trying to debug a simple query in spark SQL that is returning incorrect data.In this instance the query is a simple join between two hive tables .. The issue seems tied to the fact that a the physical plan that spark has generated (with catalyst  optimization) looks to be corrupted where some of the steps in the physical plan have not been assigned an evaluation order id and thus all evaluation on the right side of the join is not completed in the spark queryThis error is on a HDP3.1.4 cluster running   >>> sc.versionu'2.3.2.3.1.4.0-315'   here is the example query ..   from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()filter_1 = hive.executeQuery('select * from 03_score where scores = 5 or scores = 6')filter_2 = hive.executeQuery('select * from 03_score where scores = 8')joined_df = filter_1.alias('o').join(filter_2.alias('po'), filter_1.encntr_id == filter_2.

  2020-02-10 16:28:56 Starting to launch local task to process map join; maximum memory = 477626368 SELECT n.Id,t.job_id,t.job_title,n.Name,n.Email,n.Mobile_Number,n.Education,n.Total_Experiance,n.project_id,((count(n.new_skills)*100)/s.skill_count) AverageFROMrmt_demo.resume_convert nJOINrmt_demo.job_description_convert t ON n.new_skills = t.skills and n.job_position = t.job_titleJOINrmt_demo.skill_count s ON n.job_position = s.job_titleJOINrmt_demo.education e ON n.education = e.educationGROUP BYn.Id,t.job_id,t.job_title, n.Name, n.Email, n.Mobile_Number, n.Education, n.Total_Experiance,n.project_id,s.skill_count;      

Hi AllIf in connection(queue between two processors)  I switch on RoundRobin strategy flow files start to stuck there for minutes. Some investigation showed, that flow files stuck only on the primary node.No any Warns or Errors in logs.I am ready to provide any informationPlease, any help.Thanks

We have an HDP 3.1.0 cluster (single node) without Hbase.   The ambari-metrics-collector is constantly stopping with the following error     2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964) at org.apache.

Hello,We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on  affected T-Servers have any performance impact on Kudu as whole.[root@myserver ~]# ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 1546606max locked memory       (kbytes, -l) 64max memory size   &nbs

Hello I have got strange permission problem with Ambari.I have 2 stages, both have same configs.Only difference in config are information about Postgres DB server. I'm trying to stop services with curl.Curl command: curl -i -u admin:password -H "X-Requested-By: ambari" -X PUT -d "{"HostRoles": {"state": "INSTALLED"}}" https://ambari_server:8443/api/v1/clusters/DEV/hosts/slave01/host_components/NODEMANAGER In Dev stage it all working.  But in test stage Ambari throws following message whet ansible triggers service stop: INFO [ambari-client-thread-1456334] HostComponentResourceProvider:668 - Ignoring ServiceComponentHost as operation is not allowed, What option in ambari is causing this "not allowed"?   

we are running spark streaming app that consumes data from kafka topics we want to know how to tune the parameter - `yarn.scheduler.capacity.maximum-am-resource-percent`   according to documentation:   yarn.scheduler.capacity.maximum-am-resource-percent: Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running,   on some document we even see that recomneded to utilise it to `90 percent` for best results, but the default is `10%`   so my question is do we need to tune this parameter according to cluster size?   or what is the best practice in order to get good results?

I'm able to run cli.sh registry commands, but not cli.sh nifi commands for some reason. They're both secured and use the same props files (except for different baseurls) as defined in ~/.nifi-cli.config: $ cat ~/.nifi-cli.config nifi.reg.props=/appl/nifi-registry/nifi-registry-current/conf/cli-propsnifi.props=/appl/nifi/nifi-current/conf/cli-props $ ./bin/cli.sh registry list-buckets# Name Id Description - -------------- ------------------------------------ ----------- 1 abc 87959437-dfbf-4fa2-9a4d-c0509afaff17 (empty) 2 def 4864d2de-0af5-4a69-a584-d88c108b774f (empty) 3 ghi 20f8069e-60bb-437b-ac3a-f949ffa4582b (empty) But nifi commands fail:   $ ./bin/cli.sh nifi current-userERROR: Error executing command 'current-user' : Error retrieving current $ ./bin/cli.sh nifi current-user -verboseERROR: Error executing command 'current-user' : Error retrieving current

how can i drop column from csv  i only want to keep one selected feild  

Hi,  I have a database and i want to find the best way to import it into impala. Please help me.

sudo su - ${user} -c "beeline -u 'jdbc:hive2://server:port,connectionstring' -e 'insert statement'; "   above line works fine if I take out sudo, with sudo it doesnt work, any help or advice is much appreciated.   Thanks!

Recently I moved from the Oracle JDK 8 to Open JDK 8 (version: openjdk-1.8.0.242.b08). Cloudera version is 5.16.2. I have Kerberos, TLS enabled. When I am using Oracle JDK the below issue doesn't occur but when I switch to Open JDK then I am getting the below error. Please note that the proxy user is already enabled From the CM -> HDFS -> configuration.    JobHistoryServer Error starting JobHistoryServerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://XXXX.YYY.COM:8020/user/history/done]at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:680)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:616)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:577)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceI

while i executing the program i getting these error can anyone help me with that [cloudera@quickstart ~]$ spark-shell --master yarn-clientSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel).SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/avro/avro-tools-1.7.6-cdh5.13.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ vers

Hi all!I have a docker-compose created with an image of apache/nifi 1.10.0 into a linux machine.Something like this:version: '2'services: nifi-1: image: 'apache/nifi:1.10.0' hostname: nifi ports: - 8080:8080 - 8082:8082 environment: - NIFI_WEB_HTTP_PORT=8080 - NIFI_ELECTION_MAX_WAIT=1 min - TZ=America/Argentina/Buenos_Aires - LANG=es_AR.UTF-8 Into my development i have a QueryDatabaseTable that read a entire oracle view and then insert this record in other oracle database table.My problem is that this in docker not run.If i run this in a local nifi server this run with success. QueryDatabaseTable configurations: The view: The problem (Im trying with a string field and a date field); The log2020-01-30 12:26:14,004 ERROR [Timer-Driven Process Thread-10] o.a.n.p.standard.QueryDatabaseTable QueryDatabaseTable[id=72b4c9f0-e289-33d7-f82d-50bb8f23d4f1] Unable to execute SQL select query SELECT TRUNC(FECHA_EVALUACION) FECHA_E

I am trying to run a sqoop job using oozie but it log shows it stucks on connecting to hive. The Hive action is working fine and connecting to hive but sqoop action and shell action both are unable to connect to hive ... please suggest   

Dears,   I have to two different databases and the data inside them are identical: MemSql DB Postgre DB Frequently data changed In future, I would like to sync the data between those DBs to capture the data when its changed in the main DB which is the PostgreDB and reflect that changes in the memSql DB. Is there a way using Nifi to capture data changed in the Postgre DB and move the data changed to MemSql DB?   I look forward to hearing from you soon. Regards, Ahmed

Hi All,   I'm trying to enable TLS for Cloudera Manager admin console , below are the commands I followed.   1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem) 2. Copy the JDK cacerts file to jssecacerts as follows:     sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts     3. import rootca cert into JKS store     sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem     4. Created JKS and imported host certificate in keystore.     $JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048 -dname "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Singapore,ST=Singapore,C=Singapore" -ext san=dns:$(hostname -f)sudo $JAVA_HOME/bin/keytool -importcert -alias $(hostname -f) -file /opt/cl