Q&A

hi all   we are using the following HDP cluster with ambari ,   list of nodes and their RHEL version   3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines , installed on RHEL 7.2 5 kafka machines , installed on RHEL 7.2   now we want to add the following machines to the cluster but with RHEL 7.5   85 DATA-NODES machines , should be installed on RHEL 7.5 version 2 kafka brokers machines , should be installed on RHEL 7.5 version   so my question is   can we mix in HDP cluster RHEL 7.2 with RHEL  7.5 version ?          

I would like to analyze a big data (0.9 TB after unzipping) in a cluster with 14 nodes and 39 cores (Azure HDInsight/Kafka). But it's very slow. Here what I do:   1. Data is downloaded from here. 2. val data = spark.read.json(path) ---- it crashes. Data are stored in HDFS.  3. val rdd = sc.textFile(path) ... then rdd.count() .... also crashes 4. rdd.take(10) , ... these are ok 5. It was not possible to unzip the file; I read it with data.json.gz Any suggestion? How I can read it with json reader? Thanks

I am installing Cloudera Enterprise 6.3.x and I am at step 5 (https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/prepare_cm_database.html#scm_prepare_syntax) .   I could not find /opt/cloudera/cm/schema/scm_prepare_database.sh file in my installation   Please help

Hi Cloudera,  the cloudera altus director cannot create a database(s) for CM and throws this error, even if it has a root user access to the external AWS RDS posgre database:       org.postgresql.util.PSQLException: ERROR: must be member of role "cmadmin_dkndwlxo"     I could not find any hint in the docs what this exact role means and why the root user must be member of this role. Postgre error log:     2019-11-25 11:07:06 UTC:10.150.1.7(43878):dbroot@postgres:[12215]:ERROR: must be member of role "cmadmin_dkndwlxo"2019-11-25 11:07:06 UTC:10.150.1.7(43878):dbroot@postgres:[12215]:STATEMENT: create database scm_75ilop0jdikuhinujsfs7l5m1n owner cmadmin_dkndwlxo encoding 'UTF8'2019-11-25 11:07:17 UTC:10.150.1.7(43880):dbroot@postgres:[12313]:ERROR: must be member of role "cmadmin_wrhjespw"2019-11-25 11:07:17 UTC:10.150.1.7(43880):dbroot@postgres:[12313]:STATEMENT: create database scm_38kegs9qab7j5l6hgqo069h3am owner cmadmin_wrhjespw encoding 'UTF8'

Hello,   We recently did an upgrade on our Hadoop cluster from 2.6.4 to 3.1.4 - everything went decently - a few random issues that we managed to solve ourselves.But this one - it's for sure something we don't understand.We have a partitioned table with around 5b rows (1Tb+) and it's partitioned into 256 almost even partitions (and is also distributed by a certain, single column).       PARTITIONED BY ( `partfunc` string) CLUSTERED BY ( event_name) SORTED BY ( event_time ASC) INTO 10 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT

Hi,   I am having issues with the zookeeper (zookeeper-3.4.13) with latest nifi version of Nifi-1.10.0 , below is the stack trace. I am upgrading from nifi-1.9.0 .  let me know if latest version of Nifi needs different version of zookeeper ? or the issue is somewhere else.   Not : With Java 8 and nifi-1.9.0 everything works   Actual Logs using Java 11  2019-11-15 03:40:09,202 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED2019-11-15 03:40:09,241 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting2019-11-15 03:40:09,355 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.2019-11-15 03:40:09,356 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant i

Hello experts! I try to use Apache Phoenix on my Docker-based Cloudera Quickstart, for training purposes. I found some amazing blog about this : http://crazyadmins.com/install-and-configure-apache-phoenix-on-cloudera-hadoop-cdh5/ However it doesn't work at me... My current Hbase version is 1.2.0. So if I refer to Apache Phoenix website, I can get the 4.14 version but not a too recent one. Here's what I did :1. wget http://archive.apache.org/dist/phoenix/apache-phoenix-4.14.0-cdh5.13.2/bin/apache-phoenix-4.14.0-cdh5.13.2-bin.tar.gz2. tar -xvf apache-phoenix-4.14.0-cdh5.13.2-bin.tar.gz3. I copy all the content of this extracted tar.gz into my /usr/lib/hbase/lib/ directory (using : cp -a /phoenix/apache-phoenix-4.14.0-cdh5.13.2-bin/. /usr/lib/hbase/lib/apache-phoenix)4. I restart HBase (to be sure, I stop my docker container and I start it again)5. I should be able to use that kind of command : ./sqlline.py localhostBut it doesn't work. I received the following er

I wrote a Kafka producer, that sends some simulated data to a Kafka stream (replication-factor 3, one partition).   Now, I want to access this data by using Hive and/or Spark Streaming.    First approach: Using an external Hive table with KafkaStorageHandler: CREATE EXTERNAL TABLE mydb.kafka_timeseriestest ( description string, version int, ts timestamp, varname string, varvalue float)STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'TBLPROPERTIES ( "kafka.topic" = "testtopic", "kafka.bootstrap.servers"="server1:6667,server2:6667,server3:6667");-- e.g. SELECT max(varvalue) from mydb.kafka_timeseriestest; -- takes too long, and only one Tez task is running   Second approach: Writing a Spark Streaming app, that accesses the Kafka topic: // started with 10 executors, but only one executor is active...JavaInputDStream<ConsumerRecord<String, String>> stream = KafkaUtils.createDirectStream(jssc, LocationStrategies.PreferConsistent(), Consumer

Sqoop import fails while importing tables that has space and special characters in the table name.     command used: sqoop import --connect <>  --username <> --password <> --table "employee details information" --as-parquetfile --delete-target-dir --target-dir  /hdfs/dir/ --m 1 

Will Node labels be supported in the new CDP ?

Hi, I am getting the below error when trying to escape the apostrophe character in HUE Impala.   select * from hi_problem_view where state like 'Won''t fix'   won't fix is a value that I am trying to filter by for records.     Error: AnalysisException: Syntax error in line 3:undefined: state like 'Won''t fix' ^ Encountered: STRING LITERAL Expected: AND, BETWEEN, DIV, GROUP, HAVING, ILIKE, IN, IREGEXP, IS, LIKE, LIMIT, NOT, OFFSET, OR, ORDER, REGEXP, RLIKE, UNION CAUSED BY: Exception: Syntax error

Hi,    I registered to Cloudera and posted 2 questions earlier. Now whenever I login to Cloudera I get the following annoying popup our form submission has failed.This may have been caused by one of the following:Your request timed outA plugin/browser extension blocked the submission. If you have an ad blocking plugin please disable it and close this message to reload the page." I removed all my extensions in Chrome. It still kept coming. Then I uninstalled and reinstalled Chrome. No success. I installed Firefox and I get the popup. I don't know what I should do Any suggestions? ThanksGanesh

Hello community: I need you help please. You Know where exactly are all repository from Hadoop(2.6.x and 2.7.x and 3.x) and Ambari 2.6.x and 2.7.x). All URLs are broken. Thank for avance.

Good morning guys,    I have a very strong problem that I am struggling to figure out.   I installed Cloudera 6.1.1. on a set of AWS (5 hosts in total).   My /etc/hosts is the following:   127.0.0.1 localhost.localdomain localhost13.48.140.49 master.sysdatadigital.it master13.48.181.38 slave1.sysdatadigital.it slave113.48.185.39 slave2.sysdatadigital.it slave213.53.62.160 slave3.sysdatadigital.it slave313.48.18.0 slave4.sysdatadigital.it slave4     I am receiving the following error from the Master by all 4 hosts in my cluster:     2019-11-22 08:22:41,355 WARN New I/O boss #15:com.cloudera.server.cmf.HeartbeatRequester: Error requesting heartbeat of host id 072beea9-4ba3-4018-8b7b-fa11fd9eac25java.net.ConnectException: Connection refused: ec2-13-48-18-0.eu-north-1.compute.amazonaws.com/172.31.16.104:9000 at com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:133) at com.ni

I have secured my Nifi instance with help of below tutorials.   https://community.hortonworks.com/content/kbentry/170966/setting-up-a-secure-apache-nifi-registry.html   https://community.hortonworks.com/articles/171173/setting-up-a-secure-nifi-to-integrate-with-a-secur.html     All is working fine.   But now I need to remove HTTPS access and replace it with regular HTTP.   This is not as simple as I thought.   I tried changing web.http and web.https in nifi.properties and restarting nifi.   From     # web properties #nifi.web.war.directory=./libnifi.web.http.host=nifi.web.http.port=nifi.web.http.network.interface.default=nifi.web.https.host=0.0.0.nifi.web.https.port=9443       Back to     # web properties #nifi.web.war.directory=./libnifi.web.http.host=nifi.web.http.port=8888nifi.web.http.network.interface.default=nifi.web.https.host=nifi.web.https.port=       But nifi won't start, saying remote H

I am trying to write a new article, but when I put code boxes into the article, every save changes any urls to <a link tags.  Then another save does same thing to the link text, and ends up trashing the content. Can anyone help?

Hi,We are currently using HDP 3.0 with ambari and we installed 2 nifi nodes. We made some config changes on nifi node01 without restarting both nodes (i only restarted the node01 and not node02). The changes were not working properly so we decided to roll back to the previous configs but whenever i try to start node01 i am getting the following error:Failed to connect node to cluster because local flow is different than cluster flow.  My guess would be that both nodes are out of synch.... How can we fix this issue? Thank you for your help.

Hello, I'm  developing a use case using flume to save data in kafka topic. I'm traing to tail  files from a local data source  directory and  save them in the same  partition of a kafka topic. I want to use as a key of the message the absolute path of the file. I have configured a TailDir source that save the absolute path of a file but I can't configure the kafka sink in order to assign the path as the key of the message in order to save all the events from the same file in the same kafka topic partition. In more details, when I consume the message with the kafka-console-consumer, I can see that the key is 'null' and the absolute path is in the body message. In the flume documentation I didn't find much information on how to deal with this issue, could someone halps me? tnks 🙂 sourceagent3a.sources.source3a.type = TAILDIRagent3a.sources.source3a.filegroups= f1agent3a.sources.source3a.filegroups.f1 = /path/local/data/source/agent3a.sources.source3a.channels= c

Hello All,Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state.I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get.Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items.HDFS Used – 10TB / Total 90TBFSImage – 980MCM - 6.16.2 / CDH - 5.16.2The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.[root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cm

hi all  when we look on https://supportmatrix.hortonworks.com/ And choose the RHEL 7.7  HDP or AMBARI versions are not marked from some unclear reason! so we cant captured the HDP or ambari version for RHEL 7.7 is it problem in support-matrix page ?  or something else ? any way if we want to use RHEL 7.7 what are the HDP and AMBARI Versions? that fit this RHEL version ?     

Hello   I get this error when restarting components :   500 status code received on POST method for API: /api/v1/clusters/Analytics_Hadoop/requests   I deleted oozie service    I am using postegres database Thank you in advance

I am reading "Enable the Atlas Book & Bridge for Apache HBase" https://atlas.apache.org/Hook-HBase.html, and get stuck on the step belowLink Atlas hook jars in HBase classpath - 'ln -s <atlas package>/hook/hbase/* <hbase-home>/lib/'Where is the HBase classpath in HDP?I checked the file hbase_site.xml but still got lost. From the file, # Extra Java CLASSPATH elements. Optional.export HBASE_CLASSPATH=${HBASE_CLASSPATH} Is the classpath env variable specified somewhere else?

Wondering if its possible to add a GPU node to Cloudera Enterprise 5.14.4. If yes, is there any documentation available around it.    Thanks

I added an old server already used as datanode in my cluster. The process get completed, but as soon as the datanode is been added I get the following error message on cloudera manager: "The DataNode has 1 volume failure but disks"   I don't have any error message, just one warning as soon as I added the host:     5:27:37.578 PM WARN Storage Failed to add storage directory [DISK]file:/data/2/dfs/dn/org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/2/dfs/dn is in an inconsistent state: Can't format the storage directory because the current/ directory is not empty. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:495) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:600) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:279) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java

# The database type# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.com.cloudera.cmf.db.type=mysql How can that be changed to postgres. I tried so many things and now out of options.  I am doing a POC and at first started the regular installation bu installing agent,server.daemons and bringing up CM server without a DB and then unstalled all retried using the cloudera-manager-installer.bin . After a day of searching i cannot get the installtion to complete. The embedded database does not install with the below error  $$$$sudo cat /var/log/cloudera-manager-installer/7.start-cloudera-scm-server-db.logJob for cloudera-scm-server-db.service failed because the control process exited with error code. See "systemctl status cloudera-scm-server-db.service" and "journalctl -xe" for details.$$$$ systemctl status cloudera-scm-server-db.service● cloudera-scm-server-db.service - LSB: Cloudera SCM Server's Embedded DBLoaded: loaded (/etc/rc.d/init.