Q&A

Am new to Nifi and want to know, can i populate data from HBase Table to kafka TopicsI can find GetHBase Processor - In that i want to know how Hbase client service setup should be and how the out of hbase table's data to kafka.My ultimate goal is to stream the data from HBase ---> kafka ----> Spark Streaming . I thought to use Nifi for Hbase to Kafka?

I am trying to copy a table from an Oracle DB to an Impala table having the same structure, in Spark, through Kudu. This is intended to be a 1-to-1 copy of data from Oracle to Impala. We found that all the rows have been copied, but every column except for the PK (partition key) is null everywhere. Why would that happen?   I have extracted the Oracle schema of the source table and created a target Impala table with the same structure (same column names, converted to lower case, and a reasonable mapping of data types). The PK of the Oracle table is the PK and "PARTITION BY HASH" of the Impala table. The Impala table is "STORED AS KUDU". We used Spark to read the data from Oracle. Then we used a kuduContext to insert the data into Impala. No errors were raised, the row counts match, and we can find the same values in the table PK in Oracle and in Impala. But in Impala, every column except the PK is null everywhere. The PK which gets populated correctly) is a NUMBER in Oracle and an

Hello,I am trying to install CDH with cloudera manger 6.2 on 2 goodgle cloud VM instanceswhen creating the cluster it downloaded  CDH 6.2.0-1.cdh6.2.0.p0.967373 sucessfullybut then it is not doing any distrubution and just hang therefirewall is disable , SELinux is disabledi can login with ssh and copy with scp between both VMs normal when checking the agent logFile "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__fieldName='subjectAltName')WrongHost: Peer certificate subjectAltName does not match host, expected mugzy.c.essential-rider-208218.internal, got DNS:mugzy and here is hostname:[root@mugzy ~]# python -c "import socket; hostname = socket.gethostname(); print hostname;"mugzy[root@mugzy ~]# Appreciate your insight

After install mysql as per this cloudera guide , which includes changing the "/etc/mysql/my.cnf" to the one they recommend. Now when I try :sudo mysql -u root -pI get Enter password:ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)I need to do it as follows to be able to connect to mysql:mysql -u root -p -h 127.0.0.1by the way I have tried the following to solve it but with no luck .... same errorsudo mkdir -p /var/run/mysqld/sudo touch /var/run/mysqld/mysqld.pidsudo touch /var/run/mysqld/mysqld.socksudo chown mysql:mysql -R /var/run/mysqldsudo service mysql restart

Hello,  Can i run HDFS disk balancer for multiple hosts simultaneously. I tried it and it throws an error which is misleading as already one iteration is in progress. 19/05/15 07:19:24 ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException): Disk Balancer is not enabled. Thank you,Des

I'm a newbie to Nifi and learning eventually. I'm looking to create a JSON array by utilizing 5 fields of 15 fields JSON request by using JoltTransformJSON processor. JSON array must ignore the fields if the value is null/empty. @Matt BurgessInput:{ "lastName" : "cgLastName6", "gender" : "Male", "currentDate" : "20190509", "middleInitial" : "D", "address_2" : "fake addy 2", "mailingZipCode" : "55555", "provider_address_1" : "123 fake addy", "color_1" : "white", "color_2" : "", "color_3" : "black", "color_4" : "", "color_5" : "blue", "homeAddress" : "123 fake st", "primaryLanguage" : "", "country" : "us"}I'm trying to build JSON array of "color_* "attributes and current date parameter to followed by color. My output should consist of existing input + newly built JSON array which consists of 3 color as the color_2, color_4 are null.Expected Output:{ "lastName" : "cgLastName6", "gender" : "Male", "currentDate" : "20190509", "middleInitial" : "D", "address_2" : "fake ad

Hi! I'm trying to define a Scala function in Apache Spark that performs some column arithmetic, but I am running into a compilation error. Here is my code: Scala   def sample(a: Column, b: Column): Column = { var c: Column = null c = a * (1 - b) c} When I try to use this, I get the following error message: error: overloaded method value + with alternatives cannot be applied to (org.apache.spark.sql.Column) How can I fix this arithmetic expression so it works with Spark Column objects?

Hi I am trying to install cloudera manger on RHEL 7 I keept getting this error. I have tried local repository and I get same thing oracle-j2sdk1.8 installation failed. See /var/log/cloudera-manager-installer/1.install-oracle-j2sdk1.8.log for details. Click OK to revert this installation. Error from Log:----------------------Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-: managerResolving Dependencies--> Running transaction check---> Package oracle-j2sdk1.8.x86_64 0:1.8.0+update141-1 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================Package Arch Version Repository Size================================================================================Installing:oracle-j2sdk1.8 x86_64 1.8.0+update141-1 cloudera-manager 176 MTransaction Summary================================================================================Install 1 Pac

Hi , We are using mysql for our cluster database. Currently we are seeing size of -rw-rw----. 1 mysql mysql 37G May 1 18:55 ibdata1 This database has - ambaridb/rangerdb/hivedb etc.is increasing day by day. and we don't enough space on the disk.. Is there any easy way where we can reduce the size of this mysql db? Or is there a way if we can move this to some other disk? or increase the size of this mount point?Please need suggestion

  While starting kafka server its failing with below error.Tried with multiple versions. Please help with this  exception in thread "main" java.lang.UnsupportedClassVersionError:kafka/Kafka:Unsupported major.minor version 52.0

I've read the known issues for Hue in the Altus Director 6.2 docs, however according to the information in the docs, this shouldn't be applicable to CM6.2 and CentOS 7.6.  I'm seeing the following error when Hue attempts to start during the cluster's firstrun phase:  /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue makemigrations --noinputTraceback (most recent call last): File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module> from pkg_resources import load_entry_point File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module> @_call_aside File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside f(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib

as all know we can set any service from ambari GUI to maintenance modesois it possible to set any ambari service to maintenance mode by API command ?

hi allin ambari metrics configuration , we have the followinghbase.rootdir=/var/lib/ambari-metrics-collector/hbasesince /var is 97%we want to replace the path to the new disk ( that mounted to data_metrics folder )as the following:hbase.rootdir=/data_metrics/lib/ambari-metrics-collector/hbasecan we just change the path and restart the metrics service ?so all metrics data will be under - /data_metrics/lib/ambari-metrics-collector/hbaseinstead of - /var/lib/ambari-metrics-collector/hbase ?

Hi, I'm having a time out issue when trying to perform and insert operation into HIVE using the ODBC driver (v2.6.1.1001 (64 bit) Windows) as a linked server in SQL Server. When I perfrom a SELECT query using OPENQUERY function, the driver will wait till execution is complete no matter how many minutes it take. However, when I perform an INSERT query, the driver always return a timeout message when the execution time exceeds 30 seconds. I set all timeout parameters to 0, however, the issue persists. Is there a way to make INSERTS taking longer than 30 seconds succeed? From HUE, the inserts work properly and succeed after around 1:30 minutes. Appreciate your insights.------------- Below is a sample trace log of the driver when executing the insert statement:May 05 16:57:03.394 TRACE 9468 HardyTCLIServiceThreadSafeClient::ExecuteStatement: +++++ profile leave +++++May 05 16:57:03.394 TRACE 9468 HardyTCLIServiceClient::CheckCancel: +++++ enter +++++

Hello HW community,I have a problem with my HiveMetastore & Hiveserver2 dont start at all due to the database connexion with the jdbc driver.i proprelly executed the "https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-administration/content/using_hive_with_mysql.html " i have the 3 hive users on mysql, i created the hive database and linked the mysql-connector-java.jar to my ambari-server. I tried some solution kind of replace the "localhost" in the Database URL by the host name or the ip address too, but same error.PS: all the others services are turning.My error is :stderr: Traceback (most recent call last):  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 203, in <module>    HiveMetastore().execute()  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute    method(env)  File "/var/lib/ambari-agent/cache/common

Dear all,I have a flow in NiFi as the photo shows.I'm getting data from a URL and trying to put it in Hbase in a table that I made through hbase shell.here is my puthbasejson config :hbase-site.xml is already existing in the path that I gave to the hbase controller.I do not know what is wrong.I have sandbox HDP that I activated HDF on it using CDA in virtualbox and I have allocated around 12G to my VM and I allocated virtual memory to my system from my disk in my computer. My computer is 16G. Hbase, HDFS, NIFI are all active in ambari.Thanks,Best regards,

Hi All. We have an issue with cloudera navigator UI on  6.2 manual installation ..Background.. Installed the navigator audit and metadat server roles on a cluster backed by external postgresql database CDH is using auto_TLS Navigator metadata and audit roles are status green on the cloudera manager UI   However when i browse to the navigator UI we get a status 503 service unavailable message.   we have confirmed the navigator role is up and running  [root@cmhdgateway01 hue]# lsof -i:7187COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEjava 12766 cloudera-scm 532u IPv4 814278 0t0 TCP *:7187 (LISTEN)python2.7 17534 hue 31u IPv4 955050 0t0 TCP cmhdgateway01.cso.ie:50144->cmhdgateway01.cso.ie:7187 (CLOSE_WAIT)[root@cmhdgateway01 hue]#   The following is captured on the startuplogs of navigator 2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.specification.version=1.82019-05-09 12:14:42,970 INFO com.cloudera.nav.ac

Hi Everyone I use Hortonwork 3.1.1 on Centos 7, everything start nomarlly after install, but yesterday service Yarn and Mapreduce stop, i try to restart but after few second it automatically stop. Please help me ! Here is Log on /var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log2019-04-27 15:57:23,150 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1554293667897_0131,name=JavaHBaseDistributedScan demo_kafka,user=hbase,queue=default,state=FINISHED,trackingUrl=http://bigdata-01.am.local:8088/proxy/application_1554293667897_0131/,appMasterHost=N/A,submitTime=1555412153616,startTime=1555412153617,finishTime=1555412160200,finalStatus=SUCCEEDED,memorySeconds=18035,vcoreSeconds=10,preemptedMemorySeconds=18035,preemptedVcoreSeconds=10,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=SPARK,resourceSeconds=18035 MB-seconds\, 10 vcore-seconds,preemptedResourceSeconds=18035 MB-seconds\, 10 vcore-seconds

Note:  I purposefully made the title of this post the same as a 2015 post.  This is not a duplicate, the information in the old post seems to no longer be valid. Hi, I want to check which versions of every hadoop component (i.e. which version of Hive, HDFS, HBase etc.) come in newest CDH, but I can't seem to find it. I looked into release notes, but no luck, is it written down anywhere maybe? The link (http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh_package_tarball.html) provided here (https://community.cloudera.com/t5/CDH-Manual-Installation/Where-to-find-which-versions-of-components-are-in-newest-CDH/m-p/24091/highlight/true#M627) no longer works and I cannot find this information for either CDH6 or CDS.   Can someone provide updated links? Topics:CDH ManifestCDH InventoryCDH ArchitectureCDS ManifestCDS InventoryCDS Architecture Thanks.

I was waiting for the new changes to the certifications by Hortonworks to be completed in order register for HCA. But now I do not see the option to purchase the HCA certification. It says 'Currently Unavailable' even after the 31 December 2018 which was supposed to be the date after which the exam was supposed to be active again. Why is this and when will this be made available, if at all it is going to be. Or Is this going to be removed from the certifications altogether? The other certifications are available for purchase. And do any of the other certifications require HCA as a pre-requisite? ThanksNanda

In HDP Sandbox 2.6.4 I imported mysql to hdfs but when I tried to import from mysql to hive with the following command:[maria_dev@sandbox-hdp ~]$ sqoop import --connect jdbc:mysql://sandbox-hdp.hortonworks.com/azhadoop --username root --password hadoop --query 'select * from iris_mysql WHERE $CONDITIONS' --m 1 --hive-import --hive-table azhadoop.iris_hive --target-dir /tmp/hive_tempBut I got this error:19/04/27 14:22:19 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@4b8ee4de is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@4b8ee4de is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that y

I'm facing this issue when try to using GPU on YARN:Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested resource type=[yarn.io/gpu] < 0 or greater than maximum allowed allocation. Requested resource=<memory:3072, vCores:1, yarn.io/gpu: 1>, maximum allowed allocation=<memory:9216, vCores:9>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:9216, vCores:9, yarn.io/gpu: 9223372036854775807>I already enabled GPU on my cluster but some how, it still showing that the (without yarn.io/gpu) maximum allowed allocation=<memory:9216, vCores:9>

We are testing our first cluster and appear to have failing Spark jobs due to lack of disk space while the job is running. Filesystem                             Size  Used Avail Use% Mounted on/dev/mapper/centos_usnj1cldrmst1-root   50G   49G  1.5G  98% /devtmpfs                                24G     0   24G   0% /devtmpfs                                   24G     0   24G   0% /dev/shmtmpfs   &n

Hi Cloudera Community ,  How i can change  the compression codec of hive at runtime. I'm reading some table on avro format compressed with snappy and i'm triying to write a similiar table compressed on snappy but the result is compressed on "deflate", after try with multiple options the resulting files were compressed with the same codec.  Can you help me to identify my issue on the following sentences, or what can i do to define the compression codec of hive at runtime. "set hive.exec.compress.output=true;SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;SET mapred.output.compression.type=BLOCK;SET hive.exec.dynamic.partition.mode=nonstrict;  CREATE external table IF NOT EXISTS tableX partitioned by (year Int)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerO

Hi, i am new to hadoop, just start exploring in Zeppelin Hive. I had tried to create temporarily table as simple as below%jdbc(hive)CREATE TEMPORARY TABLE tmp1 (c1 string);Then I logout and re-login, I still able to view the table. How to close the HIVE session properly?