Solved

I am trying to set up a registry that pulls from a Github repo and allows a connected NiFi instance to import the process groups. I am using version 0.4.0 to do this, as it is the only release implementing NIFIREG-209. There are two obstacles I am facing, one I have a solution for and another I do not:1. Pulling down a repo and restarting will display the flow files in the Registry UI, but there are no versions and importing the Process Groups is not possible. Solution: Remove database/nifi-registry-primary.mv.db before restarting. That causes the database to get recreated on startup and allows for Process Groups to be imported in the Nifi workspace2. Even though the solution in 1 is implemented, I still cannot import a Process Group which uses another Process Group. When I try this I get the error:The Flow Registry with ID <id> reports that no Flow exists with Bucket <bucket id>, Flow <flow id>, Version NameIs this something currently being worked on, or does any

I have a very small physical cluster of 4 hosts where I want to run a CDH with Cloudera Manager. In order to have at least 3 worker hosts I'd like to install the utility and edge node along with the master node on the same host. Is this possible?Does it make sense to separete master and utility/edge nodes on two distinct hosts and go with only 2 worker hosts?Is there an alternative setup being restricted to 4 hosts? Thanks for any advice!  

In installing hive if I use the driver that came with HDP the DB connection test fails , but if I use the older driver I have(don't know the version) the check succeeds .[root@hadoop1 resources]# pwd/var/lib/ambari-server/resources[root@hadoop1 resources]# ls -al mysql-connector-java.jarlrwxrwxrwx. 1 root root 64 May 22 14:56 mysql-connector-java.jar -> /var/lib/ambari-server/resources/mysql-connector-java-8.0.15.jar[root@hadoop1 resources]#

Hi everyone ,I use Hortonwork 3.1.1 but kafka broker do not start, Here is the log i have in /var/log/kafka/server.log [2019-05-10 16:00:51,711] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)[2019-05-10 16:07:25,008] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)[2019-05-10 16:07:25,518] INFO starting (kafka.server.KafkaServer)[2019-05-10 16:07:25,519] INFO Connecting to zookeeper on am-bigdata-03.am.local:2181,am-bigdata-01.am.local:2181,am-bigdata-02.am.local:2181 (kafka.server.KafkaServer)[2019-05-10 16:07:25,532] INFO [ZooKeeperClient] Initializing a new session to am-bigdata-03.am.local:2181,am-bigdata-01.am.local:2181,am-bigdata-02.am.local:2181. (kafka.zookeeper.ZooKeeperClient)[2019-05-10 16:07:25,551] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)[2019-05-10 16:07:25,616] INFO [ZooKeeperClient] Connected. (

I am tring to add a new host.but it fails with this message at the end "Command 30(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation." the following is the "/var/log/cloudera-scm-server/cloudera-scm-server.log"  10:58:25.348 AM INFO ServiceHandlerRegistry Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=butmah, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=false, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=25, skipPackageInstall=false, skipCloudConfig=false, proxyProtocol=HTTP, proxyServer=10.4.32.3, proxyPort=8080, proxyUserName=null, proxyPassword=REDACTED, hosts=[node1, node2, node3, node4, node5], existingHosts=[]}. 10:58:25.349 AM INFO CmdStep Executing c

Guys, i`ve faced this error during Hive installation.Deep log searching gave me this: Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.the main question is: where to configure CLASSPATH ? i`ve put mysql-jar into: # ls /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql*/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql-connector-java-5.1.46.jar /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql-connector-java.jar#But no use, can anyone help me please on this ? Thanks. 

Hi, I would like to copy data from one Hbase cluster to another Hbase cluster. While copying I should copy 1000 records as first set and 1000 records as second set...and so on. For example, first set, I should copy only 1000 records, second set, I should copy 1000. How can I achieve using commands or scripts? Note: my Hbase table is huge. So, I would like to split and copy the data from one cluster to another (without moving to HDFS).

I have gone through the installation procedure several times trying to create 3 hosts at once but went back and decided to get the first host up.    This first host is up and I am trying to add additonal hosts.  On the wizard, Heardbeat is working, on the Install Parcels screen it shows CDH 6.2.0-1.cdh6.2.0.p0.967373 being downloaded/distributed/upacked/activated.   The issue is on the Host Inspector. 1 hosts are reporting with NONE CDH version   Also, none of the cloudera system accounts have been created on the new host.i.e. The user hdfs is missing on the following host   The agent seems to be running and it shows up as Managed when you show all hosts in the Manager.[root@usnj1cldrawrkr2 ~]# systemctl | grep cloudera  cloudera-scm-agent.service                                 

Hi Guys,we are trying to install the HDP 3.1.0 platform on-premise.We have finished the installation but ran into a problem setting up MariaDB for the Hive metastore.Hive explicitly lists MariaDB as supported:https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administrationand suggests using the driver class "org.mariadb.jdbc.Driver"When trying to setup "JDBC Driver Class" to "org.mariadb.jdbc.Driver" in Ambari of the HDP3.1.0 installation, then the start of the metastore fails with:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 60, in <module>HiveClient().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 966, in restartself.install(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/h

Hi All Cloudera suggests as best practice using S3 storage only for initial and final storage. The intermediate files will need to be stored in HDFS... In that case, we are still using HDFS but the cluster will only run during the batch ETL and then tore off daily. How we can pull S3 data into HDFS for each batch ETL job? and then store back the final results to S3? If Cloudera means to use distcp, how that might work for ETL batch jobs each time? It did not make sense to me using distcp... ThanksCK

i have a cluster of six nodes (centos) and installed cloudera on it. each node is having two partition of disk 30 gb  and 250 gb .but my cloudrea is using only 30 gb partition and i am running out of memory every week because of logs in namenode and data in datanode.  so how i will mount my 250gb partiotin for cloudera logs on namenode and data on datanodes.   thanks for any help.  

Hi Cloudera, I see a lot of these warnings in Impala Daemon logs: W0516 07:12:24.227567 1049 ShortCircuitCache.java:826] ShortCircuitCache(0x119fb869): could not load 1399296933_BP-76826636-10.197.31.86-1501521881839 due to InvalidToken exception.Does it indicate some bad configuration? What can I do to eliminate these warnings?Thanks 

Am new to Nifi and want to know, can i populate data from HBase Table to kafka TopicsI can find GetHBase Processor - In that i want to know how Hbase client service setup should be and how the out of hbase table's data to kafka.My ultimate goal is to stream the data from HBase ---> kafka ----> Spark Streaming . I thought to use Nifi for Hbase to Kafka?

I am trying to copy a table from an Oracle DB to an Impala table having the same structure, in Spark, through Kudu. This is intended to be a 1-to-1 copy of data from Oracle to Impala. We found that all the rows have been copied, but every column except for the PK (partition key) is null everywhere. Why would that happen?   I have extracted the Oracle schema of the source table and created a target Impala table with the same structure (same column names, converted to lower case, and a reasonable mapping of data types). The PK of the Oracle table is the PK and "PARTITION BY HASH" of the Impala table. The Impala table is "STORED AS KUDU". We used Spark to read the data from Oracle. Then we used a kuduContext to insert the data into Impala. No errors were raised, the row counts match, and we can find the same values in the table PK in Oracle and in Impala. But in Impala, every column except the PK is null everywhere. The PK which gets populated correctly) is a NUMBER in Oracle and an

Hello,I am trying to install CDH with cloudera manger 6.2 on 2 goodgle cloud VM instanceswhen creating the cluster it downloaded  CDH 6.2.0-1.cdh6.2.0.p0.967373 sucessfullybut then it is not doing any distrubution and just hang therefirewall is disable , SELinux is disabledi can login with ssh and copy with scp between both VMs normal when checking the agent logFile "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__fieldName='subjectAltName')WrongHost: Peer certificate subjectAltName does not match host, expected mugzy.c.essential-rider-208218.internal, got DNS:mugzy and here is hostname:[root@mugzy ~]# python -c "import socket; hostname = socket.gethostname(); print hostname;"mugzy[root@mugzy ~]# Appreciate your insight

After install mysql as per this cloudera guide , which includes changing the "/etc/mysql/my.cnf" to the one they recommend. Now when I try :sudo mysql -u root -pI get Enter password:ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)I need to do it as follows to be able to connect to mysql:mysql -u root -p -h 127.0.0.1by the way I have tried the following to solve it but with no luck .... same errorsudo mkdir -p /var/run/mysqld/sudo touch /var/run/mysqld/mysqld.pidsudo touch /var/run/mysqld/mysqld.socksudo chown mysql:mysql -R /var/run/mysqldsudo service mysql restart

Hello,  Can i run HDFS disk balancer for multiple hosts simultaneously. I tried it and it throws an error which is misleading as already one iteration is in progress. 19/05/15 07:19:24 ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException): Disk Balancer is not enabled. Thank you,Des

I'm a newbie to Nifi and learning eventually. I'm looking to create a JSON array by utilizing 5 fields of 15 fields JSON request by using JoltTransformJSON processor. JSON array must ignore the fields if the value is null/empty. @Matt BurgessInput:{ "lastName" : "cgLastName6", "gender" : "Male", "currentDate" : "20190509", "middleInitial" : "D", "address_2" : "fake addy 2", "mailingZipCode" : "55555", "provider_address_1" : "123 fake addy", "color_1" : "white", "color_2" : "", "color_3" : "black", "color_4" : "", "color_5" : "blue", "homeAddress" : "123 fake st", "primaryLanguage" : "", "country" : "us"}I'm trying to build JSON array of "color_* "attributes and current date parameter to followed by color. My output should consist of existing input + newly built JSON array which consists of 3 color as the color_2, color_4 are null.Expected Output:{ "lastName" : "cgLastName6", "gender" : "Male", "currentDate" : "20190509", "middleInitial" : "D", "address_2" : "fake ad

Hi! I'm trying to define a Scala function in Apache Spark that performs some column arithmetic, but I am running into a compilation error. Here is my code: Scala   def sample(a: Column, b: Column): Column = { var c: Column = null c = a * (1 - b) c} When I try to use this, I get the following error message: error: overloaded method value + with alternatives cannot be applied to (org.apache.spark.sql.Column) How can I fix this arithmetic expression so it works with Spark Column objects?

Hi I am trying to install cloudera manger on RHEL 7 I keept getting this error. I have tried local repository and I get same thing oracle-j2sdk1.8 installation failed. See /var/log/cloudera-manager-installer/1.install-oracle-j2sdk1.8.log for details. Click OK to revert this installation. Error from Log:----------------------Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-: managerResolving Dependencies--> Running transaction check---> Package oracle-j2sdk1.8.x86_64 0:1.8.0+update141-1 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================Package Arch Version Repository Size================================================================================Installing:oracle-j2sdk1.8 x86_64 1.8.0+update141-1 cloudera-manager 176 MTransaction Summary================================================================================Install 1 Pac

Hi , We are using mysql for our cluster database. Currently we are seeing size of -rw-rw----. 1 mysql mysql 37G May 1 18:55 ibdata1 This database has - ambaridb/rangerdb/hivedb etc.is increasing day by day. and we don't enough space on the disk.. Is there any easy way where we can reduce the size of this mysql db? Or is there a way if we can move this to some other disk? or increase the size of this mount point?Please need suggestion

  While starting kafka server its failing with below error.Tried with multiple versions. Please help with this  exception in thread "main" java.lang.UnsupportedClassVersionError:kafka/Kafka:Unsupported major.minor version 52.0

I've read the known issues for Hue in the Altus Director 6.2 docs, however according to the information in the docs, this shouldn't be applicable to CM6.2 and CentOS 7.6.  I'm seeing the following error when Hue attempts to start during the cluster's firstrun phase:  /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue makemigrations --noinputTraceback (most recent call last): File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module> from pkg_resources import load_entry_point File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module> @_call_aside File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside f(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib

as all know we can set any service from ambari GUI to maintenance modesois it possible to set any ambari service to maintenance mode by API command ?

hi allin ambari metrics configuration , we have the followinghbase.rootdir=/var/lib/ambari-metrics-collector/hbasesince /var is 97%we want to replace the path to the new disk ( that mounted to data_metrics folder )as the following:hbase.rootdir=/data_metrics/lib/ambari-metrics-collector/hbasecan we just change the path and restart the metrics service ?so all metrics data will be under - /data_metrics/lib/ambari-metrics-collector/hbaseinstead of - /var/lib/ambari-metrics-collector/hbase ?