Q&A

Hello, We have been facing issues with our CM Service Monitor, for the past couple of days it fails and the starts to work again, we also get the below alerts repeatedly : Concerning : The last metrics aggregation run duration is 25.4 second(s). Warning threshold: 10 second(s). Concerning : Average time spent paused was 27.9 second(s) (46.47%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. There are no configuration changes done on this node, attached is the SM log file.5:48:14.372 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2543ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2718ms5:48:14.373 AM ERROR SafeAvroResponderServlet Error procesing Avro requestorg.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator

HiWhen i do a SELECT null; at cli hive i get a good answer -> NULLwhen i do this in beeline , i get Error: Error while compiling statement: FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias or column reference 'null': (possible column names are: ) (state=42000,code=10004)this is working on one of my clusters but not in one of them (hdp 2.6.1 Connected to: Apache Hive (version 2.1.0.2.6.1.0-129)Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)Transaction isolation: TRANSACTION_REPEATABLE_READBeeline version 1.2.1000.2.6.1.0-129 by Apache Hive)i need this to work, since when i query INSERT OVERWRITE TABLE schema.customerSELECT CUST_ACCOUNTSotherschema.CUST_ACCOUNT_ID,NULL AS ACCOUNTFROM xxxx ;i really appreciate any help

Hi, i am new with hadoop,I have the following errors when i try to run the command hdfs dfs -ls on a specific directory:[t_hdhusr@node01 ~]$ hdfs dfs -ls /user/T_HDHUSRlog4j:ERROR setFile(null,true) call failed.java.io.FileNotFoundException: /var/log/hadoop/t_hdhusr/hadoop-mapreduce.jobsummary.gz (No such file or directory)at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270)at java.io.FileOutputStream.<init>(FileOutputStream.java:213)at java.io.FileOutputStream.<init>(FileOutputStream.java:133)at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)at org.apache.log4j.config.PropertySetter.setProperties(PropertySett

hi,I've two clusters running on hdp 3.1, both consists of single node and kerbeorized as well. I am trying to send a single file using distcp but failed.got error(pictures attached)even unable to list hdfs directories of other cluster on either of the cluster.Thanks.

Is there any way to parameterize Nifi DBConnection Services..? I want to use the same nifi flow to extract the data from different DataBases with different set of parameters (for example tables and data types) so i am wondering is there any way to pass polling services just like any other parameter value as a variable..?

Hi, I have a very simple question. Can hive dB have multiple ranger policies?In any circumstance can you restrict having multiple ranger policies?

Content of stock2.json file{"myid":"0001","mytype":"donut"}stock2.json file is in the path of '/warehouse/tablespace/external/hive/haijintest.db/stock2';Below is the table creation script.CREATE EXTERNAL TABLE stock4_json (myjson struct<myid:string, mytype:string>)ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION '/warehouse/tablespace/external/hive/haijintest.db/stock2';Below is the query that returned null value.0: jdbc:hive2://> select myjson.myid from stock4_json;19/05/27 20:20:18 [103d2343-eb59-4072-8a07-5469b2112093 main]: WARN optimizer.SimpleFetchOptimizer: Table haijintest@stock4_json is external table, falling back to filesystem scan.OK+-------+| myid  |+-------+| NULL  |+-------+1 row selected (0.232 seconds)0: jdbc:hive2://> select * from stock4_json;OK+---------------------+| stock4_json.myjson  |+---------------------+| NULL                |+---------------------+1 row selected (0.242 sec

how to configure hivserver2 in superset.

Hi all,We have HDP version 3.1 , we are trying to integrate cluster with corporate Active directory for kerberos , we have setup CA certificate as instruction given on ambari/hdp communities ,we have below required details with us ..AD Service account / passwordRealm nameLdap server with portDC=in , dc =xyz ,dc=com.we we have gid down may article on it but same issue we are facingbut facing issue with below error messageError message: Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://adldap.x.x.com: simple bind failed: adldap.x.xcom:636 Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore.can someone has all relevant steps which need to be follow to integrate hadoop cluster with AD for kerberos and for domain authentication

My exam for CCA 175 was scheduled on 03/03/2019 19:30:00 IST. The exam did not load on examslocal.com as it was not ready by cloudera. The chat support in examslocal suggested me to drop an email to cloudera team. As soon as i got this instruction, I dropped an email to certification@cloudera.com regarding the issue, but there is no proper response till now. All I get is, an automatic reply, which says, cloudera team is having a tech summit and would be busy between 25th feb- 3rd march, but it's been 7th march now. I am supposed to share a certificate of completion of CCA 175 before 12th march, otherwise, i'd be in trouble.Any suggestions are welcome. Thanks.

I want to read a MS SQL database. I'm using a QueryDatabaseTable to make de query and then SplitAvro, because the output of QueryDatabaseTable is an avro file. But my problem is that the query does not arrive to SplitAvro processor. Can someone help me with the configuration of this processors?qdt.PNGsplitavro.PNG

Hi, I need some help with JOLT-syntax.JSON-Input{  "nummer": "1073",  "table": [    {      "datum": "20180529",      "zn": 1    },    {      "datum": "20180530",      "zn": 2    },    {      "datum": "20190522",      "zn": 3    }  ]}JOLT-Specification[{    "operation": "shift",    "spec": {      "nummer": "Nummer",      "table": {        "*": {          "zn": "Positionen.[].ZeileNr",          "datum": "Positionen.[].Datum"        }      }    }}]JSON-Output{    "Nummer": "1073",    "Positionen": [{        "ZeileNr": 1    }, {        "Datum": "20180529"    }, {      &nbs

We have ten millions image and video files, are looking for efficient ways to store them in Hadoop (HDFS ...), and analyze them with tools available in the Hadoop ecosystem. I understand HDFS prefer big files. These image files are small, they are under ten megabytes. Please advise. Thanks very much!  

I am trying to set up a registry that pulls from a Github repo and allows a connected NiFi instance to import the process groups. I am using version 0.4.0 to do this, as it is the only release implementing NIFIREG-209. There are two obstacles I am facing, one I have a solution for and another I do not:1. Pulling down a repo and restarting will display the flow files in the Registry UI, but there are no versions and importing the Process Groups is not possible. Solution: Remove database/nifi-registry-primary.mv.db before restarting. That causes the database to get recreated on startup and allows for Process Groups to be imported in the Nifi workspace2. Even though the solution in 1 is implemented, I still cannot import a Process Group which uses another Process Group. When I try this I get the error:The Flow Registry with ID <id> reports that no Flow exists with Bucket <bucket id>, Flow <flow id>, Version NameIs this something currently being worked on, or does any

I have a very small physical cluster of 4 hosts where I want to run a CDH with Cloudera Manager. In order to have at least 3 worker hosts I'd like to install the utility and edge node along with the master node on the same host. Is this possible?Does it make sense to separete master and utility/edge nodes on two distinct hosts and go with only 2 worker hosts?Is there an alternative setup being restricted to 4 hosts? Thanks for any advice!  

In installing hive if I use the driver that came with HDP the DB connection test fails , but if I use the older driver I have(don't know the version) the check succeeds .[root@hadoop1 resources]# pwd/var/lib/ambari-server/resources[root@hadoop1 resources]# ls -al mysql-connector-java.jarlrwxrwxrwx. 1 root root 64 May 22 14:56 mysql-connector-java.jar -> /var/lib/ambari-server/resources/mysql-connector-java-8.0.15.jar[root@hadoop1 resources]#

Hi everyone ,I use Hortonwork 3.1.1 but kafka broker do not start, Here is the log i have in /var/log/kafka/server.log [2019-05-10 16:00:51,711] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)[2019-05-10 16:07:25,008] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)[2019-05-10 16:07:25,518] INFO starting (kafka.server.KafkaServer)[2019-05-10 16:07:25,519] INFO Connecting to zookeeper on am-bigdata-03.am.local:2181,am-bigdata-01.am.local:2181,am-bigdata-02.am.local:2181 (kafka.server.KafkaServer)[2019-05-10 16:07:25,532] INFO [ZooKeeperClient] Initializing a new session to am-bigdata-03.am.local:2181,am-bigdata-01.am.local:2181,am-bigdata-02.am.local:2181. (kafka.zookeeper.ZooKeeperClient)[2019-05-10 16:07:25,551] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)[2019-05-10 16:07:25,616] INFO [ZooKeeperClient] Connected. (

I am tring to add a new host.but it fails with this message at the end "Command 30(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation." the following is the "/var/log/cloudera-scm-server/cloudera-scm-server.log"  10:58:25.348 AM INFO ServiceHandlerRegistry Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=butmah, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=false, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=25, skipPackageInstall=false, skipCloudConfig=false, proxyProtocol=HTTP, proxyServer=10.4.32.3, proxyPort=8080, proxyUserName=null, proxyPassword=REDACTED, hosts=[node1, node2, node3, node4, node5], existingHosts=[]}. 10:58:25.349 AM INFO CmdStep Executing c

Guys, i`ve faced this error during Hive installation.Deep log searching gave me this: Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.the main question is: where to configure CLASSPATH ? i`ve put mysql-jar into: # ls /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql*/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql-connector-java-5.1.46.jar /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/mysql-connector-java.jar#But no use, can anyone help me please on this ? Thanks. 

Hi, I would like to copy data from one Hbase cluster to another Hbase cluster. While copying I should copy 1000 records as first set and 1000 records as second set...and so on. For example, first set, I should copy only 1000 records, second set, I should copy 1000. How can I achieve using commands or scripts? Note: my Hbase table is huge. So, I would like to split and copy the data from one cluster to another (without moving to HDFS).

I have gone through the installation procedure several times trying to create 3 hosts at once but went back and decided to get the first host up.    This first host is up and I am trying to add additonal hosts.  On the wizard, Heardbeat is working, on the Install Parcels screen it shows CDH 6.2.0-1.cdh6.2.0.p0.967373 being downloaded/distributed/upacked/activated.   The issue is on the Host Inspector. 1 hosts are reporting with NONE CDH version   Also, none of the cloudera system accounts have been created on the new host.i.e. The user hdfs is missing on the following host   The agent seems to be running and it shows up as Managed when you show all hosts in the Manager.[root@usnj1cldrawrkr2 ~]# systemctl | grep cloudera  cloudera-scm-agent.service                                 

Hi Guys,we are trying to install the HDP 3.1.0 platform on-premise.We have finished the installation but ran into a problem setting up MariaDB for the Hive metastore.Hive explicitly lists MariaDB as supported:https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administrationand suggests using the driver class "org.mariadb.jdbc.Driver"When trying to setup "JDBC Driver Class" to "org.mariadb.jdbc.Driver" in Ambari of the HDP3.1.0 installation, then the start of the metastore fails with:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 60, in <module>HiveClient().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 966, in restartself.install(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/h

Hi All Cloudera suggests as best practice using S3 storage only for initial and final storage. The intermediate files will need to be stored in HDFS... In that case, we are still using HDFS but the cluster will only run during the batch ETL and then tore off daily. How we can pull S3 data into HDFS for each batch ETL job? and then store back the final results to S3? If Cloudera means to use distcp, how that might work for ETL batch jobs each time? It did not make sense to me using distcp... ThanksCK

i have a cluster of six nodes (centos) and installed cloudera on it. each node is having two partition of disk 30 gb  and 250 gb .but my cloudrea is using only 30 gb partition and i am running out of memory every week because of logs in namenode and data in datanode.  so how i will mount my 250gb partiotin for cloudera logs on namenode and data on datanodes.   thanks for any help.  

Hi Cloudera, I see a lot of these warnings in Impala Daemon logs: W0516 07:12:24.227567 1049 ShortCircuitCache.java:826] ShortCircuitCache(0x119fb869): could not load 1399296933_BP-76826636-10.197.31.86-1501521881839 due to InvalidToken exception.Does it indicate some bad configuration? What can I do to eliminate these warnings?Thanks