Q&A

Hi, I just installed the CDH 6.1 on my VMs, and I've tried to access to oozie web ui. But I failed with the message below. { "errorMessage": "org.apache.jasper.JasperException: PWC6033: Error in Javac compilation for JSP\n\n PWC6199: Generated servlet error:\npackage javax.servlet does not exist\n\n PWC6199: Generated servlet error:\npackage javax.servlet.http does not exist\n\n PWC6199: Generated servlet error:\npackage javax.servlet.jsp does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.oozie.sla.service does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.oozie.service does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.oozie.service does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.oozie.service does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.oozie.service does not exist\n\n PWC6199: Generated servlet error:\npackage org.apache.

Using Hue™ 4.1 on CDH 5.16 I only see the Delete Forever button in File Browser but not the Move to Trash even though Use Trash is enabled in HDFS configuration. How to enable this in HUE?

Hi,After I have upgraded to CM/CDH 6.1 from 5.16.1, my hosts randomly and periodically having "Unknown Health" for about a few seconds and then go back to green. I have not seen/found any WARNING nor any ERROR in any logs from hosts or any services.The entire cluster works without any issue, I have run host inspection and network inspection without any problem. Also, synced time/date a few times just in case but still I can watch my hosts (also services because of the hosts) going grey with "Unknown Health" and back to green randomly for few seconds. Cloudera Management Service is on one server with 14 cores and 28G memory. I have checked this server activity, it is pretty idle, so the cluster is not a busy cluster. Either way, this is the heap size for the monitorings:Java Heap Size of Activity Monitor in Bytes: 2GBJava Heap Size of Alert Publisher in Bytes: 256MBJava Heap Size of EventServer in Bytes: 1GBJava Heap Size of Host Monitor in Bytes: 4GBJava Heap Size of Service Monit

We upgrade our Ambari from 2.6 to 2.7.3, after the upgrading, our dashboard metrics keep showing the loading animation without any data. However, Ambari metrics service is ok and metrics page on each components also works.I have checked log file on host of Ambari server, but no exception found. And i also tried to clear Hbase root and tmp dir, not work neither.Any suggestion is appreciate, thanks in advance!

i following bellow steps to setup sso:Install Knox.Configure Ambari Authentication for LDAP/AD.Configure an LDAP/AD Identity Provider (IdP).Enable Knox SSO using the Ambari CLI.after this step sso worked me for redirect to ambari ui.in the next step when i add the bellow property to core-site.xml, restarting hdfs failed:"hadoop.http.authentication.type": "org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler”"hadoop.http.authentication.public.key.pem": “$SSOPUBLICKEY""hadoop.http.authentication.authentication.provider.url": "$SSOPROVIDERURL"when add this property (hadoop.http.authentication.type) to hdfs happening error.hdfs log file:2019-02-26 16:28:06,366 ERROR datanode.DataNode (DataNode.java:secureMain(2870)) - Exception in secureMainjava.io.IOException: Problem starting http server Caused by:         at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165)        at org.apache.hadoop.hdfs.server.da

Hello Friends:   On a relatively new installation of CDH6.1 (parcels) with one node for CDH manager and a second node for Master and Slave services (combined), I'm getting this error:   org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"' after running this:   user$ /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/bin/parquet-tools \ cat hdfs://tmp/1.parquet Here is the output of hadoop classpath:   /etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop

Hi,I need to handle an HTTP request and I need to respond an HTML file with NiFi.My flow is likeHandleHttpRequest -> ExecuteSQL -> HandleHttpResponse.I need to query our database to show proper HTML result.My problem is HandleHttpRequest runs twice for each request when triggered by browsers such as Chrome, Firefox. HandleHttpRequest runs once if I use Postman or SoapUI.I've tried to eliminate the second call by using HashContent and DetectDuplicate but it makes some of our requests fail. I've investigated the issue and I've found that Chrome and Firefox creates an additional request for displaying favicon and sometimes some other reasons. (Link)I need to find real request and I need to eliminate other requests. The real request can be the second one depending on the browser status. Browser decides when to create new requests and their orders.So, my question is how can I run my flow exactly one time when user creates request from their browser? Because I do some operations on the

./import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=/etc/krb5.cont f -Djava.security.auth.login.config=/etc/atlas/conf/atlas_jaas.confUsing Hive configuration directory [/etc/hive/conf]Log file for import is /var/log/atlas/import-hive.loglog4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.Exception in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650)Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306) at org.apache.hadoop.security.UserGroupInformation.ensureIni

I have a GetFile processor followed by a PutSFTP. I simulate SFTP connection issue, and then would like to replay the same flow. However, I can't replay at GetFile as it shows "Cannot replay data from Provenance Event because the event does not contain the required Content Claim". PutSFTP has no data provenance event as well since the SFTP failed. Please advise.

I searched in google but not find it,is it possible to create link between HDFS folder to local folder?examplewe want to create link between folder_1 in HDFS to /home/hdfs_mirror local folderHDFS folder:su hdfs $ hdfs dfs -ls /hdfs_home/folder_1Linux local folder:ls /home/hdfs_mirror

Hello, I have been working on upgrading our cluster from CDH5.15 to CDH6.1.0 version. We have CentOS 6.10 operating system and Python version 2.6.6 on our boxes. According to the documentation the python version is compatable for the OS vesion (https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html#os_requirements) "Cloudera Enterprise 6 requires Python 2.6 or higher on RHEL 6 compatible operating systems".But, when trying to upgrade and launch services (CDH 6.1), HUE service fails with the following error message, 2019-02-25 22:20:31,504 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: ++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hadoop-kms ++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/parquet ++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/parquet ++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.

recently I have upgraded to HDP 3. If I run the stale config, it is not working(failing) to one of the master server(NN runnnig) all other nodes are working fine if I apply stale config.Whatever the services running in this host are failing.even i have restarted ambari server and ambari agent but no luck.Please helpThanks in advance..

I am encountering the error org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle: SessionHandle while running the %jdbc(hive) interpreter in Apache Zeppelin version 0.8 on HDP (Hortonworks Data Platform) version 3.0.1.

i'm currently using cloudera community editition 5.15.0 from past 90 days I had External authentication configured over the Cloudera community edition , after few days of utilization i'm noticing that external authentication is not avaliable under the Administrattion-> Security Options  Does External Authentication comes only will limited period of time ?  please advice 

I have Hadoop classpath defined but when I use it as per documentation I am getting error [root@hadoop1 conf]# hadoop classpath/usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/lib/*:/usr/hdp/2.6.5.0-292/hadoop/.//*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/./:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//*:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/*:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//*:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:ojdbc6.jar:/usr/hdp/2.6.5.0-292/tez/*:/usr/hdp/2.6.5.0-292/tez/lib/*:/usr/hdp/2.6.5.0-292/tez/conf[root@hadoop1 conf]#[root@hadoop1 conf]#[root@hadoop1 conf]#  echo $JAVA_HOME/usr/jdk64/jdk1.8.0_112[root@hadoop1 conf]#[root@hadoop1 conf]# javac -cp  `hadoop classpath`  TestHbaseTable.javaTestHbaseTable.java:3: error: package org.apache.hadoop.hbase does not existimport org.apache.had

Good evening. This message I receive after add two new hosts to the hadoop cluser.Tehnical info:Nodes - Debian GNU/Linux 8.9 (jessie)cloudera-manager-agent 5.13.0-1.cm5130.p0.55~jessie-cm5 amd64cloudera-manager-daemons 5.13.0-1.cm5130.p0.55~jessie-cm5 allEarlier we add nodes using web interface. But this 2 nodes was added by hand:1. Add repo cloudera2. Install packages from repo3. Configure /etc/cloudera-scm-agent/config.ini:server_host=hadoop-server.test.comAfter start clouder agents, I saw tho strangeness In cloudera web interface:1. All Hosts - my new host - CDH Version is None2. Error from topic: "The hostname and canonical name for this host are not consistent when checked from a Java process."I dont know is connected to each other this issues or not. I try resolve second issue:python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'regensburg.test.com 1.1.1.1This command use from /etc/cloudera-scm-agent/config.ini file. Re

Hi, I need to get the health check report of server, which is found in cloud era manager. Using APIs I need to get those details from cloud era. I need the information of list of Web services that is to be enabled to proceed on using APIs for retrieval of information from cloud era. Thanks.

We are running hive with udf vs spark comparison.The aim is to choose a faster solution for encrypting/decrypting data.Tech stack we are using is as follows:HDP 2.6.5Hive 1.2.1000Spark2 2.xYARN + MapReduce2 2.7.3Data are stored on HDF as csv files: Data set 1 (big data set): 1M rows, 44 columns, 19000 unique customers;Data set 2 (small data set): 25k rows, 44 columns, 494 unique customers;22 columns are being encrypted using unique key for unique customer.UDF on Hive is written in Java. Function on Spark is written in Scala. Encrypting decrypting functions are basically the same.To compare those two frameworks we run count (to decrypt the whole table and get one value as output).Hive is run from beeline command line. Spark is run from Zeppelin.Keys are stored in a file on HDFS.Spark (results in Zeppelin):The average time for querying encrypted small data set with decryption: 11.8 sThe average time for querying encrypted big data set with decryption: 54.8 s---The average time for queryi

Hi, Currently we are using NIFI 2.1, and one workflow is to running SQL from DBCPConnectionPool every 10 minutes to extract data, then do the following logic.Due to the source is the MySQL database in another data center, sometimes the network connection is broken due to network or firewall issues. We would like to have email notification when this kind of thing happens in NIFI.Right now, our first Executor SQL processor, when there is a network connection issue to the source MySQL DB, the only error coming out is in the bulletin or log, which hardly monitored by a human.I wonder if we can design something in the NIFI processor, or it is natively available in NIFI, will trigger Email notification when the DBCP connection pool is broken.Thanks

I can select a particular version from a table , but how can I select a range of versions ? e.g say I want to select all the rows from timestamp 1551108073557 to 1551108018724 ?hbase(main):005:0> scan 'PERSON',{NAME => 'student',VERSIONS => 5}ROW COLUMN+CELL r1 column=student:student_name, timestamp=1551108086986, value=John r1 column=student:student_name, timestamp=1551108073557, value=Mary r1 column=student:student_name, timestamp=1551108037609, value=Nancy r1 column=student:student_name, timestamp=1551108018724, value=Ram r1 column=student:student_name, timestamp=1551108002231, value=Sam1 row(s) in 0.0190 secondshbase(main):006:0>hbase(main):007:0*hbase(main):008:0* get 'PERSON'

Hello All, Need some suggestions on the exact reason of below heath tests getting failed multiple times, almost 3 times every day and generates atleast 3-4 alerts each time. 1. HIVEMETASTORE_CANARY_HEALTH2. REGION_SERVER_AUDIT_HEALTH3. IMPALAD_QUERY_MONITORING_STATUS4. HIVESERVER2_SCM_HEALTH5. NAME_NODE_AUDIT_HEALTH6. NODE_MANAGER_WEB_METRIC_COLLECTION Any help or suggestion to permanently fix these alerts would of great help. Also, if anyone could also guide to reach root cause of this would also be helpful Thankssnm1523

Hi All, We have just productionized our Nifi flows which is a real time streaming application . We have a cluster of 2 Nodes and each node has 6 processors and a high memory of 28 GB. On the Nifi side the memory has been set to 24 GB. Moreover , we tend to see a queue size reaching to around 90,000 or more. For this I have set the property nifi.queue.swap.threshold to 90000 . We also have a disk space of 200 GB. Despite having such a powerful configuration , we still get flow file repository failed to update. I understand that initially the JVM memory is uitilized for the flow file processing and if that gets overused , it spills over to disk space . Our disk space is also pretty high at 200 GB however we still keep on seeing this error.Any pointers or suggestion in this space will be of great help.

HI All , I have created 2 nifi instances on Google Cloud Platform. I now plan to bring them onto a cluster. However I am getting this errorserver.quorum.QuorumPeerConfig does not have the form host:port or host:port:port or host:port:port:typeI have checked my zoo keeper properties file again and again but not sure whats the issue. My zoo keeper properties file is as follows . I am not sure as to why this issue is cropping up. My nifi servers are by the name nifi-server2 and nifi-server1 and ports are 2888 and 3888 . I have followed the documentation by Pierre Villard.server.1=nifi-server2:2888:3888server.2=nifi-server1:2888:3888# server.3=nifi-node3-hostname:2888:3888Can someone please help me here.

I am getting error while installing namenode where it says "-safemode get | grep 'Safe mode is OFF''"Ambari:2.7.0.0HDP:3.0I got the below error and I ran :>hdfs dfsadmin -safemode leave>Output:safemode: Call From devaz01/10.161.137.4 to devaz01:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefusedthen I ran>chmod -R 644 /var/log/hadoop/hdfsSince then I am getting: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/3.0.1.0-187/hadoop/bin/hdfs --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start namenode'' returned 1. ERROR: Unable to write in /var/log/hadoop/hdfs. Aborting.019-02-08 03:31:21,394 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://machine:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From machine/xx.xxx.xxx.x to machine:8020 fail

Basic error on the created notes is that the resolve.conf fails to parse. in looking at a failed node, the resolve.conf file was poorly created - see below:    # Generated by NetworkManagernameserver [an ip address]search     Note that there's nothing after above "search". I fixed this issue by running the setup script on the Altus host.  Problem is that the generated nodes fail to create well constucted "resolve.conf" files - which then leads to the parse error.