Q&A

Hi,   We are getting the below error whenever we are trying to access HDFS from console. This error is coming after enabling the kerberos authentication in the cluster. We are getting this error when we are trying to access from a customized user 'edtuser'. But we are not getting error when we are accessing error from 'root' user. and 'hdfs' user.Please find the error below:-   [edtuser@nladfmrvu11 ~]$[edtuser@nladfmrvu11 ~]$ hadoop fs -ls /19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.2

Hi,   Is it possible to upgrade Enterprise version of Cloudera Manager to Express ? What happend if licence expired and I need to upgrade to newest Cloudera Manager that I can upgrade cluster to higher version?   

stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 43, in BeforeStartHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 37, in hook setup_unlimited_key_jce_policy() File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py", line 197, in setup_unlimited_key_jce_policy __setup_unlimited_key_jce_policy(custom_java_home=params.java_home, custom_jdk_name=params.jdk_name, custom_jce_name = params.jce_policy_zip) File "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py", line 235, in __setup_unlimited_key_jce_policy if jcePolicyInfo.is_unlimited_key_jce_policy(): File "/usr/lib/ambari-agent/lib/resource_management/core/resources/jcepolicyinfo.py", line 39, in is_unlimite

Hello All, I had a question around the upcoming CDSW 1.6 release. I understand from a recent webinar that the release would come out in the 2nd half of July of 2019. Has a more specific date been announced yet? In addition, will the new CDSW 1.6 be able to run on newer versions of RHEL other then what is currently supported by CDSW 1.5 (7.2, 7.3, 7.4, 7.5). Specifically, will it run on 7.6? Thanks,Robert

We already have a Kafka in our CDH cluster and we wish to set up a newer version of Kafka in the same cluster. I have tried adding new Kafka parcel in CM and it seems only one version of Kafka parcel can be enabled at the same time. I wonder whether it is possible. Thanks.

I am facing a StringIndexOutOfBoundsException while converting CSV to AVRO format using ConvertRecord 1.2.0 processor in NiFi. Can someone please help me.

this is the error   2019-08-21 16:13:10,828 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 408, in <module> NameNode().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 138, in start upgrade_suspended=params.upgrade_suspended, env=env) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 264, in namenode create_hdfs_directories(name_service) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 336, in create_hdfs_directori

 Caused by: java.sql.SQLException: Connections using insecure transport are prohibited while --require_secure_transport=ON. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)

in server.out file this is the error   Wed Aug 21 11:54:54 IST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.   in server.log  file this is the error   Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:146)a

Hello All, Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.    

I used configuration hive-site.xml what I already setted. than I reinstalled hadoop, hive and started hive-metastore. But I could not connect MariaDB with hive-client. Metastore log showed me MariaDB syntax error. I know hive support only MySQL. But I'm always using MariaDB and I've never gotten any problem.   I guess it is wrong espace query. I searched what is query. I thought hive ran this query. SELECT “THIS”.”NAME” AS NUCORDER0 FROM “DBS” “THIS” WHERE (LOWER(“THIS”.”NAME”) LIKE ? ESCAPE ‘\\' ) ORDER BY NUCORDER0    But I think this query using only one backslash character. so I though there is escape issue. Can I fix or avoid this problem?   This is log of exception. NestedThrowablesStackTrace:java.sql.SQLSyntaxErrorException: (conn=31) You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''\' ORDER BY NUCORDER0' at line 1at org.mariadb.jdbc.internal.util.exceptions.Exceptio

[root@sxd-cdh-nn parcels]# spark-submit --versionWelcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 2.4.0-cdh6.1.1/_/Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201Branch HEADCompiled by user jenkins on 2019-02-08T07:10:49ZRevision HEADUrlType --help for more information.     we have on top of this installed spark2 with 2.2    [root@sxd-cdh-nn parcels]# spark2-submit --versionWelcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 2.2.0.cloudera1/_/Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_201Branch HEADCompiled by user jenkins on 2017-07-13T00:28:58ZRevision 39f5a2b89d29d5d420d88ce15c8c55e2b45aeb2eUrl git://github.mtv.cloudera.com/CDH/spark.gitType --help for more information.   Could you please let me know how to uninstall spark-submit --version to 2.4?   we only want spark2 with 2.2 version    thanks Romesh

Hi   By mistake i deleted one file in ambari agent that is rm -f /run/ambari-agent/ambari-agent.pid   after that its showing this error   ambari-agent restartRestarting ambari-agentVerifying Python version compatibility...Using python /usr/bin/pythonambari-agent is not running. No PID found at /run/ambari-agent/ambari-agent.pidVerifying Python version compatibility...Using python /usr/bin/pythonChecking for previously running Ambari Agent...Checking ambari-common dir...Starting ambari-agentVerifying ambari-agent process status...ERROR: ambari-agent start failed. For more details, see /var/log/ambari-agent/ambari-agent.out:====================No handlers could be found for logger "root"Failed to start ping port listener of: Could not open port 8670 because port already used by another process:UID PID PPID C STIME TTY TIME CMDroot 23758 23754 0 Aug19 pts/4 00:00:17 /usr/bin/python /usr/lib/ambari-====================Agent out at: /var/log/ambari-agent/ambari-agent.outAgent

I would like to downgrade Ambari server (from 2.7.3 to 2.6.2). This is a new installation and since Ambari was never upgraded, there are no database back up files (as referenced here: https://community.hortonworks.com/questions/175812/downgrade-procedure-from-ambari-version-2610-to-lo.html). Therefore, I modified the instructions as follows (omitting the old database import steps and re-running ambari-server setup). Will this work without negatively affecting the other HDP 2.6.4 components? If not, is there a procedure to downgrade Ambari in a new installation?1. Stop the ambari-server.# ambari-server stop2. Stop all the ambari-agents on all hosts.# ambari-agent stop3. Download the ambari.repo of the downgraded version on all agent hosts and ambari server host.Example:# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo 4. (omitting this step) Then take a backup of "/etc/ambari-server/conf" and "/etc/ambari-agent

Good morning guys, thanks in advance for your help!I have a project that fails. I'm trying to restart all the services manually but havent been able to. I have a few questions and I'd really appreciate if you can give me some guidance because at this moment I'm kinda stuck.1. How do I check what services need to be "up and running" before restarting the next one? Is there any place where I can see the dependency?2. Do I need 2 ZooKeeper servers up and running? The first one is running in localhost but the 2nd one runs in a different machine. If I actually need them both, how can I check what was wrong in the second one?

Hi community, looking at security, I am in process of disabling any interfaces without proper authentication / authorization (or even encryption). I came across the debug web UIs of Cloudera Management services.   According to https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html, the debug WebUIs can be disabled by setting the port property to -1. This works for Reports Manager (8083), Event Server (8084), Navigator Audit Server (8089), Telemetry Publisher (10111).   This does not work, however, for  Service Monitor (8086 / 9086 TLS), Activity Monitor (8087 / 9087 TLS), Host Monitor (8091 / 9091 TLS). Setting port to -1 leads to non-starting services without a proper ERROR in the log file.   Cloudera Manager agent even tries to check, if the server successfully bound to port -1 and runs into errors: [15/Aug/2019 12:06:03 +0000] 65646 Thread-14 process ERROR [918-cloudera-mgmt-HOSTMONITOR] Failed port check: Command '['ss', '-

Hello Team, I am using RHEL 7 and followed documentation to install CFM https://docs.hortonworks.com/HDPDocuments/CFM/CFM-1.0.0/installation/content/add-nifi-service.html   CFM package has been installed as seen in screen short.  I checked both folders "/opt/cloudera/csd", "/opt/cloudera/parcel-repo" both contain the following files-rw-r-----. 1 cloudera-scm cloudera-scm 41 Aug 16 11:12 CFM-1.0.0.0-el7.parcel.sha-rw-r-----. 1 cloudera-scm cloudera-scm 1735385749 Aug 16 11:12 CFM-1.0.0.0-el7.parcel-rw-r-----. 1 cloudera-scm cloudera-scm 66350 Aug 16 11:12 CFM-1.0.0.0-el7.parcel.torrent I am stuck at Add and Configure CFM Services because I cannot see Nifi in the services list.  How should install Nifi service now? Hemanth  

We have a need to enable public IPs on the master nodes due to an requirement in Azure when adding instances to a load balancer pool.  Our cluster is accessed via private IPs only,  outside of this requirement.     Within the cluster bootstrap file, we enable public IPs for the master instance groups.  During deployment, director will throw an error similar to the following for some but not all master nodes:   ', errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.0.14.15', hostAddress=Optional.of(/10.0.14.15)}, port=Optional.absent(), url=Optional.absent()}, BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='52.232.245.168', hostAddress=Optional.of(/52.232.245.168)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}}   [2019-07-10 02:49:45.507 +0000] INFO [p-d4ed238466ce-WaitForSshToSucceed] 5f79b1c9-b7d6-41d0-a933-0c

Altus Director 6.3 Bootstrap of CDH6.3 clusters are failing due to failed activation of parcels. We've successfully deployed earlier versions of CDH 6.x.  On all nodes the agent logs show the following error for the same parcel:  [14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'spark', '-K', 'UMASK=022', '--home', '/var/lib/spark', '--comment', 'Spark', '--shell', '/sbin/nologin', 'spark'][14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Ensuring correct file permissions for new parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813.[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db hue hue[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db'][1

I have Ambari 2.7.3 set up on an Ubuntu 16.04 node and it has deployed a three-node cluster node.Ambari Metrics was not working correctly, so I removed it and then re-installed it through the interface. This appeared to clear up several problems I was seeing in error logs but I'm still not able to see any metrics from any cluster nodes.This seems to be connected to a problem showing up in the Ambari server log which keeps repeating the same error message:2019-08-08 11:56:05,936 ERROR [ambari-client-thread-1655] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 503 for URL: http://<metrics_collector_node_fqdn>:6188/ws/v1/timeline/metrics?metricNames=swap_free._avg&appId=HOST&startTime=1565286966&endTime=1565290566.So for some reason the node where the metric collector is running is reporting a 503 error. Yet if I look at the logs for the collector and also for each metrics monitor agent, they aren't reporting any errors. I

Are HBase Quotas supported in CDH 6.x?   The hbase.quota.enabled property defaults to false but this property is not listed under the HBase service configuration in Cloudera Manager 6.

Having a problem with HDFS NFS, addressed on another site where it is recommended to set hdfs-site.xml like...<property> <name>dfs.namenode.accesstime.precision</name> <value>3600000</value> <description> The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. </description> </property>Am confused about what exactly "access times for HDFS" means / is. Looking at the hadoop docs, was still not able to determine. Could someone give better understanding as to what this is doing? Also where is the nfs3 daemon log file ?

Hi All,   I am using Hue--> Hive editor to submit a query on an external table and View created on top an Hbase table. I have 3 instances of Hue running in my cluster(1 among them as Load balancer).   1.When a submit a query from Beeline on this external table and View. It works perfectly fine. 2. When i submit the same query from Hue it doesn't work.(simple query like: select * from hbase_view limit 10) 3. hbase-site.xml, hive-site.xml all have zookeeper.quorum defined correctly(I have 5 zookeeper server instances, so it has 5 nodes in zookeeper.quorum properties). Clientport:2181. That also looks fine.   But I am getting below error in hive-server2.log file when a query is submitted from Hue.   Instead of trying to connect to one of the zookeeper host, it is trying to connect to localhost4/127.0.0.1:2181. This is not the case when query runs successfully, it takes any of the zookeeper node for client connection.   2019-08-13 22:10:23,353 INFO  org.apa

Let's assume that I have a table   test_tbl which was created through impala-shell.I have a few questions :REFRESH the table only when I add new data through HIVE or HDFS commands ? That is when I am doing insert into ...through impala-shell no need for refreshing ?INVALIDATE METADATA  of the table only when I change the structure of the table (add columns, drop partitions) through HIVE? DROPping partitions of a table through impala-shell (i.e alter table  .. drop partition .. purge).  Do I have to do REFRESH or INVALIDATE METADATA?DROPping partitions of a table through impala-shell . How can I compute the new stats of the partitioned table?  Compute incremental stats OR Drop Incremental stats before dropping partition ? Thanks in advance. 

I could see the below configurations coming from desired config output from configuration dump through ambari.usersync-propertiesslider-clientranger-sitehst-common-confcapture-levelsBut when i used to get the properties output usings config.py its throwing the below error.File "/usr/lib64/python2.7/site.py", line 556, in main()File "/usr/lib64/python2.7/site.py", line 538, in mainknown_paths = addusersitepackages(known_paths)File "/usr/lib64/python2.7/site.py", line 266, in addusersitepackagesuser_site = getusersitepackages()File "/usr/lib64/python2.7/site.py", line 255, in getusersitepackagesUSER_SITE = get_path('purelib', '%s_user' % os.name)File "/usr/lib64/python2.7/sysconfig.py", line 437, in get_pathreturn get_paths(scheme, vars, expand)[name]File "/usr/lib64/python2.7/sysconfig.py", line 428, in get_pathsreturn _expand_vars(scheme, vars)File "/usr/lib64/python2.7/sysconfig.py", line 156, in _expand_vars_extend_dict(vars, get_config_vars())File "/usr/lib64/python2.7/sysconfig.py"