Community Articles

ProblemThis problem happened on HDP 2.5.3 when running Spark On HBase. Here is the error seen in the application log:... 17/04/11 10:12:04 WARN RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid17/04/11 10:12:05 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)17/04/11 10:12:05 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refusedat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) ...SolutionTo fix that problem ensur

Problem Statement: User cannot modify or start restricted components such as getHDFS in NIFI 1.1 with Ranger enabled.Solution: In Ranger you must grant /restricted_components permissions to the user in order for them to have access to the restricted components in NIFI.

To remove a already installed Grafana, following needs to be done:1. Stop the AMS service from Ambari UI.2. Execute following curl API commands to delete Grafana:# curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop HDFS via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong><CLUSTERNAME></strong>/services/AMBARI_METRICS/components/METRICS_GRAFANA# curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop HDFS via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong><CLUSTERNAME></strong>/services/AMBARI_METRICS/# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong&

How to increase Read time out value from Ambari stand alone view server to Ambari main serverStand alone Ambari view server makes below call to fetch configurations from Remote server.http://REMOTEAMBARI:8080/api/v1/clusters/CLUSTER_NAME?fields=services/ServiceInfo,hosts,ClustersIf this takes more than 10 seconds you will be exception in the logs. because of that non of the views would work.Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClie

some times if you run some curl calls expecting to fix some other issues - it may turn out to that Ambari UI does not even load the UI.Running the below call would fix the issue and load the UI.curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"\"}"}'http://AMBARIHOST:8080/api/v1/persistNote: Please note that these call should run with caution. contact Hortonworks support team.This is tried and tested with Ambari-2.4.x

QuestionWhat steps should a Cloudera Altus administrator take if a Cloudera Altus end-user leaves their company, or should no longer have access to Cloudera Altus? AnswerIf a user leaves their company, a Cloudera Altus administrator will need to delete the Cloudera Altus CLI Keys and remove user access from the Web UI to prevent unauthorized access.

QuestionWhere are Cloudera Altus Clusters created and how are Data Engineering jobs run? AnswerClusters are spun up and reside within customer's public cloud account. For example, in case of AWS, EC2 instances are spun-up and managed by Cloudera Altus in the subnet and VPC specified by customer created environment; These clusters handle specific Data Engineering workloads (Hive-on-Spark, Spark-on-YARN, MR2 on YARN, etc). 

QuestionWhat are the various Cloudera Altus interfaces available and how can I access them? AnswerCloudera Altus is accessible via command line interface (CLI), console / graphical user interface (GUI) and via a software development kit (SDK) for use with Java. The Cloudera Altus Web console is available via: https://console.altus.cloudera.com/ Instructions for installing the Cloudera Altus CLI is available here (link to Cloudera Altus CLI Documentation) Additional details for the Cloudera Altus SDK and coding examples are available here (link to Cloudera Altus SDK) Additional details on interfaces are available in the Cloudera Altus Interfaces section of the Cloudera Altus Documentation. 

QuestionAre there any current limitations with Cloudera Altus? AnswerDefault limit for commercial have the following default soft-limits: Up to 10 active clusters at any given timeLimit of 100 usersLimit of 10 defined Cloudera Altus environmentsIf there are needs required that will utilize beyond the current soft-limits, please reach out to Cloudera Support to request an increase. Users that are given access as part of a free trial will have no ability to create additional users and may have smaller limits. 

QuestionCan I change an environment once created in Cloudera Altus? AnswerBy design, Cloudera Altus environments are not modifiable once created. To make a change to the environment, we recommend to clone the environment into a new one with the appropriate changes.

QuestionCan I use a bootstrap script to further configure the cluster?  AnswerInstance bootstrap scripts are executed on all the cluster instances immediately after startup before any services are configured and started.  You can use it to install additional OS packages or add application dependencies (e.g. copy a jar to the cluster, etc). This is not intended to be a mechanism for cluster configuration.

QuestionIs logging enabled by default for jobs run in Cloudera Altus Data Engineering? AnswerWhile we recommend enabling logging for troubleshooting Data Engineering jobs, It is not enabled by default.   Details for enabling logging within the Cloudera Altus Environment Setup is available in the online doc Cloudera Altus Environment Setup:https://console.altus.cloudera.com/support/documentation.html?type=topics&page=ag_dataengr_get_start_env_setupNOTE: If logging is not enabled, cluster bundles that are automatically generated during shutdown of a Cloudera Altus-managed cluster will be lost.

QuestionWhat are the minimum and maximum cluster sizes? AnswerThe minimum cluster size is 3 worker nodes, 1 Master node, and 1 Cloudera Manager node, for a total of five nodes.  The maximum cluster size varies based on cluster type and public cloud. Further information about this could be found in this  Cloudera Altus Documentation section NOTE: Each cluster spun up by Cloudera Altus requires 2 additional (master) nodes, one for CM and the other for Master services used for each cluster.  These 2 nodes are a hard-requirement per cluster as they're required for a functional cluster and will need to be accounted for when calculating the public cloud instance limits. 

Use Cases: Seperate out hiveservers for batch jobs which can ensure user jobs or too many connections from users dont effect batch workloads or one groups workloads dont effect other group workloads.Secure hive connections through Knox which only works with http mode for hiveservers2.Custom configurations for different workloads such as different engine's MR or tez or anything else. Method:Use ambari config groups to customize the default hive.server2.zookeeper.namespace to something else other than default value “hiveserver2”Reference: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hadoop-high-availability/content/ha-hs2-service-discovery.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-operations/content/using_host_config_groups.htmlCaveat: The beeline connection string will be different from the default one. The new one needs different value for the zooKeeperNamespace which can cause confusion. Example: I will be using Ambari 2.4.2 with HDP 2.3.2 b

These checklists help ensure that you avoid unexpected problems when adding new hosts to an HDP cluster. Preparation checklist (what to do before adding the new hosts via the Ambari Add Host wizard): Have you followed the guidance in the Ambari installation documentation (e.g. example install doc is here)? It is recommended to follow the instructions for the HDP version that is installed on the cluster (go to http://docs.hortonworks.com). Does your file system partitioning scheme, storage options, memory, network, and power configurations match what is used for other nodes on the cluster? It is good to either follow the pattern of what is already on the cluster or to follow recommendations from docs.hortonworks.com (see example doc here). Compare and verify that settings and configurations match, including: THP, selinux, ssh keys, user integration, LDAP/AD setup, SSSD integration, drive layouts, etc. Have you verified that JAVA_HOME is set properly with same Java version across

QuestionWhat type of access Cloudera Altus Cluster provides for Cloudera Manager user? AnswerRead-only access. User (guest by the default) can't modify or change settings or configurations in Cloudera Manager. 

QuestionCan I choose the Linux distribution that a Cloudera Altus Cluster will be running on?  AnswerCurrently, the selected Linux distribution for deployment in Cloudera Altus Clusters is based on CentOS 7.x 64-bit.  At this time, no other Linux distrubtions are available for use with Cloudera Altus.  

QuestionDoes Cloudera save AWS user credentials (AWS Access Key ID or AWS Access Key)? AnswerNo, Cloudera doesn't save AWS user credentials. If Cloudera Altus environment is set-up using the Cloudera Altus Quickstart, the user is prompted for these credentials, but they are never sent to Cloudera Altus. They are used by the web application residing within the user's brower that sends commands directly to AWS to create necessary resources using a CloudFormation script.In case of the Environment being created via Cloudera Altus Wizard, credentials will never need to appear inside the Cloudera Altus console. User needs to create necessary resources in AWS console and then grant access to them using AWS cross account access mechanism, which does not involve explicit key management.

QuestionDoes Cloudera have access to the customer data in Cloudera Altus Clusters? Answer Spoiler (Highlight to read)  No.  Cloudera Altus clusters are setup with the expectation that data will reside on the cloud provider's object storage (AWS S3, Azure ADLS).  When data resides in this object storage (which exists inside a customer account within AWS or Azure), the Cloudera Altus service does not have access to this data. 

QuestionWhat are the available cluster types in Data Engineering Cluster? AnswerCurrently, Cloudera provides clusters that can run the following Data Engineering job types: - Hive on MapReduce- Hive on Spark- Spark on YARN- MapReduce2- PySpark Note:  Job submission for a cluster is restricted to the type of job for that various cluster.   For example, if you wanted to run just a MR2 job on a Spark-on-YARN cluster, you cannot submit this, even though Spark is configured to use YARN.   You would need to standup a separate YARN cluster for it to accept a MR2/YARN job.

QuestionI noticed the Cloudera Altus documentation is not avaialable at the standard location for Cloudera documentation at: https://www.cloudera.com/documentation.html Where is this documentation available? AnswerWhen you login with your Cloudera credentials to: https://console.altus.cloudera.com/ There is a pulldown available in the upper right corner for Support -> Documentation that links to the docummentation at: https://console.altus.cloudera.com/support/documentation.html Documentation will also be available with the standard Cloudera documentation in an upcoming release.

How within few minutes you can setup Rack Awareness through Ambari?Series1: Introduction to Rack Awareness can be viewed here:  How does my cluster look?Image below shows the hosts and the rack:You will see that the rack mentioned for each of the hosts in the test cluster is “/default-rack”. Which means Ambari (And HDFS and YARN) thinks of this cluster inside a single rack i.e. default-rack. In other words, it is not rack aware.Now let us examine the configurations that will change after we make the cluster rack-aware.Steps to verify Rack Awareness:1.Login to any node of your cluster. I am choosing node2.ssh root@node22.List the configuration files for rack awareness and view the current mapping of the system:ls –lrt /etc/hadoop/conf/topology* cat /etc/hadoop/conf/toplogy_mappings.dataQs: Why is node 2 not listed here?Because node 2 does not contain a datanode.3.As super user run the fsck and dfsadmin commandssu – hdfs hdfs fsck –racks hdfs dfsadmin –reportFor showing the relevant entr

Issue:HSI's (Tech Preview) component LLAP start fails in kerberized setup because of missing keytabs. When HSI is started, its component LLAP fails with below trace:INFO impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTEDINFO state.ConnectionStateManager: State change: CONNECTEDERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {}org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hiveat org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?]at org.a

The Cloudera Altus client (altuscli) requires Python version 3.6 or later. The reason for this requirement is because it relies on some new libraries which ships with Python 3.6 but not in earlier versions. It is not enough to have python3 installed on the machine, it also needs to be the default version, which could be checked by runningpython --version command.Documentation for CLI install 

SymptomsWhen creating an AWS Environment through Quickstart, the environment attempts to create, but shows Failed. Applies ToCloudera Altus Environments setup with Quickstart CauseThere can be several causes:Your limit of VPCs have been reached.Your Cloudwatch log group may already exist.Some other limit in AWS has been reached.You may not have proper permissions in AWS. Troubleshooting StepsBeneath the AWS Console -> CloudFormation, select the Stack that just failed to deploy.Expand Events and work from the bottom up to see what the event that failed first.If a limit was exceeded, open an issue with AWS support or request a limit increase beneath EC2 -> limits.If a authorization role is needed for your account, contact your AWS account owner to provide you with that role.Otherwise, correct any failure other failure cause you see (e.g. duplicate CloudWatch log group, etc), delete the stack, and attempt to Quickstart the environment again.  Referenceshttps://docs.aws.amazon.c