Resources

QuestionWhat are the available cluster types in Data Engineering Cluster? AnswerCurrently, Cloudera provides clusters that can run the following Data Engineering job types: - Hive on MapReduce- Hive on Spark- Spark on YARN- MapReduce2- PySpark Note:  Job submission for a cluster is restricted to the type of job for that various cluster.   For example, if you wanted to run just a MR2 job on a Spark-on-YARN cluster, you cannot submit this, even though Spark is configured to use YARN.   You would need to standup a separate YARN cluster for it to accept a MR2/YARN job.

QuestionI noticed the Cloudera Altus documentation is not avaialable at the standard location for Cloudera documentation at: https://www.cloudera.com/documentation.html Where is this documentation available? AnswerWhen you login with your Cloudera credentials to: https://console.altus.cloudera.com/ There is a pulldown available in the upper right corner for Support -> Documentation that links to the docummentation at: https://console.altus.cloudera.com/support/documentation.html Documentation will also be available with the standard Cloudera documentation in an upcoming release.

How within few minutes you can setup Rack Awareness through Ambari?Series1: Introduction to Rack Awareness can be viewed here:  How does my cluster look?Image below shows the hosts and the rack:You will see that the rack mentioned for each of the hosts in the test cluster is “/default-rack”. Which means Ambari (And HDFS and YARN) thinks of this cluster inside a single rack i.e. default-rack. In other words, it is not rack aware.Now let us examine the configurations that will change after we make the cluster rack-aware.Steps to verify Rack Awareness:1.Login to any node of your cluster. I am choosing node2.ssh root@node22.List the configuration files for rack awareness and view the current mapping of the system:ls –lrt /etc/hadoop/conf/topology* cat /etc/hadoop/conf/toplogy_mappings.dataQs: Why is node 2 not listed here?Because node 2 does not contain a datanode.3.As super user run the fsck and dfsadmin commandssu – hdfs hdfs fsck –racks hdfs dfsadmin –reportFor showing the relevant entr

Issue:HSI's (Tech Preview) component LLAP start fails in kerberized setup because of missing keytabs. When HSI is started, its component LLAP fails with below trace:INFO impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTEDINFO state.ConnectionStateManager: State change: CONNECTEDERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {}org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hiveat org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1]at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?]at org.a

The Cloudera Altus client (altuscli) requires Python version 3.6 or later. The reason for this requirement is because it relies on some new libraries which ships with Python 3.6 but not in earlier versions. It is not enough to have python3 installed on the machine, it also needs to be the default version, which could be checked by runningpython --version command.Documentation for CLI install 

SymptomsWhen creating an AWS Environment through Quickstart, the environment attempts to create, but shows Failed. Applies ToCloudera Altus Environments setup with Quickstart CauseThere can be several causes:Your limit of VPCs have been reached.Your Cloudwatch log group may already exist.Some other limit in AWS has been reached.You may not have proper permissions in AWS. Troubleshooting StepsBeneath the AWS Console -> CloudFormation, select the Stack that just failed to deploy.Expand Events and work from the bottom up to see what the event that failed first.If a limit was exceeded, open an issue with AWS support or request a limit increase beneath EC2 -> limits.If a authorization role is needed for your account, contact your AWS account owner to provide you with that role.Otherwise, correct any failure other failure cause you see (e.g. duplicate CloudWatch log group, etc), delete the stack, and attempt to Quickstart the environment again.  Referenceshttps://docs.aws.amazon.c

BackgroundCustomer attached bigger disks to expand the data node storage. If one disk physical fail, can also use this solution.Step by StepDecommission HDFS DataNode component on the hostIt takes hours to finish, depends on your existing data sizeDataNode is decommissionedTurn the host in maintenanceStop all components on the hostChange linux /etc/fstab mount new disks to existing mount points. If possible use uuid rather than disk device. uuid is much stable especially in cloud environment.Manually create the yarn log and local folders in the mount points. Because we don't reprovision the host, yarn won't create create these dirs in your configure, but try to reuse them.# for disk in /hadoop/disk-sd{d..j}/Hadoop> do> mkdir ${disk}/yarn/log && chown yarn:hadoop ${disk}/yarn/log> mkdir ${disk}/yarn/local && chown yarn:hadoop ${disk}/yarn/local> doneAfter change Linux disk mount configuration, start all components on the hostRecommission DateNodeTurn off main

Sometimes, in the face of initialize configuration and setup, it is easier to completely re-initialize an Accumulo installation than try to repair it. These steps are different than what might be found in the Apache ecosystem as the following steps are related to how Apache Ambari is used to install and configure Accumulo.Warning: the following steps (may) irreversibly remove all Accumulo related data. This includes table data, namespaces, table configuration, and Accumulo users. Do not perform these steps unless you are positive that you do not want to preserve any of the information.First, Accumulo must be stopped. This can be done via Ambari and verified using tools like `ps` on the nodes.Second, the Accumulo HDFS directory should be removed. This can be done by the HDFS superuser ("hdfs" by default) or the Accumulo user ("accumulo" by default).# sudo -u hdfs hdfs dfs -rm -R /apps/accumulo/dataNext, Accumulo needs to re-initialized using the command line tools. This command must be

Steps to Create Table in Hive on S3A with RangerCreate a bucket with a unique name, I've used "myhivebucket" and do not change any details in the permissionsComplete the "Create bucket" wizard by clicking on "create bucket" buttonMake the following entries in custom hdfs-site.xml fs.s3a.access.key = <access key>fs.s3a.secret.key = <access secret>fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystemTo retrieve the value for access key and secret, follow these steps: Login to https://aws.amazon.com/consoleClick on "Sign in to the console" tabLogin with appropriate credentialsOnce logged in, you should see your login name on the top right corner of the AWS pageClick on the drop-down arrow beside your login name and select "My Security Credentials"This should take you to a page titled "Your Security Credentials"From this page, collapse the option that says "Access Keys(Access Key ID and Secret Access Key)"You have to click on "create a new access key" because of Amazon limitati

First ... if you need background Azure Event Hubs go here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-overviewSecond, its handy to know why you need a SAS token and what you can do once you have one. See: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overviewWe will use an example configuration for and Event Hub service. The corresponding data for your service can be found on the details page of the share access policy you want to use. More information on this topic can be found here https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-shared-access-signature-authenticationFrom the shared access policy "hub-nifi"'s detail page we can get all the information we need to create our token from the connection string-primary key fieldEndpoint=sb://nifi-eventhub.servicebus.windows.net/;SharedAccessKeyName=hub-user;SharedAccessKey=2hmLYbJk2q5uZ2Yfyl0XSezXbxD+afO9ysh0Vsv4Xq8=;EntityPath=hub1A SAS token is s

When trying to add a policy that has many resource paths to Ranger using the API it can fail with the errorException [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'sort_order' at row 1Error Code: 1264Call: INSERT INTO x_policy_resource_map (ADDED_BY_ID, CREATE_TIME, sort_order, resource_id, UPDATE_TIME, UPD_BY_ID, value) VALUES (?, ?, ?, ?, ?, ?, ?) bind => [7 parameters bound]Query: InsertObjectQuery(XXPolicyResourceMap [XXDBBase={createTime={Thu Apr 13 11:42:38 UTC 2017} updateTime={Thu Apr 13 11:42:39 UTC 2017} addedByUserId={1} updatedByUserId={1} } id=null, resourceId=43, value=/tmp/129, order=128])This is caused by a limit in Ranger policies that can only contain a maximum of 128 resource paths in a single policy. The work-around would be to split the policy in to two or more policies each cont

How to clean up & recreate collections on Ambari Infra Sometimes you need to clean up an installation that went with issues and there are missing collections on Ambari Infra solr. In order to do this, go thry the next list of steps to clean up solr collections and recreate them from scratch. IMPORTANT: This steps will delete all collection data, so unless you are sure data can be removed dont perform these steps. 1) Switch adudit solr to OFF from Ranger -> Ranger Audit configuration. Restarted all affected.2) Stop Altas service3) From the ambari infra host remove all the collections using rest api$ kinit -kt /etc/security/keytabs/ambari-infra-solr.service.keytab infra-solr/....$ export SOLR_HOST=<solr host fqdn>## Ranger collection:$ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=ranger_audits"## Atlas collections:$ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=vertex_i

The goal of this article is to ingest log data from multiple servers (running MiNiFi) that push log data to a NiFi cluster. The NiFi cluster will listen for the log data on an input port and route to an HDFS directory (determined by the host name). This article will assume you are using Ambari for NiFi installation/administration.1) Set the nifi.remote.input.host and nifi.remote.input.socket.port of NiFi clusterReason: Allows NiFi cluster to use an Input Port (MiNiFi will push to a Remote Processing Group) * NiFi will listen using the Input Porta) In Ambari, go to NiFib) Choose the Configs tabc) Choose the Advanced nifi-propertiesd) Set the nifi.remote.input.host to your NiFi hostname and nifi.remote.input.socket.port to 10000e) Restart NiFi using Ambari2) In NiFi – Create a flow for incoming log data (listening [input port] for MiNiFi data)Reason: Listen for incoming log data and route the log data to an HDFS directorya) On the NiFi Flow Canvas, drag-and-drop an Input port (Name you

In non kerberized cluster the user name that we provide will be used to resolve group name. User to Group mapping will be done based on the value configured in the property hadoop.security.group.mapping. In kerberized cluster we will be passing kerberos principal instead of user name, now HDFS has to resolve the principle to a user name in order to perform User to Group mapping. This resolution of user name from kerberos principal is done based on the value configured in hadoop.security.auth_to_local. The value for hadoop.security.auth_to_local will be configured properly in a kerberized cluster. When the client is running in a kerberized cluster and trying to access non kerberized HDFS, kerberos principal is sent as username. When the username (kerberos principal in this case) is received by a non-secure NameNode, it tries to getShortName from the principal, when that happens, NameNode will try to apply the rules configured in hadoop.security.auth_to_local property.Since the cluster i

Phoenix secondary indexes are often misunderstood. Those coming from the relational world mistakenly apply the same prinples to apache phoenix. A simple data model will be used for this article. Airplane table with attributes carrier ID, Tail Number, Origin airport code, destination airport code, and flight date.The physical data model has been identified with primary access path of carreirID and TailNum, essentially the rowkey.note - it is important to understand the order the "primary access path" in which fields will be accessed. Here I have identified the first key in the access path is carrierID. If that is not the case, benefits from the underline database capabilities, HBase may be realized. Think of primary access path not as primary key but more as the core identified access pattern for reads & writes. Secondary indexes enrich & extend this functionality.What are Apache Phoenix secondary indexes? "Secondary indexes are an orthogonal way to access data from its pr

There are 4 options available for SmartSense Server to upload bundles to Hortonworks Datalake, depending on theHDP environment and customer preference. Below are more details about each options and how to configure the same.Option 1: Direct Connection from SmartSense ServerSmartSense Server (with embedded Gateway in 1.3.0 and above) having direct connection to Hortonworks Datalake.No additional configurations needed. SmartSense is preconfigured to connect to Datalake via HTTPS protocolOption 2: Dedicated Gateway Server with direct connection When using SmartSense version earlier to 1.3.0, we need a gateway host to be configured to upload bundles.Also in an SmartSense 1.3.0 environment with multiple HDP clusters running, they can be configured to share a dedicated Gateway Server to upload the bundles. If needed the Gateway Server can be located in DMZ area as well.Step 1: Install / Configure SmartSense Gateway Software.To install and configure SmartSense Gateway Software on the Dedicat

The intention of this article is to show how simple is to set up a HDP cluster which is suitable to execute neural network training on GPUs in a distributed fashion. Explanation of how distributed training works is beyond the scope of this article, but if you are interested you can read about that here. Training neural networks are computationally intensive, involves lots of matrix manipulations, such as multiplication, addition and element-wise operations, these calculations are highly parallelizible and can be significantly speed up by using GPUs instead of general purpose CPUs. Neural networks are often trained on very large datasets, where training data is too large to fit to one machine and therefore the data is partitioned across several machines. If the data is already on different machines, it might be desirable that during the training the data is not transferred across the computer network, but the processing is moved next to the data and train the neural network in a dis

ISSUE: Java Heap Space issue in Hive MR engine While working on a sample data set in hive. Query such as "select count(*)" was seen to fail with below error.Starting Job = job_1491603076412_0001, Tracking URL = http://krishna3.openstacklocal:8088/proxy/application_1491603076412_0001/Kill Command = /usr/hdp/2.4.2.0-258/hadoop/bin/hadoop job -kill job_1491603076412_0001Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 12017-04-07 22:18:09,736 Stage-1 map = 0%, reduce = 0%2017-04-07 22:18:46,065 Stage-1 map = 100%, reduce = 100%Ended Job = job_1491603076412_0001 with errorsError during job, obtaining debugging information...Examining task ID: task_1491603076412_0001_m_000000 (and more) from job job_1491603076412_0001Task with the most failures(4):-----Task ID: task_1491603076412_0001_m_000000URL: http://krishna3.openstacklocal:8088/taskdetails.jsp?jobid=job_1491603076412_0001&tipid=task_1491603076412_0001_m_000000-----Diagnostic Messages for this Task:

I’m constantly amazed by what powerful things I can do with Apache NiFi in such few steps. I often challenge myself by saying “self, I bet you couldn’t do X with NiFi”. My confidence was challenged yesterday on a long flight back from Peru to Atlanta when I realized I couldn’t perform OCR type tasks with NiFi as it stands today. Perturbed by this fact I set out to come up with a solution. Ultimately this lead me to create a NiFi Tesseract processor for performing OCR tasks natively from within Apache NiFi. It wasn’t really until I was finished that I realized the how useful this processor could be. The Apache Tesseract Processor would give me the ability to read anything from hand written doctors notes from healthcare systems to interpreting scanned children’s book images. In fact I chose to demonstrate the later by showing how to use Apache NiFi to perform OCR on an excerpt from Dr. Seuss's - "Cat in the Hat” and then feeding that resulting text from the NiFi Tesseract processor to t

Part 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-vie

Recipes framework capability to support HDFS and Hive mirroring was added in Apache Falcon 0.6.0 release and it was a client side logic. With 0.10 release its moved to server side and renamed as server side extensions as part of jira https://issues.apache.org/jira/browse/FALCON-1107.For any new mirror job to be submitted and managed, Falcon extensions should be used. Please refer https://falcon.apache.org/restapi/ExtensionEnumeration.html for more details.Supported DistCp options for HDFS mirroring in HDP 2.5:distcpMaxMapsdistcpMapBandwidthoverwriteignoreErrorsskipChecksumremoveDeletedFilespreserveBlockSizepreserveReplicationNumberpreservePermissionpreserveUserpreserveGrouppreserveChecksumTypepreserveAclpreserveXattrpreserveTimesHdfs mirroring job can be scheduled using extension as below:falcon extension -submitAndSchedule -extensionName hdfs-mirroring -file sales-monthly.propertiesContent of sales-monthly.properties file:jobName=sales-monthlyjobValidityStart=2016-06-30T00:00ZjobVali

SETUP:Kerberized cluster with Ranger installed. This article uses a latest HDP-2.6 cluster installed using Ambari -2.5Ranger based authorization is enabled with HiveZeppelin's authentication is enabled. You can use LDAP authentication, but for the purpose of this demonstration - I am using a simple authentication method and I am going to configure 2 additional users 'hive' and 'hrt_1' in zeppelin's shiro.ini .[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = admin, adminhive = hive, adminhrt_1 = hrt_1, admin# Sample LDAP configuration, for user Authentication, currently tested for single Realm[main]### A sample for configuring Active Directory Realm#activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm#activeDirectoryRealm.systemUsername = userNameA#use either systemPassword or hadoopSecurityCre

HDCloud for AWS general availability version 1.14.1 is now available, including six new HDP 2.6 and Ambari 2.5 cluster configurations and new cloud controller features.If you are new to HDCloud, you can get started using this tutorial (updated for 1.14.1). Officail HDCloud for AWS documentation is available here.HDP 2.6 and Ambari 2.5The following HDP 2.6 configurations are now available:For the list of all available HDP 2.5 and HDP 2.6 configurations, refer to Cluster Configurations documentation. Resource TaggingWhen creating a cluster, you can optionally add custom tags that will be displayed on the CloudFormation stack and on EC2 instances, allowing you to keep track of the resources that cloud controller crates on your behalf. For more information, refer to Resource Tagging documentation.Node Auto RepairThe cloud controller monitors clusters by checking for Ambari Agent heartbeat on all cluster nodes. If the Ambari Agent heartbeat is lost on a node, a failure is reported for that

The Phoenix Query Server is an HTTP server which expects very specific request data data. Sometimes, in the process of connecting different clients, the various configuration options of both client and server can create confusion about what data is actually being sent over the wire. This confusion leads to questions like "did my configuration property take effect" and "is my client operating as I expect".Linux systems often have a number of tools available for analyzing network traffic on a node. We can use one of these tools, ngrep, to analyze the traffic flowing into the Phoenix Query Server.From a host running the Phoenix Query Server, the following command would dump all traffic from any source to the Phoenix Query Server.$ sudo ngrep -t -d any port 8765The above command will listen to any incoming network traffic on the current host and filter out any traffic which is not to the port 8765 (the default port for the Phoenix Query Server). A specific network interface (e.g. eth0) can

In the event when active namenode fails over to standby node which becomes active, if the standby node is not started , users accessing hive via hive cli take long time to connect or the connection never happens.When checking debug log via following command :hive --hiveconf hive.root.logger=DEBUG,consoleFollowing message appears multiple times in the console output:INFO ipc.Client: Retrying connect to server: rp1.HORTON/172.26.97.110:8020. Already tried 0 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[500x2000ms], TryOnceThenFail]DEBUG retry.RetryUtils: RETRY 1) policy=MultipleLinearRandomRetry, exception=java.net.ConnectException: Connection refusedEven after changing the property "yarn.resourcemanager.fs.state-store.retry-policy-spec" to lower value , MultipleLinearRandomRetry policy still remains the same and hive cli does not connect or connection never happens.FIX:Inorder to fix this issue , please update yarn-site.xml file with following property. This would requi