Community Articles

BackgroundCustomer attached bigger disks to expand the data node storage. If one disk physical fail, can also use this solution.Step by StepDecommission HDFS DataNode component on the hostIt takes hours to finish, depends on your existing data sizeDataNode is decommissionedTurn the host in maintenanceStop all components on the hostChange linux /etc/fstab mount new disks to existing mount points. If possible use uuid rather than disk device. uuid is much stable especially in cloud environment.Manually create the yarn log and local folders in the mount points. Because we don't reprovision the host, yarn won't create create these dirs in your configure, but try to reuse them.# for disk in /hadoop/disk-sd{d..j}/Hadoop> do> mkdir ${disk}/yarn/log && chown yarn:hadoop ${disk}/yarn/log> mkdir ${disk}/yarn/local && chown yarn:hadoop ${disk}/yarn/local> doneAfter change Linux disk mount configuration, start all components on the hostRecommission DateNodeTurn off main

Sometimes, in the face of initialize configuration and setup, it is easier to completely re-initialize an Accumulo installation than try to repair it. These steps are different than what might be found in the Apache ecosystem as the following steps are related to how Apache Ambari is used to install and configure Accumulo.Warning: the following steps (may) irreversibly remove all Accumulo related data. This includes table data, namespaces, table configuration, and Accumulo users. Do not perform these steps unless you are positive that you do not want to preserve any of the information.First, Accumulo must be stopped. This can be done via Ambari and verified using tools like `ps` on the nodes.Second, the Accumulo HDFS directory should be removed. This can be done by the HDFS superuser ("hdfs" by default) or the Accumulo user ("accumulo" by default).# sudo -u hdfs hdfs dfs -rm -R /apps/accumulo/dataNext, Accumulo needs to re-initialized using the command line tools. This command must be

Steps to Create Table in Hive on S3A with RangerCreate a bucket with a unique name, I've used "myhivebucket" and do not change any details in the permissionsComplete the "Create bucket" wizard by clicking on "create bucket" buttonMake the following entries in custom hdfs-site.xml fs.s3a.access.key = <access key>fs.s3a.secret.key = <access secret>fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystemTo retrieve the value for access key and secret, follow these steps: Login to https://aws.amazon.com/consoleClick on "Sign in to the console" tabLogin with appropriate credentialsOnce logged in, you should see your login name on the top right corner of the AWS pageClick on the drop-down arrow beside your login name and select "My Security Credentials"This should take you to a page titled "Your Security Credentials"From this page, collapse the option that says "Access Keys(Access Key ID and Secret Access Key)"You have to click on "create a new access key" because of Amazon limitati

First ... if you need background Azure Event Hubs go here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-overviewSecond, its handy to know why you need a SAS token and what you can do once you have one. See: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overviewWe will use an example configuration for and Event Hub service. The corresponding data for your service can be found on the details page of the share access policy you want to use. More information on this topic can be found here https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-shared-access-signature-authenticationFrom the shared access policy "hub-nifi"'s detail page we can get all the information we need to create our token from the connection string-primary key fieldEndpoint=sb://nifi-eventhub.servicebus.windows.net/;SharedAccessKeyName=hub-user;SharedAccessKey=2hmLYbJk2q5uZ2Yfyl0XSezXbxD+afO9ysh0Vsv4Xq8=;EntityPath=hub1A SAS token is s

When trying to add a policy that has many resource paths to Ranger using the API it can fail with the errorException [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'sort_order' at row 1Error Code: 1264Call: INSERT INTO x_policy_resource_map (ADDED_BY_ID, CREATE_TIME, sort_order, resource_id, UPDATE_TIME, UPD_BY_ID, value) VALUES (?, ?, ?, ?, ?, ?, ?) bind => [7 parameters bound]Query: InsertObjectQuery(XXPolicyResourceMap [XXDBBase={createTime={Thu Apr 13 11:42:38 UTC 2017} updateTime={Thu Apr 13 11:42:39 UTC 2017} addedByUserId={1} updatedByUserId={1} } id=null, resourceId=43, value=/tmp/129, order=128])This is caused by a limit in Ranger policies that can only contain a maximum of 128 resource paths in a single policy. The work-around would be to split the policy in to two or more policies each cont

How to clean up & recreate collections on Ambari Infra Sometimes you need to clean up an installation that went with issues and there are missing collections on Ambari Infra solr. In order to do this, go thry the next list of steps to clean up solr collections and recreate them from scratch. IMPORTANT: This steps will delete all collection data, so unless you are sure data can be removed dont perform these steps. 1) Switch adudit solr to OFF from Ranger -> Ranger Audit configuration. Restarted all affected.2) Stop Altas service3) From the ambari infra host remove all the collections using rest api$ kinit -kt /etc/security/keytabs/ambari-infra-solr.service.keytab infra-solr/....$ export SOLR_HOST=<solr host fqdn>## Ranger collection:$ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=ranger_audits"## Atlas collections:$ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=vertex_i

The goal of this article is to ingest log data from multiple servers (running MiNiFi) that push log data to a NiFi cluster. The NiFi cluster will listen for the log data on an input port and route to an HDFS directory (determined by the host name). This article will assume you are using Ambari for NiFi installation/administration.1) Set the nifi.remote.input.host and nifi.remote.input.socket.port of NiFi clusterReason: Allows NiFi cluster to use an Input Port (MiNiFi will push to a Remote Processing Group) * NiFi will listen using the Input Porta) In Ambari, go to NiFib) Choose the Configs tabc) Choose the Advanced nifi-propertiesd) Set the nifi.remote.input.host to your NiFi hostname and nifi.remote.input.socket.port to 10000e) Restart NiFi using Ambari2) In NiFi – Create a flow for incoming log data (listening [input port] for MiNiFi data)Reason: Listen for incoming log data and route the log data to an HDFS directorya) On the NiFi Flow Canvas, drag-and-drop an Input port (Name you

In non kerberized cluster the user name that we provide will be used to resolve group name. User to Group mapping will be done based on the value configured in the property hadoop.security.group.mapping. In kerberized cluster we will be passing kerberos principal instead of user name, now HDFS has to resolve the principle to a user name in order to perform User to Group mapping. This resolution of user name from kerberos principal is done based on the value configured in hadoop.security.auth_to_local. The value for hadoop.security.auth_to_local will be configured properly in a kerberized cluster. When the client is running in a kerberized cluster and trying to access non kerberized HDFS, kerberos principal is sent as username. When the username (kerberos principal in this case) is received by a non-secure NameNode, it tries to getShortName from the principal, when that happens, NameNode will try to apply the rules configured in hadoop.security.auth_to_local property.Since the cluster i

Phoenix secondary indexes are often misunderstood. Those coming from the relational world mistakenly apply the same prinples to apache phoenix. A simple data model will be used for this article. Airplane table with attributes carrier ID, Tail Number, Origin airport code, destination airport code, and flight date.The physical data model has been identified with primary access path of carreirID and TailNum, essentially the rowkey.note - it is important to understand the order the "primary access path" in which fields will be accessed. Here I have identified the first key in the access path is carrierID. If that is not the case, benefits from the underline database capabilities, HBase may be realized. Think of primary access path not as primary key but more as the core identified access pattern for reads & writes. Secondary indexes enrich & extend this functionality.What are Apache Phoenix secondary indexes? "Secondary indexes are an orthogonal way to access data from its pr

There are 4 options available for SmartSense Server to upload bundles to Hortonworks Datalake, depending on theHDP environment and customer preference. Below are more details about each options and how to configure the same.Option 1: Direct Connection from SmartSense ServerSmartSense Server (with embedded Gateway in 1.3.0 and above) having direct connection to Hortonworks Datalake.No additional configurations needed. SmartSense is preconfigured to connect to Datalake via HTTPS protocolOption 2: Dedicated Gateway Server with direct connection When using SmartSense version earlier to 1.3.0, we need a gateway host to be configured to upload bundles.Also in an SmartSense 1.3.0 environment with multiple HDP clusters running, they can be configured to share a dedicated Gateway Server to upload the bundles. If needed the Gateway Server can be located in DMZ area as well.Step 1: Install / Configure SmartSense Gateway Software.To install and configure SmartSense Gateway Software on the Dedicat

The intention of this article is to show how simple is to set up a HDP cluster which is suitable to execute neural network training on GPUs in a distributed fashion. Explanation of how distributed training works is beyond the scope of this article, but if you are interested you can read about that here. Training neural networks are computationally intensive, involves lots of matrix manipulations, such as multiplication, addition and element-wise operations, these calculations are highly parallelizible and can be significantly speed up by using GPUs instead of general purpose CPUs. Neural networks are often trained on very large datasets, where training data is too large to fit to one machine and therefore the data is partitioned across several machines. If the data is already on different machines, it might be desirable that during the training the data is not transferred across the computer network, but the processing is moved next to the data and train the neural network in a dis

ISSUE: Java Heap Space issue in Hive MR engine While working on a sample data set in hive. Query such as "select count(*)" was seen to fail with below error.Starting Job = job_1491603076412_0001, Tracking URL = http://krishna3.openstacklocal:8088/proxy/application_1491603076412_0001/Kill Command = /usr/hdp/2.4.2.0-258/hadoop/bin/hadoop job -kill job_1491603076412_0001Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 12017-04-07 22:18:09,736 Stage-1 map = 0%, reduce = 0%2017-04-07 22:18:46,065 Stage-1 map = 100%, reduce = 100%Ended Job = job_1491603076412_0001 with errorsError during job, obtaining debugging information...Examining task ID: task_1491603076412_0001_m_000000 (and more) from job job_1491603076412_0001Task with the most failures(4):-----Task ID: task_1491603076412_0001_m_000000URL: http://krishna3.openstacklocal:8088/taskdetails.jsp?jobid=job_1491603076412_0001&tipid=task_1491603076412_0001_m_000000-----Diagnostic Messages for this Task:

I’m constantly amazed by what powerful things I can do with Apache NiFi in such few steps. I often challenge myself by saying “self, I bet you couldn’t do X with NiFi”. My confidence was challenged yesterday on a long flight back from Peru to Atlanta when I realized I couldn’t perform OCR type tasks with NiFi as it stands today. Perturbed by this fact I set out to come up with a solution. Ultimately this lead me to create a NiFi Tesseract processor for performing OCR tasks natively from within Apache NiFi. It wasn’t really until I was finished that I realized the how useful this processor could be. The Apache Tesseract Processor would give me the ability to read anything from hand written doctors notes from healthcare systems to interpreting scanned children’s book images. In fact I chose to demonstrate the later by showing how to use Apache NiFi to perform OCR on an excerpt from Dr. Seuss's - "Cat in the Hat” and then feeding that resulting text from the NiFi Tesseract processor to t

Part 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-vie

Recipes framework capability to support HDFS and Hive mirroring was added in Apache Falcon 0.6.0 release and it was a client side logic. With 0.10 release its moved to server side and renamed as server side extensions as part of jira https://issues.apache.org/jira/browse/FALCON-1107.For any new mirror job to be submitted and managed, Falcon extensions should be used. Please refer https://falcon.apache.org/restapi/ExtensionEnumeration.html for more details.Supported DistCp options for HDFS mirroring in HDP 2.5:distcpMaxMapsdistcpMapBandwidthoverwriteignoreErrorsskipChecksumremoveDeletedFilespreserveBlockSizepreserveReplicationNumberpreservePermissionpreserveUserpreserveGrouppreserveChecksumTypepreserveAclpreserveXattrpreserveTimesHdfs mirroring job can be scheduled using extension as below:falcon extension -submitAndSchedule -extensionName hdfs-mirroring -file sales-monthly.propertiesContent of sales-monthly.properties file:jobName=sales-monthlyjobValidityStart=2016-06-30T00:00ZjobVali

SETUP:Kerberized cluster with Ranger installed. This article uses a latest HDP-2.6 cluster installed using Ambari -2.5Ranger based authorization is enabled with HiveZeppelin's authentication is enabled. You can use LDAP authentication, but for the purpose of this demonstration - I am using a simple authentication method and I am going to configure 2 additional users 'hive' and 'hrt_1' in zeppelin's shiro.ini .[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = admin, adminhive = hive, adminhrt_1 = hrt_1, admin# Sample LDAP configuration, for user Authentication, currently tested for single Realm[main]### A sample for configuring Active Directory Realm#activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm#activeDirectoryRealm.systemUsername = userNameA#use either systemPassword or hadoopSecurityCre

HDCloud for AWS general availability version 1.14.1 is now available, including six new HDP 2.6 and Ambari 2.5 cluster configurations and new cloud controller features.If you are new to HDCloud, you can get started using this tutorial (updated for 1.14.1). Officail HDCloud for AWS documentation is available here.HDP 2.6 and Ambari 2.5The following HDP 2.6 configurations are now available:For the list of all available HDP 2.5 and HDP 2.6 configurations, refer to Cluster Configurations documentation. Resource TaggingWhen creating a cluster, you can optionally add custom tags that will be displayed on the CloudFormation stack and on EC2 instances, allowing you to keep track of the resources that cloud controller crates on your behalf. For more information, refer to Resource Tagging documentation.Node Auto RepairThe cloud controller monitors clusters by checking for Ambari Agent heartbeat on all cluster nodes. If the Ambari Agent heartbeat is lost on a node, a failure is reported for that

The Phoenix Query Server is an HTTP server which expects very specific request data data. Sometimes, in the process of connecting different clients, the various configuration options of both client and server can create confusion about what data is actually being sent over the wire. This confusion leads to questions like "did my configuration property take effect" and "is my client operating as I expect".Linux systems often have a number of tools available for analyzing network traffic on a node. We can use one of these tools, ngrep, to analyze the traffic flowing into the Phoenix Query Server.From a host running the Phoenix Query Server, the following command would dump all traffic from any source to the Phoenix Query Server.$ sudo ngrep -t -d any port 8765The above command will listen to any incoming network traffic on the current host and filter out any traffic which is not to the port 8765 (the default port for the Phoenix Query Server). A specific network interface (e.g. eth0) can

In the event when active namenode fails over to standby node which becomes active, if the standby node is not started , users accessing hive via hive cli take long time to connect or the connection never happens.When checking debug log via following command :hive --hiveconf hive.root.logger=DEBUG,consoleFollowing message appears multiple times in the console output:INFO ipc.Client: Retrying connect to server: rp1.HORTON/172.26.97.110:8020. Already tried 0 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[500x2000ms], TryOnceThenFail]DEBUG retry.RetryUtils: RETRY 1) policy=MultipleLinearRandomRetry, exception=java.net.ConnectException: Connection refusedEven after changing the property "yarn.resourcemanager.fs.state-store.retry-policy-spec" to lower value , MultipleLinearRandomRetry policy still remains the same and hive cli does not connect or connection never happens.FIX:Inorder to fix this issue , please update yarn-site.xml file with following property. This would requi

This is tutorial will help you get started accessing data stored on Amazon S3 from a cluster created through Hortonworks Data Cloud for AWS 1.16 (released in June 2017). The tutorial assumes no prior experience with AWS. In this tutorial: - We will use DistCp to copy sample data from S3 to HDFS and from HDFS to S3. - We will be using fs shell commands. - We will be using the Landsat 8 data that AWS makes available in the s3://landsat-pds in US West (Oregon) region. - We will also create a new S3 bucket to which we will copy data from HDFS. - In general, when specifying a path to S3, we will follow this required convention: `s3a://bucket-name/directory/`. Let's get started! Prerequisites Before starting this tutorial, your cloud controller needs to be running, and you must have a cluster running on AWS. To set up the cloud controller and cluster, refer to the following tutorial: How to set up Hortonworks Data Cloud for AWS.Accessing HDFS in HDCloud for AWS 1. SSH t

ISSUE: All of the datanodes in the cluster are shown as down in Ambari dashboard but hdfs is up, available and healthy.But the output of ps command shows that the service is actually running on the datanodes[root@node2 ~]# ps -ef|grep datanodehdfs 11150 1 0 Aug09 ? 00:03:01 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_datanode -Xmx1024m -Dhdp.version=2.3.6.0-3796 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.6.0-3796/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.6.0-3796/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.6.0-3796/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.6.0-3796 -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop-hdfs-datanode-node2.openstacklocal.log -Dhadoop.home

PROBLEM: user1 has submitted a yarn application with the application id of application_1473860344791_0001. To look at the yarn logs you would execute the following command;yarn logs -applicationId application_1473860344791_0001 If you run the command above as 'user2' you may see output similar to the following; 16/09/19 23:00:23 INFO impl.TimelineClientImpl: Timeline service address: http://mycluster.somedomain.com:8188/ws/v1/timeline/16/09/19 23:00:23 INFO client.RMProxy: Connecting to ResourceManager at mycluster.somedomain.com/192.168.1.89:8050/app-logs/user2/logs/application_1473860344791_0001 does not exist.Log aggregation has not completed or is not enabled. ROOT CAUSE: When log aggregation has been enabled each users application logs will, by default, be placed in the directory hdfs:///app-logs/<USERNAME>/logs/<APPLICATION_ID>. By default only the user that submitted the job and members of the hadoop group will have access to read the log files. In the example direct

OBSERVATIONS: After navigating to the "Register Version' we see that HDP 2.4 and 2.5 are not registered versions. We identified some INFO level messages in the logs that tell us there is an issue with one of the xml files read by Ambari during startup; 15 Feb 2017 10:52:15,526 INFO [ambari-client-thread-67] AmbariMetaInfo:1417 - Stack HDP-2.0.6.GlusterFS is not active, skipping VDF 15 Feb 2017 10:52:15,528 INFO [ambari-client-thread-67] AmbariMetaInfo:1415 - Stack HDP-2.5 is not valid, skipping VDF: Could not parse XML /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTERPRISE/configuration/arcadia-config.xml: javax.xml.bind.UnmarshalException - with linked exception: [org.xml.sax.SAXParseException; systemId: file:/var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTERPRISE/configuration/arcadia-config.xml; lineNumber: 1; columnNumber: 1; Premature end of file.]; Could not parse XML /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTE

You can use trace command for tracing requests from HBase Shell. trace 'start' command turns on tracing and trace 'stop'command turns off tracing. hbase(main):001:0> trace 'start'hbase(main):002:0> put 'test', 'row1', 'f:', 'val1' # traced commandshbase(main):003:0> trace 'stop' trace 'start' and trace 'stop' always returns boolean value representing if or not there is ongoing tracing. As a result, trace 'stop' returns false on success. trace 'status' just returns if or not tracing is turned on.hbase(main):001:0> trace 'start'=> truehbase(main):002:0> trace 'status'=> truehbase(main):003:0> trace 'stop'=> falsehbase(main):004:0> trace 'status' => false REFERECE: https://hbase.apache.org/book.html

To avoid full table scan while running hbase copytable job make use of --startrow and --stoprow parameters instead of --starttime and --endtime.Example:hbase org.apache.hadoop.hbase.mapreduce.CopyTable --startrow r1 --stoprow r12 --peer.adr=zk1,zk2,zk3:2181:/hbase-unsecure TestTablehbase -Dhbase.client.scanner.caching=1 -Dmapred.map.tasks.speculative.execution=false org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=phbaiskdc2000.phx.qa.xyz.com:2181:/hbase-unsecure --new.name=REVIEW --startrow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" --stoprow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" REVIEWReference: https://hbase.apache.org/book.html