Resources

This is tutorial will help you get started accessing data stored on Amazon S3 from a cluster created through Hortonworks Data Cloud for AWS 1.16 (released in June 2017). The tutorial assumes no prior experience with AWS. In this tutorial: - We will use DistCp to copy sample data from S3 to HDFS and from HDFS to S3. - We will be using fs shell commands. - We will be using the Landsat 8 data that AWS makes available in the s3://landsat-pds in US West (Oregon) region. - We will also create a new S3 bucket to which we will copy data from HDFS. - In general, when specifying a path to S3, we will follow this required convention: `s3a://bucket-name/directory/`. Let's get started! Prerequisites Before starting this tutorial, your cloud controller needs to be running, and you must have a cluster running on AWS. To set up the cloud controller and cluster, refer to the following tutorial: How to set up Hortonworks Data Cloud for AWS.Accessing HDFS in HDCloud for AWS 1. SSH t

ISSUE: All of the datanodes in the cluster are shown as down in Ambari dashboard but hdfs is up, available and healthy.But the output of ps command shows that the service is actually running on the datanodes[root@node2 ~]# ps -ef|grep datanodehdfs 11150 1 0 Aug09 ? 00:03:01 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_datanode -Xmx1024m -Dhdp.version=2.3.6.0-3796 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.6.0-3796/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.6.0-3796/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.6.0-3796/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.6.0-3796 -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop-hdfs-datanode-node2.openstacklocal.log -Dhadoop.home

PROBLEM: user1 has submitted a yarn application with the application id of application_1473860344791_0001. To look at the yarn logs you would execute the following command;yarn logs -applicationId application_1473860344791_0001 If you run the command above as 'user2' you may see output similar to the following; 16/09/19 23:00:23 INFO impl.TimelineClientImpl: Timeline service address: http://mycluster.somedomain.com:8188/ws/v1/timeline/16/09/19 23:00:23 INFO client.RMProxy: Connecting to ResourceManager at mycluster.somedomain.com/192.168.1.89:8050/app-logs/user2/logs/application_1473860344791_0001 does not exist.Log aggregation has not completed or is not enabled. ROOT CAUSE: When log aggregation has been enabled each users application logs will, by default, be placed in the directory hdfs:///app-logs/<USERNAME>/logs/<APPLICATION_ID>. By default only the user that submitted the job and members of the hadoop group will have access to read the log files. In the example direct

OBSERVATIONS: After navigating to the "Register Version' we see that HDP 2.4 and 2.5 are not registered versions. We identified some INFO level messages in the logs that tell us there is an issue with one of the xml files read by Ambari during startup; 15 Feb 2017 10:52:15,526 INFO [ambari-client-thread-67] AmbariMetaInfo:1417 - Stack HDP-2.0.6.GlusterFS is not active, skipping VDF 15 Feb 2017 10:52:15,528 INFO [ambari-client-thread-67] AmbariMetaInfo:1415 - Stack HDP-2.5 is not valid, skipping VDF: Could not parse XML /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTERPRISE/configuration/arcadia-config.xml: javax.xml.bind.UnmarshalException - with linked exception: [org.xml.sax.SAXParseException; systemId: file:/var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTERPRISE/configuration/arcadia-config.xml; lineNumber: 1; columnNumber: 1; Premature end of file.]; Could not parse XML /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ARCADIA-ENTE

You can use trace command for tracing requests from HBase Shell. trace 'start' command turns on tracing and trace 'stop'command turns off tracing. hbase(main):001:0> trace 'start'hbase(main):002:0> put 'test', 'row1', 'f:', 'val1' # traced commandshbase(main):003:0> trace 'stop' trace 'start' and trace 'stop' always returns boolean value representing if or not there is ongoing tracing. As a result, trace 'stop' returns false on success. trace 'status' just returns if or not tracing is turned on.hbase(main):001:0> trace 'start'=> truehbase(main):002:0> trace 'status'=> truehbase(main):003:0> trace 'stop'=> falsehbase(main):004:0> trace 'status' => false REFERECE: https://hbase.apache.org/book.html

To avoid full table scan while running hbase copytable job make use of --startrow and --stoprow parameters instead of --starttime and --endtime.Example:hbase org.apache.hadoop.hbase.mapreduce.CopyTable --startrow r1 --stoprow r12 --peer.adr=zk1,zk2,zk3:2181:/hbase-unsecure TestTablehbase -Dhbase.client.scanner.caching=1 -Dmapred.map.tasks.speculative.execution=false org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=phbaiskdc2000.phx.qa.xyz.com:2181:/hbase-unsecure --new.name=REVIEW --startrow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" --stoprow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" REVIEWReference: https://hbase.apache.org/book.html

The QueryDatabaseTable processor can easily ingest data from a table based on a incrementing key. A sequence id or primary key that is autogeneratored like Postgresql and MariaDB do is ideal. You can also do an incrementing data or Oracle Sequence ID. As long as it increments when you get a new one you can set. If your tables don't this, you could write a trigger or procedure in your database that sends it to a transaction table with such an autogenerated id and NiFi will grab that.Clearly real CDC involves reading Write Ahead Logs or Transaction logs at a deep level and grabbing all changes. That is coming and can now be done by tools like Atunity + NiFi.For use cases that I have, I just need to grab new rows when they are added to a table and I control the ID.I convert from AVRO to JSON so I can extract attributes since I want to do some routing based on column values. Based on one field in the table, I want to determine where I land the data. It can be sent to HBase (and

Many process groups have a success and failure output relationship. A common question is how to best handle these failures. For invalid data, it makes sense to output the flow files to an HDFS directory for analysis, but not when failure was caused by an external dependency (e.g. HDFS, Kafka, FTP).A simple solution might be to loop the failures back to retry, but then it may fail repeatedly without notifying an administrator. A better solution would be to retry three times, then, if it still has not succeeded, an administrator should be notified and the flow file should wait before trying again. This gives the administrator time to resolve the issue and the ability to quickly and easily retry the flow files.Below (and attached) is a simple process group that implements this logic. The failed flow files come in through the input port. The UpdateAttribute processor sets the retryCount attribute to one or increments it if it has already been set. The RouteOnAttribute processor determines

Motivation:In past, there have been many instances here in Community as well as outside, where HDP 2.5 Sandbox users wouldn't be able to enable Kerberos / install Yum packages like SmartSense, KMS, Kerberos client / Access Zeppelin UI etc. like these:1. https://community.hortonworks.com/questions/82455/zeppelin-ui-returns-error-503-hdp-25-sandbox.html2. https://community.hortonworks.com/questions/68361/fail-to-install-kerberos-client-in-sandbox-25.html3. https://community.hortonworks.com/questions/60349/enabling-kerberos-in-hdp-25-kerberos-clientpy-does.html4. https://community.hortonworks.com/questions/68234/fail-to-install-kerberos-in-hdp-sandbox-25.html5. https://community.hortonworks.com/questions/78464/ranger-kms-cannot-install-successfully-with-ambari.htmlThis article is aimed at resolving all these issues for once & all !Target Audience:All the users who are using HDP 2.5 Sandbox image for VirtualBox / VMWare.Problem:There is a known bug in Docker OverlayFS, Linux Kernel 3.1

PROBLEM:Security scan reporting clickjacking vulnerabilities for HBase ports (16010 and 16030) and AMS ports (61330 and 61310).ROOT CAUSE: HBASE-14148 RESOLUTION:Upgrade to the HDP 2.5.X

PROBLEM: HBase Master failing due to following exception.2017-03-30 20:51:13,130 ERROR [<HOST NAME>,33733,1490218864063_ChoreService_3] normalizer.RegionNormalizerChore: Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.getRegionSize(SimpleRegionNormalizer.java:172) at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.computePlanForTable(SimpleRegionNormalizer.java:99) at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1417) at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:49) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.Sche

Append following properties in /usr/hdp/current/phoenix-client/bin/log4j.properties and change log4j.threshold=TRACE log4j.logger.org.apache.phoenix=TRACE log4j.logger.org.apache.hadoop.hbase.ipc=TRACE log4j.logger.org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel=DEBUG

Cloudbreak is a popular, easy to use HDP component for cluster deployment on various cloud environments including Azure, AWS, OpenStac and GCP. This article shows how to create an Azure application for Cloudbreak using Azure CLI. Note: To do this, you need access to "Owner" account on your Azure subscription. "Developer" and other roles are not enough. Download and install Azure CLI using instructions provided here. CLI versions are available for Windows, Mac-OS and Linux https://docs.microsoft.com/en-us/cli/azure/install-azure-cli Type "az" to make sure the CLI is available and in your command path. Login to your Azure account in your web browser, and then also login from your command line:az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code HPBCSXTPJ to authenticate. Follow the instructions on the web page. When done you will see confirmation on the command line that your login was successful. Run the following command. You can free

General optimizationsDo not run HDFS balancer. It breaks data locality and data locality is important for latency-sensitive applicationsFor the very same reason disable HBase auto region balancing: balance_switch false Disable periodic automatic major compactions for time-series data. Time-series data is immutable (means no update/deletes usually). The only reason remaining for major compaction is decreasing number of store files, but we will apply different compaction policy, which limits number of files and does not require major compaction (see below)Presplit table(s) with time-series data in advance. Disable region splits completely (set DisabledRegionSplitPolicy). Region splitting results in major compaction and we do not run major compactions because it usually decrease performance, stability and increase operation latencies. Enable WAL Compression - decrease write IO. Table designDo not store data in a raw format - use time-series specific compression (refer to OpenTSDB row ke

To clear local file cache and user cache for yarn, perform the following: Find out the cache location by checking the value of the yarn.nodemanager.local-dirs property :<property> <name>yarn.nodemanager.local-dirs</name> <value>/hadoop/yarn/local</value> </property> Remove filecache and usercache folder located inside the folders that is specified in yarn.nodemanager.local-dirs.[yarn@node2 ~]$ cd /hadoop/yarn/local/ [yarn@node2 local]$ ls filecache nmPrivate spark_shuffle usercache [yarn@node2 local]$ rm -rf filecache/ usercache/ If there are more than one folder, clean them one by one.Restart YARN service.

PROBLEM: All the application_xxx_xx directories under ${yarn.nodemanager.log-dirs} are getting created with 0710. Can this be overridden to make less restrictive? For example:[root@space2 ~]# su -l hdfs -c 'hadoop jar /usr/hdp/2.5.3.0-37/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.5.3.0-37.jar randomtextwriter -write -nrFiles 10 -filesize 500' :: Running 30 maps. Job started: Wed Dec 28 00:22:55 UTC 2016 [root@space2 log]# ls -l /hadoop/yarn/logtotal 4 drwx--x---. 12 yarn hadoop 4096 Dec 28 00:24 application_1482346173510_0002 ROOT CAUSE: Currently yarn.nodemanager.default-container-executor.log-dirs.permissions property is hardcoded to permission 710 per https://issues.apache.org/jira/browse/YARN-4579. RESOLUTION: Fixed in upcoming Hadoop 2.9.

Solr core/shard backup and restore function ONLY became available after version 5.2 . See JIRA: https://issues.apache.org/jira/browse/SOLR-6637 To manually backup and restore shards, please follow the steps : Assumes a SolrCloud cluster has 4 cores and 2 node, which 2 cores on each node. (2 shards, with replicationFactor=2) Back-up each node: http://server1:8983/solr/collection/replication?command=backup&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=backup&location=location2&name=backup2 Create new collection with 2 shards and replicationFactor=2 http://server:8983/solr/admin/collections?action=CREATE&name=newCollection&numShards=2&replicationFactor=2&createNodeSet=server1:8983_solr,server2:8983_solr Restore from each backup: http://server1:8983/solr/newCollection/replication?command=restore&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=restore&location=

Following is a guide to split shards and add replicas for collection "ranger_audits" on SolrCloud , in case the initial ranger_audits is created with only 1 shard 1 replica : 1.Initial ranger_audits was created with only 1 shard and 1 replica and user would like to split the shard to balance load and add more replicas for back up. 2. In the web browser, enter URL to split the shard : http://172.25.17.51:6083/solr/admin/collections?action=SPLITSHARD&collection=ranger_audits&shard=shard1 , and user shall get a page like which means the splitting operation is successful. And in the Solr Admin UI, user can see there are two additional shards : Notice, the original shard1 still exists. This is because splitting a shard will take an existing shard and break it into two pieces which are written to disk as two (new) shards. The original shard will continue to contain the same data as-is but it will start re-routing requests to the new shards. The new shards will have as many

Cluster setup :Client hostnames : mf-amos110.novalocal , and mf-amos111.novalocalServer hostnames : node1.openstacklocal , node2.openstacklocal and node3.openstacklocal step 1. create Company CA "rootCA.key" , on host mf-amos111.novalocal : [root@mf-amos111 centos]# openssl genrsa -aes256 -out rootCA.key 4096 Step 2 . create a pem file "rootCA.pem", which will be used to sign certificates sent by other hosts :[root@mf-amos111 centos]# openssl req -x509 -new -key rootCA.key -days 1095 -out rootCA.pem Step 3. Create a truststore file "all-truststore.jks" for all hosts, and copy this keystore to /etc/hadoop/conf dir on each host.[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore all-truststore.jks -alias CARoot -import -file rootCA.pem[root@mf-amos111 centos]# cp all-truststore.jks /etc/hadoop/conf[root@mf-amos111 conf]# scp all-truststore.jks root@mf-amos110.novalocal:/etc/hadoop/confall-truststore.jks

Perform following steps to enable the verbose log for Oozie launcher Step 1.Add below property to the action configuration section in the workflow file:<configuration>.....<property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-verbose</value> </property> </configuration>Step 2.Upload the updated workflow file to workflow folder defined by oozie.wf.application.path in job property fileStep 3.Submit the workflow and now you should be able to see verbose log for the Oozie launcher such as class loading information . [Loaded java.lang.ExceptionInInitializerError from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar][Loaded org.apache.commons.logging.impl.LogFactoryImpl$2 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar][Loaded org.apache.commons.logging.impl.LogFactoryImpl$1 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-

Pipelined Sorter The pipelined sorter is a divide and conquer approach applied to the MapOutputBuffer’s sort which operates on an entire chunk of 256 or 512 Mb of data space. The primary assumption is that for a significant number of hadoop installations the CPU is under utilized by the task tracker, with the fundamental limit for the map slots being the disks available. The DefaultSorter uses two buffers ­ kvbuffer (data) and kvmeta (metadata) which grow from different parts of the large buffer allocated for operations. The PipelinedSorter uses a series of similar buffer pairs which grow only in one direction and involve no looping around back to the front. These buffer­pairs are marked as a container class named SortSpan. The allocation system pre­allocates 16Mb for the first kvmeta and then proceeds to mark the rest of the entire buffer for kvbuffer0. The collect proceeds normally until either the entire buffer is full or kvmeta0 acquires 1M items (16 byte per item, 1024x1024 items)

PROBLEM : Accessing Capacity Scheduler View give 500 internal server error message and ambari server logs and Table './ambari/alert_history' is marked as crashed and should be repaired.ERROR :ERROR [ambari-client-thread-35] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.capacityscheduler.utils.ServiceFormattedException at org.apache.ambari.view.capacityscheduler.ConfigurationService.readClusterInfo(ConfigurationService.java:162) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)ERROR [ambari-client-thread-137] ReadHandler:102 - Caught a runtime exception executing a queryjavax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.SQLException: Table './ambari/alert_history' is marked as crashed and should be repairedError Code: 145 ROOT CAUSE : alert_definition_id present

Issue:After enabling the preemption parameters in yarn-site.xml as per link, it was found the applications are still waiting to run. This is because other queues are utilising all of the available resources. With Preemption enabled, under-served queues can begin to claim their allocated cluster resources almost immediately, without having to wait for other queues' applications to finish running.Resolution:Along with the below list of parameters to enable preemption, configure this parameter "yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity" in yarn-site.xml, otherwise you won't see preemption though enabled. The doc is missing the above parameter. List of parameters to enable preemption are:1) yarn.resourcemanager.scheduler.monitor.enable2) yarn.resourcemanager.scheduler.monitor.policies3) yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval4) yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill5) yarn.resourcemanager.monitor.c

Use case: We want to control the kafka broker, producer and consumer policies using Ranger without having kerberos. "What is a recommended way to set-up policies when trying to control access to Kafka over a non-secure channel?" Original doc Demo I have defined 3 policies as shown below: Broker, Publisher and Consumer is controlled at IP level. With one click you can revoke the access from the consumer. Demo commands Happy Hadooping!!!

Data Transformation and Processing with Spark StreamingThis article will describe a little demo application that takes data from Kafka, parses it, does some some aggregations on a different time window and sends the results to Apache Phoenix. Note there is also now a Phoenix Spark plugin available but in this article we will do it the hard way. You can use the same approach to send data to pretty much any target.Note you can find the sample code about this article in the following repohttps://github.com/benleon/KafkaToPhoenixIntroduction:Spark provides three main interfaces to work with data which can more or less be transformed into each other:RDDs: Spark is built on RDDs, which are distributed datasets that can be recomputed in case of failure and are transient. This means they get only materialized if they are needed, for example to write to a file. Dataframes:Dataframes are an abstraction layer on top of RDDs that provide easy SQL style data transformations. They also use an optim