Community Articles

The QueryDatabaseTable processor can easily ingest data from a table based on a incrementing key. A sequence id or primary key that is autogeneratored like Postgresql and MariaDB do is ideal. You can also do an incrementing data or Oracle Sequence ID. As long as it increments when you get a new one you can set. If your tables don't this, you could write a trigger or procedure in your database that sends it to a transaction table with such an autogenerated id and NiFi will grab that.Clearly real CDC involves reading Write Ahead Logs or Transaction logs at a deep level and grabbing all changes. That is coming and can now be done by tools like Atunity + NiFi.For use cases that I have, I just need to grab new rows when they are added to a table and I control the ID.I convert from AVRO to JSON so I can extract attributes since I want to do some routing based on column values. Based on one field in the table, I want to determine where I land the data. It can be sent to HBase (and

Many process groups have a success and failure output relationship. A common question is how to best handle these failures. For invalid data, it makes sense to output the flow files to an HDFS directory for analysis, but not when failure was caused by an external dependency (e.g. HDFS, Kafka, FTP).A simple solution might be to loop the failures back to retry, but then it may fail repeatedly without notifying an administrator. A better solution would be to retry three times, then, if it still has not succeeded, an administrator should be notified and the flow file should wait before trying again. This gives the administrator time to resolve the issue and the ability to quickly and easily retry the flow files.Below (and attached) is a simple process group that implements this logic. The failed flow files come in through the input port. The UpdateAttribute processor sets the retryCount attribute to one or increments it if it has already been set. The RouteOnAttribute processor determines

Motivation:In past, there have been many instances here in Community as well as outside, where HDP 2.5 Sandbox users wouldn't be able to enable Kerberos / install Yum packages like SmartSense, KMS, Kerberos client / Access Zeppelin UI etc. like these:1. https://community.hortonworks.com/questions/82455/zeppelin-ui-returns-error-503-hdp-25-sandbox.html2. https://community.hortonworks.com/questions/68361/fail-to-install-kerberos-client-in-sandbox-25.html3. https://community.hortonworks.com/questions/60349/enabling-kerberos-in-hdp-25-kerberos-clientpy-does.html4. https://community.hortonworks.com/questions/68234/fail-to-install-kerberos-in-hdp-sandbox-25.html5. https://community.hortonworks.com/questions/78464/ranger-kms-cannot-install-successfully-with-ambari.htmlThis article is aimed at resolving all these issues for once & all !Target Audience:All the users who are using HDP 2.5 Sandbox image for VirtualBox / VMWare.Problem:There is a known bug in Docker OverlayFS, Linux Kernel 3.1

PROBLEM:Security scan reporting clickjacking vulnerabilities for HBase ports (16010 and 16030) and AMS ports (61330 and 61310).ROOT CAUSE: HBASE-14148 RESOLUTION:Upgrade to the HDP 2.5.X

PROBLEM: HBase Master failing due to following exception.2017-03-30 20:51:13,130 ERROR [<HOST NAME>,33733,1490218864063_ChoreService_3] normalizer.RegionNormalizerChore: Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.getRegionSize(SimpleRegionNormalizer.java:172) at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.computePlanForTable(SimpleRegionNormalizer.java:99) at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1417) at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:49) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.Sche

Append following properties in /usr/hdp/current/phoenix-client/bin/log4j.properties and change log4j.threshold=TRACE log4j.logger.org.apache.phoenix=TRACE log4j.logger.org.apache.hadoop.hbase.ipc=TRACE log4j.logger.org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel=DEBUG

Cloudbreak is a popular, easy to use HDP component for cluster deployment on various cloud environments including Azure, AWS, OpenStac and GCP. This article shows how to create an Azure application for Cloudbreak using Azure CLI. Note: To do this, you need access to "Owner" account on your Azure subscription. "Developer" and other roles are not enough. Download and install Azure CLI using instructions provided here. CLI versions are available for Windows, Mac-OS and Linux https://docs.microsoft.com/en-us/cli/azure/install-azure-cli Type "az" to make sure the CLI is available and in your command path. Login to your Azure account in your web browser, and then also login from your command line:az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code HPBCSXTPJ to authenticate. Follow the instructions on the web page. When done you will see confirmation on the command line that your login was successful. Run the following command. You can free

General optimizationsDo not run HDFS balancer. It breaks data locality and data locality is important for latency-sensitive applicationsFor the very same reason disable HBase auto region balancing: balance_switch false Disable periodic automatic major compactions for time-series data. Time-series data is immutable (means no update/deletes usually). The only reason remaining for major compaction is decreasing number of store files, but we will apply different compaction policy, which limits number of files and does not require major compaction (see below)Presplit table(s) with time-series data in advance. Disable region splits completely (set DisabledRegionSplitPolicy). Region splitting results in major compaction and we do not run major compactions because it usually decrease performance, stability and increase operation latencies. Enable WAL Compression - decrease write IO. Table designDo not store data in a raw format - use time-series specific compression (refer to OpenTSDB row ke

To clear local file cache and user cache for yarn, perform the following: Find out the cache location by checking the value of the yarn.nodemanager.local-dirs property :<property> <name>yarn.nodemanager.local-dirs</name> <value>/hadoop/yarn/local</value> </property> Remove filecache and usercache folder located inside the folders that is specified in yarn.nodemanager.local-dirs.[yarn@node2 ~]$ cd /hadoop/yarn/local/ [yarn@node2 local]$ ls filecache nmPrivate spark_shuffle usercache [yarn@node2 local]$ rm -rf filecache/ usercache/ If there are more than one folder, clean them one by one.Restart YARN service.

PROBLEM: All the application_xxx_xx directories under ${yarn.nodemanager.log-dirs} are getting created with 0710. Can this be overridden to make less restrictive? For example:[root@space2 ~]# su -l hdfs -c 'hadoop jar /usr/hdp/2.5.3.0-37/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.5.3.0-37.jar randomtextwriter -write -nrFiles 10 -filesize 500' :: Running 30 maps. Job started: Wed Dec 28 00:22:55 UTC 2016 [root@space2 log]# ls -l /hadoop/yarn/logtotal 4 drwx--x---. 12 yarn hadoop 4096 Dec 28 00:24 application_1482346173510_0002 ROOT CAUSE: Currently yarn.nodemanager.default-container-executor.log-dirs.permissions property is hardcoded to permission 710 per https://issues.apache.org/jira/browse/YARN-4579. RESOLUTION: Fixed in upcoming Hadoop 2.9.

Solr core/shard backup and restore function ONLY became available after version 5.2 . See JIRA: https://issues.apache.org/jira/browse/SOLR-6637 To manually backup and restore shards, please follow the steps : Assumes a SolrCloud cluster has 4 cores and 2 node, which 2 cores on each node. (2 shards, with replicationFactor=2) Back-up each node: http://server1:8983/solr/collection/replication?command=backup&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=backup&location=location2&name=backup2 Create new collection with 2 shards and replicationFactor=2 http://server:8983/solr/admin/collections?action=CREATE&name=newCollection&numShards=2&replicationFactor=2&createNodeSet=server1:8983_solr,server2:8983_solr Restore from each backup: http://server1:8983/solr/newCollection/replication?command=restore&location=location1&name=backup1 http://server2:8983/solr/collection/replication?command=restore&location=

Following is a guide to split shards and add replicas for collection "ranger_audits" on SolrCloud , in case the initial ranger_audits is created with only 1 shard 1 replica : 1.Initial ranger_audits was created with only 1 shard and 1 replica and user would like to split the shard to balance load and add more replicas for back up. 2. In the web browser, enter URL to split the shard : http://172.25.17.51:6083/solr/admin/collections?action=SPLITSHARD&collection=ranger_audits&shard=shard1 , and user shall get a page like which means the splitting operation is successful. And in the Solr Admin UI, user can see there are two additional shards : Notice, the original shard1 still exists. This is because splitting a shard will take an existing shard and break it into two pieces which are written to disk as two (new) shards. The original shard will continue to contain the same data as-is but it will start re-routing requests to the new shards. The new shards will have as many

Cluster setup :Client hostnames : mf-amos110.novalocal , and mf-amos111.novalocalServer hostnames : node1.openstacklocal , node2.openstacklocal and node3.openstacklocal step 1. create Company CA "rootCA.key" , on host mf-amos111.novalocal : [root@mf-amos111 centos]# openssl genrsa -aes256 -out rootCA.key 4096 Step 2 . create a pem file "rootCA.pem", which will be used to sign certificates sent by other hosts :[root@mf-amos111 centos]# openssl req -x509 -new -key rootCA.key -days 1095 -out rootCA.pem Step 3. Create a truststore file "all-truststore.jks" for all hosts, and copy this keystore to /etc/hadoop/conf dir on each host.[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore all-truststore.jks -alias CARoot -import -file rootCA.pem[root@mf-amos111 centos]# cp all-truststore.jks /etc/hadoop/conf[root@mf-amos111 conf]# scp all-truststore.jks root@mf-amos110.novalocal:/etc/hadoop/confall-truststore.jks

Perform following steps to enable the verbose log for Oozie launcher Step 1.Add below property to the action configuration section in the workflow file:<configuration>.....<property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-verbose</value> </property> </configuration>Step 2.Upload the updated workflow file to workflow folder defined by oozie.wf.application.path in job property fileStep 3.Submit the workflow and now you should be able to see verbose log for the Oozie launcher such as class loading information . [Loaded java.lang.ExceptionInInitializerError from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar][Loaded org.apache.commons.logging.impl.LogFactoryImpl$2 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar][Loaded org.apache.commons.logging.impl.LogFactoryImpl$1 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-

Pipelined Sorter The pipelined sorter is a divide and conquer approach applied to the MapOutputBuffer’s sort which operates on an entire chunk of 256 or 512 Mb of data space. The primary assumption is that for a significant number of hadoop installations the CPU is under utilized by the task tracker, with the fundamental limit for the map slots being the disks available. The DefaultSorter uses two buffers ­ kvbuffer (data) and kvmeta (metadata) which grow from different parts of the large buffer allocated for operations. The PipelinedSorter uses a series of similar buffer pairs which grow only in one direction and involve no looping around back to the front. These buffer­pairs are marked as a container class named SortSpan. The allocation system pre­allocates 16Mb for the first kvmeta and then proceeds to mark the rest of the entire buffer for kvbuffer0. The collect proceeds normally until either the entire buffer is full or kvmeta0 acquires 1M items (16 byte per item, 1024x1024 items)

PROBLEM : Accessing Capacity Scheduler View give 500 internal server error message and ambari server logs and Table './ambari/alert_history' is marked as crashed and should be repaired.ERROR :ERROR [ambari-client-thread-35] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.capacityscheduler.utils.ServiceFormattedException at org.apache.ambari.view.capacityscheduler.ConfigurationService.readClusterInfo(ConfigurationService.java:162) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)ERROR [ambari-client-thread-137] ReadHandler:102 - Caught a runtime exception executing a queryjavax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.SQLException: Table './ambari/alert_history' is marked as crashed and should be repairedError Code: 145 ROOT CAUSE : alert_definition_id present

Issue:After enabling the preemption parameters in yarn-site.xml as per link, it was found the applications are still waiting to run. This is because other queues are utilising all of the available resources. With Preemption enabled, under-served queues can begin to claim their allocated cluster resources almost immediately, without having to wait for other queues' applications to finish running.Resolution:Along with the below list of parameters to enable preemption, configure this parameter "yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity" in yarn-site.xml, otherwise you won't see preemption though enabled. The doc is missing the above parameter. List of parameters to enable preemption are:1) yarn.resourcemanager.scheduler.monitor.enable2) yarn.resourcemanager.scheduler.monitor.policies3) yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval4) yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill5) yarn.resourcemanager.monitor.c

Use case: We want to control the kafka broker, producer and consumer policies using Ranger without having kerberos. "What is a recommended way to set-up policies when trying to control access to Kafka over a non-secure channel?" Original doc Demo I have defined 3 policies as shown below: Broker, Publisher and Consumer is controlled at IP level. With one click you can revoke the access from the consumer. Demo commands Happy Hadooping!!!

Data Transformation and Processing with Spark StreamingThis article will describe a little demo application that takes data from Kafka, parses it, does some some aggregations on a different time window and sends the results to Apache Phoenix. Note there is also now a Phoenix Spark plugin available but in this article we will do it the hard way. You can use the same approach to send data to pretty much any target.Note you can find the sample code about this article in the following repohttps://github.com/benleon/KafkaToPhoenixIntroduction:Spark provides three main interfaces to work with data which can more or less be transformed into each other:RDDs: Spark is built on RDDs, which are distributed datasets that can be recomputed in case of failure and are transient. This means they get only materialized if they are needed, for example to write to a file. Dataframes:Dataframes are an abstraction layer on top of RDDs that provide easy SQL style data transformations. They also use an optim

Many of us in Hortonworks Community Connection feel most at home when we are talking about technologies and tools and the "animals in the zoo". However if we want to grow the data lake and gain support from the business we have to learn to think a little differently and use a new vocabulary to communicate. Start by meeting with the business to identifying possible use cases. Talk to the analysts about the highest priorities and pain points for the business. Before thinking about anything remotely Hadoop animal like, summarize "what" needs to be done. This may take several interviews with different business analysts to gain a full understanding of the problem. Then determine if Big Data can solve the problem. Are data silos preventing the organization from getting a complete view of the customer or logistics? Is the volume of data required to solve the problem too much or too expensive for existing systems to handle? Are the unstructured or semi-structured data required to s

Connecting to HBase throws the below exceptions: [root@hl1 hbase]# cd /usr/hdp/current/phoenix-client/bin/[root@hl1 bin]# python sqlline.py localhost:2181:/hbase-unsecuresun.misc.SignalHandler not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/etc/hbase/conf/,file:/usr/hdp/2.5.3.0-37/phoenix/bin/../phoenix-4.7.0.2.5.3.0-37-client.jar,file:./,file:/etc/hadoop/conf/,file:/usr/hdp/2.5.3.0-37/hadoop/conf/,file:/usr/hdp/2.5.3.0-37/hadoop-hdfs/./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}Setting property: [incremental, false]Setting property: [isolation, TRANSACTION_READ_COMMITTED]issuing: !connect jdbc:phoenix:localhost:2181:/hbase-unsecure none none org.apache.phoenix.jdbc.PhoenixDriverConnecting to jdbc:phoenix:localhost:2181:/hbase-unsecurejava.lang.ClassFormatError: org.apache.phoenix.jdbc.PhoenixDriver (unrecognized class file version) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java

Storage is Fundamental to Big DataStorages can be chiefly evaluated on three classes of performance metrics: Cost per GigabyteDurability - this is the measure of the permanence of data once it has been successfully written to the medium. Modern hard disks are highly durable, however given a large enough collection of disks, regular disk failures are a statistical certainty. Annual Failure Rates of disks vary between 2 – 8% for 1 – 2 year old disks as observed in a large-scale study of disk failure rates.[1]Performance - there are two key measures of storage performance: Throughput - this is the maximum raw read/write rate that the storage can support and is typically measured in MegaBits/second (MBps). This is the primary metric that batch processing applications care about.IO operations per second (IOPS) - the number of IO operations per second is affected by the workload and IO size. The rotational latency of spinning disks limits the maximum IOPS for a random IO workload which can

By default the GC logs are not enabled for Hive components. It is good to enable them to troubleshoot GC pauses on hiveserver2 instances.---------------------------------Hiveserver2 / Metastore:---------------------------------In Ambari navigate to the following pathServices -- > Hive -- > Configs -- > Advanced --> Advanced hive-env --> hive-env templateAdd the following lines at the beginning,if [[ "$SERVICE" == "hiveserver2" || "$SERVICE" == "metastore" ]]; thenHIVE_SERVERS_GC_LOG_OPTS="-Xloggc:{{hive_log_dir}}/gc.log-$SERVICE-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"export HADOOP_OPTS="$HADOOP_OPTS $HIVE_SERVERS_GC_LOG_OPTS"fi---------------------------------Webhact :---------------------------------In Ambari navigate to the following pathServices -- > Hive -- > Configs -- > Advanced --> Advanced webhcat-env --> webhcat-env templateAdd the following lines at the bottom,WEBHACAT_GC_LOG_OPTS="-Xlog

Error:ERROR [2016-12-13 00:48:04,166] ({pool-2-thread-2} Job.java[run]:189) - Job failedjava.lang.NoClassDefFoundError: org/apache/hadoop/security/UserGroupInformation$AuthenticationMethod at org.apache.zeppelin.jdbc.security.JDBCSecurityImpl.getAuthtype(JDBCSecurityImpl.java:66)....Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.UserGroupInformation$AuthenticationMethodat java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 more Solution:For systems in hdp cluster with no internet access local-repo folder doesnt get created when running zeppelin interpreters which hold the jar files needed for the interpreters.To circumvent this we would need to copy the following jars to a temporary location and then specify these in the interpreter setting from zeppelin UIcreate and copy

When using kerberos with HDP it's not uncommon to find the odd strange encryption type floating around, possibly from a badly configured AD server. By adding the following to the Ambari -> Kerberos config section under supported encryption types its possible to isolate this issue for diagnostic.While its probably not a wise idea to run with all these enabled in production having a full list of supported types can be useful for diagnostic or reference. des-cbc-crc des-cbc-md4 des-cbc-md5 des3-cbc-sha1 arcfour-hmac arcfour-hmac-exp aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha384-192 camellia128-cts-cmac camellia256-cts-cmac