Resources

Hi all,Since my first post I remark that the post's views number in the forums increase in every open or refresh of the post.Sometimes, there is a user that use F5 to have a huge number of views in his post!!I think you need a control (by membre, by IP adress and/or cookies...) in this case before the increasing.Thanks for your comprehension.

To use DPS services, i.e. DLM, for replication between HDP clusters over wired encryptions leverages the Knox gateway functionality. For that two services must be configured in Knox and one Ranger Policy created. This is the quick setup of the required Ambari services in Knox and Ranger.Adding two new services (AMBARI and AMBARIUI) to the Knox advance topology configuration <service> <role>AMBARI</role> <url>http://knox-gw:8080</url> </service> <service> <role>AMBARIUI</role> <url>http://knox-gw:8080</url> </service>In Ranger define a policy to allow the access to AmbariUI Then restart the all marked service in the cluster. Test that it works and login to Ambari by proxying Knox (default port: 8443).Now you can join the cluster in DPS.

HDP deploy with CB CLI This article provides step by step instruction for installing cloudbreak 2.5 release candidate on OSX and launch HDP Environment OSX HighSierra (laptop) Openstack environment keystone-v3 via VPN (internal facing) Pre-Requisite Homebrew Java 1.8+ If you start from scratch on OSX High Sierra # install homebrew/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"# install caskbrew tap caskroom/cask#install pre-reqbrew cask install java8 optionally - installing docker, While the official docs state docker and docker-machine are required, you can skip installing them because cbd brings it with it into ".deps" folder in cloudbreak home. brew cask install dockerbrew install docker-machine-driver-xhyve Concepts and Components We are going to use the following 3 components Cloudbreak Deployer (cbd) Cloudbreak Application (14 docker containers with cloudbreak application at the center) Cloudbreak Command

HDP deploy with CB CLI This article provides step by step instruction for installing cloudbreak 2.5 release candidate on OSX and launch HDP and HDF clusters into Openstack IaaSEnvironment OSX HighSierra (laptop) Openstack environment keystone-v3 via VPN (internal facing)Pre-Requisite Homebrew Java 1.8+ If you start from scratch on OSX High Sierra# install homebrew/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"# install caskbrew tap caskroom/cask#install pre-reqbrew cask install java8# optional - installing docker, While the official docs state docker and docker-machine are required, you can skip installing them because cbd brings it with it into ".deps" folder in cloudbreak home.brew cask install dockerbrew install docker-machine-driver-xhyve</ul><h2>Concepts and Components</h2><p>We are going to use the following 3 components</p><ol> <li>Cloudbreak Deployer (cbd) </li> <li>Cl

In previous post we have seen how to Automate HDP installation with Kerberos authentication on multi node cluster using Ambari Blueprints.In this post, we will see how to deploy multi-node node HDP Cluster with Resource Manager HA via Ambari blueprint..Below are simple steps to install HDP multi node cluster with Resource Manager HA using internal repository via Ambari Blueprints..Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications. Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Step 1: Install Ambari server using steps mentioned under below linkhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/ch_Install

With HIVE-13670Till today we need to remember the complete Hive Connection String either you are using direct 1000 port or ZK connection string. After the above Jira we can optimize that by setting up the environment variable(/etc/profile) on the Edge nodes. export BEELINE_URL_HIVE="<jdbc url>"Example:export BEELINE_URL_HIVE="jdbc:hive2://<ZOOKEEPER QUORUM>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"Now just type beeline -u HIVEEven we can setup multiple connection strings just by setting different naming connections like BEELINE_URL_BATCH, BEELIVE_URL_LLAP. Hope this helps you.

Issue: Knox Gateway fails to start with "org.apache.hadoop.gateway.services.security.KeystoreServiceException: java.io.IOException: Keystore was tampered with, or password was incorrect"Below are the startup logs.2018-03-13 05:17:47,189 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.name=knox2018-03-13 05:17:47,193 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: user.dir=/var/lib/knox2018-03-13 05:25:26,853 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.name=OpenJDK Runtime Environment2018-03-13 05:25:26,853 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.runtime.version=1.8.0_131-b112018-03-13 05:25:26,854 INFO hadoop.gateway (GatewayServer.java:logSysProp(193)) - System Property: java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64/jre2018-03-13 05:25:27,230 INFO hadoop.gateway (GatewayConfigImpl.java:loadConfigResource(322)

HelloAs some of you already know, Solr through knox on HDP platform isn't fully supported yet, however, it is possible to achieve that using an IBM IOP distribution.Here are some steps:Problems like that are usually related with kerberos issues.Pre-reqs: You should have already configured your knox with the desired authentication mode:OBS: In order to use this flag "Dsun.security.krb5.rcache", the jdk 1.8 or above must be used.Root cause: a) You may have not configured your browser for authentication ( SPNEGO ), b) You haven't included your users into the SolR Plugin on Ranger.c) You are hitting a known issue related with this parameter "Dsun.security.krb5.rcache=none", which is better described in this forumhttps://community.hortonworks.com/content/supportkb/150162/error-gssexception-failure-unspecified-at-gss-api.html1) Add the following parameters into your Hadoop core-site.xml:hadoop.proxyuser.knox.groups = *hadoop.proxyuser.knox.hosts = *OBS: You can change the impersonation requi

For security reasons, sometimes we might need to deactivate the ambari users ,rather deleting.In these scenerios make the user inactive from ambari UI, however if the users are more we might need to automate it from rest API as mentioned below.The objective is to deactivate the user "ritesh" . One of method is from UI Go to Admin --> manage users --> users --> click on user name We can see the user is in ACTIVE State. You can use this toggle button to change the value.Deactivate a user from REST API.1. If you try to curl the rest API link http://172.26.113.155:8080/api/v1/users, it will show you all the users in the ambari. To get the properties for a particular user, append the user name at the end of the API (like below)If we analyze above output, we see an element "active"="true" which controls the status of that user (active/inactive)Hence to disable the user, run this below command. Please notice that the command has argument ("Users/active":"false") for PUT method and ha

Cloudbreak 2.4.0 and newer includes a neat feature which allows you to obtain CLI commands from the web UI. This feature can save you a lot of time if you are just getting started with the CLI.PrerequisitesYou must have an instance of Cloudbreak running.You downloaded and configured Cloudbreak CLI. if you still need to do this, refer to Installing the CLI. Obtain Cluster JSON from the UI By far the most useful feature is being able to obtain cluster JSON file from the web UI. There are two easy ways to do this.Option 1 (Does not require creating a cluster)Navigate to the create cluster wizard and specify all required parameters. Once done, on the last page of the wizard there is an option to SHOW CLI COMMAND, which allows you to generate a JSON skeleton of your cluster:You can click COPY the JSON, paste it to a text edit, and save it as template.json. Next, you can copy the command listed in the web UI below the CLI skeleton to create a cluster from the template.json file: cb cluster

Steps to enable Query logging in Ambari server with Postgres Database 1. If it is Postgres as DB then add following property in ambari.properties server.jdbc.properties.loglevel=2 After restart you can see SQL query logging to Ambari-server.out file 2.If it is MySQL as DB then add/update following property in ambari.properties. append below value in JDBC URL ?logger=com.mysql.jdbc.log.Slf4JLogger&profileSQL=trueEx:server.jdbc.rca.url=jdbc:mysql://apappudb.hdp.com:3306/ambariupgrade?logger=com.mysql.jdbc.log.Slf4JLogger&profileSQL=trueandserver.jdbc.url=jdbc:mysql://apappudb.hdp.com:3306/ambariupgrade?logger=com.mysql.jdbc.log.Slf4JLogger&profileSQL=true After re-start you will find the logs in ambari-server.log

Zeppelin is A web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.The latest version of zeppelin is 0.6.2 when this article is written, although the community has made lots of effort to improve it. Sometimes you would still meet some weird issues due to environment issue, wrong configuration or zeppelin’s bug of itself. This article would try to illustrate to you to how to diagnose zeppelin if you meet some issues that you can't figure out what’s wrong.Zeppelin Architecture Before I go to details I’d like to give an illustration of zeppelin’s architecture. So that we can understand where to diagnose. The above is a diagram of zeppelin’s diagram. Overall it has 3 layers: FrontendZeppelin ServerInterpreter Process I would not talk about the details, but just want you to have an overall picture of what components zeppelin has, and usually we hit issues on zeppelin server and in

AbstractThis article objective is to cover the end to end lineage capture with in bigdata platformIntroduction A visual representation of data lineage helps to track data from its origin to its destination. It explains the different processes involved in the data flow and their dependencies. Metadata management is the key input to capturing enterprise dataflow and presenting data lineage from end to end and especially in banking industry due to CCAR requirements and regulations. When it comes to Hortonworks platforms ( HDP Hortonworks Data Platform , Hortonworks Data Flow) there are several components involved beginning from data ingestion, cleansing/transformation all the way through data analytics using Kaka, Nifi, Spark, Hive, Hive LLAP etc in solving enterprise bigdata problems.Problem Statement Apache Atlas provides a scalable and extensible set of core foundational governance services including lineage. Capturing lineage from end to end is complex problem to solve for

1. Kafka Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day. Appender: org.apache.log4j. RollingFileAppender Roll criteria: Every 100MB. Retention: 9 backup logs. Steps: Login to Ambari Goto the config tab of the Kafka broker. Expand Advanced kafka-log4j Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender. Comment out the appender’s date pattern: #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH #log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender Add the following lines for each appender (change the appender name): log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender log4j.appender.kafkaAppender.MaxFileSize=100MB log4j.appender.kafkaAppender.MaxBackupIndex=9 Repeat the steps for all the appenders in the section. Rolling restart Kafk

In this article series, part 1 , part 2 , part 3, part 4 covered various Hbase tuning parameters, scenarios, system side of things etc, in last and part 5 of this series, I will discuss little bit about Phoenix performance parameters and general tips for tuning. I am taking an example of a query which was performing very slow and how we investigated this situation. We will start by reading explain plan of this query. I cannot quote exact query (customer's data) here but it was a select query with some where clause and finally order by conditions. The explain plan of the query is as follows: +--------------------------------------------------------------------------------------------------------------------------------------------------------+ | CLIENT 5853-CHUNK 234762256 ROWS 1671023360974 BYTES PARALLEL 5853-WAY FULL SCAN OVER MESSAGE_LOGS.TEST_MESSAGE | | SERVER FILTER BY ((ROUTING IS NULL OR ROUTING = ‘TEST') AND TESTNAME = 'createTest' AND TESTID = ’TEST’ AND ERRORCODE | | SERVER

In my articles part 1 and part 2 , I explained various parameters which could be tuned for achieving optimized performance from Hbase. In part 3, we discussed some scenarios and aspects to focus while investigating performance issues. Continuing this series, part 4 covers some system and network level investigations.   DISKS Along with investigating potential issues at HBASE and HDFS layer, we must not ignore system side of thingslike OS, network and disks. We see several cases everyday when severe issues at this layer are identified.Detailed investigation is beyond the scope of this article , but we should know where to point fingers. The triggering factor to look at system side are messages such as following in datanode logs at the time of performance issue. WARN datanode.DataNode (BlockReceiver.java:receivePacket(694)) - Slow BlockReceiver write data to disk cost:317ms (threshold=300ms) Following are some of several tests we could do to ascertain disk performance: - Run dd test

As we understood important tuning parameters of Hbase in part 1 and part 2 of this article series, this article focuses on various areas which should be investigated when handling any Hbase performance issue. Locality By locality we mean the physical HDFS blocks related to Hbase Hfiles need to be local to the region server node where this respective region is online. This locality is important because Hbase prefers to use short circuit reads directly from physical disks bypassing HDFS. If a region’s Hfiles are not local to it, it will incur cost to a read latency by doing reads across the node over network. One can monitor locality of a table / region from Hbase Master UI specifically on table’s page by clicking on the listed table name. The value of “1”on the locality column means 100% block locality. Overall Hbase locality is visible on Ambari metrics section of Hbase ( in percentage). Here, Major compaction tries to bring back all Hfiles related to a region on a single region server

TIBCO Enterprise Message Servicehttps://www.tibco.com/products/tibco-enterprise-message-serviceI tested this against the most recent release of TIBCO Enterprise Message Service and their JMS driver available via trial download. I followed the very easy install directions. I downloaded it to a Centos 7 server.Expanded my download to TIB_ems-dev_8.4.0_linux_x86_64Then made it executable and ran TIBCOUniversalInstaller-lnx-x86-64.bin --console.I used all the defaults (I picked server and client) and then quickly ran the finished install server.Running Tibco on Centos 7cd /opt/tibco/ems/8.4/bin/./tibemsd64 -config ~/TIBCO_HOME/tibco/cfgmgmt/ems/data/tibemsd.confExample JMS Queue Settings URL: tcp://servername:7222 class: com.tibco.tibjms.TibjmsQueueConnectionFactory Directory: /opt/tibco/ems/8.4/lib/I believe it just uses these files from that directory: tibjms.jar jms-2.0.jar Once I have my server and port shown, it's easy to add those settings to Apache NiFi.The setting

Cloudbreak 2.5.0 Technical Preview is available now. Here are the highlights: Creating HDF ClustersYou can use Cloudbreak to create HDF clusters from base images on AWS, Azure, Google Cloud and OpenStack. In the Cloudbreak web UI, you can do this by selecting "HDF 3.1" under Platform Version and then selecting an HDF blueprint.Cloudbreak includes one default HDF blueprint "Flow Management: Apache NiFi" and supports uploading your own custom HDF 3.1.1 NiFi blueprints.Note the following when creating NiFi clusters:When creating a cluster, open 9091 TCP port on the NiFi host group. Without it, you will be unable to access the UI.Enabling kerberos is mandatory. You can either use your own kerberos or select for Cloudbreak to create a test KDC.Although Cloudbreak includes cluster scaling (including autoscaling), scaling is not fully supported by NiFi. Downscaling NiFi clusters is not supported - as it can result in data loss when a node is removed that has not yet processed all the data on

In large clusters, where there are multiple services which makes use of single Zookeeper quorum, the state store is maintained as znodes. Hence the count of such znodes are directly proportional to the services that are deployed and also the activity on the cluster. If LLAP apps are deployed in such clusters, it is imperative that slider is enabled (by setting the property, "hadoop.registry.rm.enabled"), this will introduce an overhead in the Znode scans for all the application containers that are created and destroyed on timely basis. The behavior of the scans are as described below, If the property is set in core-site.xml or yarn-site.xml, the YARN Resource Manager will behave as follows: 1. On startup: create the initial root paths of /, /services and /users. On a secure cluster, access will be restricted to the system accounts (see below). 2. When a user submits a job: create the user path under /users. 3. When a container is completed: delete from the registry all service recor

As we understood basic parameters of Hbase in Part 1 , lets try and understand some advanced parameters which should be tried after consultation with experts or by those who know what they are doing. hbase.hstore.blockingStoreFiles Default value of this parameter is 10. We know that each memstore flush creates an Hfile (hbase.hregion.memstore.flush.size), now the purpose this parameter serves is to send a message along the write pipeline that unless these many Hfiles are compacted using minor compaction, we should not go ahead with any more flushes. One would see messages in logs such as “Too many HFiles, delaying flush” . But like it says, it can only delay flush up to certain seconds, and even if the writes continue to happen, memstore could stretch only up to the size guided by : hbase.hregion.memstore.flush.size X hbase.hregion.memstore.block.multiplier Once this limit reaches , no more writes will be accepted by region server for this region and you will see messages like "org.apa

Following bullets provide, vantage point from which Applications on Hive has to be analyzed for performance Application typesIngestion intensive Staging/Storage intensive ETL intensive Consumption intensive Data Model used Level of NormalizationStar schema Table design Storage format usedcompressionUsage of collection data types (struct,array,map)Partition Is there a possibility of over partitionWhether dynamic partition enabled Bucket (Review join conditions on bucketed column ) Functions Usage of UDF,UDAF Query pattern Select with where ( map only)Group by (map shuffle reduce)Order by (map shuffle single reduce )Analytical functionsSort by (map shuffle multi reduce )JoinMap join ( these are mapper only but would seek heavier memory )Sort merge joinPartition column usage -( Especially For huge transaction tables )From source table - Usage of multi passTable size : For huge table

https://roaringelephant.org/2018/04/10/episode-82-dataworks-summit-berlin-2018-preview/ Next week is DataWorks Summit Berlin week! Your two hosts will be in attendance and in this episode we go over the agenda and plan which sessions we want to attend and why. Peppered throughout we add further insights and experiences from previous years. Audio Player 00:00 00:00 Play in new window | Download (Duration: 47:38 — 33.0MB)Unfortunately, Dave’s network was a little unstable and there are a couple audio glitches in this episode.For some session statistics or if you can use some help deciding what sessions you want to attend, you can use the dashboard we created: DSW2018 Berlin dashboard (http://aka.ms/DWS2018)Click the screenshot above or go to http://aka.ms/DWS2018 to access the dashboard. It is a dynamic report: clicking on graph elements (bars of pie slices) will apply filters on all the visualizations and the session list. Use control-click to combine filters.The Summit ag

https://roaringelephant.org/2018/04/03/episode-81-roaring-news/In this installment of Big Data News, we talk about the recent Facebook leak, how everybody is still doing it wrong (according to some at least) and installing Hadoop “the old-fashioned way”. Also briefly covered is Elastic’s X-Pack, now even more “open” than before, but still rather closed it would seem. Breaking News Audio Player 00:00 00:00 Play in new window | Download (Duration: 26:19 — 18.3MB) Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Short Description:This article covers how to improve the performance of the NiFi UI. Article:Over time it has been seen that the users of NiFi have been building very large dataflows consisting of many thousands of components (processor, reporting tasks, controller services, etc). While NiFi in no way limits to any degree the number of components that can be added to the NiFi canvas, the more components a user adds, the less responsive the UI becomes. This processor explosion not only affects the responsiveness of the UI, but can also lead to unexpected node disconnections.--- What are the various states a component can have?NiFi components have multiple states that consist of stopped, started, enabled, and disabled. Beyond these states exists one of two statuses:Valid: Component configuration was successfully validated. This means that all required properties have been configured and in the case of processors all required connections have been accounted for (connected to another com