Resources

HDP 2.5.3.0 and Ambari 2.4.2.0 with SmartSense 1.3.1 is now available for immediate use.Being maintenance releases, there are no features introduced. These releases contain bug fixes. HDP 2.5.3.0 has about 300 bugs since HDP 2.5.0.0. Ambari 2.4.2.0 contains about 140 fixes since Ambari 2.4.0.1 was released.Please consult the HDP 2.5.3.0 documentation and release notes. Also the Ambari 2.4.2.0 documentation and release notes.Thank you

Quick tips on how to find low level hardware performance stats. I use it often for NiFi/Spark/Hadoop. This is not limited to those use services. Additionally, this is not a exhaustive list nor am I advocating one tool over the another. Just a few I have ran over the years during my implementations/POCs experience. These give me insights whether I have allocated enough physical resources to run the services. I highly recommend not assuming what your hardware can or can't do. Benchmark it! How? Read my article here.Lets get to it.CPU stats iostat -c 1 3 will provide you cpu stats every 1 second 3 times. Output of the report (sourced right from here😞 CPU Utilization ReportThe first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:%userShow the percentage of CPU utilization that occurred while executing at the user level (application).

After completing Ambari upgrade from Ambari 1.7 to Ambari 2.0.2, Ambari failed to start because of below error(s):[root@xyz ~]# service ambari-server start Using python /usr/bin/python2.6 Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... ERROR: Exiting with exit code -1. REASON: Ambari Server java process died with exitcode 255. Check /var/log/ambari-server/ambari-server.out for more information. .If you check /var/log/ambari-server/ambari-server.out logs, you get below error(s):[root@xyz ~]# cat /var/log/ambari-server/ambari-server.out [EL Warning]: metadata: 2016-12-01 10:01:23.436--ServerSession(306824177)--The reference column name [resource_type_id] mapped on the element [field perm

This article describes how to launch Apache NiFi 1.0.0 on docker. To launch Apache NiFi 1.1.0 on docker go here.During my development of the Json2CSV processor here, I quickly found a need for an environment to test my processor. I don't want to build and install NiFi from my laptop since I need all my applications isolated from each other for ease of maintenance. Docker to the rescue! Similar to how I launch a PyCharm IDE from a docker image here which will rerender back to my laptop. Isolation! I like to keep it simply. Put everything in a Dockerfile and allow myself to quickly launch a NiFi Docker image. Here are the steps to get you up and runningPrerequisitesDownload latest virtualbox from here.To run docker containers or build images a docker machine is required. Download docker machine from here.First pull the prebuilt and complied docker image (https://hub.docker.com/r/sunileman/dockernifi/) by running this command:docker pull sunileman/dockernifiNow you have the docker

There are many ways to run Hadoop on virtual machines. Earlier this year I tried several approaches, and ended up using a helpful Quick Start Guide written by Yusaku Sako. The Quick Start uses VirtualBox, Vagrant, and predefined scripts to set up a multi-node HDP cluster. You can choose which version of Ambari to install, and then choose and install an associated version of the HDP stack. For anyone new to virtual machines, there is now a Quick Start for New VM Users. The extended version adds background information and additional details for installing Ambari and the HDP stack. Topics include: Terminology Prerequisites Installing VirtualBox and Vagrant Starting Linux Virtual Machines Accessing Virtual Machines Installing Ambari Installing the HDP Stack Troubleshooting Reference information for basic Vagrant commands

INTRODUCING THE HORTONWORKS CONNECTED DATA CLOUD TECHNICAL PREVIEW To this end, we are introducing the “ Hortonworks Connected Data Cloud” Technical Preview. This Technical Preview gives you a way to quickly spin up Apache Hive and Apache Spark clusters that are ready to run ephemeral workloads in your Amazon Web Services (AWS) environment.Source Step 1 Follow the video in this http://hortonworks.github.io/hdp-aws/launch/ Step 2 Create and Terminate a cluster. Ambari Services starting Install Completed GUI access different views

Over the last year, Oracle has continued to update and add support for Hortonworks HDP. Below is a list of products which support using Hortonworks HDP 2.5.0.0Big Data SQL https://www.oracle.com/database/big-data-sql/index.htmlBig Data Connector https://www.oracle.com/database/big-data-connectors/certifications.html Includes -Oracle SQL Connector for HDFS-Oracle Loader for Hadoop-Oracle Data Integrator-Oracle XQuery for Hadoop-Oracle R Advanced Analytics for Hadoop-Oracle Datasource for HadoopSpatial and Graph https://www.oracle.com/database/spatial/index.htmlGoldenGate for Big Data https://www.oracle.com/middleware/data-integration/goldengate/big-data/index.htmlOracle Data Integrator Enterprise Edition https://www.oracle.com/middleware/data-integration/enterprise-edition-big-data/index.htmlBig Data Discovery https://www.oracle.com/big-data/big-data-discovery/index.html

Prerequisites1) Service Ambari Infra installed -> Ranger will use Ambari Infra's SolrCloud for Ranger Audit2) MySQL installed and running (I'll use Hive's Metastore MySQL instance * MySQL is one of the many DB options)Installing Apache Ranger using Ambari Infra (SolrCloud) for Ranger Audit1) Find the location of mysql-connector-java.jar (assume /usr/share/java/mysql-connector-java.jar) run the following command on the Ambari serversudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 2) In Ambari, click Add Service 3) Choose Ranger and click Next4) Choose "I have met all the requirements above." and click Proceed (this was done in #1 above)5) Assign master(s) for "Ranger Usersync" and "Ranger Admin" and click Next6) Assign Slaves and Clients - since we did not install Apache Atlas, Ranger TagSync is not required and click Next7) Customize Services -> Ranger Audit, click on "OFF" to enable SolrCloudBefore clicking:After clicking:😎 Customi

- Suppose if we have a webserver configured in front of the ambari server something like following "/etc/httpd/conf/httpd.conf"<VirtualHost erie1.example.com:80> ServerAdmin erie1.example.com ProxyPass / http://erie1.example.com:8080/ </VirtualHost>Means any request coming to "http://erie1.example.com" will be passed to "http://erie1.example.com:8080/" now with this setup we will see that ambari is working fine and accessible.However when we try to access the PigView then we will see that the PigView is not able to load the scripts:Also we will see that in the debugger tool that the script URL is not accessible (404 http response)http://erie1.example.com/api/v1/views/PIG/versions/1.0.0/instances/Pig_A/resources/file/%2Fuser%2Fadmin%2Fpig%2Fscripts%2Fsmalldatatest-2016-11-25_09-26.pigAbove is the URL using which PigView tries to read the script that is stored in the HDFS. Even when we open this URL in the browser we get the same 404 error. However we see that the

Usually Ambari server generates certs with 1 year validity. after an year all Agent would fail to communicate with Ambari-server. Agent and Server certs would be expired. below steps can be followed to replace/resolve the expired certs.  1. stop ambari-server2. take a back of existing /var/lib/ambari-server/keys folder and empty it.3. download the attached keys.zip file and copy it to /var/lib/ambari-server/ , your new folder structure should be like /var/lib/ambari-server/keys/ca.config,/var/lib/ambari-server/keys/db/, - basically this is a fresh keys folder ( this is what you get when you install ambari-server )4. Take a back up of all the Agent certs located at /var/lib/ambari-agent/keys/ in all the hosts.5. Delete all the files under /var/lib/ambari-agent/keys/ folder6. restart ambari-server.Note: ambari-server should create new certs under /var/lib/ambari-server/keys/ca.crt , /var/lib/ambari-server/keys/ca.key ....7. restart ambari-agentNote: ambari-agent should create

I.OverviewHadoop can use a lot of network bandwidth as well as disk storage. Typical configuration options such as data replication settings to 3x can further increase demands to the network as can normally desirable technologies such as network storage. For example, loading a 1 TB file with a 3x replication factor can end up consuming 3 TB of network traffic to successfully load the file and another 3 TB of data movement if network drives are utilized. The sections below highlight an aggregation of previously identified best practices to best setup the network to support the deployment of an HDP cluster.II.Network Topology Machines should be on an isolated network from the rest of the data center. This means that no other applications or nodes should share network I/O with the Hadoop infrastructure. This is recommended as Hadoop is I/O intensive, and all other interference should be removed for a performant cluster.Machines should have static IPs. This will enable stability in the

Here is the scenario:1. I have workflow.xml which contains hive action..2. I have added <job-xml> tag inside hive action and provided path to the hive-site.xml (say /tmp/hive-site.xml).3. I have added hive-site.xml to the ${wf.application.path}/lib directory as well..4. I have hive-site.xml to the oozie sharelib under:/user/oozie/sharelib/lib_<timestamp>/oozie/hive-site.xml/user/oozie/sharelib/lib_<timestamp>/sqoop/hive-site.xml/user/oozie/sharelib/lib_<timestamp>/hive/hive-site.xml.5. My simple hive workflow is failing with below error:Oozie Hive action configuration ================================================================= Using action configuration file /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/action.xml ------------------------ Setting env property for mapreduce.job.credentials.binary to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_144311159

Grafana username and password is stored in a sqlite3 database. One of the way is to reset the password back to admin first and then can be changed in Grafana Dashboard. To do the same, following steps could be used:1. Logon to the node where Grafana is installed and invoke Grafana sqlite3 database as follows:# sqlite3 /var/lib/ambari-metrics-grafana/grafana.dbsqlite> select salt, password from user;pyaUhfDzYg|54c7d1ce2eeaa6000bd84407d0f8ab4663dfa575e0a326bc70dc5cab4b864f6677b21879dbf5e33427c88f9160f744b625bfsqlite> update user set password = '59acf18b94d7eb0694c61e60ce44c110c7a683ac6a8f09580d626f90f4a242000746579358d77dd9e570e83fa24faa88a8a6', salt = 'F3FAxVm33R' where login = 'admin';sqlite> .exit2. Once done, edit Ambari Metrics Server-Configs and update Grafana Password to "admin"3. Restart the Ambari Metrics Server4. Access Grafana page using the Quick Links under Ambari Metric Server Dashboard5. Click on the Grafana Symbol in the top left corner of the screen and Sign-in

Sometimes, after disabling Kerberos successfully, you can get below error while accessing NN or RM UIs..Because of this, all the alerts, Ambari metrics data can be broken because JMX data won't be available to Ambari Metrics or Alerts via HTTP..Resolution:Modify below propery in core-site.xml via Ambari and restart required services.hadoop.http.authentication.simple.anonymous.allowed=trueNote - Most of the times issue happens because of this property being set as false after disabling Kerberos.Please also compare below property and see if it is correct. hadoop.security.authentication=simple.Issue is simple but it takes lot of time to troubleshoot. Hence posting this article to reduce someone's time 🙂Please comment if you face any issues or need any further help on this. Happy Hadooping!! 🙂

HDP 2.5 GA'd phoenix query server. This makes connecting to phoenix much easier. Article will walk through steps on how to connect to phoenix Phoenix Query Server via DBVisualizer. Grab the phoenix thin jdbc driver onto your desktop. On the HDP 2.5 here is the location of the jdbc driverStart up DBVisualizer. From the top menu bar select Tools->Driver ManagerPopoluate the fields:Name:Apache Phoenix Thin ClientI used phoenixthin.URL Format: jdbc:phoenix:thin:url=<scheme>://<server-hostname>:<port>[...]Client on the folder iconLocate your phoenix thin jdbc driver you downloaded to your desktop. Once selected click on ok and and now you have the driver loaded.Lets connect to phoenix. Click on the icon shown below which creates a new db connection. Then select "Use Wizard".Enter a connection name. I used phoenix-QPSNow select the driver you loaded in the previous steps. I named my driver phoenixthinNext enter your userid & password. For this example I use

The objective of this article is to have a discussion and capture critical points for consideration when developing a Streaming Application using Storm, Spark or any other streaming technology. Intent is to add additional points from discussion and comments to evolve this article into a comprehensive guideline for Stream Engineering.Here are a few points synthesized from my experience developing streaming applications using Storm and Flume:1. Select Streaming Application’s SLAs requirementsMost Streaming Applications can be classified based on their SLA requirements into:At-most once (data drops are acceptable)At-least once (data drops are NOT acceptable)Exactly once (idempotent computations)These requirements tend to heavily drive design requirements like whether or not to ack tuples (in Storm), or if downsampling is acceptable (sensor data processing)2. Minimal and careful data replaysIn Guaranteed Processing SLA use cases, data replay must be minimized by application logic to avoid

BehaviorThe number of cells returned to the client are normally filtered based on the table configuration; however, when using the RAW => true parameter, you can retrieve all of the versions kept by HBase, unless there was a major compaction or a flush to disk event in meanwhile.DemonstrationCreate a table with a single column family:create 't1', 'f1' Configure it to retain a maximum version count of 3:alter 't1',NAME=>'f1',VERSIONS=>3Perform 4 puts:put 't1','r1','f1:c1',1put 't1','r1','f1:c1',2put 't1','r1','f1:c1',3put 't1','r1','f1:c1',4Scan with RAW=>true. I used VERSIONS as 100 for a catch-all. It could have been anything greater than 3 (number of versions set previously). Unless specified, only the latest version is returned by scan command.scan 't1',{RAW=>true,VERSIONS=>100}The above scan returns all four versions.ROW COLUMN+CELL r1 column=f1:c1,timestamp=1479950685181, value=4 r1 column=f1:c1,timestamp=1479950685155, value=3 r1 column=f1:c1,timestamp=14799

Hivemall: Machine Learning on Hive, Pig and Spark SQL Install HiveMall https://github.com/myui/hivemall/wiki/InstallationPick latest release https://github.com/myui/hivemall/releases # Setup Your Environment $HOME/.hivercadd jar /home/myui/tmp/hivemall-core-xxx-with-dependencies.jar; source /home/myui/tmp/define-all.hive;# Create a directory in HDFS for the JARhadoop fs -mkdir -p /apps/hivemallhdfs dfs -chmod -R 777 /apps/hivemallcp hivemall-core-0.4.2-rc.2-with-dependencies.jar hivemall-with-dependencies.jarhdfs dfs -put hivemall-with-dependencies.jar /apps/hivemall/hdfs dfs -put hivemall-with-dependencies.jar /apps/hive/warehouse/hdfs dfs -put hivemall-core-0.4.2-rc.2-with-dependencies.jar /apps/hivemallshow functions "hivemall.*";+-----------------------------------------+--+| tab_name |+-----------------------------------------+--+| hivemall.add_bias || hivemall.add_feature_index || hivemall.amplify

Download URLCrazy (http://www.morningstarsecurity.com/downloads/urlcrazy-0.5.tar.gz) An Example Command Line Run for URLCrazy [root@tspanndev13 security]# ./url.sh dataflowdeveloper.comTypo Type,Typo,Valid,Pop,DNS-A,CC-A,Country-A,DNS-MX,ExtnCharacter Omission,daaflowdeveloper.com,true,,,?,,com,Character Omission,dataflodeveloper.com,true,,,?,,com,Character Omission,dataflowdeeloper.com,true,,,?,,com,Character Omission,dataflowdeveloer.com,true,,,?,,com,Character Omission,dataflowdevelope.com,true,,,?,,com,Character Omission,dataflowdeveloper.cm,true,,,?,,cm,Character Omission,dataflowdeveloper.co,false,,,?,,,Character Omission,dataflowdeveloper.om,false,,,?,,,Character Omission,dataflowdevelopercom,false,,,?,,,... Shell Script to Call From Apache NiFi /opt/demo/security/urlcrazy-0.5/urlcrazy -i -f csv -p $@ An Example Command Line Run for NSLookup Non-authoritative answer:sparkdeveloper.com text = "v=spf1 ip4:00.000.0.0/24 ip4:00.000.00.0/24 ip4:11.111.111.0/19 ?all" The Final JSON O

BackgroundWhen we use NiFi flow to load Adobe ClickStream tsv file into hive, we found around 3% rows are in wrong format or missed.Source Data Quality$ awk -F "\t" '{print NF}' 01-weblive_20161014-150000.tsv | sort | uniq -c | sort 1 154 1 159 1 162 1 164 1 167 1 198 1 201 1 467 2 446 2 449 2 569 6 13 10 3 13 146 13 185 15 151 16 54 18 433 21 432 22 238 23 102 26 2 34 138 179 1319412 670After clean the tsv$ awk -F "\t" 'NF == 670' 01-weblive_20161014-150000.tsv >> cleaned.tsv$ awk -F "\t" '{print NF}' cleaned.tsv | sort | uniq -c | sort319412 670Still missed a few percent rows.Root Cause and SolutionWe are using ConvertCSVToAvro and ConvertAvroToORC.The clickstrem tsv files have " in them and the ConvertCSVtoAvro processor uses " as the value for the "CSV quote Character" processor configuration property by default. As a result many tabbed fields end up in the same record. We can get good output by changing this configuration property to anothe

ISSUE :When we click on Datanodes tab on Namenode UI, nothing happens. ROOT CAUSE : This is happening because of https://issues.apache.org/jira/browse/HDFS-10888 and https://issues.apache.org/jira/browse/HDFS-9193 which is fixed in Hadoop version 2.8. But HDP 2.5 is on Hadoop 2.7. RESOLUTION/WORKAROUND : To resolve this, we can work around by editing the below file on both the namenodes/usr/hdp/current/hadoop-hdfs-client/webapps/hdfs/dfshealth.js and changing } else if (u.usedPercentage < 85) {to } else if (v < 85) {We should be able to access datanode tabs after restarting namenode.

Special thanks to Michael Young for the help to be my mentor.Step1:cd /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs/conf/screen-shot-2016-11-21-at-100524-am.pngStep2:vi managed-schema: add these 3 lines<field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" />screen-shot-2016-11-21-at-100929-am.pngStep3:vi solrconfig.xml on the same directory.Replace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor />as<updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor> <str name="fieldName"&g

The scope of this article is to explore ways to simulate Slowly changing dimension (SCD-Type2), given the append-only nature of HAWQ, the idea is to capture delta from source systems and have a flag in each row which shows whether the row is A-Append, U-Update or D-Delete, this will ensure that we can capture this information in our base tables in HAWQ. In an Enterprise Data Lake, you will come across scenarios where you have to capture updates/deletes from Source Systems, since HAWQ is append-only, we do need an alternative way to simulate this, which we discuss in this article. For each source table, we will create a corresponding base table, and then create view on top of it for access. The base table will have 2 additional attributes “update_version” and “delete_flag”, the “update_version” of type “bigserial” which will be an auto increment number which will track which version of the row is the latest, largest number would signify most recent record. The “delete_flag” would dete

Known issue :There is a known issue with this configuration. When Ranger service is installed using custom user (Service user’s name can be updated during installation of a service on a Misc Tab on Ambari) and location of PID file was changed by the user (i.e. value of below given property is changed from its default.), there is an issue with service startup. On Ambari Configs section of Ranger : ranger-env / ranger_pid_dir = /var/run/ranger ## is a default valueIf user changes value of the above property to be a different folder, there is an issue with usersync service startup. When usersync start command is called, it invokes /grid/0/hdp/<version>/ranger-usersync/ranger-usersync-services.sh script, which has hardcoded value of PID file for usersync process and the owner as ‘ranger’. Because of this, usersync restart will fail as custom service user does not have permission to create or write pid file on the directory owned by user ‘ranger’.WorkaroundIn order to solve above

Question:I have installed HDP cluster using Ambari which has Ranger service installed and working properly. I enabled the plugin Kafka for Ranger. I noticed something a little bit annoying. If Ranger Admin is down, Kafka will take a long time to start, because it tries to connect to Ranger Admin to get the repository. The error log is something like as shown below -### Will retry 74 time(s), caught exception: Connection failed to Ranger Admin. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s) ### Is there a way to decrease this number of retry or the duration of sleep between retry ? Ranger Admin down should not have any impact on the components for which the plugins are enabled, right ? Findings: Ambari uses the following Script to return the ranger admin login check response. /usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) /usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_funct