Community Articles

Prerequisites1) Service Ambari Infra installed -> Ranger will use Ambari Infra's SolrCloud for Ranger Audit2) MySQL installed and running (I'll use Hive's Metastore MySQL instance * MySQL is one of the many DB options)Installing Apache Ranger using Ambari Infra (SolrCloud) for Ranger Audit1) Find the location of mysql-connector-java.jar (assume /usr/share/java/mysql-connector-java.jar) run the following command on the Ambari serversudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 2) In Ambari, click Add Service 3) Choose Ranger and click Next4) Choose "I have met all the requirements above." and click Proceed (this was done in #1 above)5) Assign master(s) for "Ranger Usersync" and "Ranger Admin" and click Next6) Assign Slaves and Clients - since we did not install Apache Atlas, Ranger TagSync is not required and click Next7) Customize Services -> Ranger Audit, click on "OFF" to enable SolrCloudBefore clicking:After clicking:😎 Customi

- Suppose if we have a webserver configured in front of the ambari server something like following "/etc/httpd/conf/httpd.conf"<VirtualHost erie1.example.com:80> ServerAdmin erie1.example.com ProxyPass / http://erie1.example.com:8080/ </VirtualHost>Means any request coming to "http://erie1.example.com" will be passed to "http://erie1.example.com:8080/" now with this setup we will see that ambari is working fine and accessible.However when we try to access the PigView then we will see that the PigView is not able to load the scripts:Also we will see that in the debugger tool that the script URL is not accessible (404 http response)http://erie1.example.com/api/v1/views/PIG/versions/1.0.0/instances/Pig_A/resources/file/%2Fuser%2Fadmin%2Fpig%2Fscripts%2Fsmalldatatest-2016-11-25_09-26.pigAbove is the URL using which PigView tries to read the script that is stored in the HDFS. Even when we open this URL in the browser we get the same 404 error. However we see that the

Usually Ambari server generates certs with 1 year validity. after an year all Agent would fail to communicate with Ambari-server. Agent and Server certs would be expired. below steps can be followed to replace/resolve the expired certs.  1. stop ambari-server2. take a back of existing /var/lib/ambari-server/keys folder and empty it.3. download the attached keys.zip file and copy it to /var/lib/ambari-server/ , your new folder structure should be like /var/lib/ambari-server/keys/ca.config,/var/lib/ambari-server/keys/db/, - basically this is a fresh keys folder ( this is what you get when you install ambari-server )4. Take a back up of all the Agent certs located at /var/lib/ambari-agent/keys/ in all the hosts.5. Delete all the files under /var/lib/ambari-agent/keys/ folder6. restart ambari-server.Note: ambari-server should create new certs under /var/lib/ambari-server/keys/ca.crt , /var/lib/ambari-server/keys/ca.key ....7. restart ambari-agentNote: ambari-agent should create

I.OverviewHadoop can use a lot of network bandwidth as well as disk storage. Typical configuration options such as data replication settings to 3x can further increase demands to the network as can normally desirable technologies such as network storage. For example, loading a 1 TB file with a 3x replication factor can end up consuming 3 TB of network traffic to successfully load the file and another 3 TB of data movement if network drives are utilized. The sections below highlight an aggregation of previously identified best practices to best setup the network to support the deployment of an HDP cluster.II.Network Topology Machines should be on an isolated network from the rest of the data center. This means that no other applications or nodes should share network I/O with the Hadoop infrastructure. This is recommended as Hadoop is I/O intensive, and all other interference should be removed for a performant cluster.Machines should have static IPs. This will enable stability in the

Here is the scenario:1. I have workflow.xml which contains hive action..2. I have added <job-xml> tag inside hive action and provided path to the hive-site.xml (say /tmp/hive-site.xml).3. I have added hive-site.xml to the ${wf.application.path}/lib directory as well..4. I have hive-site.xml to the oozie sharelib under:/user/oozie/sharelib/lib_<timestamp>/oozie/hive-site.xml/user/oozie/sharelib/lib_<timestamp>/sqoop/hive-site.xml/user/oozie/sharelib/lib_<timestamp>/hive/hive-site.xml.5. My simple hive workflow is failing with below error:Oozie Hive action configuration ================================================================= Using action configuration file /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/action.xml ------------------------ Setting env property for mapreduce.job.credentials.binary to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_144311159

Grafana username and password is stored in a sqlite3 database. One of the way is to reset the password back to admin first and then can be changed in Grafana Dashboard. To do the same, following steps could be used:1. Logon to the node where Grafana is installed and invoke Grafana sqlite3 database as follows:# sqlite3 /var/lib/ambari-metrics-grafana/grafana.dbsqlite> select salt, password from user;pyaUhfDzYg|54c7d1ce2eeaa6000bd84407d0f8ab4663dfa575e0a326bc70dc5cab4b864f6677b21879dbf5e33427c88f9160f744b625bfsqlite> update user set password = '59acf18b94d7eb0694c61e60ce44c110c7a683ac6a8f09580d626f90f4a242000746579358d77dd9e570e83fa24faa88a8a6', salt = 'F3FAxVm33R' where login = 'admin';sqlite> .exit2. Once done, edit Ambari Metrics Server-Configs and update Grafana Password to "admin"3. Restart the Ambari Metrics Server4. Access Grafana page using the Quick Links under Ambari Metric Server Dashboard5. Click on the Grafana Symbol in the top left corner of the screen and Sign-in

Sometimes, after disabling Kerberos successfully, you can get below error while accessing NN or RM UIs..Because of this, all the alerts, Ambari metrics data can be broken because JMX data won't be available to Ambari Metrics or Alerts via HTTP..Resolution:Modify below propery in core-site.xml via Ambari and restart required services.hadoop.http.authentication.simple.anonymous.allowed=trueNote - Most of the times issue happens because of this property being set as false after disabling Kerberos.Please also compare below property and see if it is correct. hadoop.security.authentication=simple.Issue is simple but it takes lot of time to troubleshoot. Hence posting this article to reduce someone's time 🙂Please comment if you face any issues or need any further help on this. Happy Hadooping!! 🙂

HDP 2.5 GA'd phoenix query server. This makes connecting to phoenix much easier. Article will walk through steps on how to connect to phoenix Phoenix Query Server via DBVisualizer. Grab the phoenix thin jdbc driver onto your desktop. On the HDP 2.5 here is the location of the jdbc driverStart up DBVisualizer. From the top menu bar select Tools->Driver ManagerPopoluate the fields:Name:Apache Phoenix Thin ClientI used phoenixthin.URL Format: jdbc:phoenix:thin:url=<scheme>://<server-hostname>:<port>[...]Client on the folder iconLocate your phoenix thin jdbc driver you downloaded to your desktop. Once selected click on ok and and now you have the driver loaded.Lets connect to phoenix. Click on the icon shown below which creates a new db connection. Then select "Use Wizard".Enter a connection name. I used phoenix-QPSNow select the driver you loaded in the previous steps. I named my driver phoenixthinNext enter your userid & password. For this example I use

The objective of this article is to have a discussion and capture critical points for consideration when developing a Streaming Application using Storm, Spark or any other streaming technology. Intent is to add additional points from discussion and comments to evolve this article into a comprehensive guideline for Stream Engineering.Here are a few points synthesized from my experience developing streaming applications using Storm and Flume:1. Select Streaming Application’s SLAs requirementsMost Streaming Applications can be classified based on their SLA requirements into:At-most once (data drops are acceptable)At-least once (data drops are NOT acceptable)Exactly once (idempotent computations)These requirements tend to heavily drive design requirements like whether or not to ack tuples (in Storm), or if downsampling is acceptable (sensor data processing)2. Minimal and careful data replaysIn Guaranteed Processing SLA use cases, data replay must be minimized by application logic to avoid

BehaviorThe number of cells returned to the client are normally filtered based on the table configuration; however, when using the RAW => true parameter, you can retrieve all of the versions kept by HBase, unless there was a major compaction or a flush to disk event in meanwhile.DemonstrationCreate a table with a single column family:create 't1', 'f1' Configure it to retain a maximum version count of 3:alter 't1',NAME=>'f1',VERSIONS=>3Perform 4 puts:put 't1','r1','f1:c1',1put 't1','r1','f1:c1',2put 't1','r1','f1:c1',3put 't1','r1','f1:c1',4Scan with RAW=>true. I used VERSIONS as 100 for a catch-all. It could have been anything greater than 3 (number of versions set previously). Unless specified, only the latest version is returned by scan command.scan 't1',{RAW=>true,VERSIONS=>100}The above scan returns all four versions.ROW COLUMN+CELL r1 column=f1:c1,timestamp=1479950685181, value=4 r1 column=f1:c1,timestamp=1479950685155, value=3 r1 column=f1:c1,timestamp=14799

Hivemall: Machine Learning on Hive, Pig and Spark SQL Install HiveMall https://github.com/myui/hivemall/wiki/InstallationPick latest release https://github.com/myui/hivemall/releases # Setup Your Environment $HOME/.hivercadd jar /home/myui/tmp/hivemall-core-xxx-with-dependencies.jar; source /home/myui/tmp/define-all.hive;# Create a directory in HDFS for the JARhadoop fs -mkdir -p /apps/hivemallhdfs dfs -chmod -R 777 /apps/hivemallcp hivemall-core-0.4.2-rc.2-with-dependencies.jar hivemall-with-dependencies.jarhdfs dfs -put hivemall-with-dependencies.jar /apps/hivemall/hdfs dfs -put hivemall-with-dependencies.jar /apps/hive/warehouse/hdfs dfs -put hivemall-core-0.4.2-rc.2-with-dependencies.jar /apps/hivemallshow functions "hivemall.*";+-----------------------------------------+--+| tab_name |+-----------------------------------------+--+| hivemall.add_bias || hivemall.add_feature_index || hivemall.amplify

Download URLCrazy (http://www.morningstarsecurity.com/downloads/urlcrazy-0.5.tar.gz) An Example Command Line Run for URLCrazy [root@tspanndev13 security]# ./url.sh dataflowdeveloper.comTypo Type,Typo,Valid,Pop,DNS-A,CC-A,Country-A,DNS-MX,ExtnCharacter Omission,daaflowdeveloper.com,true,,,?,,com,Character Omission,dataflodeveloper.com,true,,,?,,com,Character Omission,dataflowdeeloper.com,true,,,?,,com,Character Omission,dataflowdeveloer.com,true,,,?,,com,Character Omission,dataflowdevelope.com,true,,,?,,com,Character Omission,dataflowdeveloper.cm,true,,,?,,cm,Character Omission,dataflowdeveloper.co,false,,,?,,,Character Omission,dataflowdeveloper.om,false,,,?,,,Character Omission,dataflowdevelopercom,false,,,?,,,... Shell Script to Call From Apache NiFi /opt/demo/security/urlcrazy-0.5/urlcrazy -i -f csv -p $@ An Example Command Line Run for NSLookup Non-authoritative answer:sparkdeveloper.com text = "v=spf1 ip4:00.000.0.0/24 ip4:00.000.00.0/24 ip4:11.111.111.0/19 ?all" The Final JSON O

BackgroundWhen we use NiFi flow to load Adobe ClickStream tsv file into hive, we found around 3% rows are in wrong format or missed.Source Data Quality$ awk -F "\t" '{print NF}' 01-weblive_20161014-150000.tsv | sort | uniq -c | sort 1 154 1 159 1 162 1 164 1 167 1 198 1 201 1 467 2 446 2 449 2 569 6 13 10 3 13 146 13 185 15 151 16 54 18 433 21 432 22 238 23 102 26 2 34 138 179 1319412 670After clean the tsv$ awk -F "\t" 'NF == 670' 01-weblive_20161014-150000.tsv >> cleaned.tsv$ awk -F "\t" '{print NF}' cleaned.tsv | sort | uniq -c | sort319412 670Still missed a few percent rows.Root Cause and SolutionWe are using ConvertCSVToAvro and ConvertAvroToORC.The clickstrem tsv files have " in them and the ConvertCSVtoAvro processor uses " as the value for the "CSV quote Character" processor configuration property by default. As a result many tabbed fields end up in the same record. We can get good output by changing this configuration property to anothe

ISSUE :When we click on Datanodes tab on Namenode UI, nothing happens. ROOT CAUSE : This is happening because of https://issues.apache.org/jira/browse/HDFS-10888 and https://issues.apache.org/jira/browse/HDFS-9193 which is fixed in Hadoop version 2.8. But HDP 2.5 is on Hadoop 2.7. RESOLUTION/WORKAROUND : To resolve this, we can work around by editing the below file on both the namenodes/usr/hdp/current/hadoop-hdfs-client/webapps/hdfs/dfshealth.js and changing } else if (u.usedPercentage < 85) {to } else if (v < 85) {We should be able to access datanode tabs after restarting namenode.

Special thanks to Michael Young for the help to be my mentor.Step1:cd /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs/conf/screen-shot-2016-11-21-at-100524-am.pngStep2:vi managed-schema: add these 3 lines<field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" />screen-shot-2016-11-21-at-100929-am.pngStep3:vi solrconfig.xml on the same directory.Replace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor />as<updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor> <str name="fieldName"&g

The scope of this article is to explore ways to simulate Slowly changing dimension (SCD-Type2), given the append-only nature of HAWQ, the idea is to capture delta from source systems and have a flag in each row which shows whether the row is A-Append, U-Update or D-Delete, this will ensure that we can capture this information in our base tables in HAWQ. In an Enterprise Data Lake, you will come across scenarios where you have to capture updates/deletes from Source Systems, since HAWQ is append-only, we do need an alternative way to simulate this, which we discuss in this article. For each source table, we will create a corresponding base table, and then create view on top of it for access. The base table will have 2 additional attributes “update_version” and “delete_flag”, the “update_version” of type “bigserial” which will be an auto increment number which will track which version of the row is the latest, largest number would signify most recent record. The “delete_flag” would dete

Known issue :There is a known issue with this configuration. When Ranger service is installed using custom user (Service user’s name can be updated during installation of a service on a Misc Tab on Ambari) and location of PID file was changed by the user (i.e. value of below given property is changed from its default.), there is an issue with service startup. On Ambari Configs section of Ranger : ranger-env / ranger_pid_dir = /var/run/ranger ## is a default valueIf user changes value of the above property to be a different folder, there is an issue with usersync service startup. When usersync start command is called, it invokes /grid/0/hdp/<version>/ranger-usersync/ranger-usersync-services.sh script, which has hardcoded value of PID file for usersync process and the owner as ‘ranger’. Because of this, usersync restart will fail as custom service user does not have permission to create or write pid file on the directory owned by user ‘ranger’.WorkaroundIn order to solve above

Question:I have installed HDP cluster using Ambari which has Ranger service installed and working properly. I enabled the plugin Kafka for Ranger. I noticed something a little bit annoying. If Ranger Admin is down, Kafka will take a long time to start, because it tries to connect to Ranger Admin to get the repository. The error log is something like as shown below -### Will retry 74 time(s), caught exception: Connection failed to Ranger Admin. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s) ### Is there a way to decrease this number of retry or the duration of sleep between retry ? Ranger Admin down should not have any impact on the components for which the plugins are enabled, right ? Findings: Ambari uses the following Script to return the ranger admin login check response. /usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) /usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_funct

PROBLEM STATEMENT: Ambari not displaying and service action buttons after login. It displays "Move Master Wizard In Progress". Not able to perform any operation from ambari and same is affecting for all users.ERROR: ROOT CAUSE: The issue seems that the user who logged inn to ambari UI[acting as admin], has tried to perform operation which was left in middle and just logged off from UI. The operation was cached in ambari UI and hence is reflected for other users in UI.RESOLUTION: Take ambari server backup and run below command from Ambari node - curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"<username>\",\"controllerName\":\"<controller_name>\"}"}'

PROBLEM STATEMENT: We have recently added 40 Datnodes added to cluster and they went down immediately after adding. Below was the exception found - ERROR:ROOT CAUSE:Customer added the 40nodes to the cluster running rack topology script as per link given - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/ch05.htmlThe topology script which was copied by customer from above link was missing "fi" on last line and hence was giving above exception. RESOLUTION: Populated corrected script rack-topology.sh on namenode and datanodes and tried starting the datanodes after which datanode service was able to startup.

PROBLEM STATEMENT: Customer deleted hdfs repository from ranger ui. Tried re-enabling plugin but the repository was getting created. Reinstalled ranger, but still no luck.There was alert saying - "Ranger Admin Password check"Text is: This alert is used to ensure that the Ranger Admin password in Ambari is correct. Response is: User:amb_ranger_admin credentials on Ambari UI are not in sync with Ranger Doing a test and changing ranger_admin_password to admin in Ambari. Configs->Advanced Restarting Ranger, not prompted but doing so anyway.Also changed local password on ranger admin host: [root@ip-172-53-51-18 admin]# passwd amb_ranger_admin Changing password for user amb_ranger_admin. New password: Retype new password: passwd: all authentication tokens updated successfully. Did restart of HDFS, but now HDFS was not coming up. ERROR: Ranger admin log error:2016-08-26 11:06:22,917 [http-bio-6080-exec-4] INFO org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:311) - Operation er

PROBLEM STATEMENT: We have a strange problem with ranger. When I do a "select * from <table>; "I can see in ranger, on hive audit, that my user (dnid) is getting logged correctly. But when I look at the same situation on hdfs audit it shows that another user did the request. This Is to me very strange, I´ve tried with different users and the same problem happens again. ERROR: ROOT CAUSE: This is known issue and a BUG -https://issues.apache.org/jira/browse/HIVE-13120http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/fixed_issues.html BUG-53108HIVE-13120 RESOLUTION: Changed the below property in hiveserver2 configs and restarted hiveserver2 after which the Ranger HDFS audit was showing user as hive in HDFS audit.From:"hive.server2.enable.doAs"=trueTO:"hive.server2.enable.doAs"=false

PROBLEM STATEMENT: There are frequent email alerts from HiveServer2 Metastore process which are misleading. When check the HiveServer2 Metastore process is up and running fine without issue. ERROR: Below is sample email alert - Ambari Alerts log output shows -/var/log/ambari-server/ambari-alerts.log ROOT CAUSE: The issue was with kerberos credentials for ambari-qa user. RESOLUTION:HiveServer2 Metastore alert is triggered from the node on which the service is installed. "ambari-qa" user is used by ambari to trigger the alert. If you are using kerberized cluster make sure you have "ambari-qa" ticket in place else false alert will be triggered.You can use below command to check -$su - ambari-qa $klist <--[make sure you have valid ticket]$hive --hiveconf hive.metastore.uris=thrift://<hive_mestastore_host>:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore

PROBLEM STATEMENT: I would like to change the default password for rangertagsync. According to the manual https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-installation/content/ch13s05s04.htmlI should just run the python updatetagadminpassword.py after changing the password in ranger. However I get error as shown below -ERROR: [root@test ranger-tagsync]# pwd/usr/hdp/current/ranger-tagsync[root@test ranger-tagsync]# python updatetagadminpassword.py 2016-11-18 09:30:08,311 [E] Required file not found: [/usr/hdp/2.5.0.0-1245/ranger-tagsync/conf/ranger-tagsync-site.xml][root@test ranger-tagsync]# Debugging Steps: From the above logs its clear that "ranger-tagsync-site.xml" is missing.But when check using "rpm -qa |grep tagsync" I see the rpm was installed [root@thakur2 ranger-tagsync]# rpm -qa |grep tagsyncranger_2_5_0_0_1245-tagsync-0.6.0.2.5.0.0-1245.el6.x86_64 2. Check the "rpm -ql" on package to make sure "ranger-tagsync-site.xml" comes from same package, but

PROBLEM STATEMENT:#hdfs dfsadmin -report shows "Blocks with corrupt replica : 20 "& #hdfs fsck / shows " Corrupt blocks : 0 "But.. "Corrupted block" shows as 20 from Ambari and Summary of HDFS services shows " Block 20 corrupt / 0 missing / ... "ERROR: Please find the screenshot below -Ambari Output:FSCK output:DFSADMIN output: ROOT CAUSE: Ambari shows its "Corrupted Block" but its "Corrupted Replica" RESOLUTION: This is a BUG -https://hortonworks.jira.com/browse/BUG-41958, which is fixed in Ambari version-2.2.2