Resources

PROBLEM STATEMENT: Ambari not displaying and service action buttons after login. It displays "Move Master Wizard In Progress". Not able to perform any operation from ambari and same is affecting for all users.ERROR: ROOT CAUSE: The issue seems that the user who logged inn to ambari UI[acting as admin], has tried to perform operation which was left in middle and just logged off from UI. The operation was cached in ambari UI and hence is reflected for other users in UI.RESOLUTION: Take ambari server backup and run below command from Ambari node - curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"<username>\",\"controllerName\":\"<controller_name>\"}"}'

PROBLEM STATEMENT: We have recently added 40 Datnodes added to cluster and they went down immediately after adding. Below was the exception found - ERROR:ROOT CAUSE:Customer added the 40nodes to the cluster running rack topology script as per link given - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/ch05.htmlThe topology script which was copied by customer from above link was missing "fi" on last line and hence was giving above exception. RESOLUTION: Populated corrected script rack-topology.sh on namenode and datanodes and tried starting the datanodes after which datanode service was able to startup.

PROBLEM STATEMENT: Customer deleted hdfs repository from ranger ui. Tried re-enabling plugin but the repository was getting created. Reinstalled ranger, but still no luck.There was alert saying - "Ranger Admin Password check"Text is: This alert is used to ensure that the Ranger Admin password in Ambari is correct. Response is: User:amb_ranger_admin credentials on Ambari UI are not in sync with Ranger Doing a test and changing ranger_admin_password to admin in Ambari. Configs->Advanced Restarting Ranger, not prompted but doing so anyway.Also changed local password on ranger admin host: [root@ip-172-53-51-18 admin]# passwd amb_ranger_admin Changing password for user amb_ranger_admin. New password: Retype new password: passwd: all authentication tokens updated successfully. Did restart of HDFS, but now HDFS was not coming up. ERROR: Ranger admin log error:2016-08-26 11:06:22,917 [http-bio-6080-exec-4] INFO org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:311) - Operation er

PROBLEM STATEMENT: We have a strange problem with ranger. When I do a "select * from <table>; "I can see in ranger, on hive audit, that my user (dnid) is getting logged correctly. But when I look at the same situation on hdfs audit it shows that another user did the request. This Is to me very strange, I´ve tried with different users and the same problem happens again. ERROR: ROOT CAUSE: This is known issue and a BUG -https://issues.apache.org/jira/browse/HIVE-13120http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/fixed_issues.html BUG-53108HIVE-13120 RESOLUTION: Changed the below property in hiveserver2 configs and restarted hiveserver2 after which the Ranger HDFS audit was showing user as hive in HDFS audit.From:"hive.server2.enable.doAs"=trueTO:"hive.server2.enable.doAs"=false

PROBLEM STATEMENT: There are frequent email alerts from HiveServer2 Metastore process which are misleading. When check the HiveServer2 Metastore process is up and running fine without issue. ERROR: Below is sample email alert - Ambari Alerts log output shows -/var/log/ambari-server/ambari-alerts.log ROOT CAUSE: The issue was with kerberos credentials for ambari-qa user. RESOLUTION:HiveServer2 Metastore alert is triggered from the node on which the service is installed. "ambari-qa" user is used by ambari to trigger the alert. If you are using kerberized cluster make sure you have "ambari-qa" ticket in place else false alert will be triggered.You can use below command to check -$su - ambari-qa $klist <--[make sure you have valid ticket]$hive --hiveconf hive.metastore.uris=thrift://<hive_mestastore_host>:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore

PROBLEM STATEMENT: I would like to change the default password for rangertagsync. According to the manual https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-installation/content/ch13s05s04.htmlI should just run the python updatetagadminpassword.py after changing the password in ranger. However I get error as shown below -ERROR: [root@test ranger-tagsync]# pwd/usr/hdp/current/ranger-tagsync[root@test ranger-tagsync]# python updatetagadminpassword.py 2016-11-18 09:30:08,311 [E] Required file not found: [/usr/hdp/2.5.0.0-1245/ranger-tagsync/conf/ranger-tagsync-site.xml][root@test ranger-tagsync]# Debugging Steps: From the above logs its clear that "ranger-tagsync-site.xml" is missing.But when check using "rpm -qa |grep tagsync" I see the rpm was installed [root@thakur2 ranger-tagsync]# rpm -qa |grep tagsyncranger_2_5_0_0_1245-tagsync-0.6.0.2.5.0.0-1245.el6.x86_64 2. Check the "rpm -ql" on package to make sure "ranger-tagsync-site.xml" comes from same package, but

PROBLEM STATEMENT:#hdfs dfsadmin -report shows "Blocks with corrupt replica : 20 "& #hdfs fsck / shows " Corrupt blocks : 0 "But.. "Corrupted block" shows as 20 from Ambari and Summary of HDFS services shows " Block 20 corrupt / 0 missing / ... "ERROR: Please find the screenshot below -Ambari Output:FSCK output:DFSADMIN output: ROOT CAUSE: Ambari shows its "Corrupted Block" but its "Corrupted Replica" RESOLUTION: This is a BUG -https://hortonworks.jira.com/browse/BUG-41958, which is fixed in Ambari version-2.2.2

PROBLEM STATEMENT:Running the python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py on the ambari metrics server I get every service running again, except the ambar-metrics-collector: The process is running, but there are two alerts left: Metrics Collector Process - Connection failed: [Errno 111] Connection refused to XXXXX:6188 Metrics Collector - HBase Master Process - Connection failed: [Errno 111] Connection refused to XXXXXX:61310 ERROR: 0x15741734f740003, negotiated timeout = 12000007:57:42,262 INFO [main] ZooKeeperRegistry:107 - ClusterId read in ZooKeeper is null07:57:42,341 WARN [main] HeapMemorySizeUtil:55 - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size07:58:13,170 INFO [main-SendThread(localhost:61181)] ClientCnxn:1142 - Unable to read additional data from server sessionid 0x15741734f740001, likely server has closed socket, closing socket connection and attempting reconnect07:58:13,170 INFO [main-

SmartSense 1.3 includes Activity Explorer, which hosts prebuilt notebooks that visualize cluster utilization data related to user, queue, job duration, and job resource consumption, including an HDFS Dashboard notebook. This dashboard helps operators better understand how HDFS is being used and which users and jobs are consuming the most resources within the file system.It's important to note that the source data for ACTIVITY.HDFS_USER_FILE_SUMMARY comes from fsimage, which does not contain file- and directory-level information. Many operators are also interested in more fine-grained analytics regarding cluster data use, which can drive decisions such as storage tiering using HDFS heterogeneous storage.Since these data are not available in fsimage, we will use the Ranger audit data for HDFS which the Ranger plugin writes during authorization events. The best practice is for the plugin to write these data to both Solr (for short-term use, driving performance in the UI) as well as HDFS f

Problem Statement: Created a user in Ranger. After sometime the user is not reflecting in ranger ui. But he user is reflecting in Ranger DB table x_user and in usersync logs we see the user is getting synchronized all the time.The user was LDAP user and there was no issue with other users.ERROR: Below is the issue snap -"testuser" is not displayed in Ranger UI but its reflected in Ranger DB as shown below- ROOT CAUSE: It seems the database for the particular user was corrupted.RESOLUTION: Inserted below value to the table "x_portal_user_role" after which issue was resolved. INSERT INTO x_portal_user_role VALUES(NULL,'2016-09-09 00:00:00','2016-09-09 00:00:00',1,1,(SELECT id FROM x_portal_user WHERE login_id='XXXX'),'ROLE_USER',1); ### NOTE: Replace XXXX with the login_id(username used for login into ranger portal) of the user ('XXXX') You can replace 'ROLE_USER' with 'ROLE_SYS_ADMIN' if you want it to be an admin

Problem Statement: Downgrading HDP which failed on restarting Namenode service and struck on below error -resource_management.core.exceptions.Fail: The NameNode None is not listed as Active or Standby, waiting...ERROR:===Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 420, in <module>NameNode().execute()File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in executemethod(env)File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restartself.start(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 101, in startupgrade_suspended=params.upgrade_suspended, env=env)File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunkreturn fn(*args, **kwargs)File "/var/lib/ambari-agent/cache/common-services/HD

ISSUE: 1. While performing HDP upgrade from 2.4.2 to 2.5.0 and which failed on last step Finalize upgrade saying few hosts are not able to upgrade to latest version.2. Tried to revert the upgrade and proceeded with Downgrade option.3. While Downgrade it prompt for Atlas and Kafka to be deleted from Cluster. 4. Deleted Atlas and Kafka from cluster.5. Proceeding further the stopping of service failed on to stop KAFKA6. The downgrade screen was paused and not it was showing in Ambari UI details tab that it "Failed to start KAFKA_BROKER" ROOT CAUSE: Looks like there are two tasks stuck in the PENDING state from the downgrade RESOLUTION: 1. You need to check at what step the upgrade is stuck using below command -

IntroductionThis article is a continuation of Geo-spatial Queries with Hive Using ESRI Geometry Libraries article published a few months ago.ObjectiveDemonstrate how to use Hive context and invoke built-in ESRI UDFs for Hive from Spark SQL.Pre-requisites HDP 2.4.2Steps documented on Geo-spatial Queries with Hive Using ESRI Geometry Libraries Steps1. Launch spark-shell with--jarsas its parameter:spark-shell --jars /home/spark/esri/esri-geometry-api.jar,/home/spark/esri/spatial-sdk-hive-1.1.1-SNAPSHOT.jarI placed the dependency jars to /home/spark/esri path, but you can store them in hdfs or local filesystem and grant proper privileges to your spark user.2. Instantiate sqlContext:val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);3. From spark-shell, define temporary functions:sqlContext.sql("""create temporary function st_point as 'com.esri.hadoop.hive.ST_Point'""");sqlContext.sql("""create temporary function st_x as 'com.esri.hadoop.hive.ST_X'""");4. From spark-shell, invok

IntroductionTrident processes a stream in batches of tuples for a defined topology. Currently, there is no windowing support of tuples in a trident stream. This section describes how windowing is supported in trident.Below types of windows can be defined for batches of tuples based on processing time or count. Tumbling windowSliding windowTumbling windowTuples are grouped in a single window based on processing time or count. Any tuple belongs to only one of the windows. Below is the new API on org.apache.storm.trident.Stream./**Returns a stream of tuples which are aggregated results of a tumbling window with every {@code windowCount} of tuples.*/public Stream tumblingWindow(intwindowCount, WindowsStoreFactory windowStoreFactory,Fields inputFields,Aggregator aggregator,Fields functionFields)/** * Returns a stream of tuples which are aggregated results of a window that tumbles at duration of {@code windowDuration} */public Stream tumblingWindow(BaseWindowedBolt.Duration windowDuration

ISSUE: Hive view is not working. ERROR: H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ROOT CAUSE: Issue was with mysql pool connection size limit exceeded.Check using -mysql> SHOW VARIABLES LIKE "max_connections";+-----------------+-------+| Variable_name | Value |+-----------------+-------+| max_connections | 100 |+-----------------+-------+1 row in set (0.00 sec) RESOLUTION: Modified mysql pool size limit from 100 to 500 and restarted mysql which resolved the issue.mysql> SET GLOBAL max_connections = 500;Query OK, 0 rows affected (0.00 sec)mysql> SHOW VARIABLES LIKE "max_connections";+-----------------+-------+| Variable_name | Value |+-----------------+-------+| max_connections | 500 |+-----------------+-------+1 row in set (0.00 sec)

SYMPTOM: Standby NN crashing due to edit log corruption and complaining that OP_CLOSE cannot be applied because the file is not under-construction ERROR:2016-09-30T06:23:25.126-0400 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation CloseOp [length=0, inodeId=0, path=/appdata/148973_perfengp/TARGET/092016/tempdb.TARGET.092016.hdfs, replication=3, mtime=1475223680193, atime=1472804384143, blockSize=134217728, blocks=[blk_1243879398_198862467], permissions=gsspe:148973_psdbpe:rwxrwxr-x, aclEntries=null, clientName=, clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE, txid=1585682886] java.io.IOException: File is not under construction: /appdata/148973_perfengp/TARGET/092016/tempdb.TARGET.092016.hdfs at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:436) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:230) at org.apache.hadoop.hdfs.serve

Check out the HDC on AWS technical preview documentation and learn how to use HDC to create clusters on AWS. The steps are pretty simple, even if you are new to AWS:Log in to your AWS account or create one.Create an AWS key pair.Launch a cloud controller instance. If you're just getting started with AWS, select the first deployment option.Log in to the cloud controller and create clusters.Use the link on the dashboard to log in to Ambari.This documentation is work in progress. We would love to hear your feedback and suggestions for improving it. Feel free to comment below.

Pre-RequisitesThis is a continuation of of "How to setup a multi user (Active Directory backed) zeppelin integrated with ldap and using Livy Rest server"You should have a running HDP cluster, Zeppelin with Livy, Ranger.Tested on HDP 2.5.I will use 3 user accounts for demoing security: hadoopadmin, sales1, hr1. You can substitute accordingly to your environment. These users are synced from an LDAP/AD in my use caseSpark and HDFS securityWe will Configure Ranger policies to: Protect /sales HDFS dir - so only sales group has access to itProtect sales hive table - so only sales group has access to itCreate /sales dir in HDFS as hadoopadminCreate /sales dir in HDFS as hadoopadminsudo -u hadoopadmin hadoop fs -mkdir /salessudo -u hadoopadmin hadoop fs -chown sales:hadoop /salesNow login as sales1 and attempt to access it before adding any Ranger HDFS policysudo su - sales1hdfs dfs -ls /salesIt fails with authorization error:Permission denied: user=sales1, access=READ_EXECUTE, inode="/sales":

Hadoop Installation on RHEL 7.2 TipsCustom Install Tips: Read this article and have it available.Make sure you know your Hadoop Mount PointsFollow the networking Best Practices Make sure you follow the Hortonworks HDP minimum requirements.Make sure you have root access and make sure you have yum, rpm, scp, curl, wget, unzip, tar, yum-utils, createrepo, reposync installed, working and in your path. Make sure networking, iptables, proxies and firewalls are all open enough for you to access the Hortonworks repos and the bandwidth is decent.Download the OpenJDK 1.8 64-bit update 51 or higherMake sure you setup passwordless SSH to all machines, including the current machine you are starting from which is the Ambari Server. You may need to SSH to yourself. Make sure you have 20G+ of /var space, 20G+ /usr space and plenty of /tmp space. Many things will go here: /usr/hdpOne time I needed to manually install MySQL via sudo yum -y install mysqlAny space where you will install your dat

ISSUE: While performing unkerberizing cluster all services were down and nothing was coming up. Also the unkeberized cluster step failed. The start of services was failed.Tried to manually start Namenodes which came up but the status was not displayed correctly in Ambari UI. The journal node were not able to start and was failing with error as shown below.ERROR: Screenshot is attached below Journal node error:ROOT CAUSE: There were multiple issue as below -1. From the JN error it says "missing spnego keytab". From the error It seems the kerberos was not properly disabled on cluster.2. As checked in hdfs-site.xml the property "hadoop.http.authentication.type" was set to kerberos.3. Oozie was not able to detect active namenode, since the property "hadoop.http.authentication.simple.anonymous.allowed" was set to false.RESOLUTION:1. Setting hadoop.http.authentication.type to simple in hdfs-site.xml, HDFS was able to restart2. Setting the property hadoop.http.authentication.simple.anonymous.

updateddeleteuser.zipISSUE: Ranger ldap integration was working fine. Customer delete user from ranger UI and was facing issue while re-importing user in Ranger. ROOT CAUSE: Customer removed the users from ranger UI and expected that the user should be automatically imported from ranger usersync processBelow are sample screenshot -User named 'testuser' is deleted from Ranger UI.But below you can see the user is still available in database. RESOLUTION: There are multiple tables which has the entry for the user. You need to run delete script to delete the user entries from database and re-start ranger usersync process to re-import the user.Please find attach delete script- Syntax to run the script - $ deleteUser.sh -f input.txt -u ranger_user -p password -db ranger [-r <replaceUser>]

ISSUE: After enabling Ambari SSL Hive views stopped working.ERROR: 08 Nov 2016 11:32:23,330 WARN [qtp-ambari-client-263] nio:720 - javax.net.ssl.SSLException: Received fatal alert: certificate_unknown08 Nov 2016 11:32:23,331 ERROR [qtp-ambari-client-256] ServiceFormattedException:100 - org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambariorg.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambariat org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:176)at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:142)at org.apache.ambari.view.utils.ambari.AmbariApi.getHostsWithComponent(AmbariApi.java:99)at org.apache.ambari.view.hive.client.ConnectionFactory.getHiveHost(ConnectionFactory.java:79)at org.apache.ambari.view.hive.client.ConnectionFactory.create(ConnectionFactory.java:68)at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLoc

SYMPTOM:During HDP upgrade from 2.3 to 2.5 YARN check is failing due to NoSuchMethodError org.apache.hadoop.yarn.api.records.Resource.getMemorySize()JERROR: Below was the error in application logs -16/11/14 10:30:12 FATAL distributedshell.ApplicationMaster: Error running ApplicationMasterjava.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.records.Resource.getMemorySize()Jat org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:585)at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:298)ROOT CAUSE: There was issue with classpath where the nodemanager on which the job was running was pointing to older version[ie. 2.3] classpath.RESOLUTION: There are two solutions as below -1. Skip this step in Ambari upgrade UI and proceed. Ambari will take care of setting up the classpath.2. Modify the classpath manually and confirm the set classpath using "hadoop classpath" command and re-run the service c

PostgreSQL extension PG-Strom, allows users to customize the data scan and run queries faster. CPU-intensive work load is identified and transferred to the GPU to take advantage of the powerful GPU parallel execution ability to complete the data task. The combination of few number of core processors, RAM bandwidth, and the GPU has a unique advantage. GPUs typically have hundreds of processor cores and RAM bandwidths that are several times larger than CPUs. They can handle large numbers of computations in parallel, so their operations are very efficient.PG-Storm based on two basic ideas: On-the-fly native GPU code generation.Asynchronous pipeline execution mode.Below figure shows how query is submitted to execution engine and during query optimization phase, PG-Storm detects whether a given query is fully or partially executable on the GPU, and then determines whether the query can be transferred. If the query can be transferred, PG-Storm creates the source code for the GPU native binar

Link to series of videos and blog posts that cover: Introduction to HDF HDF 2.0 + Ambari - How ToDemos of creating dataflows with NiFi, Kafka and StormIn-depth explanation of new features of HDF 2.0 - new UI, multi-tenancy, zero master clustering, Apache MiNiFi77 questions and answers about HDF - examples: how to connect NiFi with enterprise schedulers like ESP or Zeke, how per processor authorization can be utilized, "scheduling" of flowsLink to all here: http://hortonworks.com/blog/harness-power-data-motion/