Community Articles

PROBLEM STATEMENT:Running the python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py on the ambari metrics server I get every service running again, except the ambar-metrics-collector: The process is running, but there are two alerts left: Metrics Collector Process - Connection failed: [Errno 111] Connection refused to XXXXX:6188 Metrics Collector - HBase Master Process - Connection failed: [Errno 111] Connection refused to XXXXXX:61310 ERROR: 0x15741734f740003, negotiated timeout = 12000007:57:42,262 INFO [main] ZooKeeperRegistry:107 - ClusterId read in ZooKeeper is null07:57:42,341 WARN [main] HeapMemorySizeUtil:55 - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size07:58:13,170 INFO [main-SendThread(localhost:61181)] ClientCnxn:1142 - Unable to read additional data from server sessionid 0x15741734f740001, likely server has closed socket, closing socket connection and attempting reconnect07:58:13,170 INFO [main-

SmartSense 1.3 includes Activity Explorer, which hosts prebuilt notebooks that visualize cluster utilization data related to user, queue, job duration, and job resource consumption, including an HDFS Dashboard notebook. This dashboard helps operators better understand how HDFS is being used and which users and jobs are consuming the most resources within the file system.It's important to note that the source data for ACTIVITY.HDFS_USER_FILE_SUMMARY comes from fsimage, which does not contain file- and directory-level information. Many operators are also interested in more fine-grained analytics regarding cluster data use, which can drive decisions such as storage tiering using HDFS heterogeneous storage.Since these data are not available in fsimage, we will use the Ranger audit data for HDFS which the Ranger plugin writes during authorization events. The best practice is for the plugin to write these data to both Solr (for short-term use, driving performance in the UI) as well as HDFS f

Problem Statement: Created a user in Ranger. After sometime the user is not reflecting in ranger ui. But he user is reflecting in Ranger DB table x_user and in usersync logs we see the user is getting synchronized all the time.The user was LDAP user and there was no issue with other users.ERROR: Below is the issue snap -"testuser" is not displayed in Ranger UI but its reflected in Ranger DB as shown below- ROOT CAUSE: It seems the database for the particular user was corrupted.RESOLUTION: Inserted below value to the table "x_portal_user_role" after which issue was resolved. INSERT INTO x_portal_user_role VALUES(NULL,'2016-09-09 00:00:00','2016-09-09 00:00:00',1,1,(SELECT id FROM x_portal_user WHERE login_id='XXXX'),'ROLE_USER',1); ### NOTE: Replace XXXX with the login_id(username used for login into ranger portal) of the user ('XXXX') You can replace 'ROLE_USER' with 'ROLE_SYS_ADMIN' if you want it to be an admin

Problem Statement: Downgrading HDP which failed on restarting Namenode service and struck on below error -resource_management.core.exceptions.Fail: The NameNode None is not listed as Active or Standby, waiting...ERROR:===Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 420, in <module>NameNode().execute()File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in executemethod(env)File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restartself.start(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 101, in startupgrade_suspended=params.upgrade_suspended, env=env)File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunkreturn fn(*args, **kwargs)File "/var/lib/ambari-agent/cache/common-services/HD

ISSUE: 1. While performing HDP upgrade from 2.4.2 to 2.5.0 and which failed on last step Finalize upgrade saying few hosts are not able to upgrade to latest version.2. Tried to revert the upgrade and proceeded with Downgrade option.3. While Downgrade it prompt for Atlas and Kafka to be deleted from Cluster. 4. Deleted Atlas and Kafka from cluster.5. Proceeding further the stopping of service failed on to stop KAFKA6. The downgrade screen was paused and not it was showing in Ambari UI details tab that it "Failed to start KAFKA_BROKER" ROOT CAUSE: Looks like there are two tasks stuck in the PENDING state from the downgrade RESOLUTION: 1. You need to check at what step the upgrade is stuck using below command -

IntroductionThis article is a continuation of Geo-spatial Queries with Hive Using ESRI Geometry Libraries article published a few months ago.ObjectiveDemonstrate how to use Hive context and invoke built-in ESRI UDFs for Hive from Spark SQL.Pre-requisites HDP 2.4.2Steps documented on Geo-spatial Queries with Hive Using ESRI Geometry Libraries Steps1. Launch spark-shell with--jarsas its parameter:spark-shell --jars /home/spark/esri/esri-geometry-api.jar,/home/spark/esri/spatial-sdk-hive-1.1.1-SNAPSHOT.jarI placed the dependency jars to /home/spark/esri path, but you can store them in hdfs or local filesystem and grant proper privileges to your spark user.2. Instantiate sqlContext:val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);3. From spark-shell, define temporary functions:sqlContext.sql("""create temporary function st_point as 'com.esri.hadoop.hive.ST_Point'""");sqlContext.sql("""create temporary function st_x as 'com.esri.hadoop.hive.ST_X'""");4. From spark-shell, invok

IntroductionTrident processes a stream in batches of tuples for a defined topology. Currently, there is no windowing support of tuples in a trident stream. This section describes how windowing is supported in trident.Below types of windows can be defined for batches of tuples based on processing time or count. Tumbling windowSliding windowTumbling windowTuples are grouped in a single window based on processing time or count. Any tuple belongs to only one of the windows. Below is the new API on org.apache.storm.trident.Stream./**Returns a stream of tuples which are aggregated results of a tumbling window with every {@code windowCount} of tuples.*/public Stream tumblingWindow(intwindowCount, WindowsStoreFactory windowStoreFactory,Fields inputFields,Aggregator aggregator,Fields functionFields)/** * Returns a stream of tuples which are aggregated results of a window that tumbles at duration of {@code windowDuration} */public Stream tumblingWindow(BaseWindowedBolt.Duration windowDuration

ISSUE: Hive view is not working. ERROR: H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ROOT CAUSE: Issue was with mysql pool connection size limit exceeded.Check using -mysql> SHOW VARIABLES LIKE "max_connections";+-----------------+-------+| Variable_name | Value |+-----------------+-------+| max_connections | 100 |+-----------------+-------+1 row in set (0.00 sec) RESOLUTION: Modified mysql pool size limit from 100 to 500 and restarted mysql which resolved the issue.mysql> SET GLOBAL max_connections = 500;Query OK, 0 rows affected (0.00 sec)mysql> SHOW VARIABLES LIKE "max_connections";+-----------------+-------+| Variable_name | Value |+-----------------+-------+| max_connections | 500 |+-----------------+-------+1 row in set (0.00 sec)

SYMPTOM: Standby NN crashing due to edit log corruption and complaining that OP_CLOSE cannot be applied because the file is not under-construction ERROR:2016-09-30T06:23:25.126-0400 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation CloseOp [length=0, inodeId=0, path=/appdata/148973_perfengp/TARGET/092016/tempdb.TARGET.092016.hdfs, replication=3, mtime=1475223680193, atime=1472804384143, blockSize=134217728, blocks=[blk_1243879398_198862467], permissions=gsspe:148973_psdbpe:rwxrwxr-x, aclEntries=null, clientName=, clientMachine=, overwrite=false, storagePolicyId=0, opCode=OP_CLOSE, txid=1585682886] java.io.IOException: File is not under construction: /appdata/148973_perfengp/TARGET/092016/tempdb.TARGET.092016.hdfs at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:436) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:230) at org.apache.hadoop.hdfs.serve

Check out the HDC on AWS technical preview documentation and learn how to use HDC to create clusters on AWS. The steps are pretty simple, even if you are new to AWS:Log in to your AWS account or create one.Create an AWS key pair.Launch a cloud controller instance. If you're just getting started with AWS, select the first deployment option.Log in to the cloud controller and create clusters.Use the link on the dashboard to log in to Ambari.This documentation is work in progress. We would love to hear your feedback and suggestions for improving it. Feel free to comment below.

Pre-RequisitesThis is a continuation of of "How to setup a multi user (Active Directory backed) zeppelin integrated with ldap and using Livy Rest server"You should have a running HDP cluster, Zeppelin with Livy, Ranger.Tested on HDP 2.5.I will use 3 user accounts for demoing security: hadoopadmin, sales1, hr1. You can substitute accordingly to your environment. These users are synced from an LDAP/AD in my use caseSpark and HDFS securityWe will Configure Ranger policies to: Protect /sales HDFS dir - so only sales group has access to itProtect sales hive table - so only sales group has access to itCreate /sales dir in HDFS as hadoopadminCreate /sales dir in HDFS as hadoopadminsudo -u hadoopadmin hadoop fs -mkdir /salessudo -u hadoopadmin hadoop fs -chown sales:hadoop /salesNow login as sales1 and attempt to access it before adding any Ranger HDFS policysudo su - sales1hdfs dfs -ls /salesIt fails with authorization error:Permission denied: user=sales1, access=READ_EXECUTE, inode="/sales":

Hadoop Installation on RHEL 7.2 TipsCustom Install Tips: Read this article and have it available.Make sure you know your Hadoop Mount PointsFollow the networking Best Practices Make sure you follow the Hortonworks HDP minimum requirements.Make sure you have root access and make sure you have yum, rpm, scp, curl, wget, unzip, tar, yum-utils, createrepo, reposync installed, working and in your path. Make sure networking, iptables, proxies and firewalls are all open enough for you to access the Hortonworks repos and the bandwidth is decent.Download the OpenJDK 1.8 64-bit update 51 or higherMake sure you setup passwordless SSH to all machines, including the current machine you are starting from which is the Ambari Server. You may need to SSH to yourself. Make sure you have 20G+ of /var space, 20G+ /usr space and plenty of /tmp space. Many things will go here: /usr/hdpOne time I needed to manually install MySQL via sudo yum -y install mysqlAny space where you will install your dat

ISSUE: While performing unkerberizing cluster all services were down and nothing was coming up. Also the unkeberized cluster step failed. The start of services was failed.Tried to manually start Namenodes which came up but the status was not displayed correctly in Ambari UI. The journal node were not able to start and was failing with error as shown below.ERROR: Screenshot is attached below Journal node error:ROOT CAUSE: There were multiple issue as below -1. From the JN error it says "missing spnego keytab". From the error It seems the kerberos was not properly disabled on cluster.2. As checked in hdfs-site.xml the property "hadoop.http.authentication.type" was set to kerberos.3. Oozie was not able to detect active namenode, since the property "hadoop.http.authentication.simple.anonymous.allowed" was set to false.RESOLUTION:1. Setting hadoop.http.authentication.type to simple in hdfs-site.xml, HDFS was able to restart2. Setting the property hadoop.http.authentication.simple.anonymous.

updateddeleteuser.zipISSUE: Ranger ldap integration was working fine. Customer delete user from ranger UI and was facing issue while re-importing user in Ranger. ROOT CAUSE: Customer removed the users from ranger UI and expected that the user should be automatically imported from ranger usersync processBelow are sample screenshot -User named 'testuser' is deleted from Ranger UI.But below you can see the user is still available in database. RESOLUTION: There are multiple tables which has the entry for the user. You need to run delete script to delete the user entries from database and re-start ranger usersync process to re-import the user.Please find attach delete script- Syntax to run the script - $ deleteUser.sh -f input.txt -u ranger_user -p password -db ranger [-r <replaceUser>]

ISSUE: After enabling Ambari SSL Hive views stopped working.ERROR: 08 Nov 2016 11:32:23,330 WARN [qtp-ambari-client-263] nio:720 - javax.net.ssl.SSLException: Received fatal alert: certificate_unknown08 Nov 2016 11:32:23,331 ERROR [qtp-ambari-client-256] ServiceFormattedException:100 - org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambariorg.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambariat org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:176)at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:142)at org.apache.ambari.view.utils.ambari.AmbariApi.getHostsWithComponent(AmbariApi.java:99)at org.apache.ambari.view.hive.client.ConnectionFactory.getHiveHost(ConnectionFactory.java:79)at org.apache.ambari.view.hive.client.ConnectionFactory.create(ConnectionFactory.java:68)at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLoc

SYMPTOM:During HDP upgrade from 2.3 to 2.5 YARN check is failing due to NoSuchMethodError org.apache.hadoop.yarn.api.records.Resource.getMemorySize()JERROR: Below was the error in application logs -16/11/14 10:30:12 FATAL distributedshell.ApplicationMaster: Error running ApplicationMasterjava.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.records.Resource.getMemorySize()Jat org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:585)at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:298)ROOT CAUSE: There was issue with classpath where the nodemanager on which the job was running was pointing to older version[ie. 2.3] classpath.RESOLUTION: There are two solutions as below -1. Skip this step in Ambari upgrade UI and proceed. Ambari will take care of setting up the classpath.2. Modify the classpath manually and confirm the set classpath using "hadoop classpath" command and re-run the service c

PostgreSQL extension PG-Strom, allows users to customize the data scan and run queries faster. CPU-intensive work load is identified and transferred to the GPU to take advantage of the powerful GPU parallel execution ability to complete the data task. The combination of few number of core processors, RAM bandwidth, and the GPU has a unique advantage. GPUs typically have hundreds of processor cores and RAM bandwidths that are several times larger than CPUs. They can handle large numbers of computations in parallel, so their operations are very efficient.PG-Storm based on two basic ideas: On-the-fly native GPU code generation.Asynchronous pipeline execution mode.Below figure shows how query is submitted to execution engine and during query optimization phase, PG-Storm detects whether a given query is fully or partially executable on the GPU, and then determines whether the query can be transferred. If the query can be transferred, PG-Storm creates the source code for the GPU native binar

Link to series of videos and blog posts that cover: Introduction to HDF HDF 2.0 + Ambari - How ToDemos of creating dataflows with NiFi, Kafka and StormIn-depth explanation of new features of HDF 2.0 - new UI, multi-tenancy, zero master clustering, Apache MiNiFi77 questions and answers about HDF - examples: how to connect NiFi with enterprise schedulers like ESP or Zeke, how per processor authorization can be utilized, "scheduling" of flowsLink to all here: http://hortonworks.com/blog/harness-power-data-motion/

Here are the Requirements:Total Data Size - Uncompressed: 13.5TB; Compressed: 2 TBLarge Virtual Fact Table, View containing a Union All of 3 Large Tables, 11 Billion Records in Total SizeAnother view taking the large virtual fact table, with consecutive Left Out Joins on 8 Dimension Tables, so that no matter what 11 Billion records is always the result.There is timestamp data that you can use to filter rows by.Suppose you were given the following. How would you begin configuring Hortonworks for Hive? Would you focus on storage? How can we configure for compute?Lets assume:Platform: AWSData Node Instance: r3_4xlargeCores: 16RAM: 122 GBEBS Storage: 2 x 1TB DisksSo where do we begin?First Some Quick Calculations:Memory per Core: 122GB/16 = 7.625; Approximate 8 GB per CPU CoreThis means our largest Container Size PER Node per core is 8 GB However we should not reserve all 16 Cores to Hadoop. Some Cores are need for OS and other processes.Let's Assume 14 Cores is reserved for YARN.Memory

It's easier than I would have thought to add images to your SQL results tables in Apache Zeppelin.It's pretty simple to do this in HDP 2.5's version of Apache Zeppelin. You use the %html tag to output HTML instead of text.Use Case: Displaying Image with TensorFlow Inception Image Recognition Results in Same ListExample SQL:SELECT user_name, handle, concat('%html <img width=50 height=60 src="', media_url, '">') as media,substring(inception,0,150) as inception, msg, sentiment, stanfordsentiment, location, timeFROM twitterorc where inception not like '%Not found%' and inception is not null and trim(inception)!= ''

Using Ignition version 7.8.4 (b2016082217) | 64-bitCopy dependencies needed to make a connection to the Apache Phoneix Server werecopied into Ignition\libEdit Igntion\data\ignition.conf to register the dependacy with Ignition's boot strap processRestart Nifi and then begin the process of configuring JDBC drivers by selecting the drivers link in the Databases section of Ignition management consoleSelect create new JDBC Driver on the Drivers Home PageComplete and submit the driver properties page using the following as a guideNow select connections from the data bases menuand create new database connection from the connections home pageComplete and submit the connection properties dialog After successfully submitting the configuration the connection should appear on the connections home pageIf there are any issues with the driver or connection they can be found in the log of the system consoleWith the connection configured we can now added data to the HMI application from queries of the

ErrorRanger Tagsync shows lots of KafkaException in log file, which causes disk space alert in Ambari. Also it used out all of the client port./var/log/ranger/tagsync/tagsync.log10 Nov 2016 11:46:43 ERROR TagSynchronizer [main] - 262 tag-source:atlas initialization failed with javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, b ut the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user kafka.common.KafkaException: fetching topic metadata for topics [Set(ATLAS_ENTITIES)] from broker [ArrayBuffer(BrokerEndPoint(1001,host.domain,6667))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

Ambari 2.2.2.18 is released and available for immediate use. The team fixed 60 issues, many of which customers have experienced. No features are included, so an upgrade should be seamless for those using Ambari 2.2.2.0 or are considering upgrading to Ambari 2.2.2.xRefer to the Product Documentation and Release Notes for further information about this release.This product rolls up all issues in private releases less than 18.

ObjectiveIf you are using Docker on CentOS 7, you may have run into issues importing the Docker version of the Hortonworks Sandbox for HDP 2.5. The default configuration of Docker on CentOS 7 limits your Docker virtual machine image to 10GB in size. The Docker HDP sandbox is 13GB in size. This will cause the loading process of the sandbox to fail.This tutorial will guide you through the process of installing Docker on CentOS 7 using Vagrant and modifying the configuration of Docker to move the location of and increase the size of Docker virtual machine image.While we are using Vagrant + Virtualbox, this process should work for any install of CentOS (Amazon, etc) with small changes because VirtualBox and related plugins are not needed.PrerequisitesYou should already have downloaded the Docker HDP 2.5 Sandbox. Read more here: Docker HDP SandboxYou should already have installed VirtualBox 5.1.x. Read more here: VirtualBoxYou should already have installed Vagrant 1.8.6. Read more here: Vag

Introduction Apache Phoenix is a relational database "layer" built on top of Apache HBase, a NoSQL key-value data store. Until recently, the only provided entry-point for users to Phoenix was by their JDBC driver. This implicitly limited client applications to JVM-based languages. The Phoenix QueryServer was built using a sub-project of the Apache Calcite project named Avatica. Avatica is a JDBC-over-HTTP library. It includes both an HTTP server and a JDBC driver. The HTTP server has its own wire API which can be leveraged by clients in any language communicate with the backend JDBC server.Deployment One of the goals of the Phoenix QueryServer was to follow typical deployment strategies of HTTP servers. The primary adopted goal is that the QueryServer aims to be stateless. While this is not entirely the case as QueryServer does maintain internal state, the server and Java client are written in such a way that any missing state in the PQS instance can be automatically rec