Resources

Does HBase replication guarantee consistency or snapshot-in-time?HBase replication works by propagating the WAL from the source cluster to the destination and is replayed asynchronously. Thus, it is eventually consistent.For snapshot functionality, consider taking a Snapshot on the source cluster and exporting that to the destination using the ExportSnapshot tool.Is there an option of enabling replication at the GLOBAL level rather than individually at the table level?HBase replication can be enabled at the table level or column family level onlyDoes enabling replication cause the table to be disabled?Check that the hbase.online.schema.update.enable in hbase-site.xmlis true. If so, then the table will not be disabled when turning on table replication.The default value is true so if the property is not set in hbase-site.xml, it is 'true'. Does HBase replication solve the problem of replicating a cluster which already has data to an empty cluster?Replication alone does not solve replica

Symptom:After adding additional configurations to Solr, the service fails with both 'no servers hosting shard' and 'OutOfMemoryError' errors. A restart will resolve the issue for some time and it will eventually hit the error again.Root Cause:In some versions, the default setting for max user processes for the infra-solr user is 1024, which may be set too low for certain use cases. Resolution:Increase the max user process user limit. This can be done by creating the file /etc/security/limits.d/infra-solr.conf on the solr node and setting the nproc property. For example if the new value chosen is 6000, the contents would be: infra-solr - nproc 6000

Symptom:The oldWALs directory stores WALs that are no longer needed for recovery. We have come across situations where the directory continually increases in size and the older WALs are not getting cleaned up.Root Cause:This can get filled up due to a bug where hbase.backup.enable is true by default in HDP-2.5.3Resolution:- Add the property to HBase Custom hbase-site:hbase.backup.enable=false- Restart HBase

ROOT CAUSE:kafka.metrics.reporters in Advanced Kafka-broker was pointing to Ganglia metrics reporter. kafka.metrics.reporters=kafka.ganglia.KafkaGangliaMetricsReporterRESOLUTION: Kafka.metrics.reporters is pointing to GangliaMetrics. It should be pointing to Ambari metrics like this: kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter You can modify the property via Ambari -> Kafka -> configs -> Advanced kafka-broker-> kafka.metrics.reporters and modify the property. Please save the changes and restart required services

SYMPTOM: Storm nimbus fails to come up after Ambari was upgraded to 2.4 version while on HDP 2.42016-11-22 13:29:43.066 [timer] b.s.d.nimbus [ERROR] Error when processing eventjava.lang.NullPointerException at clojure.lang.Numbers.ops(Numbers.java:961) ~[clojure-1.6.0.jar:?] at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.6.0.jar:?] at backtype.storm.util$partition_fixed.invoke(util.clj:892) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?] at clojure.core$partial$fn__4228.doInvoke(core.clj:2468) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.invoke(RestFn.java:408)

We can configure multiple listeners by giving comma-separated list of URIs that Kafka will listen on. Please follow the steps below to implement this:1. Add the listeners as comma separated value in Ambari ->kafka->configs->listeners, for example:listeners=PLAINTEXT://myhost:6667, PLAINTEXTSASL://myhost:66682. Add ACL for 'Anonymous' user, because In PLAINTEXT connections user's identity is set to Anonymous. For example :$ bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=ambari-server:2181 --add --allow-principal User:Anonymous --producer --topic topic-oct3. Run the producer with security protocol set to PLAINTEXT to listen to PLAINTEXT and set it to PLAINTEXTSASL to listen to other listener, something like this:$ bin/kafka-console-producer.sh --broker-list ambari-server.support.com:6667 --topic topic-oct --security-protocol PLAINTEXT$ bin/kafka-console-producer.sh --broker-list ambari-server.support.com:6668 --t

PROBLEM: Enable GC logging for Zookeeper SOLUTION: When Using Ambari web UI: 1. Click on the Zookeeper Service 2. Click on Configs tab 3. Navigate to 'Advanced Zookeeper-env'4. Locate the setting 'zookeeper-env template' 5. Append the following to 'export SERVER_JVMFLAGS=-Xmx1024m' : -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$ZOO_LOG_DIR/zookeeper.gc.`date +'%Y%m%d%H%M'`" To be precise it should look like :export SERVER_JVMFLAGS="-Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$ZOOKEEPER_LOG_DIR/zookeeper.gc.`date +'%Y%m%d%H%M'`" 6. Save the changes and restart the Zookeeper service when prompted When the cluster is managed outside Ambari: 1. On application cluster node open zookeeper-env.sh 2. Append the above mentioned parameter to SERVER_JVMFLAGS values. You can find the zookeeper-env.sh at /etc/zookeeper/conf/

SYMPTOMS: When using the SelectHiveQL processor in Nifi to run hive queries, it does not use the specific yarn queue ('nifi') if it is configured as in the following HiveConnectionPool settings:jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;tez.queue.name=nifiFollowing is an example of queue setting in yarn capacity scheduler:yarn.scheduler.capacity.root.queues=default,nifiyarn.scheduler.capacity.root.default.user-limit-factor=1yarn.scheduler.capacity.root.default.state=RUNNINGyarn.scheduler.capacity.root.default.maximum-capacity=40yarn.scheduler.capacity.root.default.capacity=40yarn.scheduler.capacity.root.default.acl_submit_applications=yarnyarn.scheduler.capacity.root.capacity=100yarn.scheduler.capacity.root.acl_administer_queue=yarnyarn.scheduler.capacity.root.accessible-node-labels=*yarn.scheduler.capacity.node-locality-delay=40yarn.scheduler.capacity.maximum-applications=10000yar

From HDP-2.5 onwards Ranger installation with Ambari 2.4.0, supports uploading of solrconfig to zookeeper during start of Ranger Admin component. However once default solrconfig.xml is uploaded the modified solrconfig.xml is not uploaded again for the next Ranger Admin start.To upload modified solrconfig.xml, below manual steps needs to be followed.1) Edit solrconfig.xml: Ranger solrconfig.xml is located at /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml2) Upload modified solrconfig.xml under znode used by Ranger Audit: Znode can be found in property ranger.audit.solr.zookeepers as {zookeeper_host}:2181/znode under Ranger Audit Tab.NOTE: If Ranger is using Ambari Infra solr cloud then default znode is /infra-solr otherwise if it is using externally managed solr cloud znode is /ranger_audits Use below command to upload, need to also export JAVA_HOME=/usr/jdk64/jdk1.8.0_77/usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string {zookee

PROBLEM : Cloudbreak failed to start. SYMPTOM : We get the below error "bind already in use"ERROR :ERROR: for consul Cannot start container 8a0b07e6b35e7350656b54314f7023107f417c0e0031db28e91f6bfb7a245d6f: failed to create endpoint cbreak_consul_1 on network bridge: Error starting userland proxy: listen udp X.X.X.X: bind: address already in use Traceback (most recent call last): File "<string>", line 3, in <module> File "compose/cli/main.py", line 63, in main AttributeError: 'ProjectError' object has no attribute 'msg' docker-compose returned -1 ROOT CAUSE : Cloudbreak may have stale process and because there are some container still running from the previous run. RESOLUTION : Please run the following : cbd kill # check if all containers have stopped cbd start This should fix the issue.

screen-shot-2017-03-09-at-44439-pm.pngscreen-shot-2017-03-09-at-44200-pm.pngPlease use the REST-plugin tool to perform the deletion. Please find the attached screenshot on how to use it, and the instruction on how to use encoded password (string to the word Basic) REST-plugin : https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo/related?hl=en-USYou can encode your password using this tool : http://base64-encoding.online-domain-tools.com/ Type in the format username:password to get the encoded password.

Why-Hive-rolling-restart-doesn-t-change-the-configurations-all-the-time?This completely depends on what configuration are you trying to change.For example : We changed the value of hive.exec.compress.output to true and restarted 1 hiveserver2 and when we checked beeline set hive.exec.compress.output the value got changed. 2. Then we changed the log4j property hive.root.logger value to Debug and then just restarted hiveserver2 but the change didn't take effect until we restart Hive metastore on the host 3. Then we changed hadoop.security.credential.provider.path this value but this didn't take effect until we refresh the client configs first and then restart metastore and Hiveserver2. (restart order is important)

STEPS :Take a backup of your Ambari database before executing the below command as you are issuing a delete command.curl --insecure -u admin:<password> -H 'X-Requested-By: ambari' -X DELETE http://<ambari-server-fqdn>:8080/api/v1/groups/<group name you wanted to delete>

An OutputCommitter that commits files specified in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}. in mapred-site.xml The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1 The file output committer has three phases 1.Commit task Recover task Commit Job If you choose version value to be Version 1 : 1. Commit task will rename the directory from $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/_temporary/$appAttemptID/$taskID/ , to put it in a simple way it just reduces one subdirectory. 2. RecoverTask will also do a rename of $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/_temporary/($appAttemptID + 1)/$taskID/ 3. Commit Job will merge every task output in $joboutput/_temporary/$appAttemptID/$taskID/ to the path which is specified in the mapreduce.output.fileoutputformat.outputdir $joboutput/, then it will delete $joboutput/_temporary/ and write $joboutput/_SUCCESS It has a perfor

HDF Version: 2.1.2Test Environment: Single AWS VPC with multiple subnets, EC2 Security Groups with port rules controlling access between subnets, AWS LogFlow to track accepted/rejected traffic, 1 EC2 instance for Management, 3 EC2 instances for worker nodes, Laptop on public IP for remote access.Deployment / Testing Method: Installed baseline of Ambari, Ambari Infra, Ambari Metrics, NiFi Certificate Authority, Zookeeper Cluster, NiFi Cluster, ClientsAdded RangerEnabled SSL across all servicesTested all commonly used interfaces, checked for rejected trafficNot Tested: Using External LDAP/AD servicesUsing external Certificate AuthorityConnecting to Auxiliary servicesDeployment TopologyZoneM(anagement)D(ata)R(emote)P(ublic)MembersManagement NodeWorker Nodes 1,2,3Other NiFi ClusterUsersServicesAmbari Infra (Infra Solr), Ranger, Metrics Collector, Grafana, Metrics Monitor, Nifi Certificate Authority, Clients (Infra Solr, ZooKeeper)NiFi, Zookeeper, Metrics Monitor, Clients (Zookeper)NiFiBro

STATEMENT: Ambari audit log file gets generated from Ambari ver. 2.4 - https://issues.apache.org/jira/browse/AMBARI-15241. PROBLEM: How can I enable ambari audit in Ambari ver. 2.2.2? SOLUTION: To get ambari 2.2.2 having the audit, follow up the below instruction. This would be to change log level to DEBUG for this PersistKeyValueService class. - log into ambari node- edit /etc/ambari-server/conf/log4j.properties- set (or add) the following property:log4j.logger.org.apache.ambari.server.api.services.PersistKeyValueService=DEBUG- save the changes- run: $ ambari-server restart

It is known that in a secure Kafka environment, we need to explicitly authorize user/principal to either read/write to a Kafka topic as shown belowGrant Read/Write Access to a TopicTo add the following ACL:"Principals user:bob and user:alice are allowed to perform Operation Read and Write on Topic Test-Topic from Host1 and Host2"run the CLI with the following options:bin/kafka-acls.sh --add --allow-principal <strong><em>User</em></strong>:bob --allow-principal User:alice --allow-host host1 --allow-host host2 --operation Read --operation Write --topic test-topicGrant Full Access to Topic, Cluster, and Consumer GroupTo add ACLs to a topic, specify --topic <topic-name> as the resource option. Similarly, to add ACLs to cluster, specify --cluster; to add ACLs to a consumer group, specify --consumer-group <group-name>.The following examples grant full access for principal bob to topic test-topic and consumer group 10, across the cluster. Substitute your ow

A often requested feature has popped up in HDP 2.5, Phoenix namespace mapping AKA Phoenix schemas. This feature allows creation of schemas and development within a schema work space. This guide will walk through steps to enable this feature and how to use it.Use Case:Create 2 schemea named "schema1" and "scheama2"Within both schemas create table customer.Create another schema named "schema3". Drop schema "schema3"Note - A schema can only be dropped if it is empty (does not host any tables).1. Through Ambari go to HBase. Select Advanced tab 2. Add a parameter to hbase-site.xml. To do this go to "Custom hbase-site" and click on "Add Property"3. Add the property Key: phoenix.schema.isNamespaceMappingEnabled Value: trueClick on Add button. A restart hbase service will be required. 4. Test the functionality, use sqlline.py. It is located under /usr/hdp/current/phoenix/binNote - I am using /usr/hdp/2.5.0.0-1245/phoenix/bin since I want to prove to the audience I am in fact using HDP 2.

When Ranger service is configured for using custom PID file location and custom user using HDP-2.5, below steps need to be followed.1) Edit ranger-admin-services.sh script file: This file is located at /usr/hdp/current/ranger-admin/ews/ranger-admin-services.sh which have hard-coded default path as pidf=/var/run/ranger/rangeradmin.pid. Need to update the hard-coded path with the custom value specified from Ambari in ranger_pid_dir property under Advanced ranger-env section. Also update the hard-coded user used for chown {pidf} with the service user’s name given during installation of Ranger service on a Misc Tab from Ambari.Update PID location:pidf={custom_pid_path}/rangeradmin.pidUpdate service user:chown {custom_user} ${pidf}2) Edit ranger-usersync-services.sh: This file is located at /usr/hdp/current/ranger-usersync/ranger-usersync-services.sh. Similar changes are required to update the hard-coded pid location and hard-coded service user for Ranger Usersync component.Update PID locat

Here is the steps to access HDFS UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/hdfsui/2.7.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/hdfsui/2.7.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/hdfsui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>HDFSUI</role> <url>http://NNHOST:50070</url> </service>7.Restart Knox service.8. You should be able to access Name node UI from the below URL https://Knxo-host:8443/gateway/default

In the past, if Select returned no rows, hadoop would still create an empty file. Some customer would use this file to help with their work flow. In https://issues.apache.org/jira/browse/HIVE-13040 this was fixed, because anyone using cloud based solutions could be charged for empty files, so it was determined that hadoop should not write the files.

SYMPTOM Concurrency issues hit during multi-threaded moveFile issued when processing queries such as "INSERT OVERWRITE TABLE ... SELECT .." The following pattern is displayed in stack trace:Loading data to table testdb.test_table from hdfs://xyz/ra_hadoop/.hive-staging_hive_2017-01-31_14-09-52_561_8101886747064006778-4/-ext-10000 ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3140) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1727) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:353) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.h

OverviewAmbari's Kerberos Descriptor is JSON-formatted document used to help Ambari enable Kerberos for installed services. The descriptor contains the information needed to create the required principals and keytab files. It also declares configuration changes needed by the serviced so they a are configured for Kerberos properly. The Kerberos Descriptor is comprised of the compiled Kerberos descriptors found in the relevant service definitions with user-specified changes applied to it. This combination of data is known as the "Composite Kerberos Descriptor", where the separate parts are known as the "Stack-level Kerberos Descriptor" and the "User-specified Kerberos Descriptor". Each of these descriptors may be obtained from Ambari via is REST API:GET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/COMPOSITEGET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/STACKGET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/USERNOTE: Be sure to replace {CLUSTER_NAME} when the name o

How to set up a Ranger enabled cluster using Ambari-blueprint. In order to setup a cluster using Ambari blueprints we will need a minimum three node cluster.Lets say this cluster contains three hosts as below:simple-ranger-a.comsimple-ranger-b.comsimple-ranger-c.com Pre-requisites: To setup a Ranger enabled cluster we need to install ambari-server on one of the hosts can be the called the gateway node let and ambari-agents should be installed on all the nodes.All the ambari-agents should be configured to point to the ambari-server on the gateway node.Note : The operating system used for this setup is Cent-OS.Lets consider the gateway node to be simple-ranger-a.com, to install ambari-server on this host we need to add ambari repository to the OS repositories. To update the OS repository we add ambari.repo file to /etc/yum.repos.d/ directory. Next we do a refresh of the OS repositories using command yum clean allThen we run the command yum install -y ambari-server ambari-agent, this wi

For users having hive insert query with dynamic partition, and partitions (> 10) on a column, you may notice that your query is generating too many small files per partition. INSERT OVERWRITE TABLE dB.Test partition(column5)select column1 ,column2 ,column3 ,column4 ,column5from Test2;For Example, if your table has 2000 partitions, and your query is generating 1009 reducers (hive.exec.reducers.max), then you might end up with 2 million small files. To Understand "How Does Tez determine the number of reducers" refer: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html)This could result into issues with:1. HDFS Namenode performance:Refer: https://community.hortonworks.com/articles/15104/small-files-in-hadoop.html2. File Merge Operation failing due java.lang.OutOfMemoryError: GC overhead limit exceeded .» File Merge , java.lang.OutOfMemoryError: GC overhead limit exceeded   at java.lang.StringCoding$StringDecoder.decode(StringC