PROBLEM: When the hostname is mixed like: 172.26.93.148 GRAFANA-hdp253-s1.openstacklocal GRAFANA-hdp253-s1 172.26.93.149 GRAFANA-hdp253-s2.openstacklocal GRAFANA-hdp253-s2 172.26.93.150 GRAFANA-hdp253-s3.openstacklocal GRAFANA-hdp253-s3 Ambari creates the datasource with a lower case hostname: Once you enter the hostname with mixed case, Grafana works: RESOLUTION Grafana 2.6.0 backend uses Go 1.5. Go's DNS lookup had a bug where the look up is case sensitive: https://github.com/golang/go/issues/12806 We can manual workaround by using the original casing in the Grafana Data Source URL as mentioned in the description. Another workaround is to update the /etc/hosts file to have all lowercase patterns as well. This bug was fixed in Go 1.6, so we will no longer have this issue once we upgrade Grafana to a later version (say 4.1.x) in a future version of Ambari. ... View more

PROBLEM: For example, while running from shell, we usually append the single quotes around the date type: hive> select * from students where datestamp='2014-09-23'; OK fred flintstone351.282014-09-23 Time taken: 0.761 seconds, Fetched: 1 row(s) But in case of Hue, internally it doesn't append these single quotes and throws error on browse data. STEPS TO REPRODUCE: 1. Create table: CREATE TABLE students(name varchar(64), age int, gpa decimal(3,2)) PARTITIONED BY ( datestamp date); 2. Insert: INSERT INTO TABLE students PARTITION (datestamp = '2014-09-23') VALUES ('fred flintstone', 35, 1.28); 3. Login into hue -> Go to HCatalog -> Tables -> select 'students' -> Click on Browse data 4. This will generate error: RESOLUTION: There is an internal bug reported. Please reach out to support. ... View more

PROBLEM: We see below message on initial runs: 2017-02-28 19:37:48,681 INFO [main] impl.TimelineClientImpl: Timeline service address: http://<timeline-server-hostname>:8188/ws/v1/timeline/ 2017-02-28 19:37:48,823 INFO [main] client.AHSProxy: Connecting to Application History server at <history-server-hostname>/<history-server-ip>:10200 2017-02-28 19:37:49,016 WARN [main] ipc.Client: Failed to connect to server: <resource-manager-A>/<resource-manager-A-ip>:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ROOT CAUSE: In Yarn configs -> custom yarn-site.xml, you might have configuration for rm1 as resource-manager-A. This is the reason first attempt to connect was made to this server and then got connection refused. Then it goes to active resource manager which is resource-manager-B. This warning is not seen when resource-manager-A is active, because the property rm1 pointing to this server, first connect attempt is successful. First connection attempt is always made to the resource manager which is specified in rm1. RESOLUTION: This warning cannot be suppressed as of now. There is a jira open to change the logging: https://issues.apache.org/jira/browse/YARN-6145 ... View more

PROBLEM: For an external hive table created based on hbase, if there are any missing mappings or any other issues (Syntactical), create table statement is executed successfully. You can see the table is created. However, while trying to insert data in that table, following error is seen: 08S01: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1495472057323_3947_1_00, Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:800) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:170) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:555) ROOT CAUSE Per: https://wiki.apache.org/hadoop/Hive/HBaseIntegration?action=diff&rev1=13&rev2=14 If there are any issues with the section: WITH SERDEPROPERTIES ( "hbase.columns.mapping" = "cf1:val", "hbase.table.name" = "xyz" ); while creating the table, then the CREATE TABLE will succeed, but attempts to insert data will fail with this internal error: {{{ java.lang.RuntimeException: org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector }}} RESOLUTION Create table should have clear definition of mappings to hbase table. Refer: https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration ... View more

This is for HDP 2.5 only. If you are seeing the same error HDP 2.6, there could be something else that has failed before this stage. Please check the full log.   After enabling Hive LLAP, it fails to start with: ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {} org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] Steps to fix: 1. /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server `hostname` 2. create /llap-sasl "" sasl:hive:cdrwa,world:anyone:r 3. create /llap-sasl/user-hive "" sasl:hive:cdrwa,world:anyone:r 4. create /llap-sasl/user-hive/llap0 "" sasl:hive:cdrwa,world:anyone:r 5. create /llap-sasl/user-hive/llap0/workers "" sasl:hive:cdrwa,world:anyone:r Note: If Kerberos is enabled: su as zookeeper kinit as hive ... View more

PROBLEM: When we go to Grafana UI: Under HBase - Tables: 1. NUM FLUSHES 2. NUM WRITE REQUESTS 3. NUM READ REQUESTS Under HBase - Users: 1. Num Get Requests 2. Num Scan Next Requests We just see: Problem! java.lang.Exception: Invalid number of functions specified. grafana.log show: [I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 7653us [I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 3316us [I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 1734us RESOLUTION: 1. Login with Grafana admin. 2. Set transform=none in panels. ... View more

SYMPTOMS: Sometimes the Hive CLI hangs and the gives no response. At the same time, the /var/log/hive/hivemetastore.log reports timeouts ERROR [<hostname>-47]: txn.TxnHandler (TxnHandler.java:getDbConn(984)) - There is a problem with aconnection from the pool, retrying(rc=9): Timed out waiting for a free available connection. (SQLState=08001,ErrorCode=0) java.sql.SQLException: Timed out waiting for a free available connection. ROOT CAUSE: The Hive Metastore service is waiting for the database connections to get free. At the time of error, not enough concurrent connections were available. RESOLUTION: (Note: This is for mysql DB only). To fix this problem, the number of max database connection should be increased. 1. Open the /etc/my.cnf in the text editor. vi /etc/my.cnf 2. Under [mysqld] section, add: max_connections = 250 3. Save the file and restart the mysqld service service mysqld restart ... View more

HDP Stack Version: 2.4.0 SYMPTOMS: WARN Error while fetching metadata [{TopicMetadata for topic <topic-name> -> No partition metadatafor topic <topic-name> due to kafka.common.TopicAuthorizationException}] for topic <topic-name>: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) .... ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: <topic-name> (kafka.producer.async.DefaultEventHandler) ROOT CAUSE: At the moment we can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it is not possible to assert client’s identity over the non-secure channel. It is ip based in a non-secure environment. Reference: https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin See: Authorizing Kafka access over non-authenticated channel via Ranger RESOLUTION: The policy doesn't work if, IP address is not configured. This address will be of producer and consumer. ... View more

Run hive shell debug so it prints detailed error: hive -hiveconf hive.log.file=hivecli_tez.log -hiveconf hive.log.dir=/tmp/hivecli -hiveconf hive.execution.engine=tez -hiveconf hive.root.logger=DEBUG,DRFA SYMTOMS: DEBUG [main]: amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1152)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: null; Status Code: 403; Error Code: 403 Forbidden; Request ID: 85BA6566D33A519B), S3 Extended Request ID: 228pqAjcCjTHo+ExpZ+86INHAhkIeE+DQoicPLkan8GDaraxsklIuHwK3f+QmjtIBzw/z5OSWaM= WARN [main]: avro.AvroSerDe (AvroSerDe.java:determineSchemaOrReturnErrorSchema(169)) - Encountered AvroSerdeException determining schema. Returning signal schema to indicate problem org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read schema from given path: s3a://<file-path> WORKAROUND: 01. mkdir joda-backup 02. cd /tmp 03. wget http://central.maven.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar 04. mv /usr/hdp/current/hive/lib/joda-time-2.5.jar /joda-backup 05. mv /usr/hdp/current/hive2/lib/joda-time-2.5.jar /joda-backup 06. cp /tmp/joda-time-2.8.1.jar /usr/hdp/2.5.0.0-1245/hive/lib/ 07. cp /tmp/joda-time-2.8.1.jar /usr/hdp/2.5.0.0-1245/hive2/lib/ 08. unzip joda-time-2.8.1.jar -d /unzip-joda 09. cd ./unzip-joda Take a backup of hive-exec-*.jar 10. jar -uf /usr/hdp/current/hive/lib/hive-exec-1.2.1000.2.5.0.0-1245.jar ./org 11. jar -uf /usr/hdp/current/hive2/lib/hive-exec-2.1.0.2.5.0.0-1245.jar ./org Permissions on this jar should be: -rw-r--r--. 1 root root joda-time-2.8.1.jar ... View more

SYMTOMPS: Ambari server log shows: WARN [qtp-client-41858] ObjectGraphWalker:209 - The configured limit of 1,000 object references was reached while attempting to calculate the size of the object graph. Severe performance degradation could occur if the sizing operation continues. This can be avoided by setting the CacheManger or Cache <sizeOfPolicy> elements maxDepthExceededBehavior to "abort" or adding stop points with @IgnoreSizeOf annotations. If performance degradation is NOT an issue at the configured limit, raise the limit value using the CacheManager or Cache <sizeOfPolicy> elements maxDepth attribute. For more information, see the Ehcache configuration documentation. WORK-AROUND Disable cache by setting below property in in /etc/ambari-server/conf/ambari.properties server.timeline.metrics.cache.disabled = true ROOT CAUSE See: AMBARI-13517 Disabling cache is not going have adverse effect. It was introduced in Ambari 2.1.2 to create a Caching layer that provides sliding window behavior for metric requests to Ambari. ... View more