Member since
10-28-2016
392
Posts
7
Kudos Received
20
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2792 | 03-12-2018 02:28 AM | |
| 4343 | 12-18-2017 11:41 PM | |
| 3096 | 07-17-2017 07:01 PM | |
| 2157 | 07-13-2017 07:20 PM | |
| 6503 | 07-12-2017 08:31 PM |
12-11-2017
09:43 PM
hello - i've a parquet file, and i've created an EXTERNAL Hive table on top of the parquet file. When i try to query the table, it give 0 rows, any ideas what the issue might be ? hdfs dfs -ls hdfs://abc/apps/hive/warehouse/amp.db/power/year=2017/month=12/day=01
-rw-r--r-- 2 pstl hdfs141913174 2017-12-01 22:33 hdfs://abc/apps/hive/warehouse/amp.db/power/year=2017/month=12/day=01/part-00023-e749dbd1-63a9-499d-932e-a6eadf03a67c.c000.snappy.parquet Table created : CREATE EXTERNAL TABLE power_k1(topic_k varchar(255), partition_k int, offset_k bigint, timestamp_k timestamp, deviceid bigint, devicename varchar(50), deviceip varchar(128), peerid int, objectid int, objectname varchar(256), objectdesc varchar(256), oid varchar(50), pduoutlet varchar(50), pluginid int,pluginname varchar(255), indicatorid int, indicatorname varchar(255), format int, snmppollvalue varchar(128) COMMENT 'value in sevone kafka avsc',time double, clustername varchar(50) COMMENT 'rpp or power', peerip varchar(50)) COMMENT 'external table at /apps/hive/warehouse/amp.db/sevone_power' PARTITIONED BY ( year int, month int, day int) STORED AS PARQUET LOCATION '/apps/hive/warehouse/amp.db/power' select count(1) from power_k1 -> returns 0 records Any ideas what the issue might be & how to debug this ?
... View more
Labels:
- Labels:
-
Apache Hive
08-14-2017
03:25 AM
@gnovak - Thanks, how did you conclude that my cluster has 12 GB RAM .. i mean - what parameter indicates that ?
... View more
08-07-2017
06:44 AM
@gnovak - here is what i see in YARN Capacity Scheduler yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.maximum-capacity=60 yarn.scheduler.capacity.root.default.capacity=60 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.acl_administer_jobs=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.maximum-am-resource-percent=0.9 yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=40
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.llap.capacity=40
... View more
08-07-2017
03:35 AM
This is what i see in the Ambari (YARN capacity scheduer config) --------------- yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.maximum-capacity=60 yarn.scheduler.capacity.root.default.capacity=60 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.acl_administer_jobs=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.maximum-am-resource-percent=0.9 yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=40
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.llap.capacity=40
... View more
08-07-2017
03:15 AM
@smanjee, @mqureshi, @Neeraj Sabharwal, - any ideas on this ?
... View more
08-06-2017
07:41 PM
Hello - i've HDP 2.5.x and i'm trying to launch spark-shell .. ApplicationMaster gets launched, but YARN is not able to assign containers. Command -> ./bin/spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m Error -> [Sun Aug 06 19:33:29 +0000 2017] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:2048, vCores:1>; Queue Resource Limit for AM = <memory:6144, vCores:1>; User AM Resource Limit of the queue = <memory:6144, vCores:1>; Queue AM Resource Usage = <memory:6144, vCores:3>; Any ideas on what parameters to change ? Pls note -> In YARN, the parameter - yarn.scheduler.capacity.maximum-am-resource-percent = 0.9 , AM should have access sufficient to assign container
... View more
Labels:
07-26-2017
07:11 AM
@Daniel Kozlowski - re-attaching snippet of the controller.log file at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,073] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,406] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,725] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:01,045] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
... View more
07-26-2017
07:02 AM
@Daniel Kozlowski - here are the steps i did to enable SSL : 1) create SSL certificates, create CA using openSSL, sign the certificates using the CA & import the certificate and the signed certificate into the keystore (Steps shown below) 1) GENERATE THE JKS FILE
(Generate the key and certificate for a component process)
keytool -genkey -keyalg RSA -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -keystore server.keystore.jks -storepass changeit -validity 360 -keysize 2048
2) Get the certificate signed (using OpenSSL) - DONE
CREATE THE CERTIFICATE AUTHORITY (CA)
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
ADD THE GENERATED CA TO THE SERVER'S TRUSTSTORE:
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
ADD THE GENERATED CA TO THE CLIENT'S TRUSTSTORE, SO THAT CLIENTS KNOW THAT THEY CAN TRUST THIS CA
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
3) Sign all certificates generated in Step 1 with the CA generated in Step 2a:
-> EXPORT THE CERTIFICATE FROM THE KEYSTORE
keytool -keystore server.keystore.jks -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -certreq -file cert-file
-> SIGN THE CERTIFICATE WITH THE CA
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 360 -CAcreateserial -passin pass:changeit
-> IMPORT THE CA CERTIFICATE AND THE SIGNED CERTIFICATE INTO THE KEYSTORE.
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed 2) created client.keystore.jks STEPS 1) GENERATE THE JKS FILE :
keytool -genkey -keyalg RSA -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -keystore client.keystore.jks -storepass changeit -validity 360 -keysize 2048
2) EXPORT THE CERTIFICATE FROM THE KEYSTORE :
keytool -keystore client.keystore.jks -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -certreq -file cert1-file
3) SIGN THE CERTIFICATE WITH THE CA
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert1-file -out cert1-signed -days 360 -CAcreateserial -passin pass:changeit
4) IMPORT THE CA CERTIFICATE AND THE SIGNED CERTIFICATE INTO THE KEYSTORE.
keytool -keystore client.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.keystore.jks -alias localhost -import -file cert1-signed 3) Made changes to server.properties (shared earlier) Restarted Zookeeper & Kafka. wrt changing the log4j.rootLogger parameter in /etc/kafka/conf/tools-log4j.properties, i'd changed the mode to DEBUG, but it seems to be getting reverted back to "WARN" when i restart the Broker .. How do i ensure it doesn't get reverted back? Currently, the error message in the controller.log is same as shared in earlier post.
... View more
07-26-2017
06:16 AM
@Daniel Kozlowski - when i telnet (controller to broker, i.e 1001 -> 1001), here is what i seem.. seems connectivity on the ssl port is Not an issue [root@nwk2-bdp-kafka-04 kafka]# telnet nwk2-bdp-kafka-04.gdcs-qa.apple.com 6668
Trying 10.78.130.143...
Connected to nwk2-bdp-kafka-04.gdcs-qa.apple.com.
Escape character is '^]'.
... View more
07-26-2017
06:08 AM
@Daniel Kozlowski - here is the update on the this. with -> security.inter.broker.protocol = PLAINTEXT, i'm able to start the Console Producer & consumer and publish & read the messages published. However, when i set -> security.inter.broker.protocol = SSL, the errors start. (in both PLAINTEXT & SSL modes) when i start the Console Producer in PLAINTEXT Mode : /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-04.gdcs-qa.ale.com:6667 --topic sslTopic3 --security-protocol PLAINTEXT
[2017-07-26 05:53:26,172] WARN Error while fetching metadata with correlation id 17 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-07-26 05:53:26,277] WARN Error while fetching metadata with correlation id 18 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-07-26 05:53:26,388] WARN Error while fetching metadata with correlation id 19 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) when i start the Console Producer in SSL Mode : /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-04.gdcs-qa.ale.com:6668 --topic sslTopic3 --producer.config /tmp/ssl-kafka/client-ssl.properties --security-protocol SSL
hi
HELLO
[2017-07-26 05:59:31,888] ERROR Error when sending message to topic sslTopic3 with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. Error in controller.log file : [2017-07-26 05:58:49,535] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 05:58:49,853] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
when i describe the topic, i see that the leader is 1001 & Isr has only 1001 /usr/hdp/2.5.3.0-37/kafka/bin/kafka-topics.sh --describe --zookeeper nwk2-bdp-kafka-05.gdcs-qa.apple.com:2181,nwk2-bdp-kafka-04.gdcs-qa.apple.com:2181,nwk2-bdp-kafka-06.gdcs-qa.apple.com:2181 --topic sslTopic3
Topic:sslTopic3PartitionCount:3ReplicationFactor:3Configs:
Topic: sslTopic3 Partition: 0 Leader: 1001 Replicas: 1003,1001,1002 Isr: 1001
Topic: sslTopic3 Partition: 1 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1001
Topic: sslTopic3 Partition: 2 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001 It seems setting the parameter -> security.inter.broker.protocol = SSL causes connectivity issues between the Controller (in this case 1001) & the Brokers (1001, 1002, 1003) The question is why & what needs to be done to fix this ?
... View more