Member since
03-14-2016
4721
Posts
1110
Kudos Received
874
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1723 | 04-27-2020 03:48 AM | |
3380 | 04-26-2020 06:18 PM | |
2694 | 04-26-2020 06:05 PM | |
2098 | 04-13-2020 08:53 PM | |
3122 | 03-31-2020 02:10 AM |
09-17-2018
07:45 AM
@Saurabh You can pass your own "log4j.properties" path to log messages and pass it to your spark shell command. Example: # spark-shell --master yarn --deploy-mode client --files /your/path/to/log4j.properties --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" --driver-java-options "-Dlog4j.configuration=file:/your/path/to/log4j.properties"
... View more
09-17-2018
05:57 AM
@Manikandan Jeyabal
It is hard to provide the exact values based on the data size. However you can refer to the following article to understand Executor Memory/Core/Resource Optimization. https://community.hortonworks.com/articles/42803/spark-on-yarn-executor-resource-allocation-optimiz.html https://dzone.com/articles/apache-spark-on-yarn-resource-planning
... View more
09-15-2018
11:48 PM
@Ankita Ghate Thank you for opening a new HCC thread for the current issue. For the "UNKNOWN_TOPIC_OR_PARTITION" issue i have responded to: https://community.hortonworks.com/questions/221979/not-able-to-produce-messages-on-kafka-broker-after-1.html
... View more
09-15-2018
11:46 PM
@Ankita Ghate Can you please check if "kafka.acl" is setup proplery? This is just to verify if you have provided proper read permission on the group and topic? as we see "UNKNOWN_TOPIC_OR_PARTITION" https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/kafka-acl-examples.html # kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --list Also have you enabled Kafka Ranger Plugin? In that case have you setup the ranger policies correctly? - Also can you please try restarting your Kafka Brokers and try once again?
... View more
09-15-2018
12:24 AM
@Ankita Ghate Good to know that your original issue reported in the thread is resolved. I see that your current issue is slightly different. It is always recommended to open separate HCC thread for different issues so that One HCC thread has one Specific solution / answer. This helps us HCC users to quickly find the solution for originally reported issues. Long threads with multiple issues addressed in Single thread causes users to get easily confused. I will suggest you to please mark this HCC thread as Answered (Accepted) by clicking the Accept button and then open a new thread for the recent issue.
... View more
09-14-2018
04:59 AM
@Shantanu
Sharma
As you are getting JAVA_HOME not set issue while attempting to manually start HS2 so please try this: # su $HIVE_USER
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_112
# nohup /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=/tmp/hiveserver2HD.out 2 /tmp/hiveserver2HD.log . lets start the HS2 properly now (by using the manual startup command) and then we will troubleshoot the Ambari Side HS2 startup issue.
... View more
09-14-2018
03:22 AM
@Shantanu
Sharma
On the host where it is failing to start please check the XML file if it has the correct entry? # grep -A2 -B2 'hive.server2.support.dynamic.service.discovery' '/etc/hive/conf/hive-site.xml'
<property>
<name>hive.server2.support.dynamic.service.discovery</name>
<value>true</value>
</property> Also please try starting the HS2 server manually to see if it works. As described in : https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/administration/content/starting_hdp_services.html # su $HIVE_USER
# nohup /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=/tmp/hiveserver2HD.out 2 /tmp/hiveserver2HD.log If manually starting HiveSerevr2 works then in that case we might need to see if cleaning the Ambari Agent cached data "/var/lib/ambari-agent/cache" "data" ..etc and restarting agent fixes it or not?
... View more
09-14-2018
02:17 AM
Please login to Ambari UI and then check the Hive Configuration Param if the following Checkbix is ticket (Selected) or not? hive.server2.support.dynamic.service.discovery = true It can be found at: Ambari UI --> Hive --> Configs --> Advanced -- > "Advanced hive-site" Please make sure that the mentioned property is selected and then restart the Hive Services.
... View more
09-13-2018
08:39 AM
@Michael Bronson Adding @Tarun Parimi . Tarun, Can you please share your thoughts on this.
... View more
09-13-2018
06:43 AM
@Michael Bronson The yarn.nodemanager.resource.cpu-vcores value sets the
number of CPU cores that can be allocated for containers.You should set the number of vcores to match the number of physical CPU cores on the
NodeManager hosts. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_yarn-resource-management/content/enabling_cpu_scheduling.html So depends on how many CPU cores do you have? Following is a good reference point for vcores: https://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/ .
... View more