Member since
04-13-2016
422
Posts
150
Kudos Received
55
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1937 | 05-23-2018 05:29 AM | |
| 4972 | 05-08-2018 03:06 AM | |
| 1686 | 02-09-2018 02:22 AM | |
| 2716 | 01-24-2018 08:37 PM | |
| 6175 | 01-24-2018 05:43 PM |
10-26-2016
02:29 PM
2 Kudos
@ARUN It's very high The Hadoop RPC server consists of a single RPC queue per port and multiple handler (worker) threads that dequeue and process requests. If the number of handlers is insufficient, then the RPC queue starts building up and eventually overflows. You may start seeing task failures and eventually job failures and unhappy users. It is recommended that the RPC handler count is set to 20 * log2(Cluster Size) with an upper limit of 200. e.g. for a 64 node cluster you should initialize this to 20 * log2(64) = 120. The RPC handler count can be configured with the following setting in hdfs-site.xml <property>
<name>dfs.namenode.handler.count</name>
<value>120</value>
</property> This heuristic is from the excellent Hadoop Operations book. If you are using Ambari to manage your cluster this setting can be changed via a slider in the Ambari Server Web UI. Link. Hope this helps your.
... View more
10-20-2016
08:27 PM
@zhao xiaojing: Can you please check gateway and hst server logs?
... View more
10-16-2016
04:56 PM
2 Kudos
If you are not using any Hive Authorization like SQL Standard Authorization or Ranger and you don't want other to read/write data and allow the only owner to read & write, and allowing the group to read. Changing the warehouse directory permission. 1.From the command line in the Ambari server node, edit the file /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py Search for hive_apps_whs_dir which should go to this block: params.HdfsResource(params.hive_apps_whs_dir,
type="directory",
action="create_on_execute",
owner=params.hive_user,
group=params.user_group,
mode=0755
)
Modify the value for mode from 0755 to the desired permission, for example, 0777. Save and close the file. Restart the Ambari server to propagate the change to all nodes in the cluster: ambari-server restart It may take a few seconds to update the file in the Ambari agents on all nodes. To verify if the change has been applied on a particular node, check the content of hive.py in /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py From the Ambari UI, restart HiveServer2 to apply the new permission to the warehouse directory. If multiple HiveServer2 instances are configured, any one instance can be restarted. Hope this helps you.
... View more
Labels:
10-11-2016
09:12 PM
2 Kudos
Hi, Is it possible to install two version of Spark on the same cluster? I know that in HDP 2.5 it supports Spark 1.6 and Spark 2.0 but I want them on either HDP 2.3.6 to install Spark 1.5 and Spark 1.6. The reason for above requirement is we are integrating the external tool with Hadoop and Spark, where one tool support only Spark 1.5 and other tool require Spark 1.6. Any help is highly appricated.
... View more
Labels:
- Labels:
-
Apache Spark
10-03-2016
03:03 PM
Hi, I'm doing POC on HBase and Phoenix Query Server for next project. We would like to tune both the services for the best outcome for operations on 100million records.
Right now we are using most of the default parameters which come with Hortonworks installation. Can anyone provide the best set of tuning properties operations on 100million records? Any help is highly appreciated and thanks in advance.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Phoenix
09-10-2016
01:37 AM
1 Kudo
@mliem It's always recommended to have 3 zookeeper. Are you installing HBase? When setting up the zookeeper quorum the reasonable numbers are 1, 3 and 5 nodes. 1 is useful if you don't want redundancy at all. This happens, for instance on the sandbox version where you have only a single node in the cluster. 3 is useful for failure tolerance, but it is sensitive to hardware failure during maintenance when you might have one machine down. 5 is used in large, high-value clusters which need to stay up at all costs. It is very rare to use more than 5 Zookeeper nodes in a cluster.
... View more
09-07-2016
05:22 PM
@Saurabh Kumar Changing the warehouse directory permission
1.From the command line in the Ambari server node, edit the file /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
Search for hive_apps_whs_dir which should go to this block: params.HdfsResource(params.hive_apps_whs_dir,
type="directory",
action="create_on_execute",
owner=params.hive_user,
group=params.user_group,
mode=0755
)
Modify the value for mode from 0755 to the desired permission, for example 0777. Save and close the file.
Restart the Ambari server to propagate the change to all nodes in the cluster: ambari-server restart
It may take a few seconds to update the file in the Ambari agents on all nodes. To verify if the change has been applied on a particular node, check the content of hive.py in /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
From the Ambari UI, restart HiveServer2 to apply the new permission to the warehouse directory. If multiple HiveServer2 instances are configured, any one instance can be restarted. Hope this helps you.
... View more
09-01-2016
04:23 PM
@Josh Elser @Randy Gelhausen Thanks for the quick reply. If I don't install Phoneix Query Server then is it possible to establish JDBC connection? @Ted Yu
... View more
09-01-2016
03:32 PM
Hi, When I'm trying to establish connection between HBase, It looks like when you try to do the native HBase connection in metadata, HBase server requires the JDBC connection to go through the Phoenix driver so that we can establish the connection.
Is there anyway if we can establish a JDBC connection with your HBase bypassing the Phoenix driver? If yes, please share me the steps or document where I can get process to establish connection between HBase and Talend etc., Thanks in advance.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Phoenix
08-30-2016
02:53 PM
@Frank Lu @Kit Menke Below is the error message I'm getting: Processor is: 0s
16/08/25 15:58:40 ERROR teradata.TeradataSqoopImportHelper:
Exception running Teradata import job
com.teradata.connector.common.exception.ConnectorException:
Import Hive table's column schema is missing
at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)
at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)
at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)
at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)
at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)
16/08/25 15:58:40 INFO teradata.TeradataSqoopImportHelper:
Teradata import job completed with exit code 1
16/08/25 15:58:40 ERROR tool.ImportTool: Encountered
IOException running import job: java.io.IOException: Exception running Teradata
import job
at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:373)
at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)
at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by:
com.teradata.connector.common.exception.ConnectorException: Import Hive table's
column schema is missing
at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)
at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)
at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)
... 9 more Hope this helps you.
... View more