Greg's answer applies to you as well for incremental import/export operation. Also if you have some column in your source table which is an sequential index etc then you can be used for --split-by clause for distribution of data per mapper to scale parallelism and reduce runtime of app. My understanding is random numbers in a column if used for split key , can cause skew as well leading to different runtimes for map tasks. ... View more

Try disabling vectorization for this job alone, I remember this being a bug in hive-1.2.1 i.e set hive.vectorized.execution.enabled=false; as workaround. ... View more

Are you trying to setup hive metastore for the first time? Based on below it either looks like you are run init from hive client of a different version or you have run it multiple times that entry already committed into mysql db due to first run. I faced below problem when I did hive upgrade from 1.2.x to 2.0 as part of migration effort to Ambari from the Apache hive. Error: Duplicate key name 'PCS_STATS_IDX' (state=42000,code=1061) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace. *** schemaTool failed *** One thing I figured out is rather than running upgrade scripts individually better to do it using below approach. export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/conf.server [root@myhost ~]# /usr/hdp/current/hive-server2-hive2/bin/schematool -upgradeSchema -dbType mysql -userName hive_user -passWord 'XXXXXXXXXX' SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.0-157/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.0-157/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL: jdbc:mysql://XXXXX/hive_db Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive_user Starting upgrade metastore schema from version 1.2.0 to 2.1.0 Upgrade script upgrade-1.2.0-to-1.2.1000.mysql.sql Completed upgrade-1.2.0-to-1.2.1000.mysql.sql Upgrade script upgrade-1.2.1000-to-2.0.0.mysql.sql Completed upgrade-1.2.1000-to-2.0.0.mysql.sql Upgrade script upgrade-2.0.0-to-2.1.0.mysql.sql Completed upgrade-2.0.0-to-2.1.0.mysql.sql schemaTool completed note: If your hive metadata is important, before you do any upgrade , I strongly suggest to take backup of it using mysqldump utility. ... View more

For your first question explore param ipc.maximum.data.length that should help, on the other hand, value of 65536 for dfs.datanode.max.xcievers seems way high. Basically, I feel your datanode block reports are not reaching namenode because of length limitation and so namenode is missing enough blocks to exit safemode. Thus it makes sense why its reporting missing blocks upon forceful safemode exit. For namenode heap configuration visit https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/configuring-namenode-heap-size.html ... View more

If you would like to just check if your hive server2 port is accepting connections or not you can run checks like below. nc -zv <hiveserver2 hostname> 10000 Connection to localhost 10000 port [tcp/ndmp] succeeded! hive metastore connections: netstat -all|grep 9083|wc -l 314 But note above doesn't guarantee whether your hive service is really running any tasks or not. Best way for figuring that out is to enable hive metrics which you can poll every few minutes as per your requirement. If you have Grafana setup it should help as well. ps: I'm not sure what distribution you are using, however, if you are on HDP, ambari already has run service check option which you can see if you can invoke it via rest api. ... View more

For me adding the line below to spark-defaults.conf helped based on packages installed on my test cluster. spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native/:/usr/hdp/current/share/lzo/0.6.0/lib/native/Linux-amd64-64/ ... View more

As mentioned before Hive version 0.12 ships varchar datatype, based on your cdh4.3.0 it looks like hive version is 0.10 (ref: https://archive.cloudera.com/cdh4/cdh/4/hive-0.10.0-cdh4.3.0.releasenotes.html ) ... View more

Probably your version of hive is old. Check below. Varchar datatype was introduced in Hive 0.12.0 (HIVE-4844). ... View more