Hi: I have locally in a postgres database the hivemetastore, but now we will migrate to a remote metastore with a postgres database also -Is necesary to migrate the hive metastore database to the new postgres instance? -Is necesary to shut down the cluster? Thanks ... View more

Hi: We have this error when I do a quey from Hive: errorMessage='java.io.IOException: java.io.IOException: Cannot obtain block length for LocatedBlock{BP-277687468-192.168.11.18-1438941203340:blk_1091826778_18212579; getBlockSize()=636546; corrupt=false Also the HDFS is HEALTHY, any suggestion? Thanks ... View more

HI: Could I Install Sqoop and FLume in HDF or just i can install it on HDP? Thanks ... View more

Hi: I am indexing table between hive and elasticsearch and iam receiving this error, any sugestion: here is my claspath /usr/hdp/current/hive-client/lib ls -lrt total 110288 -rw-r--r-- 1 root root 65261 ago 17 15:27 oro-2.0.8.jar -rw-r--r-- 1 root root 41123 ago 17 15:27 commons-cli-1.2.jar -rw-r--r-- 1 root root 284220 ago 17 15:27 commons-lang-2.6.jar -rw-r--r-- 1 root root 105112 ago 17 15:28 servlet-api-2.5.jar -rw-r--r-- 1 root root 185140 ago 17 15:29 commons-io-2.4.jar -rw-r--r-- 1 root root 588337 ago 17 15:29 commons-collections-3.2.2.jar -rw-r--r-- 1 root root 445288 ago 17 15:29 antlr-2.7.7.jar -rw-r--r-- 1 root root 62050 ago 17 15:30 commons-logging-1.1.3.jar -rw-r--r-- 1 root root 58160 ago 17 15:30 commons-codec-1.4.jar -rw-r--r-- 1 root root 33031 ago 17 15:30 jsr305-3.0.0.jar -rw-r--r-- 1 root root 714194 ago 17 15:30 javassist-3.18.1-GA.jar -rw-r--r-- 1 root root 62983 ago 17 15:30 activation-1.1.jar -rw-r--r-- 1 root root 94672 ago 17 15:30 xz-1.0.jar -rw-r--r-- 1 root root 533455 ago 17 15:30 protobuf-java-2.5.0.jar -rw-r--r-- 1 root root 29555 ago 17 15:30 paranamer-2.3.jar -rw-r--r-- 1 root root 45944 ago 17 15:33 json-20090211.jar -rw-r--r-- 1 root root 25429 ago 17 15:56 regexp-1.3.jar -rw-r--r-- 1 root root 21879 ago 17 16:05 asm-tree-3.1.jar -rw-r--r-- 1 root root 32693 ago 17 16:05 asm-commons-3.1.jar -rw-r--r-- 1 root root 250546 ago 17 16:23 plexus-utils-1.5.6.jar -rw-r--r-- 1 root root 1208356 ago 17 16:23 netty-3.7.0.Final.jar -rw-r--r-- 1 root root 481535 ago 17 16:23 log4j-1.2.16.jar -rw-r--r-- 1 root root 2189117 ago 17 16:23 guava-14.0.1.jar -rw-r--r-- 1 root root 60527 ago 17 16:24 jcommander-1.32.jar -rw-r--r-- 1 root root 26514 ago 17 16:42 stax-api-1.0.1.jar -rw-r--r-- 1 root root 48557 ago 17 16:48 pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar -rw-r--r-- 1 root root 611863 ago 17 16:48 janino-2.7.6.jar -rw-r--r-- 1 root root 18482 ago 17 16:48 eigenbase-properties-1.1.5.jar -rw-r--r-- 1 root root 96221 ago 17 16:48 commons-pool-1.5.4.jar -rw-r--r-- 1 root root 160519 ago 17 16:48 commons-dbcp-1.4.jar -rw-r--r-- 1 root root 30595 ago 17 16:48 commons-compiler-2.7.6.jar -rw-r--r-- 1 root root 19827 ago 17 16:50 opencsv-2.3.jar -rw-r--r-- 1 root root 1251514 ago 17 16:50 snappy-java-1.0.5.jar -rw-r--r-- 1 root root 448794 ago 17 16:53 apache-log4j-extras-1.2.17.jar -rw-r--r-- 1 root root 213854 ago 17 16:53 jline-2.12.jar -rw-r--r-- 1 root root 279781 ago 17 16:53 commons-httpclient-3.0.1.jar -rw-r--r-- 1 root root 588001 ago 17 16:53 joda-time-2.5.jar -rw-r--r-- 1 root root 15071 ago 17 16:54 jta-1.1.jar -rw-r--r-- 1 root root 201124 ago 17 16:54 jdo-api-3.0.1.jar -rw-r--r-- 1 root root 110600 ago 17 16:54 bonecp-0.8.0.RELEASE.jar -rw-r--r-- 1 root root 148627 ago 17 16:54 stringtemplate-3.2.1.jar -rw-r--r-- 1 root root 164368 ago 17 16:54 antlr-runtime-3.4.jar -rw-r--r-- 1 root root 236660 ago 17 16:54 ST4-4.0.4.jar -rw-r--r-- 1 root root 35907 ago 17 16:54 metrics-jvm-3.1.0.jar -rw-r--r-- 1 root root 15823 ago 17 16:55 metrics-json-3.1.0.jar -rw-r--r-- 1 root root 111908 ago 17 16:55 metrics-core-3.1.0.jar -rw-r--r-- 1 root root 38605 ago 17 16:55 jackson-annotations-2.4.0.jar -rw-r--r-- 1 root root 6998724 ago 17 16:55 groovy-all-2.4.4.jar -rw-r--r-- 1 root root 447676 ago 17 16:56 mail-1.4.1.jar -rw-r--r-- 1 root root 12131 ago 17 16:56 jpam-1.1.jar -rw-r--r-- 1 root root 1681148 ago 17 16:56 jetty-all-7.6.0.v20120127.jar -rw-r--r-- 1 root root 93210 ago 17 16:59 super-csv-2.2.0.jar -rw-r--r-- 1 root root 1683027 ago 17 17:00 jetty-all-server-7.6.0.v20120127.jar -rw-r--r-- 1 root root 1075759 ago 17 20:22 jackson-databind-2.4.2.jar -rw-r--r-- 1 root root 225316 ago 17 20:22 jackson-core-2.4.2.jar -rw-r--r-- 1 root root 69569 ago 18 00:21 HikariCP-1.3.9.jar -rw-r--r-- 1 root root 1888550 ago 18 00:21 datanucleus-rdbms-4.1.7.jar -rw-r--r-- 1 root root 2009163 ago 18 00:21 datanucleus-core-4.1.6.jar -rw-r--r-- 1 root root 366565 ago 18 00:21 datanucleus-api-jdo-4.2.1.jar -rw-r--r-- 1 root root 15071 ago 18 00:21 transaction-api-1.1.jar -rw-r--r-- 1 root root 249790 ago 18 00:21 javax.jdo-3.2.0-m3.jar -rw-r--r-- 1 root root 15935 ago 18 00:21 dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar -rw-r--r-- 1 root root 392124 ago 26 03:37 velocity-1.5.jar -rw-r--r-- 1 root root 30359 ago 26 03:38 apache-curator-2.6.0.pom -rw-r--r-- 1 root root 794085 ago 26 03:38 zookeeper-3.4.6.2.5.0.0-1245.jar -rw-r--r-- 1 root root 234201 ago 26 03:38 libthrift-0.9.3.jar -rw-r--r-- 1 root root 321639 ago 26 03:38 httpcore-4.4.jar -rw-r--r-- 1 root root 719304 ago 26 03:38 httpclient-4.4.jar -rw-r--r-- 1 root root 185245 ago 26 03:38 curator-framework-2.6.0.jar -rw-r--r-- 1 root root 68866 ago 26 03:38 curator-client-2.6.0.jar -rw-r--r-- 1 root root 832410 ago 26 03:38 commons-math-2.1.jar -rw-r--r-- 1 root root 112821 ago 26 03:38 hive-shims-common-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 32394 ago 26 03:38 hive-shims-0.20S-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 400680 ago 26 03:39 avro-1.7.5.jar -rw-r--r-- 1 root root 1475955 ago 26 03:39 htrace-core-3.1.0-incubating.jar -rw-r--r-- 1 root root 248171 ago 26 03:39 curator-recipes-2.6.0.jar -rw-r--r-- 1 root root 241367 ago 26 03:39 commons-compress-1.4.1.jar -rw-r--r-- 1 root root 61961 ago 26 03:39 hive-shims-0.23-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 13531 ago 26 03:39 hive-shims-scheduler-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 10387 ago 26 03:39 hive-shims-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 18336 ago 26 03:39 ant-launcher-1.9.1.jar -rw-r--r-- 1 root root 1997485 ago 26 03:39 ant-1.9.1.jar -rw-r--r-- 1 root root 345828 ago 26 03:39 hive-common-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 2902379 ago 26 03:39 parquet-hadoop-bundle-1.8.1.jar -rw-r--r-- 1 root root 924676 ago 26 03:40 hive-serde-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 313702 ago 26 03:40 libfb303-0.9.3.jar -rw-r--r-- 1 root root 2838580 ago 26 03:40 derby-10.10.2.0.jar -rw-r--r-- 1 root root 6038452 ago 26 03:40 hive-metastore-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 49710 ago 26 03:40 hive-ant-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 1282424 ago 26 03:40 ivy-2.4.0.jar -rw-r--r-- 1 root root 446274 ago 26 03:41 calcite-linq4j-1.2.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 3537018 ago 26 03:41 calcite-core-1.2.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 20137 ago 26 03:41 avatica-metrics-1.8.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 5223335 ago 26 03:41 avatica-1.8.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 21667449 ago 26 03:42 hive-exec-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 12452 ago 26 03:42 geronimo-annotation_1.0_spec-1.1.1.jar -rw-r--r-- 1 root root 16030 ago 26 03:42 geronimo-jta_1.1_spec-1.1.1.jar -rw-r--r-- 1 root root 30548 ago 26 03:42 geronimo-jaspic_1.0_spec-1.0.jar -rw-r--r-- 1 root root 1983070 ago 26 03:42 hive-service-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 69858 ago 26 03:43 maven-scm-provider-svnexe-1.4.jar -rw-r--r-- 1 root root 40066 ago 26 03:43 maven-scm-provider-svn-commons-1.4.jar -rw-r--r-- 1 root root 94421 ago 26 03:43 maven-scm-api-1.4.jar -rw-r--r-- 1 root root 16532 ago 26 03:43 accumulo-trace-1.7.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 64528 ago 26 03:43 accumulo-start-1.7.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 110607 ago 26 03:43 accumulo-fate-1.7.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 415578 ago 26 03:43 commons-vfs2-2.0.jar -rw-r--r-- 1 root root 4923758 ago 26 03:43 accumulo-core-1.7.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 122811 ago 26 03:43 hive-accumulo-handler-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 106364 ago 26 03:43 hive-jdbc-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 20087903 ago 26 03:45 hive-jdbc-1.2.1000.2.5.0.0-1245-standalone.jar -rw-r--r-- 1 root root 144514 ago 26 03:45 hive-beeline-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 40089 ago 26 03:45 hive-cli-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 122696 ago 26 03:46 hive-contrib-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 118677 ago 26 03:46 hive-hbase-handler-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 29090 ago 26 03:46 hive-hwi-1.2.1000.2.5.0.0-1245.jar -rw-r--r-- 1 root root 16837 ago 26 05:02 ranger-plugin-classloader-0.6.0.2.5.0.0-1245.jar -rw-r--r-- 1 root root 17191 ago 26 05:02 ranger-hive-plugin-shim-0.6.0.2.5.0.0-1245.jar drwxr-xr-x 2 root root 4096 oct 14 11:32 ranger-hive-plugin-impl lrwxrwxrwx 1 root root 47 oct 14 11:33 hive-accumulo-handler.jar -> hive-accumulo-handler-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 34 oct 14 11:33 hive-ant.jar -> hive-ant-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 38 oct 14 11:33 hive-beeline.jar -> hive-beeline-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 34 oct 14 11:33 hive-cli.jar -> hive-cli-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 37 oct 14 11:33 hive-common.jar -> hive-common-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 35 oct 14 11:33 hive-exec.jar -> hive-exec-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 38 oct 14 11:33 hive-contrib.jar -> hive-contrib-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 44 oct 14 11:33 hive-hbase-handler.jar -> hive-hbase-handler-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 38 oct 14 11:33 hive-service.jar -> hive-service-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 36 oct 14 11:33 hive-serde.jar -> hive-serde-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 40 oct 14 11:33 hive-metastore.jar -> hive-metastore-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 46 oct 14 11:33 hive-jdbc.jar -> hive-jdbc-1.2.1000.2.5.0.0-1245-standalone.jar lrwxrwxrwx 1 root root 34 oct 14 11:33 hive-hwi.jar -> hive-hwi-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 46 oct 14 11:33 hive-shims-scheduler.jar -> hive-shims-scheduler-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 42 oct 14 11:33 hive-shims.jar -> hive-shims-0.20S-1.2.1000.2.5.0.0-1245.jar lrwxrwxrwx 1 root root 43 oct 14 11:33 hive-shims-common.jar -> hive-shims-common-1.2.1000.2.5.0.0-1245.jar drwxr-xr-x 5 root root 88 oct 14 11:33 php drwxr-xr-x 10 root root 142 oct 14 11:33 py -rw-r--r-- 1 root root 990927 oct 14 11:36 mysql-connector-java.jar 2016-11-06 11:27:30,675 ERROR [HiveServer2-Background-Pool: Thread-50]: operation.Operation (SQLOperation.java:run(256)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 2, vertexId=vertex_1478365570188_0018_2_01, diagnostics=[Task failed, taskId=task_1478365570188_0018_2_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException at org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57) at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82) at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:360) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:363) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:160) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.URIException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 27 more ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159) ... View more