Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while loading a hive data table in pig

Error while loading a hive data table in pig

New Contributor
grunt> A1 = load 'gurjinder.empl' Using org.apache.hive.hcatalog.pig.HCatLoader(); 2016-03-03 21:12:36,157 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] Details at logfile: /usr/lib/hive-0.13.1-bin/lib/pig_1456947801495.log

On Running :

pig -useHCatalog I am getting

ls: cannot access /usr/lib/hive-0.13.1-bin/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /usr/lib/hive-0.13.1-bin/hcatalog//share/hcatalog/hcatalog-core-*.jar: No such file or directory
ls: cannot access /usr/lib/hive-0.13.1-bin/hcatalog//share/hcatalog/hcatalog-*.jar: No such file or directory
ls: cannot access /usr/lib/hive-0.13.1-bin/hcatalog//share/hcatalog/hcatalog-pig-adapter-*.jar: No such file or directory
2016-03-03 21:36:42,549 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.1-SNAPSHOT (rexported) compiled Mar 22 2014, 07:41:29
2016-03-03 21:36:42,565 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/lib/hive-0.13.1-bin/lib/pig_1457021202539.log
2016-03-03 21:36:43,199 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/edureka/.pigbootup not found
2016-03-03 21:36:46,584 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-03-03 21:36:46,585 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-03-03 21:36:46,586 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:8020
2016-03-03 21:36:46,633 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hbase-0.96.2-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2016-03-03 21:36:48,854 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-03-03 21:36:55,320 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-03-03 21:36:55,610 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.df.interval is deprecated. Instead, use fs.df.interval
2016-03-03 21:36:55,611 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.http.address is deprecated. Instead, use mapreduce.tasktracker.http.address
2016-03-03 21:36:55,611 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2016-03-03 21:36:55,611 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.retain.hours is deprecated. Instead, use mapreduce.job.userlog.retain.hours
2016-03-03 21:36:55,615 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2016-03-03 21:36:55,615 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacestart is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacestart
2016-03-03 21:36:55,616 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.read.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.read.timeout
2016-03-03 21:36:55,616 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.sort.spill.percent is deprecated. Instead, use mapreduce.map.sort.spill.percent
2016-03-03 21:36:55,616 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.parallel.copies is deprecated. Instead, use mapreduce.reduce.shuffle.parallelcopies
2016-03-03 21:36:55,617 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
2016-03-03 21:36:55,626 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacekill is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacekill
2016-03-03 21:36:55,626 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile is deprecated. Instead, use mapreduce.task.profile
2016-03-03 21:36:55,627 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2016-03-03 21:36:55,627 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.heartbeats.in.second is deprecated. Instead, use mapreduce.jobtracker.heartbeats.in.second
2016-03-03 21:36:55,627 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2016-03-03 21:36:55,627 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.healthChecker.interval is deprecated. Instead, use mapreduce.tasktracker.healthchecker.interval
2016-03-03 21:36:55,627 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout
2016-03-03 21:36:55,628 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir
2016-03-03 21:36:55,628 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - jobclient.completion.poll.interval is deprecated. Instead, use mapreduce.client.completion.pollinterval
2016-03-03 21:36:55,628 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.active
2016-03-03 21:36:55,628 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2016-03-03 21:36:55,629 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2016-03-03 21:36:55,629 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.shuffle.merge.percent is deprecated. Instead, use mapreduce.reduce.shuffle.merge.percent
2016-03-03 21:36:55,630 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
2016-03-03 21:36:55,630 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.input.buffer.percent
2016-03-03 21:36:55,630 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.task.cache.levels is deprecated. Instead, use mapreduce.jobtracker.taskcache.levels
2016-03-03 21:36:55,630 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
2016-03-03 21:36:55,637 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.instrumentation is deprecated. Instead, use mapreduce.jobtracker.instrumentation
2016-03-03 21:36:55,637 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.limit.kb is deprecated. Instead, use mapreduce.task.userlog.limit.kb
2016-03-03 21:36:55,637 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-03-03 21:36:55,637 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use mapreduce.job.speculative.slownodethreshold
2016-03-03 21:36:55,637 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2016-03-03 21:36:55,638 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.map.max.skip.records is deprecated. Instead, use mapreduce.map.skip.maxrecords
2016-03-03 21:36:55,638 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.block.size is deprecated. Instead, use dfs.blocksize
2016-03-03 21:36:55,644 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2016-03-03 21:36:55,645 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size
2016-03-03 21:36:55,645 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.hours
2016-03-03 21:36:55,645 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.handler.count is deprecated. Instead, use mapreduce.jobtracker.handler.count
2016-03-03 21:36:55,649 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2016-03-03 21:36:55,649 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
2016-03-03 21:36:55,650 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.maps is deprecated. Instead, use mapreduce.task.profile.maps
2016-03-03 21:36:55,650 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2016-03-03 21:36:55,650 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
18 REPLIES 18

Re: Error while loading a hive data table in pig

Mentor

@Gurjinder Singh what version of HDP are you using? Your package for HCatLoader is correct for the latest HDP but I see you're using Hive 0.13 and I believe the HCatLoader in 0.13 still needs org.apache.hcatalog.pig.HCatLoader() and not org.apache.hive.hcatalog.pig.HCatLoader(). Please confirm.

Re: Error while loading a hive data table in pig

Guru

He is using the right HCatalog class.

Re: Error while loading a hive data table in pig

Mentor

@Rich Raposa not in earlier release it's not. in Hive 0.13.1 the correct path was https://hive.apache.org/javadocs/r0.13.1/api/hcatalog/hcatalog-pig-adapter/org/apache/hcatalog/pig/H... no .hive. HCatalog was merged as part of Hive in later release, I think 0.14, that's why we get so many questions about wrong path.

Re: Error while loading a hive data table in pig

Mentor

@Gurjinder Singh please rerun your script with

using org.apache.hcatalog.pig.HCatLoader()

Re: Error while loading a hive data table in pig

New Contributor

still same error is there

grunt> A = load 'gurjinder.empl' using org.apache.hcatalog.pig.HCatLoader(); 2016-03-03 21:52:49,038 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.] Details at logfile: /home/edureka/pig_1457022011095.log

Re: Error while loading a hive data table in pig

Super Guru

Gurjinder Singh - It looks like you are using Apache Hadoop(not HDP). can you please make sure that you have hcat-*.jar files at /usr/lib/hive-0.13.1-bin/hcatalog/share/hcatalog/ location?

Re: Error while loading a hive data table in pig

New Contributor

@Kuldeep Kulkarni

yes I am using apache hadoop

Re: Error while loading a hive data table in pig

New Contributor

@ Kuldeep

yes I have hcat jars as below

[edureka@dhcppc2 hcatalog]$ ls -lrt total 664 -rwxrwxrwx. 1 edureka edureka 80165 Sep 23 2014 hive-hcatalog-pig-adapter-0.13.1.jar -rwxrwxrwx. 1 edureka edureka 467819 Sep 23 2014 hive-hcatalog-core-0.13.1.jar -rwxrwxrwx. 1 edureka edureka 67410 Sep 23 2014 hive-hcatalog-server-extensions-0.13.1.jar -rwxrwxrwx. 1 edureka edureka 51825 Sep 23 2014 hive-hcatalog-streaming-0.13.1.jar drwxrwxrwx. 3 edureka edureka 4096 Sep 23 2014 storage-handlers [edureka@dhcppc2 hcatalog]$

Re: Error while loading a hive data table in pig

Super Guru

@Gurjinder Singh

Please use option suggested by @Artem Ervits