Json SerDe not found

[#1]

Hi

As part of this tutorial on the Hortonworks site, I used a maven to compile the Json SerDe as instructed. I placed the Jar file into /usr/hdp/2.4.3.0-227/hive/lib. The tutorial says to place the same file in hive2/lib folder but there is no such folder in my HDP 2.4 box.

Interestingly, if I just do a select on the table that explicitly uses Serde, I get no error. The error only comes when Tez is doing a Join condition.

CREATE EXTERNAL TABLE IF NOT EXISTS tweets_text(
  tweet_id bigint,
  created_unixtime bigint,
  created_time string,
  lang string,
  displayname string,
  time_zone string,
  msg string)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION '/tmp/tweets_staging';

No Errors. Results are returned successfully. The following query or any other join query, gives the error message below.

-- Compute sentiment
create view IF NOT EXISTS l1 as select tweet_id, words from tweets_text lateral view explode(sentences(lower(msg))) dummy as words;
create view IF NOT EXISTS l2 as select tweet_id, word from l1 lateral view explode( words ) dummy as word;
create view IF NOT EXISTS l3 as select
    tweet_id,
    l2.word,
    case d.polarity
      when  'negative' then -1
      when 'positive' then 1
      else 0 end as polarity
 from l2 left outer join dictionary d on l2.word = d.word;

Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:265) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.openx.data.jsonserde.JsonSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:347) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:382) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:227) ... 15 more Caused by: java.lang.ClassNotFoundException: Class

I would be most grateful for a quick resolution to this.

Thanks

Regards,

Adeel

[#2 · Accepted solution]

@Adeel Hashmi

Have you already placed the "json-serde-*.jar" jar into folder "/usr/hdp/current/hive-client/auxlib" (in each hiveserver2/hive cli node) ?

Also you should create an "auxlib" folder as following in the hive installation and put all jars that are needed in there. These are then available to the Server and the executed tez jobs.

.

mkdir /usr/hdp/current/hive/auxlib
cp -f /tmp/json-serde-*.jar  /usr/hdp/current/hive/auxlib

.

[#3]

Hi Jay,

Sincere thanks for solving this problem for me. Have a great day!

Thanks

Adeel