Reply
New Contributor
Posts: 6
Registered: ‎01-08-2014

hive cannot talk to hdfs after switching to HA

We've run into a rather odd problem today, curious if anyone else has seen something like this. 

 

Night before last, we upgraded to CDH 4.5, in preperation for switching to HA mode. Yesterday we had no issues. 

Last night, we upgraded to HA using the Cloudera Manager wizard. 

 

Today, we're seeing a couple of very wierd issues with Hive, and aren't sure where the problem might be. 

 

In order of least problematic to most: 

 

1. Hive no longer appears to recognize the hive.aux.jars.path configuration key. We have some custom serdes we store outside of the parcel lib tree, and after switching to HA, the hive clients don't recognize the JARs in this configured path. If we set the environment variable HIVE_AUX_JARS_PATH the Serde jars are read in, and if we use "add jar" from within the hive client it works, but the hive-site.xml directive seems to be ignored. 

2. More problematic, when trying to create a new EXTERNAL table in Hive, hive cannot read the Avro schema files from either hdfs or a local file. For example, if I run the following: 

CREATE EXTERNAL TABLE avrocme_kdh                              
     ROW FORMAT SERDE                                               
       'org.apache.hadoop.hive.serde2.avro.AvroSerDe'               
     STORED AS INPUTFORMAT                                          
       'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' 
     OUTPUTFORMAT                                                   
       'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
     LOCATION                                                       
     '/cmefix/20140107'
     TBLPROPERTIES (
  'avro.schema.url'='hdfs://nameservice1/schemas/cmefix/cmefixmsg.avsc' );

 That create table directive returns without error, but trying to query the table gives me the following fault: 

    > select * from avrocme_kdh;
OK
Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: Found cmefixmsg.avro.CmeFixMsg, expecting org.apache.hadoop.hive.CannotDetermineSchemaSentinel

 Both the external table location (and files) and the schema exist in the specified locations on hdfs, and I can access them via hdfs, but HIVE fails. When I look at my log, I'm getting the following java error: 

2014-01-08 13:36:06,779 WARN  avro.AvroSerdeUtils (AvroSerdeUtils.java:determineSchemaOrReturnErrorSchema(89)) - Encountered AvroSerdeException determining schema. Returning signal schema to indicate problem
org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read schema from HDFS: hdfs://nameservice1/schemas/cmefix/cmefixmsg.avsc
        at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:72)
        at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
        at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:214)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:556)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3719)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:254)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.io.FileNotFoundException: File does not exist: /schemas/cmefix/cmefixmsg.avsc
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:39)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1439)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1391)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1367)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:413)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44938)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
        at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:972)
        at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:960)
        at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:238)
        at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
        at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:746)
        at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromHDFS(AvroSerdeUtils.java:105)
        at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:70)
        ... 25 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /schemas/cmefix/cmefixmsg.avsc
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:39)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1439)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1391)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1367)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1340)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:413)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44938)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

        at org.apache.hadoop.ipc.Client.call(Client.java:1238)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)

 Now, if this were simply a unix filesystem, that second layer of java exception (file oes not exist) might indicate a permissions problem, but I've checked and the entire tree down to the file (and including the file) is world-readable and traversable. 

 

Again, the only configuration change that has been made was switching from a single zookeeper/namenode setup to zookeeper quorum and HA HDFS. All services that do not rely on hive (hbase, mapred, etc) are functioning properly, but HIVE appears to now have issues communicating with hdfs in some cases. What's got me quite stumped is that we can read and query from EXISTING tables and databases, but are getting this error when trying to create NEW tables. 

 

We're running the CDH 4.5.0 parcel on cloudera manager (standard) 4.6.1. 

Any thoughts or ideas would be welcome. 

 

Cheers,

Eric

Cloudera Employee
Posts: 508
Registered: ‎07-30-2013

Re: hive cannot talk to hdfs after switching to HA

Hi Eric,

 

After running the HA wizard, there are a couple manual steps that may need to be done, one of which is to back up the Hive metastore and run a command to update the metastore to use the nameservice names instead of host names. This is mentioned in the HA wizard and in the docs, here:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Cloudera-Manager-Enterpr...

 

Did you run that step?

 

Thanks,

Darren

New Contributor
Posts: 6
Registered: ‎01-08-2014

Re: hive cannot talk to hdfs after switching to HA

[ Edited ]

Yes, we did take that action. 

 

One of the things I was looking at today is that when I run metatool -listFSRoot, I get no result: 

./metatool -listFSRoot
Initializing HiveMetaTool..
14/01/08 15:27:23 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/01/08 15:27:23 INFO metastore.ObjectStore: ObjectStore, initialize called
14/01/08 15:27:24 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
14/01/08 15:27:25 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/01/08 15:27:25 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
14/01/08 15:27:25 INFO metastore.ObjectStore: Initialized ObjectStore
Listing FS Roots..

Which is interesting, as I would expect to see "hdfs://nameservice1" as a result.  

 

New Contributor
Posts: 6
Registered: ‎01-08-2014

Re: hive cannot talk to hdfs after switching to HA

One other update/clarification: 

 

After talking through it with our users, I was given some incorrect information earlier. All Avro SerDE tables are non-functional at this time. The tables that are functional are actually using an JSON SerDE we added. So, it appears that we are both unable to read from existing tables using the avro SerDE and to create new tables using Avro Schema. 

Posts: 416
Topics: 51
Kudos: 83
Solutions: 49
Registered: ‎06-26-2013

Re: hive cannot talk to hdfs after switching to HA

I have moved this thread to the Hive board in hopes that some additional Hive experts can pipe in.

 

Also, if I recall a past issue correctly, you may need to add your aux jars to the "safety valve section" of that service for them to be picked up.

New Contributor
Posts: 6
Registered: ‎01-08-2014

Re: hive cannot talk to hdfs after switching to HA

The safety valve section is where we set the aux jars value. 

 

I have one correction of a correction: Our users were mistaken, we can query existing tables using the avro serde. A couple of users had failed to update their avro.schema.url definitions to use the nameservice name rather than the old single nameserver host:port pair. 

 

Cloudera Employee
Posts: 508
Registered: ‎07-30-2013

Re: hive cannot talk to hdfs after switching to HA

Running ./metatool -listFSRoot will use the hive-site.xml in your /etc/hive/conf, which does not normally list the database login information (for security reasons). You need to set the conf dir to the directory in /var/run/cloudera-scm-agent/process/<something METASTORE something>/, which will actually have the db login info.

As for the serde issues, I'm not sure why those aren't getting picked up. You can try setting the hive client safety valve with the following:
export AUX_CLASSPATH=${AUX_CLASSPATH}:/path/to/jar

Then re-deploy hive client configuration, and inspect /etc/hive/conf/hive-env.sh to make sure it looks correct.

Note that in CM5 beta 2 we'll make it easy to specify a directory for custom Hive jars.
New Contributor
Posts: 6
Registered: ‎01-08-2014

Re: hive cannot talk to hdfs after switching to HA

Ah, that's interesting. 

 

Ok, after updating the config, metatool is reporting back the FS Roots. They are all correctly referencing the new nameservice: 

Initializing HiveMetaTool..
14/01/09 08:00:38 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/01/09 08:00:38 INFO metastore.ObjectStore: ObjectStore, initialize called
14/01/09 08:00:38 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
14/01/09 08:00:39 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/01/09 08:00:39 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
14/01/09 08:00:39 INFO metastore.ObjectStore: Initialized ObjectStore
Listing FS Roots..
hdfs://nameservice1/user/hive/warehouse/cmefix.db
hdfs://nameservice1/user/hive/warehouse/hadooppresentation.db
hdfs://nameservice1/flume/Viper
hdfs://nameservice1/user/hive/warehouse/cme.db
hdfs://nameservice1/user/hive/warehouse/itch.db
hdfs://nameservice1/user/hive/warehouse/erictest.db
hdfs://nameservice1/user/hive/warehouse/eventlogs.db
hdfs://nameservice1/user/hive/warehouse/abacus_export.db
hdfs://nameservice1/user/hive/warehouse/viper_20131216.db
hdfs://nameservice1/user/hive/warehouse/captures.db
hdfs://nameservice1/user/hive/warehouse/sethtest.db
hdfs://nameservice1/user/hive/warehouse/keith.db
hdfs://nameservice1/user/hive/warehouse/feed.db
hdfs://nameservice1/user/hive/warehouse
hdfs://nameservice1/user/hive/warehouse/jmost.db
hdfs://nameservice1/user/hive/warehouse/benchmark.db
hdfs://nameservice1/user/hive/warehouse/viperresults.db
hdfs://nameservice1/user/hive/warehouse/regsho.db
hdfs://nameservice1/user/hive/warehouse/valuations.db

 

We've already implemented the env var for the aux libs as a workaround, so we're "good" there. I was mainly curious as to why the configuration parameter would stop functioning after the HA transition. 

 

New Contributor
Posts: 6
Registered: ‎01-08-2014

Re: hive cannot talk to hdfs after switching to HA

Just wanted to post an update -- we've managed to solve our problem. 


I don't know if this is a generally-acceptable solution, but it worked for us. :) 

 

We were planning to migrate our metastore from postgres to mysql in the near future, anyway, so our resolution path was: 

 

1. Write a script to pull all the HIVE DDL from the existing metastore (as opposed to an actual dump of the metastore DB) and parse it into loadable SQL. This was pretty kludgy on our end, but worked. 

2. Create a new, empty metastore on a new mysql instance. 

3. Update the HIVE configuration to use the new, empty metastore. 

4. Re-create all of the metadata using our script. 

 

Presto, problem solved. 

 

Cheers,

Eric

Highlighted
Cloudera Employee
Posts: 30
Registered: ‎12-09-2014

Re: hive cannot talk to hdfs after switching to HA

Sorry as this is coming a bit late, but the problem is fixed in HIVE-10324 and will be in CDH5.4.1

 

The issue is that Hive Metatool doesn't update the avroSchemaURL correctly if they were specified using 'create table... tblProperties', but it does update if it they were specified in 'create table ... serdeProperties'.

Announcements