Reply
New Contributor
Posts: 7
Registered: ‎08-27-2015

Not able to see Hive transformations in Cloudera Navigator

Hi, I am trying to see a Hive transformation in Cloudera Navigator, but I can't see any.

I have created a table in Hive, and populated it from another table using an "insert overwrite from select" statement. After some minutes, I am able to see in Cloudera Navigator the new entities (table, fields), but not the operation execution. Am I missing any configuration?

What I am trying to achieve is to see in lineage the two tables linked by the operation.

 

Thanks, Elena

Cloudera Employee
Posts: 13
Registered: ‎01-28-2014

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Elena,

 

Navigator has a number of "extractors" the connect to each compute engine and data store, such as Hive, Impala, HDFS, etc.  These extractors run several times an hour. The net result is that you have to wait about 20-30 minutes before full lineage appears in the UI. Navigator is zero-config for lineage so aside from the brief wait, there's nothing else you need to do for column-level lineage to automatically be collected. If it still doesn't show up, here are a few other tips:

 

- make sure you're running a supported version of CDH - the supported versions are listed in the documentation

- check the Navigator log files from CM if all else fails - you might see some clues there, too 

 

Mark 

New Contributor
Posts: 7
Registered: ‎08-27-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Mark,

 

thank you for your reply. I went for the third option because it doesn't seem to be a problem of time (nothing shows up even after days) and the version we are using is Cloudera Enterprise 5.3.3. 

 

Looking in the Navigator log files I found this error:

 

NoSuchObjectException(message:There is no database named cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362)
at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:516)
at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:527)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at com.sun.proxy.$Proxy0.getDatabase(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)
at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
at com.sun.proxy.$Proxy5.get_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7696)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7680)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 

 

Unfortunaely it doesn't mean much to me. Can you help me to understand if this could be the cause of my problem?

Thank you very much, regards

Elena

Cloudera Employee
Posts: 2
Registered: ‎08-28-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Have you set up Hive Metastore?

I would check if you are getting a message saying "Neglecting query <your query> as metastore is getting bypassed."

 

If that's not the case, then it is possible that there are pasing errors (which will be fixed in next release). 

For this I would check for message saying "Unable to parse query..."

 

If you don't see any of those messages then if you don't mind posting your log file that will be great.

Cloudera Employee
Posts: 8
Registered: ‎08-18-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Elena,

 

As the tables are showing up that means interaction with hive is working. Navigator gets the queries from the MR job that was launched by Hive.

What is the execution engine for HIVE in your setup?

 

Vikas

Cloudera Employee
Posts: 2
Registered: ‎08-28-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Seems like my reply did not show up.

So replying again.

 

1) Are you using Hive Metastore?

To check that I would look for following messages in the log

"Neglecting query ... as metastore is getting bypassed."

 

2) Are there any parsing errors (these should be fixed in next release)

To check that I would look for following message in the log

"Unable to parse query..."

 

If you dont see any of the message, can you please post your log file and I can take a look.

Cloudera Employee
Posts: 8
Registered: ‎08-18-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Elena,

 

Couple of more questions:

 

1) Are you using Hive Metastore?

To check that I would look for following messages in the log

"Neglecting query ... as metastore is getting bypassed."

 

2) Are there any parsing errors (these should be fixed in next release)

To check that I would look for following message in the log

"Unable to parse query..."

 

Vikas

New Contributor
Posts: 7
Registered: ‎08-27-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi sudhanshu,

I answer to both you and Vikas questions.

 

Hive MapReduce service is set to YARN (MR2 Included).

In the log named hadoop-cmf-hive-HIVEMETASTORE-fisv-bdvm2.gft.com.log I didn't find any of the two messages you suggested.

 

Is there a way to attach the file to the post? The complete log is approximately 160MB.

Anyway, the content of the log repeates every 5 minutes the same as follows:

 

2015-08-27 00:03:14,006 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,006 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,006 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-08-27 00:03:14,007 INFO org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called
2015-08-27 00:03:14,008 INFO org.apache.hadoop.hive.metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: ERROR: syntax error at or near "@@"
  Position: 5
2015-08-27 00:03:14,011 INFO DataNucleus.Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
2015-08-27 00:03:14,012 INFO org.apache.hadoop.hive.metastore.ObjectStore: Initialized ObjectStore
2015-08-27 00:03:14,014 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,014 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,016 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: NoSuchObjectException(message:There is no database named cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362)
 at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:516)
 at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:527)
 at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
 at com.sun.proxy.$Proxy0.getDatabase(Unknown Source)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)
 at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)
 at com.sun.proxy.$Proxy5.get_database(Unknown Source)
 at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7696)
 at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database.getResult(ThriftHiveMetastore.java:7680)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
 at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
 at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
 at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

2015-08-27 00:03:14,016 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362, parameters:null)
2015-08-27 00:03:14,016 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362, parameters:null) 
2015-08-27 00:03:14,016 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,016 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,042 INFO org.apache.hadoop.hive.common.FileUtils: Creating directory if it doesn't exist: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,171 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,171 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,175 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 create_table: Table(tableName:CM_TEST_TABLE, dbName:cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362, owner:null, createTime:0, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:s, type:string, comment:test string), FieldSchema(name:f, type:float, comment:test float), FieldSchema(name:a, type:array<map<string,struct<p1:int,p2:int>>>, comment:test complex type)], location:null, inputFormat:null, outputFormat:null, compressed:false, numBuckets:1, serdeInfo:SerDeInfo(name:CM_TEST_TABLE, serializationLib:null, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[FieldSchema(name:p1, type:string, comment:partition-key-1), FieldSchema(name:p2, type:int, comment:partition-key-2)], parameters:null, viewOriginalText:null, viewExpandedText:null, tableType:null)
2015-08-27 00:03:14,175 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 create_table: Table(tableName:CM_TEST_TABLE, dbName:cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362, owner:null, createTime:0, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:s, type:string, comment:test string), FieldSchema(name:f, type:float, comment:test float), FieldSchema(name:a, type:array<map<string,struct<p1:int,p2:int>>>, comment:test complex type)], location:null, inputFormat:null, outputFormat:null, compressed:false, numBuckets:1, serdeInfo:SerDeInfo(name:CM_TEST_TABLE, serializationLib:null, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[FieldSchema(name:p1, type:string, comment:partition-key-1), FieldSchema(name:p2, type:int, comment:partition-key-2)], parameters:null, viewOriginalText:null, viewExpandedText:null, tableType:null) 
2015-08-27 00:03:14,184 INFO org.apache.hadoop.hive.common.FileUtils: Creating directory if it doesn't exist: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362/cm_test_table
2015-08-27 00:03:14,319 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,319 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,343 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 add_partition : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,343 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 add_partition : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,371 INFO org.apache.hadoop.hive.common.FileUtils: Creating directory if it doesn't exist: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362/cm_test_table/p1=p0/p2=420
2015-08-27 00:03:14,415 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 add_partition : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,415 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 add_partition : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,438 INFO org.apache.hadoop.hive.common.FileUtils: Creating directory if it doesn't exist: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362/cm_test_table/p1=p1/p2=421
2015-08-27 00:03:14,581 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,581 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,597 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 drop_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,597 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 drop_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,597 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE
2015-08-27 00:03:14,597 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_table : db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 tbl=CM_TEST_TABLE 
2015-08-27 00:03:14,680 INFO hive.metastore.hivemetastoressimpl: deleting  hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362/cm_test_table
2015-08-27 00:03:14,685 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
2015-08-27 00:03:14,693 INFO hive.metastore.hivemetastoressimpl: Moved to trash: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362/cm_test_table
2015-08-27 00:03:14,796 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,796 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_database: cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,799 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 drop_database: cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,799 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 drop_database: cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,800 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_all_tables: db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,800 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_all_tables: db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 
2015-08-27 00:03:14,802 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: source:/10.238.11.182 get_functions: db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 pat=*
2015-08-27 00:03:14,802 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=source:/10.238.11.182 get_functions: db=cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 pat=* 
2015-08-27 00:03:14,804 INFO org.apache.hadoop.hive.metastore.ObjectStore: Dropping database cloudera_manager_metastore_canary_test_db_hive_hivemetastore_7e0851c027bdcf6130fe57333e855362 along with all tables
2015-08-27 00:03:14,809 INFO hive.metastore.hivemetastoressimpl: deleting  hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,819 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
2015-08-27 00:03:14,830 INFO hive.metastore.hivemetastoressimpl: Moved to trash: hdfs://fisv-bdvm2.gft.com:8020/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_7e0851c027bdcf6130fe57333e855362
2015-08-27 00:03:14,935 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: Shutting down the object store...
2015-08-27 00:03:14,935 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=Shutting down the object store... 
2015-08-27 00:03:14,935 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 133: Metastore shutdown complete.
2015-08-27 00:03:14,935 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: ugi=hue ip=/10.238.11.182 cmd=Metastore shutdown complete.

 

 

Is this the log you need?

Thank you, Elena

 

Cloudera Employee
Posts: 8
Registered: ‎08-18-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Elena,

 

You need to look into Navigator Metastore Server logs (not Hive server logs). The logs are located in /var/log/cloudera-scm-navigator directory of the

host where Navigator Metastore server is running. Please let us know if you find anything there.

 

Vikas

New Contributor
Posts: 7
Registered: ‎08-27-2015

Re: Not able to see Hive transformations in Cloudera Navigator

Hi Vikas,

sorry for the mistake. In the mgmt-cmf-mgmt-NAVIGATORMETASERVER log I found the following log. It is repeated every 10 minutes.

 

2015-09-01 09:22:16,908 INFO com.cloudera.nav.extract.CdhClientExtractionTask: Enqueuing 4 extraction tasks for service 'cluster::oozie'...
2015-09-01 09:22:16,911 INFO com.cloudera.nav.extract.CdhClientExtractionTask: Enqueuing 4 extraction tasks for service 'cluster::hdfs'...
2015-09-01 09:22:16,915 INFO com.cloudera.nav.extract.CdhClientExtractionTask: Enqueuing 4 extraction tasks for service 'cluster::yarn'...
2015-09-01 09:22:16,915 INFO com.cloudera.nav.extract.CdhClientExtractionTask: Enqueuing 4 extraction tasks for service 'cluster::hive'...
2015-09-01 09:22:17,143 INFO com.cloudera.nav.extract.CdhClientExtractionTask: All cdh-client extraction tasks for service 'cluster::oozie' finished.
2015-09-01 09:22:17,323 INFO com.cloudera.nav.extract.CdhClientExtractionTask: All cdh-client extraction tasks for service 'cluster::yarn' finished.
2015-09-01 09:22:18,262 INFO com.cloudera.nav.hive.extractor.HiveExtractor: Skipping Hive extraction due to non-zero wait period.
2015-09-01 09:22:18,348 INFO com.cloudera.nav.extract.CdhClientExtractionTask: All cdh-client extraction tasks for service 'cluster::hive' finished.
2015-09-01 09:22:20,134 INFO com.cloudera.nav.extract.CdhClientExtractionTask: All cdh-client extraction tasks for service 'cluster::hdfs' finished.

 

Thanks,

Elena

Announcements
New solutions