Member since
07-30-2020
6
Posts
0
Kudos Received
0
Solutions
08-10-2021
11:52 PM
Lock on /hadoopfs/fs1/hdfs/namenode/in_use.lock acquired by nodename 8360@xxxx-080060091.xxxx.com 2021-08-11 06:30:25,079 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(716)) - Encountered exception loading fsimage java.io.IOException: NameNode is not formatted. When i tried to format the namenode by using hadoop namenode -format getting the below error as, 21/08/11 06:45:12 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 2 exceptions thrown: 10.80.60.88:8485: Directory /hadoopfs/fs1/hdfs/journalnode/dev-30 is in an inconsistent state: Can't format the storage directory because the current directory is not empty. In hdfs-site.xml: <property> <name>dfs.data.dir</name> <value>/hadoop/hdfs/data</value> <final>true</final> </property> <property> <name>dfs.name.dir</name> <value>/hadoopfs/fs1/hdfs/namenode</value> <final>true</final> </property> Please help me on above issue.
... View more
Labels:
- Labels:
-
Apache Hadoop
04-05-2021
03:04 AM
Sure, we can track in another thread, Please find the attached hiveserver2 logs and Query ID, queryId=hive_20210405095656_39e4aac2-c7ba-4e8a-ac98-7bb5733ae522
... View more
03-19-2021
03:07 AM
Thanks Asish, now facing some other issue, I have been reindex the corrupted indexes successfully(hl_txnid_index and tc_txnid_index), but now unable to query the any table's in Hive, In Postgresql logs showing 2021-03-19 09:00:15 UTC::@:[9941]:LOG: skipping vacuum of "hive_locks" --- lock not available 2021-03-19 09:00:31 UTC::@:[10049]:LOG: skipping vacuum of "hive_locks" --- lock not available 2021-03-19 09:00:46 UTC::@:[10178]:LOG: skipping vacuum of "hive_locks" --- lock not available SELECT * FROM pg_locks WHERE relation = 'HIVE_LOCKS'::regclass; please check attached file for locks, And also in Hive logs showing Unable to rename temp file 2021-03-19T09:59:27,909 ERROR [json-metric-reporter]: metrics2.JsonFileMetricsReporter (:()) - Unable to rename temp file /tmp/hmetrics8066666106938386459json to /tmp/report.json 2021-03-19T09:59:27,909 ERROR [json-metric-reporter]: metrics2.JsonFileMetricsReporter (:()) - Exception during rename java.nio.file.FileSystemException: /tmp/hmetrics8066666106938386459json -> /tmp/report.json: Operation not permitted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.8.0_232] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_232] at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) ~[?:1.8.0_232] at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) ~[?:1.8.0_232] at java.nio.file.Files.move(Files.java:1395) ~[?:1.8.0_232] at org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter.run(JsonFileMetricsReporter.java:175) ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_232] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_232] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_232] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
... View more
03-18-2021
12:32 AM
Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore (state=42000,code=10) Unable to create Hive Table, while creating a table getting error in acquiring locks: Error communicating with the metastore, ERROR : FAILED: Error in acquiring locks: Error communicating with the metastore org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the metastore at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:178) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:447) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:463) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:278) at org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.acquireLocks(HiveTxnManagerImpl.java:76) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:95) at org.apache.hadoop.hive.ql.Driver.acquireLocks(Driver.java:1651) at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:1838) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2008) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1752) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:324) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:342) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.TApplicationException: Internal error processing lock at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:5510) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:5497) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2745) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) at com.sun.proxy.$Proxy58.lock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2956) at com.sun.proxy.$Proxy58.lock(Unknown Source) at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:103) ... 25 more Error: Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore (state=42000,code=10) tried few solutions, it didn't work, set hive.support.concurrency= true to false set hive.lock.numretries=100; set hive.lock.sleep.between.retries=50; hive.server2.metrics.enabled = true to false hive.metastore.metrics.enabled = true to false hive.exec.compress.intermediate = false to true and also show locks; show locks extended; it's not showing any locks in hive, Hope you understand the issue,
... View more
Labels:
- Labels:
-
Apache Hive
10-30-2020
12:46 AM
Hi All, While trying to take Atlas metadata Backup in HDP 3.1.0.0.0-78 using CLI getting below error, Followed this official link: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/installing-atlas/content/grate_atlas_metadata_when_upgrading_to_hdp-3_0.html atlas@xxxx-0x00x00xx:~$ python /usr/hdp/3.1.0.0-78/atlas/tools/migration-exporter/atlas_migration_export.py -d /atlas_metadata Using Java version: 1.8.0_232 and Python 2.7.5 Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: com/tinkerpop/blueprints/Graph at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: com.tinkerpop.blueprints.Graph at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 7 more Please help me on this issue, Thanks in advance
... View more
Labels:
- Labels:
-
Apache Atlas