Member since
05-22-2016
29
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1682 | 10-16-2017 08:40 PM | |
6289 | 02-24-2017 01:23 AM | |
2908 | 05-23-2016 11:40 PM |
08-28-2019
05:49 PM
Dear all.
I have a hbase cluster using JDK 1.7 and G1GC and CentOS6.10.
Sometimes a region server shutdown unexpectedly.
There is no log about the shutdown in the hbase region server log file.
But it remains crash log in the /var/run/cloudera-scm-agent/process/3643-REGIONSERVER/hs_err_pid142004.log
Do you have any idea of this symtoms?
Is it okay to change one of region servers JDK version from 1.7 to 1.8?
Below is the head of crash log.
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007ffa5d888c36, pid=142004, tid=140713136191232 # # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x522c36] G1BlockOffsetArray::forward_to_block_containing_addr_slow(HeapWord*, HeapWord*, void const*)+0x1d6 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp #
--------------- T H R E A D ---------------
Current thread (0x00007ffa58058800): GCTaskThread [stack: 0x00007ffa546f7000,0x00007ffa547f8000] [id=142226]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000000000000a8
Registers: ...........
..........
...........
... View more
Labels:
08-28-2019
04:19 AM
Deal all.
CM has below query to draw "Total slow get operations across RegionServers"
SELECT integral(total_slow_get_rate_across_regionservers) WHERE entityName = "hbase" AND category = SERVICE
What are the criteria for slow? 1sec ? or hbase.ipc.warn.response.time?
Thanks in advance
... View more
Labels:
05-29-2019
12:46 AM
dear all. I'm trying to initiate replication. My situation is same as below. https://www.cloudera.com/documentation/enterprise/5-13-x/topics/cdh_bdr_hbase_replication.html#topic_20_11_7 Start replication. Add the destination cluster as a peer and immediately disable it using disable_peer. On the source cluster, take a snapshot of the table and export it. The snapshot command flushes the table from memory for you. On the destination cluster, import and restore the snapshot. Run enable_peer to re-enable the destination cluster. Source cluster has 40TB of table and to export snapshot to target cluster it will take 15 hours. Is the changes (wal or replication queue) for this 15 hours will not be deleted? Thanks.
... View more
- Tags:
- HBase
- replication
- wal
Labels:
09-19-2018
12:32 AM
@bgooley Thank you for your feedback. It was mistake inputting password. Now solved the problem. Sorry and Thank you again.
... View more
09-11-2018
02:28 AM
Hello. I have configured hue to integrate ldap login. And suceeded 'Test LDAP Configuration' from Hue's Action Menu. But When I try to login with ldap user on hue webui login I failed with 'incorrect username or password'. What should I have to check ? And the login page does not display LDAP select box. Just only displayed username and password box. Thanks.
... View more
Labels:
07-03-2018
08:23 PM
Got it! Thank you for the explanation.
... View more
07-03-2018
07:43 PM
Dear Harsh J Thank you very much. Very helpful. One more questions about changing hfile format. After change hfile format version, is it rewrite all of hfile to new version format? Or Just be applied only newly created hfile? Thanks.
... View more
07-03-2018
02:54 AM
1 Kudo
Hello ~ I'm looking for the solution of below symtoms. Anyone who knows the solution ? Thanks . Environment : CDH 5.14 (HBase 1.2) -- create test table
create 'jkkim','f'
-- put an sample data
put 'jkkim','row2','f:name','jonggyun Kim2'
-- put another sample data with TTL
put 'jkkim','row4','f:name','jonggyun Kim',{TTL => 10000}
-- scan after 15 seconds
scan 'jkkim'
-- row4 does not shown result (expected)
ROW COLUMN+CELL
row2 column=f:name, timestamp=1530609778641, value=jonggyun Kim2
1 row(s) in 0.0120 seconds
--put sample data again with TTL
put 'jkkim','row4','f:name','jonggyun Kim',{TTL => 10000}
-- Run manually flush command
flush 'jkkim'
--scan after 15 seconds
scan 'jkkim'
-- row4 is shown result (not expected)
ROW COLUMN+CELL
row2 column=f:name, timestamp=1530609778641, value=jonggyun Kim2
row4 column=f:name, timestamp=1530609975940, value=jonggyun Kim
... View more
- Tags:
- TTL
Labels:
02-08-2018
10:37 PM
Hello ~
I can't solve the below issue, plz help me.
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 10000
Number of Maps = 10
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
18/02/09 15:23:14 INFO client.RMProxy: Connecting to ResourceManager at kerberos001-hbase-db.testinfra-dev.com/10.127.86.45:8032
18/02/09 15:23:14 INFO hdfs.DFSClient: Created token for vincent: HDFS_DELEGATION_TOKEN owner=vincent@TEST.COM, renewer=yarn, realUser=, issueDate=1518157394188, maxDate=1518762194188, sequenceNumber=9, masterKeyId=117 on 10.127.86.45:8020
18/02/09 15:23:14 INFO security.TokenCache: Got dt for hdfs://kerberos001-hbase-db.testinfra-dev.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.127.86.45:8020, Ident: (token for vincent: HDFS_DELEGATION_TOKEN owner=vincent@TEST.COM, renewer=yarn, realUser=, issueDate=1518157394188, maxDate=1518762194188, sequenceNumber=9, masterKeyId=117)
18/02/09 15:23:14 INFO input.FileInputFormat: Total input paths to process : 10
18/02/09 15:23:14 INFO mapreduce.JobSubmitter: number of splits:10
18/02/09 15:23:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1518080546056_0001
18/02/09 15:23:14 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 10.127.86.45:8020, Ident: (token for vincent: HDFS_DELEGATION_TOKEN owner=vincent@TEST.COM, renewer=yarn, realUser=, issueDate=1518157394188, maxDate=1518762194188, sequenceNumber=9, masterKeyId=117)
18/02/09 15:23:15 INFO impl.YarnClientImpl: Submitted application application_1518080546056_0001
18/02/09 15:23:15 INFO mapreduce.Job: The url to track the job: http://kerberos001-hbase-db.testinfra-dev.com:8088/proxy/application_1518080546056_0001/
18/02/09 15:23:15 INFO mapreduce.Job: Running job: job_1518080546056_0001
18/02/09 15:23:27 INFO mapreduce.Job: Job job_1518080546056_0001 running in uber mode : false
18/02/09 15:23:27 INFO mapreduce.Job: map 0% reduce 0%
18/02/09 15:23:27 INFO mapreduce.Job: Job job_1518080546056_0001 failed with state FAILED due to: Application application_1518080546056_0001 failed 2 times due to AM Container for appattempt_1518080546056_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://kerberos001-hbase-db.testinfra-dev.com:8088/proxy/application_1518080546056_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1518080546056_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:373)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Shell output: main : command provided 1
main : run as user is vincent
main : requested yarn user is vincent
Writing to tmp file /dfs/yarn/nm/nmPrivate/application_1518080546056_0001/container_1518080546056_0001_02_000001/container_1518080546056_0001_02_000001.pid.tmp
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
18/02/09 15:23:27 INFO mapreduce.Job: Counters: 0
Job Finished in 13.602 seconds
java.io.FileNotFoundException: File does not exist: hdfs://kerberos001-hbase-db.testinfra-dev.com:8020/user/vincent/QuasiMonteCarlo_1518157392185_124408720/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1844)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
user : vincent
-- os account
cat /etc/passwd |grep vincent vincent:x:11002:11102::/home/vincent:/bin/bash
-- hdfs user directory
hdfs dfs -ls /user |grep vincent drwxr-xr-x - vincent supergroup 0 2018-02-09 15:23 /user/vincent
klist Ticket cache: FILE:/tmp/krb5cc_p29242 Default principal: vincent@TEST.COM
Valid starting Expires Service principal 02/09/18 15:22:57 02/10/18 15:22:57 krbtgt/TEST.COM@TEST.COM renew until 02/16/18 15:22:57
Kerberos 4 ticket cache: /tmp/tkt0 klist: You have no tickets cached
... View more
12-31-2017
04:21 AM
Dear all. I'd like to know it is possible to force PARALLEL RANGE SCAN. Now I have execution plan below. And this plan takes about 10 seconds, So I hope to run this with parallel 80-way range scan. How can I fix this problem? Help me! Thanks. +-------------------------------------------------------------------------------------------------------------+ | PLAN | +-------------------------------------------------------------------------------------------------------------+ | CLIENT 80-CHUNK 4683260417 ROWS 377487413197 BYTES SERIAL 80-WAY RANGE SCAN OVER _LOCAL_IDX_DEVICE_HISTORY | | SERVER FILTER BY FIRST KEY ONLY | | SERVER 1000 ROW LIMIT | | CLIENT MERGE SORT | | CLIENT 1000 ROW LIMIT | +-------------------------------------------------------------------------------------------------------------+
... View more
Labels:
10-16-2017
07:52 PM
Dear all. I'm trying to enable kerberos for cluster but I have got some errors below. Anyone who know what is wrong ? /usr/share/cmf/bin/import_credentials.sh failed with exit code 127 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf3579156749802764271.keytab
+ USER=cloudera-scm/admin@HBASE.ITSC.COM
+ PASSWD='REDACTED'
+ KVNO=1
+ SLEEP=0
+ RHEL_FILE=/etc/redhat-release
+ '[' -f /etc/redhat-release ']'
+ set +e
+ grep Tikanga /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'CentOS release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ '[' 0 -eq 0 ']'
+ grep 'Scientific Linux release 5' /etc/redhat-release
+ '[' 1 -eq 0 ']'
+ set -e
+ '[' -z /etc/krb5.conf ']'
+ echo 'Using custom config path '\''/etc/krb5.conf'\'', contents below:'
+ cat /etc/krb5.conf
+ IFS=' '
+ read -a ENC_ARR
+ for ENC in '"${ENC_ARR[@]}"'
+ ktutil
+ echo 'addent -password -p cloudera-scm/admin@HBASE.ITSC.COM -k 1 -e aes256-cts:normal'
+ '[' 0 -eq 1 ']'
+ echo 'REDACTED'
+ echo 'wkt /var/run/cloudera-scm-server/cmf3579156749802764271.keytab'
ktutil: relocation error: ktutil: symbol krb5_enctype_to_name, version k5crypto_3_MIT not defined in file libk5crypto.so.3 with link time reference
... View more
Labels:
05-08-2017
03:29 AM
Dear all. I'm tring to select some data from phoenix by python 2.6.6 The source code is very simple. import phoenixdb
database_url = 'http://xx.xx.xx.xx:8765/'
conn = phoenixdb.connect(database_url, autocommit=True)
cursor = conn.cursor()
cursor.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, username VARCHAR)")
cursor.execute("UPSERT INTO users VALUES (?, ?)", (1, 'admin'))
cursor.execute("SELECT * FROM users")
print cursor.fetchall() When executes it return errors below python test.py
Traceback (most recent call last):
File "test.py", line 4, in <module>
conn = phoenixdb.connect(database_url, autocommit=True)
File "/usr/lib/python2.6/site-packages/phoenixdb/__init__.py", line 64, in connect
return Connection(client, **kwargs)
File "/usr/lib/python2.6/site-packages/phoenixdb/connection.py", line 47, in __init__
self.open()
File "/usr/lib/python2.6/site-packages/phoenixdb/connection.py", line 64, in open
self._client.open_connection(self._id, info=self._connection_args)
File "/usr/lib/python2.6/site-packages/phoenixdb/avatica.py", line 329, in open_connection
response_data = self._apply(request)
File "/usr/lib/python2.6/site-packages/phoenixdb/avatica.py", line 202, in _apply
message.name = 'org.apache.calcite.avatica.proto.Requests${}'.format(request_name)
ValueError: zero length field name in format
Exception ValueError: 'zero length field name in format' in <bound method Connection.__del__ of <phoenixdb.connection.Connection object at 0x213bf10>> ignored It seems to not good combination of python, phoenixdb, phoenix version. Now I'm testing with this. * CLABS_PHOENIX 4.7.0-1.clabs_phoenix1.3.0.p0.000 * Python 2.6.6 * phoenixdb (0.6) Thanks.
... View more
- Tags:
- python phoenix
Labels:
04-20-2017
05:56 PM
Hello, I'm jonggyun. When I execute below Query#1, it returns null result. But actually, there are 3,122 rows between '2015-04-12 01:00:00' and '2015-04-21 01:10:00'. I have checked the result of Query#2 by changing the date value to '2015-04-21 01:00:00.001' . I'd like to know if this is a bug or there is any way to fix this problem. Query#1 0: jdbc:phoenix:lnxxxxn1503:2181> select count(*) as cnt from SNS_ID_HISTORY where operated between to_date('2015-04-21 01:00:00') and to_date('2015-04-21 01:10:00'); +------+ | CNT | +------+ +------+ No rows selected (0.013 seconds) Query#2 0: jdbc:phoenix:lnxxxxn1503:2181> select count(*) as cnt from SNS_ID_HISTORY where operated between to_date('2015-04-21 01:00:00.001') and to_date('2015-04-21 01:10:00');
+-------+
| CNT | +-------+
| 3122 | +-------+ 1 row selected (0.081 seconds) Query#3 0: jdbc:phoenix:lnxxxxn1503:2181> select operated from SNS_ID_HISTORY where operated = to_date('2015-04-21 01:00:00');
+--------------------------+ | OPERATED | +--------------------------+ | 2015-04-21 01:00:00.000 |
| 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | +--------------------------+ 5 rows selected (0.063 seconds) Thanks in advance.
... View more
Labels:
04-20-2017
01:32 AM
Hello, I'm jonggyun. I had posted this problem before but not solved yet. (you can check this link. https://community.cloudera.com/t5/Cloudera-Labs/Phoenix-date-column-value-does-not-compare-properly/td-p/51372 ) When I execute below Query#1, it returns null result. But actually, there are 3122 rows between '2015-04-12 01:00:00' and ' 2015-04-21 01:10:00'. I have checked the result of Query#2 by changing the date value to '2015-04-21 01:00:00.001' . I'd like to know if this is a bug or there is any way to fix this problem. Query#1 0: jdbc:phoenix:lnactnn1503:2181> select count(*) as cnt from SNS_ID_HISTORY where operated between to_date('2015-04-21 01:00:00') and to_date('2015-04-21 01:10:00'); +------+ | CNT | +------+ +------+ No rows selected (0.013 seconds) Query#2 0: jdbc:phoenix:lnactnn1503:2181> select count(*) as cnt from SNS_ID_HISTORY where operated between to_date('2015-04-21 01:00:00.001') and to_date('2015-04-21 01:10:00'); +-------+ | CNT | +-------+ | 3122 | +-------+ 1 row selected (0.081 seconds) Query#3 0: jdbc:phoenix:lnactnn1503:2181> select operated from SNS_ID_HISTORY where operated = to_date('2015-04-21 01:00:00'); +--------------------------+ | OPERATED | +--------------------------+ | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | | 2015-04-21 01:00:00.000 | +--------------------------+ 5 rows selected (0.063 seconds)
... View more
02-24-2017
01:23 AM
1 Kudo
I tested this again. "Restarting JN which is in the same machine of NN occurs NN fail" was my misunderstanding. The cause of problem is below I thnik. When the JN is restarted, the editlog file known to NN's QJM is no longer available (invalid) You can not record changes to this JN until the edit log is rolled and a new log file is created. After the JN restart, it takes about 3 minutes for the editlog to roll and transition to a usable state. When restarting JN, restart one of them and immediately restart the other. If two JNs can not write, they will stop NN. So, If I restarted JN with interval (about 5 minutes), this can not happen.
... View more
02-22-2017
11:03 PM
Dear all. Is there anyone who knows why below query return null? 0: jdbc:phoenix:lctsthb1502:2181> select count(operated) as cnt from user_history where operated between to_date('2015-02-24') and to_date('2015-02-24 00:10') ; +------+ | CNT | +------+ +------+ Actually the count is 21915. I can show you the count as like below. 0: jdbc:phoenix:lctsthb1502:2181> select count(operated) as cnt from user_history where operated between to_date('2015-02-24 00:00:00.001') and to_date('2015-02-24 00:10') ; +--------+ | CNT | +--------+ | 21915 | +--------+ 1 row selected (0.197 seconds) phoenix version is 4.7 Thanks in advance
... View more
Labels:
02-20-2017
01:30 AM
dear all. My cluster is configured as below 2 NN 3 JN 2 Failovercontroller 20 DN For maintainence reason, I had to restart all component of our hadoop cluster. When I restarted a JN (this JN is in the same machine of NN), NN failovered. below is the NN's log This log says it failed because it failed writing to majority. I'd like to know why can't write to JN. Thanks. 2017-02-17 11:18:04,266 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.192:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:04,267 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.191:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:07,267 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.189:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:30,896 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
2017-02-17 11:18:30,898 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond( s).
2017-02-17 11:19:00,897 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
2017-02-17 11:19:00,898 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond( s).
2017-02-17 11:19:01,314 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 8 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 2 Number of syncs: 5 SyncTimes(ms): 23 6
2017-02-17 11:19:01,354 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal xxx.xxx.xx.188:8485 failed to write txns 679604-679604. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException): Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy18.journal(Unknown Source)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:167)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:385)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:378)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-02-17 11:19:01,776 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal xxx.xxx.xx.190:8485 failed to write txns 679604-679604. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException): Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy18.journal(Unknown Source)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:167)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:385)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:378)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-02-17 11:19:01,779 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [xxx.xxx.xx.188:8485, xxx.xxx.xx.190:8485, xxx.xxx.xx.193:8485], stream=QuorumOutputStream starting at txid 679597))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:
xxx.xxx.xx.193:8485: null [success]
2 exceptions thrown:
xxx.xxx.xx.190:8485: Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
xxx.xxx.xx.188:8485: Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:651)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2752)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2624)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:599)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:112)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:401)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java )
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
2017-02-17 11:19:01,781 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid 679597
2017-02-17 11:19:01,788 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2017-02-17 11:19:01,795 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at abc.db.co/xxx.xxx.xx.190
... View more
Labels:
02-19-2017
06:47 PM
Dear all. I have a cluster CDH 5.10.0. - 1 active NN - 1 standby NN - 3 JN - 2 failovercontroller - 20 DN For maintainance reason, I have to restart all components. When I restarted a JN (actually this JN is in same server with NameNode), NN failovered. Restarting in other JN there was no problem. I'm concerned about why this happen --> "Can't write, no segment open" Below is the namenode logs 2017-02-17 11:18:04,266 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.192:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:04,267 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.191:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:07,267 INFO BlockStateChange: BLOCK* BlockManager: ask xxx.xxx.xx.189:50010 to delete [blk_1073833801_92991]
2017-02-17 11:18:30,896 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
2017-02-17 11:18:30,898 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond( s).
2017-02-17 11:19:00,897 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
2017-02-17 11:19:00,898 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond( s).
2017-02-17 11:19:01,314 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 8 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 2 Number of syncs: 5 SyncTimes(ms): 23 6
2017-02-17 11:19:01,354 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal xxx.xxx.xx.188:8485 failed to write txns 679604-679604. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException): Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy18.journal(Unknown Source)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:167)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:385)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:378)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-02-17 11:19:01,776 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal xxx.xxx.xx.190:8485 failed to write txns 679604-679604. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException): Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy18.journal(Unknown Source)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:167)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:385)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$7.call(IPCLoggerChannel.java:378)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-02-17 11:19:01,779 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [xxx.xxx.xx.188:8485, xxx.xxx.xx.190:8485, xxx.xxx.xx.193:8485], stream=QuorumOutputStream starting at txid 679597))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:
xxx.xxx.xx.193:8485: null [success]
2 exceptions thrown:
xxx.xxx.xx.190:8485: Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
xxx.xxx.xx.188:8485: Can't write, no segment open
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:354)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:651)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2752)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2624)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:599)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:112)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:401)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java )
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
2017-02-17 11:19:01,781 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid 679597
2017-02-17 11:19:01,788 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2017-02-17 11:19:01,795 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at abc.db.co/xxx.xxx.xx.190
... View more
Labels:
02-13-2017
01:17 AM
Dear all. Now I'm using CDH5.9.1 and I hope to upgrade the new version (5.10). How can I upgrade CDH 5.10 from 5.9.1 ? I tried by "cluster upgrade" pull down menu of Cluster but didn't appeared cdh 5.10 in the list of candidate. Thanks
... View more
- Tags:
- Upgrade 5.10
Labels:
01-17-2017
05:21 PM
Dear surajacharya. Thanks , very helpful!!! Have a good day!
... View more
01-17-2017
12:30 AM
Dear all. I want to know about whether if cdh 5.9 applied the patches for HBASE-16132 bug . https://issues.apache.org/jira/browse/HBASE-16132 And Can I check what the list of patches for some CDH versions somewhere ? Thanks.
... View more
- Tags:
- HBASE-16132
- Scan
Labels:
12-19-2016
02:37 AM
Deal all. I want to dump hue's workflow data. I run as like below. [root@HOSTNAMEXX:/opt/cloudera/parcels/CDH/lib/hue/build/env/bin]# hue dumpdata desktop.Document2 --indent 2 --natural > data.json but failed with this error messages. Is there anyone who knows this problem? Error: Password not present
Traceback (most recent call last):
File "./hue", line 12, in <module>
load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/manage_entry.py", line 59, in entry
execute_from_command_line(sys.argv)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 261, in fetch_command
commands = get_commands()
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 107, in get_commands
apps = settings.INSTALLED_APPS
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 49, in _setup
self._wrapped = Settings(settings_module)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 128, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/settings.py", line 328, in <module>
"PASSWORD" : desktop.conf.get_database_password(),
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/conf.py", line 1278, in get_database_password
password = DATABASE.PASSWORD_SCRIPT.get()
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get
return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value
return self._coerce_type(raw_val, prefix)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type
return self.type(raw)
File "/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hue/desktop/core/src/desktop/lib/conf.py", line 714, in coerce_password_from_script
raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/212-hue-HUE_SERVER/altscript.sh sec-5-password' returned non-zero exit status 1
... View more
Labels:
11-28-2016
09:59 PM
Dear all. Today , I'm trying to adopt G1GC for region server GC. But someone told me below problem may occur. And apache hbase patched this at 1.4 Is this problems are still in CDH 5.9 ? https://issues.apache.org/jira/browse/HBASE-16616 https://issues.apache.org/jira/browse/HBASE-16146 https://issues.apache.org/jira/browse/HBASE-17072 Thanks in advance.
... View more
Labels:
05-23-2016
11:40 PM
I solved this problem. I restarted cloudera-scm-agent service at all nodes.
... View more
05-22-2016
08:12 PM
hi, all Now I'm trying to install CLABS_PHONEIX on my CDH 5.5. I read some cdh manual and did as manual And downloaded parcels sucessfully and distributed successfully. But not unpacked at some servers. (status : 2 node successful, 8 node not unpacked) It's been some hours. I did checking cloudera-scm-agent.log but there is no specical log about it. Is there anyone who knows this problem ? thanks.
... View more