Member since
01-25-2017
396
Posts
28
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
830 | 10-19-2023 04:36 PM | |
4355 | 12-08-2018 06:56 PM | |
5450 | 10-05-2018 06:28 AM | |
19822 | 04-19-2018 02:27 AM | |
19844 | 04-18-2018 09:40 AM |
02-18-2017
10:42 AM
My 3rd ZK was on the same VM but after i got to this issue i moved the ZK to another OpenStack servers and moved the spark history server one of the NNs to to reduce the load from the VM and increased the Vcores for the Vm to 6 cores but still have the same issue.
... View more
02-18-2017
10:25 AM
Is it familair to add JN on DataNode/NodeManager server? In my cluster, the 2NNs are physical, the CM and the application server that hosts mysql and oozie are VMs servers, all other DataNodes are physical ones.
... View more
02-18-2017
10:17 AM
Indeed it's happening for few seconds and then the Vm get Sync, it happened from time to time so sometimes i suspect that one job or hive query that writes alot of blocks and files that may cause the issue. Do you think i should examine this again? should i check the content of the file itself? do you think if migrate the JN role from the vm to a stronger node with 12 vcores can solve the issue?
... View more
02-18-2017
01:07 AM
Hi, I have 3 JNs, 2 on physical servers and the 3rd on virtual server with 6 Vcores. Recently from time to time the vm server get out of sync for few seconds, I checked the vm resources and parmeters and nothing looks out of the rodinary, what is see in Cloudera manager metrics that the journal write bytes sometime are higher than different times here what i see: The active NameNode was out of sync with this JournalNode. =============== org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException: Can't write txid 1659311573 expecting nextTxId=1659311555
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkSync(Journal.java:485)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:371)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:149)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:158)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25421)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera Manager
-
HDFS
-
Security
02-17-2017
11:40 PM
Any other ideas? The more intersting in the issue that it's happens only for the output of specific jobs and notf or all the HDFS. Is there any way to set that the new written files to specific dir to be with specific replication factor?
... View more
02-14-2017
02:18 AM
Looking at one of the running jobs conf and see the following with replication factor 3: mapreduce.client.submit.file.replication s3.replication kfs.replication dfs.namenode.replication.interval ftp.replication s3native.replication
... View more
02-14-2017
01:18 AM
Changed at all the cluster nodes and restarted all services at the cluster after. It didn't solve the issue.
... View more
02-13-2017
11:08 AM
No, i'm not using CM
... View more
02-13-2017
10:58 AM
i have the dir /etc/oozie/conf but there is no hdfs-site.xml. Yes this cluster is CDH4.
... View more