Member since
02-10-2016
36
Posts
14
Kudos Received
0
Solutions
12-22-2017
11:15 AM
Thank you
... View more
12-19-2017
01:31 PM
Thanks for the response. I wanted to know if the memory assignment could be done, without providing these while submitting jobs.
... View more
12-12-2017
01:18 PM
I had tried executing the mapreduce example program in HDP2.6 cluster, and I got the below warning in the logs: 2017-12-12 14:51:53,816 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties Please provide any suggestions to resolve this warning message.
... View more
06-29-2017
11:34 AM
The DN process is running if do a check in the machine using ps -ef. But Ambari incorrectly shows the DataNode process as stopped.
... View more
06-27-2017
01:13 PM
In Ambari UI, the data node is in stopped state few seconds after starting it. As mentioned in the earlier reply, with hdfs fsck command the newly added nodes are also listed, though Ambari doesnt recognize the addition.
... View more
06-27-2017
10:49 AM
I'm trying to add 2 new datanodes to an existing HDP2.3 cluster through Ambari. The existing 36 data nodes have configuration 10CPU's, 56GB RAM and 8.5 TB disk size. The data node heap size is set as 1 GB.The 2 new ones to be added have configuration of 6 CPU's, 25GB RAM and 1 TB disk size. The HDFS disk usage is 7%. I'm able to start the NodeManager and AmbariMetrics service in the new nodes, but the datanode service goes down immediately after starting. Below are the logs from hadoop-hdfs-datanode-worker1.log 2017-06-27 12:07:30,047 INFO datanode.DataNode (BPServiceActor.java:blockReport(488)) - Successfully sent block report 0x2235b2b47bf3a, containing 1 storage report(s), of which we sent 1. The reports had 19549 total blocks and used 1 RPC(s). This took 10 msec to generate and 695 msecs for RPC and NN processing. Got back no commands.
2017-06-27 12:07:36,003 ERROR datanode.DataNode (DataXceiver.java:run(278)) - worker1.bigdata.net.net:50010:DataXceiver error processing unknown operation src: /10.255.yy.yy:49656 dst: /10.255.xx.xx:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-06-27 12:08:00,180 INFO datanode.DataNode (DataXceiver.java:writeBlock(655)) - Receiving BP-1320493910-10.255.zz.zz-1479412973603:blk_1100238956_26515824 src: /10.254.yy.yy:45293 dest: /10.255.xx.xx:50010
2017-06-27 12:08:00,326 INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1432)) - src: /10.254.yy.yy:45293, dest: /10.255.xx.xx:50010, bytes: 26872748, op: HDFS_WRITE, cliID: DFSClient_attempt_1498498030455_0521_r_000001_0_-908535141_1, offset: 0, srvID: f148bbe2-8f2a-489b-b03d-c8322aecd43e, blockid: BP-1320493910-10.255.zz.zz-1479412973603:blk_1100238956_26515824, duration: 122445075
2017-06-27 12:08:00,326 INFO datanode.DataNode (BlockReceiver.java:run(1405)) - PacketResponder: BP-1320493910-10.255.12.202-1479412973603:blk_1100238956_26515824, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
Thanks in advance.
... View more
Labels:
06-16-2017
10:58 AM
I have a multi-tenanted HDP2.3 cluster. It has been configured with an S3 end-point in custom hdfs-site.xml. Is it possible to add another S3 end-point for another tenant? If so, what should be the property name?
Thanks in Advance.
... View more
Labels:
05-18-2017
05:48 AM
@Mike Riggs Thanks for the response. I'm looking for the availability of a backup and recovery option for Hive tables without much scripting work to be done. There is an option to mirror HDFS data to S3 from Falcon Web UI, is something similar available for Hive table?
... View more
05-16-2017
09:58 AM
I have an HDP2.3 cluster. I need to setup backup and restore of hive tables in S3. Could you please suggest the best way to do the same? Can Falcon Web UI be used? Can I schedule the replication activity from Falcon?
... View more
Labels:
09-08-2016
11:16 AM
I have a HDP 2.3 cluster, setup using Ambari 2.2.1. I need to modify "hive.in.test" parameter at runtime. When I try to set this from Hive CLI I'm getting the below exception: Query returned non-zero code: 1, cause: Cannot modify hive.in.test at runtime. It is in the list of parameters that can't be modified at runtime
I have added this parameter to "hive.security.authorization.sqlstd.confwhitelist.append" to custom hive-site.xml and hiveserver2-site.xml and have restarted Hive-server2. I'm still facing the same issue. Please suggest a solution.
... View more
Labels:
06-15-2016
07:08 AM
1 Kudo
I have a HDP 2.0 cluster where I'm executing a mapreduce program which takes Hive(0.14) table as input. There are a large number of small files for the Hive table and hence large number of mapper containers are being requested. Please let me know if there is a way to combine small files before being input to mapreduce job?
... View more
Labels:
04-28-2016
05:36 AM
Yes, all the data nodes are healthy. I could do a intra cluster copy for the same file successfully. I have 6 data nodes, both at source and destination.
... View more
04-28-2016
05:07 AM
Yes, I'm able to communicate to nn2.cluster2 on port 8020 and with the datanode on port 50010 from cluster1. In fact, the folder structures are getting created in the destination cluster successfully. I'm also able to copy zero size files.
... View more
04-27-2016
10:55 AM
I'm trying to copy data between two clusters( source :HDP2.2 and destination HDP2.3) using below command: hadoop distcp hdfs://nn1.cluster1:8020/apps/hive/warehouse/tmo/file.txt hdfs://nn2.cluster2:8020/tmp/ I'm getting the below exception while using the above command: 16/04/27 13:43:13 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3454)
16/04/27 13:43:13 WARN hdfs.DFSClient: Failed to connect to /x.x.x.x:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3454)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)
I could do a intra cluster copy for the same file successfully. Kindly suggest a solution.
... View more
Labels:
04-27-2016
10:29 AM
Thank you, I'm planning to upgrade my cluster to use a higher version of Hive. I have one more query, would Hive queries or UDF's cause PermGen error. I had noticed below statement in error logs relating to Hive UDF: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask.PermGen space
... View more
04-25-2016
01:55 PM
Both PermGen size and Max PermGen size is set to 512MB. I have not set PermGen specifically for Hive client, so as per my understanding, the Hadoop settings are used by default.
... View more
04-25-2016
12:56 PM
Thanks for the response. The Hive heap space was set to 40GB, as lesser value was throwing OOM error . I'm aware of the memory leak error in the Hive version used, but I would like to understand the cause of PermGen space error in Hive server or how I could fine tune the configurations to avoid it.
... View more
04-25-2016
12:25 PM
Could you please clarify which parameter you are referring to?
... View more
04-25-2016
12:08 PM
I have a 35 node Hadoop(HDP2.2 ) cluster with 24 data nodes. I
have dedicated Hive server( 0.14) with a memory of 68 GB, and 40GB heap
space. The PermGen space for Name node has been configured as 512 MB. Below are the mapreduce configurations: Default virtual memory for job's map-task: 5120MB
Default virtual memory for job's reduce-task: 5120MB
Map-side sort buffer memory: 1024 MB Occasionally, after executing Hive queries, I get the below
error in Hive server logs: 2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: ql.Driver (SessionState.java:printError(833)) - FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space
2016-04-24 22:12:41,529 ERROR [HiveServer2-Background-Pool: Thread-212461]: operation.Operation (SQLOperation.java:run(199)) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask. PermGen space
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:314)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:146)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
Any suggestions to resolve this issue is greatly appreciated. Thanks
... View more
Labels:
04-14-2016
10:17 AM
Thanks for the suggestions.Two of the data nodes in the cluster had to be replaced, as it didn't have enough disk space. I have also set the below in hdfs configuration and the jobs started executing fine even though I have noticed "Premature end of fail" error in data node logs. dfs.client.block.write.replace-datanode-on-failure.policy=ALWAYS
... View more
04-12-2016
12:48 PM
I'm trying to execute a MapReduce streaming job in a 10 node Hadoop cluster(HDP2.2). There are 5 datanodes in the cluster. When the reduce phase reaches almost 100% completion, I'm getting the below error in client logs: Error: java.io.IOException: Failed to replace a bad
datanode on the existing pipeline due to no more good datanodes being available
to try. (Nodes: current=[x.x.x.x:50010], original=[x.x.x.x:50010]).
The current failed datanode replacement policy is DEFAULT, and a client may
configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'
in its configuration The data node on which the jobs were executing contained below logs: INFO datanode.DataNode (BlockReceiver.java:run(1222)) - PacketResponder:
BP-203711345-10.254.65.246-1444744156994:blk_1077645089_3914844,
type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203)
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
2016-04-10 08:12:14,477 WARN datanode.DataNode
(BlockReceiver.java:run(1256)) - IOException in BlockReceiver.run():
java.io.IOException: Connection reset by peer
016-04-10 08:13:22,431 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(816)) - Exception for
BP-203711345-x.x.x.x -1444744156994:blk_1077645082_3914836
java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/XX.XXX.XX.XX:50010 remote=/XX.XXX.XX.XXX:57649]
The NameNode logs contained the below warning: WARN blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseTarget(383)) - Failed to place enough
replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK],
storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK],
creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more
information, please enable DEBUG log level on
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy I had tried setting the below parameters in hdfs-site.xml dfs.datanode.handler.count =10
dfs.client.file-block-storage-locations.num-threads = 10
dfs.datanode.socket.write.timeout=20000
But still the error persists. Kindly suggest a solution. Thanks
... View more
Labels:
03-22-2016
06:00 AM
I have upgraded to Hadoop 2.7 now. I have done configurations changes for s3a and the queries are executing successfully. Thank you.
... View more
02-26-2016
06:49 AM
1 Kudo
Though have not yet upgraded to Hadoop 2.7, I made the configuration changes for s3a as per the documentation. On executing Hive create query, I got the below exception: FAILED: AmazonClientException Unable to execute HTTP request: Connect to hive-bucket.s3.amazonaws.com:443 timed out
... View more
02-22-2016
10:09 AM
1 Kudo
@Artem Ervits Copied jets3t.properties to all data nodes. Currently I'm getting below exception: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.ServiceException: S3 Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/hive-bucket</Resource><RequestId></RequestId></Error>
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:470)
... View more
02-17-2016
04:34 PM
1 Kudo
I'm using Hadoop 2.6.
... View more
02-17-2016
01:08 PM
1 Kudo
Thanks for the response. Yes, I'm able to access S3 through simple Hive queries.From the logs, I could see that the map-reduce job is trying to connect to "hive-bucket.s3.amazonaws.com:443", which doesn't exist. I need to connect to a custom S3 endpoint, which is "s3-customlocation.net". I have gone through the hdfs-site configuration,but I couldnt find any parameter to set custom endpoint.
... View more
02-17-2016
12:41 PM
1 Kudo
I'm using a custom S3 for Eucalyptus, not the AWS one. I have been trying to resolve this since past few weeks.
... View more
02-17-2016
12:35 PM
1 Kudo
I have a Hadoop cluster(HDP 2.2) set-up in Eucalyptus environment. I have created an external table in Hive(0.14), using the below query: CREATE EXTERNAL TABLE tempbatting (col_value STRING) LOCATION 's3n://hive-bucket/';
I'm using a custom S3 location, so I have set jets3t property in Hive configuration directory as below: set s3service.https-only = true;
set s3service.s3-endpoint = s3-customlocation.net;
set s3service.s3-endpoint-http-port = 80;
set s3service.s3-endpoint-https-port = 443;
set s3service.disable-dns-buckets = true;
set s3service.enable-storage-classes = false;
Though I'm able to execute simple select queries on the table successfully, the aggregate queries are failing. Below are the logs: Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to hive-bucket.s3.amazonaws.com:443 timed out
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:416)
From the logs, the map-reduce job seems to access Amazon S3. I have tried using the set command for Hive(set fs.s3n.endpoint=s3-customlocation.net), but it didn't seem to work. Is there a way to specify custom end-point?
... View more
Labels: