I have a HDP 2.0 cluster where I'm executing a mapreduce program which takes Hive(0.14) table as input. There are a large number of small files for the Hive table and hence large number of mapper containers are being requested. Please let me know if there is a way to combine small files before being input to mapreduce job? ... View more

I'm trying to execute a MapReduce streaming job in a 10 node Hadoop cluster(HDP2.2). There are 5 datanodes in the cluster. When the reduce phase reaches almost 100% completion, I'm getting the below error in client logs: Error: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[x.x.x.x:50010], original=[x.x.x.x:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration The data node on which the jobs were executing contained below logs: INFO datanode.DataNode (BlockReceiver.java:run(1222)) - PacketResponder: BP-203711345-10.254.65.246-1444744156994:blk_1077645089_3914844, type=HAS_DOWNSTREAM_IN_PIPELINE java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203) java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194) 2016-04-10 08:12:14,477 WARN datanode.DataNode (BlockReceiver.java:run(1256)) - IOException in BlockReceiver.run(): java.io.IOException: Connection reset by peer 016-04-10 08:13:22,431 INFO datanode.DataNode (BlockReceiver.java:receiveBlock(816)) - Exception for BP-203711345-x.x.x.x -1444744156994:blk_1077645082_3914836 java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XXX.XX.XX:50010 remote=/XX.XXX.XX.XXX:57649] The NameNode logs contained the below warning: WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(383)) - Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy I had tried setting the below parameters in hdfs-site.xml dfs.datanode.handler.count =10 dfs.client.file-block-storage-locations.num-threads = 10 dfs.datanode.socket.write.timeout=20000 But still the error persists. Kindly suggest a solution. Thanks ... View more

I have a Hadoop cluster(HDP 2.2) set-up in Eucalyptus environment. I have created an external table in Hive(0.14), using the below query: CREATE EXTERNAL TABLE tempbatting (col_value STRING) LOCATION 's3n://hive-bucket/'; I'm using a custom S3 location, so I have set jets3t property in Hive configuration directory as below: set s3service.https-only = true; set s3service.s3-endpoint = s3-customlocation.net; set s3service.s3-endpoint-http-port = 80; set s3service.s3-endpoint-https-port = 443; set s3service.disable-dns-buckets = true; set s3service.enable-storage-classes = false; Though I'm able to execute simple select queries on the table successfully, the aggregate queries are failing. Below are the logs: Error: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to hive-bucket.s3.amazonaws.com:443 timed out at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:416) From the logs, the map-reduce job seems to access Amazon S3. I have tried using the set command for Hive(set fs.s3n.endpoint=s3-customlocation.net), but it didn't seem to work. Is there a way to specify custom end-point? ... View more