I have a HDP 2.0 cluster where I'm executing a mapreduce program which takes Hive(0.14) table as input. There are a large number of small files for the Hive table and hence large number of mapper containers are being requested. Please let me know if there is a way to combine small files before being input to mapreduce job? ... View more

I'm trying to execute a MapReduce streaming job in a 10 node Hadoop cluster(HDP2.2). There are 5 datanodes in the cluster. When the reduce phase reaches almost 100% completion, I'm getting the below error in client logs: Error: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[x.x.x.x:50010], original=[x.x.x.x:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration The data node on which the jobs were executing contained below logs: INFO datanode.DataNode (BlockReceiver.java:run(1222)) - PacketResponder: BP-203711345-10.254.65.246-1444744156994:blk_1077645089_3914844, type=HAS_DOWNSTREAM_IN_PIPELINE java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203) java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194) 2016-04-10 08:12:14,477 WARN datanode.DataNode (BlockReceiver.java:run(1256)) - IOException in BlockReceiver.run(): java.io.IOException: Connection reset by peer 016-04-10 08:13:22,431 INFO datanode.DataNode (BlockReceiver.java:receiveBlock(816)) - Exception for BP-203711345-x.x.x.x -1444744156994:blk_1077645082_3914836 java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/XX.XXX.XX.XX:50010 remote=/XX.XXX.XX.XXX:57649] The NameNode logs contained the below warning: WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(383)) - Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy I had tried setting the below parameters in hdfs-site.xml dfs.datanode.handler.count =10 dfs.client.file-block-storage-locations.num-threads = 10 dfs.datanode.socket.write.timeout=20000 But still the error persists. Kindly suggest a solution. Thanks ... View more

I have a Hadoop cluster(HDP 2.2) set-up in Eucalyptus environment. I have created an external table in Hive(0.14), using the below query: CREATE EXTERNAL TABLE tempbatting (col_value STRING) LOCATION 's3n://hive-bucket/'; I'm using a custom S3 location, so I have set jets3t property in Hive configuration directory as below: set s3service.https-only = true; set s3service.s3-endpoint = s3-customlocation.net; set s3service.s3-endpoint-http-port = 80; set s3service.s3-endpoint-https-port = 443; set s3service.disable-dns-buckets = true; set s3service.enable-storage-classes = false; Though I'm able to execute simple select queries on the table successfully, the aggregate queries are failing. Below are the logs: Error: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to hive-bucket.s3.amazonaws.com:443 timed out at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:416) From the logs, the map-reduce job seems to access Amazon S3. I have tried using the set command for Hive(set fs.s3n.endpoint=s3-customlocation.net), but it didn't seem to work. Is there a way to specify custom end-point? ... View more

I'm trying to set up a single-node Hadoop cluster (HDP2.3 version) using Ambari 2.2 in Eucalyptus environment. I have used Private IP address as recommended while registering the VM's (RHEL 6.5) with Ambari. When I tried to bring up Hadoop services (MR2, Hive, Yarn) , I'm getting the below error: <LI id=L_11001_11>Error Code 11001: Host not found <LI id=L_11001_12>Background: This error indicates that the gateway could not find the IP address of the website you are trying to access. This is usually due to a DNS-related error. <LI id=L_11001_13>Date: 2/9/2016 5:57:11 AM [GMT] <LI id=L_11001_14>Server: FIESPRX004.xxx.net <LI id=L_11001_15>Source: DNS error </UL> ... View more