Hi all, I have table in HIVE with insertDTM and id. My job is to find out those member whose last insertDTM is 30 days older than the present date. I am using Datediff UFD for that. My query is something like .. select * from finaldata where datediff(current_date,select max(insertdtm) from finaldata group by memberid)>30; But it is giving some error. Its looks like datediff does not support any SQL as param. One more thing , can any one explain why this is not working ? when I am calling datediff with SQL , how HIVE handle this ? My assumption was that first it will execute the SQL in between (in my case which is "select max(insertdtm) from finaldata group by memberid") then call the Datediff with current_date and the output of this SQL. But it looks like it is not working that way. Thanks a lot ... ... View more

Hi All , I have a simple mapper which read some data from a log file and do some join operation with a another file data and send that combined output to reducer for further processing. In mapper I have used DistributedCache as the file is small one. Its working properly. Now I have to write some MRUnit test cases for that mapper. Can any one help me out with some code example how to write MRUnit with DistributedCache support. I am using Hadoop2 and MRUnit version is as follows .... <dependency> <groupId>org.apache.mrunit</groupId> <artifactId>mrunit</artifactId> <version>1.1.0</version> <classifier>hadoop2</classifier> </dependency> In Driver class I have added for DistributedCache (this is just to explain how I added cache in MR)Job job = Job.getInstance(conf);job.setJarByClass(ReportDriver.class); job.setJobName("Report"); job.addCacheFile(new Path("zone.txt").toUri()); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.setMapperClass(ReportMapper.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); job.setReducerClass(ReportReducer.class); job.setNumReduceTasks(3); //job.setCombinerClass(ReportReducer.class); logger.info("map job started ---------------"); System.exit(job.waitForCompletion(true) ? 0 : 1); In Mapper class I am fetching the cases file like this .... @Override protected void setup(Context context) throws IOException, InterruptedException { URI[] localPaths = context.getCacheFiles(); } Please help me out if any one use DistributedCache with MRUnit with some code example... Thanks a lot .... ... View more

Hi All, I have a hadoop cluster setup with one master and two slave. I want to installed hbase in the cluster. But when running hbase I hbase I am getting one error in zookeeper log file <code>> 2016-07-11 22:49:18,199 WARN [QuorumPeer[myid=0]/0.0.0.0:2181] quorum.QuorumCnxManager: Cannot open channel to 2 at election address /10.0.1.105:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762) 2016-07-11 22:49:18,201 WARN [QuorumPeer[myid=0]/0.0.0.0:2181] quorum.QuorumCnxManager: Cannot open channel to 1 at election address /10.0.1.103:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762) 2016-07-11 22:49:18,201 INFO [QuorumPeer[myid=0]/0.0.0.0:2181] quorum.FastLeaderElection: Notification time out: 25600 hduser@hadoopmaster:/home/$ But no error in salve. My HBase-site.xml configuration is as follows ... In Master <code><configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase:rootdir</name> <value>hdfs://hadoopmaster:9000/hbase</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>10.0.1.104,10.0.1.103,10.0.1.105</value> </property> <property> <name>hbase.master</name> <value>hadoopmaster:60000</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>hdfs://hadoopmaster:9000/zookeeper</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.zookeeper.leaderport</name> <value>3888</value> </property> In Slave <code><configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase:rootdir</name> <value>hdfs://hadoopmaster:9000/hbase</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.zookeeper.leaderport</name> <value>3888</value> </property> </configuration> IP Address Details 10.0.1.104 --- hadoopmaster 10.0.1.103 --- hadoopslave2 10.0.1.105 --- hadoopslave1 Please note, I do not have firewall enabled in setup. I am not using separate zookeepr installation rather I am using embed zookeeper installation that comes with HBASE. Any one face this issue ? Please suggest the way to resolve this issue. Any help will be highly appropriated, Thanks in advance.... ... View more

Hi All, Any one can help me in HBASE installation and configuration in cluster environment(I have 10 node cluster with yarn and working properly)? I spent a lot of time but nothing came out. I tried many salutations which got from google , but unfortunately noting works. Present situation, All services are running in master and slave but not able to "list" or create table in Hbase. I guess I am doing something wrong in some where. Please provide any URL or document for step by step installation process and setup process for HBASE in cluster. One more point do I need to install/run zookeeper in master and slave node separately for HBASE.? Any help will be highly appropriated..... Thanks a lot ..... ... View more