@Praveen PentaReddy agent.sources = Twitter agent.channels = MemChannel agent.sinks = HDFS agent.sources.Twitter.type = com.orienit.kalyan.flume.source.KalyanTwitterSource agent.sources.Twitter.channels = MemChannel agent.sources.Twitter.consumerKey = xxxx agent.sources.Twitter.consumerSecret = xxx agent.sources.Twitter.accessToken = xxxx agent.sources.Twitter.accessTokenSecret = xxxx agent.sources.Twitter.keywords = hadoop,spark,kafka,flume,spark steaming,NIFI,Bigdata,hortonworks,oozie,sqoop,hive,mapreduce,pig,scala agent.sinks.HDFS.type = hdfs agent.sinks.HDFS.channel = MemChannel agent.sinks.HDFS.hdfs.path =/flume/tweets/%y/%m/%d/%H/%M agent.sinks.HDFS.hdfs.fileType = DataStream agent.sinks.HDFS.hdfs.writeFormat = Text agent.sinks.HDFS.hdfs.batchSize = 100 agent.sinks.HDFS.hdfs.rollSize = 0 agent.sinks.HDFS.hdfs.rollCount = 100 agent.sinks.HDFS.hdfs.useLocalTimeStamp = true agent.channels.MemChannel.type = memory agent.channels.MemChannel.capacity = 1000 agent.channels.MemChannel.transactionCapacity = 100 You can try this configuration Praveen ... View more

@ INDRANIL ROY Hi INDRANIL ROY, Are you able to get the continuously streaming data (flat file) into hadoop. What are the ecosystems you have used to get the real time data into hadoop. Please provide the ecosystems details or the steps you followed to get the flat files in to hadoop. ... View more

Hi Neeraj Sabharwal, @Neeraj Sabharwal @Rushikesh Deshmukh This are the steps i followed for incremental import in sqoop for hbase table. Step 1: Importing a Table To HBase sqoop import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --hbase-table employee --hbase-create-table --column-family cf --hbase-row-key id -m 1 Step 2: SQOOP HBASE INCREMENTAL IMPORT sqoop import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --incremental append --check-column id --last-value 71 -m 1 Step 3: SQOOP JOB CREATION FOR HBASE INCREMENT sqoop job --create incjobsnew -- import --connect "jdbc:sqlserver://x.x.x.x:1433;database=test" --username sa -P --table employee --incremental append --check-column id --last-value 71 -m 1. When i execute sqoop job sqoop job --exec incjobsnew. Sqoop command runs successfully and it show the exact number of records retrieved successfully. When i check in hbase for the records. It doesn't show the retrieved results. Could you tell where is the mistake done. I need to automate this sqoop job in Oozie to run a particular time interval daily. ... View more