No. Consumed files are either deleted or renamed to "originalname.COMPLETED". ... View more

Yes, you can connect Flume agents. "To setup a multi-tier flow, you need to have an avro/thrift sink of first hop pointing to avro/thrift source of the next hop." (Configuring a multi agent flow) Agent1 Agent2 [Any source -> channel -> Avro Sink] --> [Avro Source -> channel -> HDFS Sink] ... View more

Please attach you Flume configuration (don't share your Twitter api key) ... View more

There is no retry option in distributed shell client. Open ticket in Apache Jira: YARN-815 - Add container failure handling to distributed-shell ... View more

@Wael Horchani That is strange. Maybe I was wrong and the table is there. Is security enabled? Are you using HDP, HDP Sandbox, CDH or vanilla Hadoop? Please edit the question to add details about your setup. ... View more

Link of the tutorial for the record: http://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/#section_4 ... View more

You get this error if you drop the backup table in hbase namespace. Check if you have the namespace and table: hbase(main):001:0> list_namespace NAMESPACE default hbase 3 row(s) in 0.0220 seconds hbase(main):002:0> list_namespace_tables 'hbase' TABLE acl backup meta namespace 4 row(s) in 0.0280 seconds If you have a backup of any table then you can run restore of that table and hbase will recreate the backup table: [hbase@sandbox ~]$ hadoop dfs -ls /user/hbase/backup Found 2 items drwxr-xr-x - hbase hdfs 0 2017-02-11 17:43 /user/hbase/backup/backup_1486835033442 drwxr-xr-x - hbase hdfs 0 2017-02-11 18:09 /user/hbase/backup/backup_1486836579046 [hbase@sandbox ~]$ hbase restore hdfs://sandbox.hortonworks.com:8020/user/hbase/backup/ backup_1486836579046 iemployee -overwrite [...]2017-02-11 18:44:50,038 INFO [main] impl.RestoreClientImpl: Restore for [iemployee] are successful! Or you can explicitly issue the create command. (describe 'hbase:backup' gives this definition but have to change TTL from 'FOREVER' to '2147483647'): create 'hbase:backup', {NAME => 'meta', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'session', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} ... View more

Do you have any exceptions when you run the above client? Try running the Flume agent with the extra option -Dflume.root.logger=DEBUG,console ... View more

Do you want to write a program which continuously polls a web server and writes that data to HDFS? If this is the case then you could add dependency of groupId: org.apache.hadoop, artifactId: hadoop-client and call append on HDFS FileSystem api directly (without the use of flume). Different approach would be to start an embedded flume agent inside your application. That way you do not have to setup the flume source but you could directly put events to the flume channel. ... View more

Please give some background like which tutorial, user guide or github repo you are following. Also attaching your flume conf file might help answering your question. ... View more

@Magesh Kumar, please edit the question and format the stack trace as code with line feeds and truncate it to contain the exception only once. Could also mark this question as resolved? ... View more

Change handler to blob handler if you want to send the whole response to Flume as is. a1.sources.r1.type = http a1.sources.r1.handler = org.apache.flume.sink.solr.morphline.BlobHandler a1.sources.r1.port = 44444 Python code import requests flume_url = 'http://127.0.0.1:44444' api_key = 'put your api key here' source_url = 'http://data.fmi.fi/fmi-apikey/{0}/wfs?request=describeStoredQueries&storedquery_id=fmi::observations::weather::multipointcoverage&place=helsinki'.format(api_key) response = requests.get(source_url) requests.post(flume_url, data=response.text) ... View more

Exception 1: The path might be different from "D:\gprs\" when you access your ftp server, for example: "/gprs" depending on what is the root directory. Exception 2: File roll sink (agent.sinks.k1.type = file_roll) will not save data to hdfs but to the local filesystem where the agent runs. ... View more

Agent configuration is loaded by java.util.Properties where the backslash is the escape caracter. In your configuration the following two lines: agent.sources.ftp1.folder =D:\gprs\ agent.sources.ftp1.file.name = gprs_20161201 Loaded as one property: Name = "agent.sources.ftp1.folder" Value = "D:gprsagent.sources.ftp1.file.name = gprs_20161201" This happens because at the line end the backslash character escapes the newline causing that everything until the next newline character will belong to the first property value. To fix this you have to escape the backslashes like this: agent.sources.ftp1.folder = D:\\gprs\\ agent.sources.ftp1.file.name = gprs_20161201 ... View more

Executable files with setuid bit set are listed with red background so it is normal. Reason of permission denied might be that yarn user is not part of hadoop group. # verify yarn's groups id yarn # add hadoop to yarn's groups usermod -g hadoop yarn ... View more