Please Explain how to do the data transfer from Local file system to Hdfs using Taildir Flume Source. My use case is to deal with real time data  so data in the source directory is keep updating ... View more

you are correct, i tested. i thought it shows min.insynch.replicas also,thanks for help. ... View more

This seems to be indicating that the jute.maxbuffer has been exceeded. You can increase this on the command line side by exporting the following: export ZKCLI_JVM_FLAGS=-Djute.maxbuffer=4194304 You may need to confirm on the ZK service configuration, if the jute.maxbuffer size is also 4 MB -pd ... View more

ok @pdvorak, just like I thought, thanks a lot for the confirmation! 😉 Kind regards! Alex ... View more

Also, I ran the flume in debug mode : [cloudera@quickstart flume-ng]$ ./bin/flume-ng agent --conf conf -conf-file conf/flumekafka.conf --name agent1 -Dflume.root.logger=DEBUG,console   Getting the below snippet repeatedaly with 'Updated cluster metadata version' value changing 2018-03-16 12:01:51,797 (lifecycleSupervisor-1-4) [DEBUG - org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendGroupCoordinatorRequest(AbstractCoordinator.java:470)] Sending coordinator request for group flume to broker quickstart.cloudera:9092 (id: 33) 2018-03-16 12:01:51,801 (lifecycleSupervisor-1-4) [DEBUG - org.apache.kafka.clients.consumer.internals.AbstractCoordinator.handleGroupMetadataResponse(AbstractCoordinator.java:483)] Received group coordinator response ClientResponse(receivedTimeMs=1521226911801, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@10f78e4b, request=RequestSend(header={api_key=10,api_version=0,correlation_id=15,client_id=consumer-1}, body={group_id=flume}), createdTimeMs=1521226911798, sendTimeMs=1521226911798), responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}) 2018-03-16 12:01:51,879 (lifecycleSupervisor-1-4) [DEBUG - org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:627)] Sending metadata request ClientRequest(expectResponse=true, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=16,client_id=consumer-1}, body={topics=[Airports]}), isInitiatedByNetworkClient, createdTimeMs=1521226911878, sendTimeMs=0) to node 33 2018-03-16 12:01:51,891 (lifecycleSupervisor-1-4) [DEBUG - org.apache.kafka.clients.Metadata.update(Metadata.java:180)] Updated cluster metadata version 10 to Cluster(nodes = [quickstart.cloudera:9092 (id: 33)], partitions = [Partition(topic = Airports, partition = 6, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 10, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 1, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 11, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 13, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 5, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 7, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 12, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 8, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 17, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 16, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 4, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 3, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 14, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 2, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 15, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 19, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 18, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 0, leader = 33, replicas = [33,], isr = [33,], Partition(topic = Airports, partition = 9, leader = 33, replicas = [33,], isr = [33,]]) S ... View more

https://stackoverflow.com/questions/34844209/consumer-not-receiving-messages-kafka-console-new-consumer-api-kafka-0-9/48821014#48821014 ... View more

We are tryng the same thing. However when we execute our code, it asks for the Kereros Credentials. Is there anyway to authenticate solr user using Keytab files? ... View more

HI Any update for the last question I cannot get the correctly numfound after i run:     HADOOP_OPTS="-Djava.security.auth.login.config=jaas.conf" \ hadoop --config /etc/hadoop/conf jar /opt/cloudera/parcels/CDH/lib/hbase-solr/tools/hbase-indexer-mr-1.5-cdh5.11.2-job.jar --conf /etc/hbase/conf/hbase-site.xml -Dmapreduce.job.queuename=root.hadoop.plarch --hbase-indexer-zk oddev03:2181,oddev04:2181,oddev05:2181 --hbase-indexer-name onedata_order_orderIndexer --go-live   Is this a known issue? if so, how to workaround? if no, how to correct the above command line?      Thanks for your reply. BR Paul ... View more

where should we add this warn?? ... View more