About pdvorak

pdvorak · ‎05-19-2016

What is the value of your "ZooKeeper Root" set to in the kafka service configuration? If it is set to something like "/kafka", you'll need to specify that in your zk string: bin/kafka-console-consumer --zookeeper localhost:2181 /kafka --topic kafkatest --from-beginning What version of kafka parcel are you using? -pd

pdvorak · ‎05-19-2016

If your edge node is part of the cluster, and you are using parcels, then you won't have start and stop scripts, and the recommended method to run flume is by setting up a flume service in CM to run on the edge node. The only difference between an edge node and a cluster node, is that the edge nodes generally don't run hadoop services. Have you installed the flume rpms on this edge node or are you using parcels? Where are you running the flume-ng command from: which flume-ng alternatives --display flume-ng -pd

pdvorak · ‎05-18-2016

If you are using flume to deliver to hdfs, it is recommended to have that flume agent run on a node in your cluster. If you are using flume to collect events from other applications and send downstream to another agent which then delivers to its final destination (hdfs, solr, etc), then you can run that agent on a cluster node, or on the machine where the events are being generated. If it is not running on a CDH node, you can use packages to install flume, and then use the stop and start scripts to start it and keep it running as a daemon. -pd

pdvorak · ‎05-17-2016

No, initialize solr usually happens when the service is installed, but it may not have if you installed from director. It is in the Actions drop down menu, and grayed out when solr is running

pdvorak · ‎05-17-2016

Have you ever had solr up and running in this cluster? Do you have any active collections? One thing to try is to perform an "Initialize Solr" action from CM (from within the solr service), when the solr service is shut down. If solr has not been initialized in ZK, you could get your observed error. -pd

pdvorak · ‎05-12-2016

Taildir source is documented here: http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#taildir-source -pd

pdvorak · ‎05-12-2016

You need some method to forward the logs to the flume agents. You could do something like rsyslog with the imfile input module (http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html ) to forward to a syslog source on the flume agents, or you could just install the standalone flume agent ( without the rest of the CDH) via rpms or tarball: http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download.html If you run the standalone flume agents, they could use spooldir or the new taildir source (in flume for CDH5.7) to monitor the files and forward via avro to flume agents within your cluster. -pd

pdvorak · ‎05-12-2016

Add 3 more hdfs sinks, all using the same channel. Be sure to add hdfs.filePrefix with a unique value per sink to avoid filename collision. Hopefully that will deliver the events fast enough to keep up. -pd

pdvorak · ‎05-06-2016

As noted in your other similar post, one of two thing happened: 1. Your single sink is not keeping up with the source, you need to add more sinks to pull from the same channel or 2. You had an error in your sink that caused it to stop delivering to hdfs. You should review the logs for the first error (prior to the channel full exceptions). Often times restarting will resolve the issue. If you add additional sinks, that will help as well, because failure of one sink won't prevent the other sinks from pulling events off the channel. -pd

pdvorak · ‎05-06-2016

The error is reporting a problem when trying to write into the "/solr" directory, as indicated here: Unable to create core [live_logs] Caused by: Permission denied: user=solr, access=WRITE, inode="/solr":hdfs:supergroup:drwxr-xr-x at You need to give the solr user ownership of the "/solr" directory in hdfs. sudo -u hdfs hdfs dfs -chown solr:solr /solr Additionally, you hvae changed permissions/ownership on the /user folder and /tmp, which is not recommended: sudo -u hdfs hdfs dfs -chown solr:solr /user sudo -u hdfs hdfs dfs -mkdir /tmp sudo -u hdfs hdfs dfs -chown solr:solr /tmp -pd

Online	Offline
Last Visited	‎01-08-2020 04:37 PM

Member Since	‎01-09-2014 08:15 AM
Last Visited	‎01-08-2020 04:37 PM
Posts	283
Kudos received	70

Cloudera Community

Re: spooldir channel error - too many files. - how...

Re: How to configure Flume with Kafka channel with...

Re: How to configure Flume with Kafka channel with...

Re: Solrcloud Replica Names

Re: flume kafkasource, hdfs sink remove avro field

Re: No brokers found in ZK

Re: Flume agent on edge node

Re: Flume agent on edge node

Re: Failed to configure urlScheme property for Sol...

Re: Failed to configure urlScheme property for Sol...

Re: Flume without agents on web server

Re: Flume without agents on web server

Re: While running flume agent facing some error

Re: While running flume agent facing some error

Re: Permission error while creating solr collectio...