hadoop logs push to Kafka topic using Ambari LogSearch

[#1]

Ambari 2.4.0.0 officially supports LogSearch Component [ Tech Preview] . To learn more about Log Search component please refer to link Ambari LogSearch . While LogSearch component does support pushing the logs to Kafka topic [ based on which real time log analytics can be performed ] this is official not supported in Ambari 2.4.0.0. This might get addressed in Ambari 2.5.0.0 probably.

This article provides the details on how to configure LogSearch [ LogFeeder component ] to push to Kafka topic if there is a need to capture and perform real time analytics based on logs in your cluster.

1.After installing LogSearch component from Ambari 2.4.0.0, go to the LogSearch component config screen and add below property under "Advanced logfeeder-properties" to property "logfeeder.config.files"

{default_config_files},kafka-output.json

2. Create the kafka-output.json file with below content under directory /etc/ambari-logsearch-logfeeder/conf/ on the nodes which has logfeeder [ ideally all the nodes in your cluster ]

{
  "output": [
    {
      "is_enabled": "true",
      "destination": "kafka",
      "broker_list": "ctr-e25-1471039652053-0001-01-000006.test.domain:6667",
      "topic": "log-streaming",
      "conditions": {
        "fields": {
          "rowtype": [
            "service"
          ]
        }
      }
    }
  ]
}

3. Configure Kafka PLAINTEXT listener as below if the cluster is Kerberozied because workaround to push to PLAINTEXTSASL is not available.Make sure broker endpoint configured in Step#2 is PLAINTEXT

PLAINTEXT://localhost:6667,PLAINTEXTSASL://localhost:6668

4. Create Kafka topic and provide ACLs to ANONYMOUS user. Below command help in the same.

./bin/kafka-topics.sh --zookeeper zookeeper-node:2181 --create --topic log-streaming --partitions 1 --replication-factor 1
./bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper-node:2181 --add --allow-principal User:ANONYMOUS --operation Read --operation Write --operation Describe --topic log-streaming

5. Restart LogSearch service from Ambari and thats it ! logs should be pushing by now. Below is the command to check the same.

/bin/kafka-console-consumer.sh --zookeeper zookeeper-node:2181 --topic log-streaming --from-beginning --security-protocol PLAINTEXT