Please find the log message below,2018-04-06 11:10:34,253 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2018-04-06 11:10:34,433 - Stack Feature Version Info: stack_version=2.5, version=None, current_cluster_version=None -> 2.5 2018-04-06 11:10:34,440 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf User Group mapping (user_group) is missing in the hostLevelParams 2018-04-06 11:10:34,441 - Group['metron'] {} 2018-04-06 11:10:34,443 - Group['livy'] {} 2018-04-06 11:10:34,443 - Group['elasticsearch'] {} 2018-04-06 11:10:34,443 - Group['spark'] {} 2018-04-06 11:10:34,444 - Group['zeppelin'] {} 2018-04-06 11:10:34,444 - Group['hadoop'] {} 2018-04-06 11:10:34,444 - Group['kibana'] {} 2018-04-06 11:10:34,444 - Group['users'] {} 2018-04-06 11:10:34,444 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,445 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,446 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,447 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,448 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']} 2018-04-06 11:10:34,448 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop']} 2018-04-06 11:10:34,449 - User['metron'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,450 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,450 - User['elasticsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,451 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,452 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']} 2018-04-06 11:10:34,453 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,453 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,454 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,455 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,456 - User['kibana'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,456 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,457 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,458 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']} 2018-04-06 11:10:34,459 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-04-06 11:10:34,460 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-04-06 11:10:34,465 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2018-04-06 11:10:34,466 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-04-06 11:10:34,467 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-04-06 11:10:34,468 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-04-06 11:10:34,472 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2018-04-06 11:10:34,473 - Group['hdfs'] {} 2018-04-06 11:10:34,473 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']} 2018-04-06 11:10:34,474 - FS Type: 2018-04-06 11:10:34,474 - Directory['/etc/hadoop'] {'mode': 0755} 2018-04-06 11:10:34,487 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2018-04-06 11:10:34,487 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2018-04-06 11:10:34,503 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2018-04-06 11:10:34,509 - Skipping Execute[('setenforce', '0')] due to not_if 2018-04-06 11:10:34,510 - Directory['/appl/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'} 2018-04-06 11:10:34,511 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'} 2018-04-06 11:10:34,512 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'} 2018-04-06 11:10:34,515 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2018-04-06 11:10:34,517 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2018-04-06 11:10:34,523 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2018-04-06 11:10:34,531 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'} 2018-04-06 11:10:34,532 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2018-04-06 11:10:34,533 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2018-04-06 11:10:34,537 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2018-04-06 11:10:34,541 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2018-04-06 11:10:34,734 - User['elasticsearch'] {'action': ['create'], 'groups': [u'elasticsearch']} 2018-04-06 11:10:34,736 - Directory['/appl/log/elasticsearch'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755} 2018-04-06 11:10:34,737 - Directory['/var/run/elasticsearch'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755} 2018-04-06 11:10:34,737 - Directory['/etc/elasticsearch/'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755} 2018-04-06 11:10:34,737 - Directory['/appl/elasticsearch/lmm/es_data'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755} 2018-04-06 11:10:34,738 - Directory['/etc/elasticsearch//scripts'] {'owner': 'elasticsearch', 'create_parents': True, 'group': 'elasticsearch', 'mode': 0755} Master env: /etc/elasticsearch//elastic-env.sh 2018-04-06 11:10:34,742 - File['/etc/elasticsearch//elastic-env.sh'] {'content': InlineTemplate(...), 'owner': 'elasticsearch', 'group': 'elasticsearch'} Master yml: /etc/elasticsearch//elasticsearch.yml 2018-04-06 11:10:34,749 - File['/etc/elasticsearch//elasticsearch.yml'] {'owner': 'elasticsearch', 'content': Template('elasticsearch.master.yaml.j2'), 'group': 'elasticsearch'} Master sysconfig: /etc/sysconfig/elasticsearch 2018-04-06 11:10:34,753 - File['/etc/sysconfig/elasticsearch'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'root'} Start the Master 2018-04-06 11:10:34,754 - Execute['service elasticsearch start'] {} Command failed after 1 tries ... View more

@Simon Elliston Ball Ok thanks Simon. I will check it. As you mentioned above i used SplitText and now it is working without any issue. Cheers, Suresh ... View more

@Simon Elliston Ball it is end like below. CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-............. aid=393wdw18BABCAB3JFk8LDzQ\=\= ... View more

@Simon Elliston Ball I was trying to send data in CEF from using "\n" demarcator. But it doesnot parsed correctly in metron CEF paser. Could you please send me a proper demarcator which is working for CEF format. Sample log format : CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4624 CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4634 Cheers, Suresh ... View more

Hi Simon, Thanks a lot. I added "\n" to message demarcator. Now it is working with out any issue. thanks a lot for your quick response. Really appreciate that. Cheers, Suresh ... View more

@Ss i managed to convert timestamp in Kibana in metron, what i did was added below Paser config in metron sensor settings. PARSER CONFIG timestampField - timestamp if you need more details feel free to contact me. ... View more

@Lee Adrian i managed to convert timestamp in metron Kibana, what i did was add below Paser config in metron sensor settings. PARSER CONFIG timestampField - timestamp if you need more details feel free to contact me. ... View more