Member since
06-13-2017
10
Posts
0
Kudos Received
0
Solutions
04-06-2018
06:09 AM
Please find the log message below,2018-04-06 11:10:34,253 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2018-04-06 11:10:34,433 - Stack Feature Version Info: stack_version=2.5, version=None, current_cluster_version=None -> 2.5
2018-04-06 11:10:34,440 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2018-04-06 11:10:34,441 - Group['metron'] {}
2018-04-06 11:10:34,443 - Group['livy'] {}
2018-04-06 11:10:34,443 - Group['elasticsearch'] {}
2018-04-06 11:10:34,443 - Group['spark'] {}
2018-04-06 11:10:34,444 - Group['zeppelin'] {}
2018-04-06 11:10:34,444 - Group['hadoop'] {}
2018-04-06 11:10:34,444 - Group['kibana'] {}
2018-04-06 11:10:34,444 - Group['users'] {}
2018-04-06 11:10:34,444 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,445 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,446 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,447 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,448 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 11:10:34,448 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop']}
2018-04-06 11:10:34,449 - User['metron'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,450 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,450 - User['elasticsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,451 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,452 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users']}
2018-04-06 11:10:34,453 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,453 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,454 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,455 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,456 - User['kibana'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,456 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,457 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,458 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
2018-04-06 11:10:34,459 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-06 11:10:34,460 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-04-06 11:10:34,465 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2018-04-06 11:10:34,466 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2018-04-06 11:10:34,467 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-04-06 11:10:34,468 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2018-04-06 11:10:34,472 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2018-04-06 11:10:34,473 - Group['hdfs'] {}
2018-04-06 11:10:34,473 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': [u'hadoop', u'hdfs']}
2018-04-06 11:10:34,474 - FS Type:
2018-04-06 11:10:34,474 - Directory['/etc/hadoop'] {'mode': 0755}
2018-04-06 11:10:34,487 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 11:10:34,487 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-04-06 11:10:34,503 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-04-06 11:10:34,509 - Skipping Execute[('setenforce', '0')] due to not_if
2018-04-06 11:10:34,510 - Directory['/appl/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2018-04-06 11:10:34,511 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2018-04-06 11:10:34,512 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2018-04-06 11:10:34,515 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2018-04-06 11:10:34,517 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2018-04-06 11:10:34,523 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2018-04-06 11:10:34,531 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 11:10:34,532 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2018-04-06 11:10:34,533 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2018-04-06 11:10:34,537 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2018-04-06 11:10:34,541 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2018-04-06 11:10:34,734 - User['elasticsearch'] {'action': ['create'], 'groups': [u'elasticsearch']}
2018-04-06 11:10:34,736 - Directory['/appl/log/elasticsearch'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755}
2018-04-06 11:10:34,737 - Directory['/var/run/elasticsearch'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755}
2018-04-06 11:10:34,737 - Directory['/etc/elasticsearch/'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755}
2018-04-06 11:10:34,737 - Directory['/appl/elasticsearch/lmm/es_data'] {'owner': 'elasticsearch', 'group': 'elasticsearch', 'create_parents': True, 'mode': 0755}
2018-04-06 11:10:34,738 - Directory['/etc/elasticsearch//scripts'] {'owner': 'elasticsearch', 'create_parents': True, 'group': 'elasticsearch', 'mode': 0755}
Master env: /etc/elasticsearch//elastic-env.sh
2018-04-06 11:10:34,742 - File['/etc/elasticsearch//elastic-env.sh'] {'content': InlineTemplate(...), 'owner': 'elasticsearch', 'group': 'elasticsearch'}
Master yml: /etc/elasticsearch//elasticsearch.yml
2018-04-06 11:10:34,749 - File['/etc/elasticsearch//elasticsearch.yml'] {'owner': 'elasticsearch', 'content': Template('elasticsearch.master.yaml.j2'), 'group': 'elasticsearch'}
Master sysconfig: /etc/sysconfig/elasticsearch
2018-04-06 11:10:34,753 - File['/etc/sysconfig/elasticsearch'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'root'}
Start the Master
2018-04-06 11:10:34,754 - Execute['service elasticsearch start'] {}
Command failed after 1 tries
... View more
04-06-2018
05:40 AM
Elasticsearch master get fails during the restart. Seems log file is not updating. Please help me slove this.
... View more
Labels:
11-24-2017
01:00 PM
@Simon Elliston Ball Ok thanks Simon. I will check it. As you mentioned above i used SplitText and now it is working without any issue. Cheers, Suresh
... View more
11-24-2017
12:44 PM
@Simon Elliston Ball it is end like below. CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-............. aid=393wdw18BABCAB3JFk8LDzQ\=\=
... View more
11-24-2017
11:16 AM
@Simon Elliston Ball
I was trying to send data in CEF from using "\n" demarcator. But it doesnot parsed correctly in metron CEF paser. Could you please send me a proper demarcator which is working for CEF format. Sample log format : CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4624 CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4634 Cheers, Suresh
... View more
11-22-2017
03:45 PM
Hi Simon, Thanks a lot. I added "\n" to message demarcator. Now it is working with out any issue. thanks a lot for your quick response. Really appreciate that. Cheers, Suresh
... View more
11-22-2017
03:04 PM
I configured nifi process to ingest data from a file to
kafka broker to metron. Nifi send data to kafka without any error. But it will
not push to metron parsers. Please help me on this. But if I run below two commands it send data parsers
without any issues. cat
/var/log/squid/access.log |${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh
--broker-list $BROKERLIST --topic squid ${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh
--zookeeper $ZOOKEEPER --topic squid --from-beginning
... View more
Labels:
11-17-2017
01:08 PM
@Ss i managed to convert timestamp in Kibana in metron, what i did was added below Paser config in metron sensor settings. PARSER CONFIG
timestampField - timestamp if you need more details feel free to contact me.
... View more
11-17-2017
01:00 PM
@Lee Adrian i managed to convert timestamp in metron Kibana, what i did was add below Paser config in metron sensor settings. PARSER CONFIG
timestampField - timestamp if you need more details feel free to contact me.
... View more
06-13-2017
06:38 AM
Hi @asubramanian,
Is it recommended to run Metron on Docker?
... View more