METRON FAILING TO START AFTER RESETING AMBARI SERVER AND CHANGING THE CLUSTER NAME

[#1]

esource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hcp/1.4.1.0-18/metron/bin/zk_load_configs.sh --zk_quorum node02.local:2181,node03.local:2181,node01.local:2181 --mode PATCH --config_type GLOBAL --patch_file /tmp/metron-global-config-patch.json' returned 1. 2018-03-07 13:00:41 ERROR ConfigurationManager:314 - Unable to apply patch to Zookeeper config

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /metron/topology/global

[#2]

Hi @abu ameen,

Are you able to access to Zookeeper CLI and run a ls?

What do you mean about "reseting Ambari Server"?

Gonçalo

[#3]

Hi @Gonçalo Cunha

i reseted ambari server using ambari-server reset everything was gone then i deployed the cluster from scratch using ambari and encounterd the above issue

ZK cli:

[zk: localhost:2181(CONNECTED) 6] ls

[zk: localhost:2181(CONNECTED) 7]

[#4]

@abu ameen,

The command is wrong note the slash

 [zk: localhost:2181(CONNECTED) 6] ls /

[#5]

@abu ameen

I think there's the problem. Zookeeper doesn't have any paths for Metron like -> /metron/topology/global.

You can create this specific path but you can face further issues such as ACLs, etc. or you can reinstall Metron.

Gonçalo

[#6]

@abu ameen

Did you get this to work?
I'm expriencing the same issue - with HDP 2.6 and HCP 1.4.2.

Metron Enrichment fails to start.

Showing this same error.

Reinstalled Metron - still the same.

,

@abu ameen

Did you ever get this to work?
I tried reinstalling metron, no luck.

[#7]

@Mark Kotlyar

yes i did.

could you try running the Zookeeper CLI (usr/hdp/current/zookeeper-client/zkCli.sh ) and run a (ls /) or (ls /metron/topolgoy) ?

what do you get ?

[#8]

Yeah. Did that. Nothing was there.

I tried this:

1. ./zk_load_configs.sh -i ../config/zookeeper -m PUSH -z ${zookeeper}

So far, it seems I got past the error you had here (and me) in the Enrichment.

Now waiting for the rest of the services to startup.

Fingers crossed 😄

[#9]

@abu ameen

Most probably there's too many open connections to Zookeeper on that host. Zookeeper limits the maximum possible connections from a single host. Per default this is 60. If there are 60 open connections to zookeeper from a single host, no further connections are opened. This is why the script fails.

To check from where the connections are opened execute:

netstat -nape | awk '{if ($5 == "ADD_MY_ZOOKEEPER_IP_HERE:2181") print $4, $9;}' | sort | uniq -c

Identify the process ids of the applications opening connections:

1 192.168.123.1:45162 577/java

In this case it's 577

Check which application this is. It might be the Metron REST server. In this case you can kill it and restart it.