Quick command to find the total number of partitions in a Kafka cluster, it could help for example in Mirror Maker sizing. Please replace ZK_SERVER values with your cluster details. cd /tmp zookeeper="ZK_SERVER1:2181,ZK_SERVER2:2181,ZK_SERVER3:2181" sum=0 for i in $(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list --zookeeper $zookeeper ); do count=$(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper $zookeeper --topic $i |grep Leader | wc -l); sum=`expr $sum + $count` ; echo 'total partitions is ' $sum; done If you want to count partitions with specific filter on the name for Topics zookeeper="ZK_SERVER1:2181,ZK_SERVER2:2181,ZK_SERVER3:2181" sum=0 for i in $(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list --zookeeper $zookeeper | grep 'FILTER'); do count=$(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper $zookeeper --topic $i |grep Leader | wc -l); sum=`expr $sum + $count` ; echo 'total partitions is ' $sum; done ... View more

How to get the number of documents indexed : curl -o /tmp/result.txt --negotiate -u : -X GET "SOLR_SERVER:8886/solr/ranger_audits_shard1_replica1/select?q=*:*&distrib=false" How to run a delete command via curl ( delete data older than 24h 😞 curl --negotiate -u: "SOLR_SERVER:8886/solr/ranger_audits/update?commit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>evtTime:[* TO NOW-24HOURS]</query></delete>" How to run optimize command via curl : curl --negotiate -u: "SOLR_SERVER:8886/solr/ranger_audits/update?optimize=true" ... View more

##install Luna client, Unzip Luna client for example under /opt/LUNAHSM under /opt/LUNAHSM/linux/64/ run sh install.sh all Follow the the instructions below, for the questions asked, please answer as below : Accept conditions (y/n) y Products Choose Luna Products to be installed [1]: Luna SA [2]: Luna PCI-E [3]: Luna G5 [4]: Luna Remote Backup HSM [N|n]: Next [Q|q]: Quit Enter selection: 1 Products Choose Luna Products to be installed *[1]: Luna SA [2]: Luna PCI-E [3]: Luna G5 [4]: Luna Remote Backup HSM [N|n]: Next [Q|q]: Quit Enter selection: n Advanced Choose Luna Components to be installed [1]: Luna Software Development Kit (SDK) *[2]: Luna JSP (Java) *[3]: Luna JCProv (Java) [B|b]: Back to Products selection [I|i]: Install [Q|q]: Quit Enter selection: i List of Luna Products to be installed: - Luna SA List of Luna Components to be installed: - Luna JSP (Java) - Luna JCProv (Java) ... installation complete <br>#now to swap the certificate : copy SERVER.pem from LUNA server to your KMS server /tmp cp /tmp/SERVER.pem /usr/safenet/lunaclient/cert/server #under lunaClient [root@XXXXX lunaclient]# pwd /usr/safenet/lunaclient #get the local IP where the client is installed YY.YY.YY.YY (YY.YY.YY.YY is your local IP) [root@XXXXX lunaclient]# bin/vtl createCert -n YY.YY.YY.YY Private Key created and written to: /usr/safenet/lunaclient/cert/client/SERVERkey.pem Certificate created and written to: /usr/safenet/lunaclient/cert/client/xx.xx.xx.xx.pem #add a Luna SA Server to the trusted list of servers [root@XXXXX lunaclient]# bin/vtl addServer -n xx.xx.xx.xx -c /usr/safenet/lunaclient/cert/server/SERVER.pem New server xx.xx.xx.xx successfully added to server list. transfer the pem generated to the Luna server. SWAP COMPLETED. [root@XXXXX lunaclient]# bin/vtl verify ... View more

Quick post to add an auto fix for Solr infra lock issue. Ranger server under : /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf Edit the file solrconfig.xml Uncomment and change <unlockOnStartup>false</unlockOnStartup> to <unlockOnStartup>true</unlockOnStartup> Submit the new xml: /usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string XXXX:2181/infra-solr --upload-config --config-dir /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf --config-set ranger_audits --jaas-file Increase the sleep time from 5 to 30 seconds in /opt/lucidworks-hdpsearch/solr/bin/solr sed -i 's/(sleep 5)/(sleep 30)/g'/opt/lucidworks-hdpsearch/solr/bin/solr Or in the following : sed -i 's/(sleep 5)/(sleep 30)/g' /usr/lib/ambari-infra-solr/bin/solr you can also add in the script the following command : hadoop fs -rm /user/infra-solr/ranger_audits/core_node1/data/index/write.lock ... View more

Quick tips to optimise your infra Solr for Ranger audits using SolrCloud. 1) Change the SolrCloud retention period of the audit. On the server of ranger under : /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf #### Edit the file or use sed to replace the 90 Days in the solrconfig.xml , choose the right retention period, here is 6 hours sed -i 's/+90DAYS/+6HOURS/g' solrconfig.xml sed -i 's/86400/7200/g' solrconfig.xml 2) Change ZK config, by submitting the xml again /usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string XXXXXX:2181/infra-solr --upload-config --config-dir /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf --config-set ranger_audits --jaas-file /usr/hdp/current/ranger-admin/conf/ranger_solr_jaas.conf Check that we loaded it correctly, in the Solr UI or with the following command #Download the solrconfig.xml from Zookeeper /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh --zkhost XXXXXX:2181 -cmd getfile /infra-solr/configs/ranger_audits/solrconfig.xml /tmp/solrconfig.xml 3) Restart Infra ... View more