How to clean up & recreate collections on Ambari Infra Sometimes you need to clean up an installation that went with issues and there are missing collections on Ambari Infra solr. In order to do this, go thry the next list of steps to clean up solr collections and recreate them from scratch. IMPORTANT: This steps will delete all collection data, so unless you are sure data can be removed dont perform these steps. 1) Switch adudit solr to OFF from Ranger -> Ranger Audit configuration. Restarted all affected. 2) Stop Altas service 3) From the ambari infra host remove all the collections using rest api $ kinit -kt /etc/security/keytabs/ambari-infra-solr.service.keytab infra-solr/.... $ export SOLR_HOST=<solr host fqdn> ## Ranger collection: $ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=ranger_audits" ## Atlas collections: $ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=vertex_index" $ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=edge_index" $ curl -i -v --negotiate -u : "http://$SOLR_HOST:8886/solr/admin/collections?action=DELETE&name=fulltext_index" 4) Stop the Ambari Infra 5) Remove the /infra-solr znode IMPORTANT: This next command will remove all the configuration of all the collections on zookeeper for infra-solr. $ zookeeper-client > rmr /infra-solr 6) Start Ambari Infra again (once started the znode /infra-solr should be created again) 7) Switch adudit solr back ON from Ranger -> Ranger Audit configuration 8) Restart all affected 9) Still need to restart all Ranger components one more time 10) Start Atlas 11) After all services come up on zookeeper you should see the following znodes have been created: [vertex_index, edge_index, fulltext_index, ranger_audits] HTH ... View more

Introduction This article will address some of the most least documented issues you may find while importing data using oozie-sqoop-hcatalog and respective solutions. Heart beat, follow by read-timed-out with thrift server SYMPTOM While trying to execute an oozie workflow, on the stdout of the oozie sqoop job you can see: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI thrift://host:port Heart beat Heart beat Heart beat ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1221)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ROOT CAUSE Missing hive-site.xml for the sqoop action caused the error RESOLUTION Save the hive-site.xml on the hdfs and reference from the sqoop action file tag: <file>${nameNode}/user/${wf:user()}/${appRoot}/env/${cluster}/hive-site.xml#hive-site.xml</file> Import failed: Can not create a Path from an empty string SYMPTOM While trying to execute an oozie workflow, on the stdout of the oozie sqoop job you can see: ERROR org.apache.sqoop.tool.ImportTool - Imported Failed: Can not create a Path from an empty string ERROR [main] tool.ImportTool (ImportTool.java:run(607)) - Imported Failed: Can not create a Path from an empty string ROOT CAUSE Missing skip-dist-cache argument for the sqoop action RESOLUTION Add --skip-dist-cache argument to sqoop action Got exception running Sqoop: java.lang.NullPointerException SYMPTOM While trying to execute an oozie workflow, on the stdout of the oozie sqoop job you can see: 4591 [main] ERROR org.apache.sqoop.Sqoop - Got exception running Sqoop: java.lang.NullPointerException ERROR [main] sqoop.Sqoop (Sqoop.java:runSqoop(186)) - Got exception running Sqoop: java.lang.NullPointerException Intercepting System.exit(1) and on the stderr you can see: java.lang.NullPointerException at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) at java.lang.Runtime.exec(Runtime.java:620) at java.lang.Runtime.exec(Runtime.java:528) at org.apache.sqoop.util.Executor.exec(Executor.java:76) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.executeExternalHCatProgram(SqoopHCatUtilities.java:1145) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.launchHCatCli(SqoopHCatUtilities.java:109 ROOT CAUSE NPE is happening because HCAT_HOME is not set when running Sqoop import through Oozie for hcatalog. RESOLUTION To fix this issue, please set the hcatalog-home in Sqoop import workflow.xml: ... <arg>--hcatalog-home</arg> <arg>/usr/hdp/current/hive-webhcat</arg> ... ... View more