About hosako

hosako · ‎03-26-2018

Hi @Olivér Szabó, I see the new dates in solrconfig.xml in the Zookeeper, but newly added docs still used previous days, so that I posted this question. After unloading ranger_audit core multiple times, it eventually worked. But I was wondering if unload/reload is required? If yes, why Ambari doesn't do that?

hosako · ‎03-26-2018

Looks like updating "Max Retention Days" from Ambari Web UI does not update not only existing docs (which is expected) but also newly added docs. I unloaded the ranger audit core multiple times and restarted Ranger to recreate the collection, then it eventually worked. Is this expected behaviour? If so, why Ambari doesn't unload (or reload if that should work)? (a bug?)

hosako · ‎03-26-2018

As the fix is "prop_value = prop_value.replace("/usr/lib/python2.6/site-packages", "/usr/lib/ambari-server/lib")", how about creating a symlink? ln -s /usr/lib/ambari-server/lib /usr/lib/python2.6/site-packages

hosako · ‎03-22-2018

Thanks a lot! java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -Ddruid.storage.storageDirectory=hdfs://`hostname -f`:8020/tmp/data/index/output -Ddruid.storage.type=hdfs -classpath /usr/hdp/current/druid-overlord/extensions/druid-hdfs-storage/*:/usr/hdp/current/druid-overlord/lib/*:/usr/hdp/current/druid-overlord/conf/_common:/etc/hadoop/conf/ io.druid.cli.Main index hadoop ./hadoop_index_spec.json Above worked. Mine is sandbox so using `hostname -f`.

hosako · ‎03-19-2018

The core-site.xml under /etc/hadoop/conf shows: <property> <name>fs.defaultFS</name> <value>hdfs://sandbox-hdp.hortonworks.com:8020</value> <final>true</final> </property> So... I guess my config is OK? Do I need to add "druid.indexer.fork.property.druid.indexer.task.hadoopWorkingPath" in some property file and add this in the -cp?

hosako · ‎03-15-2018

Thank you, @Nishant Bangarwa I sent those by email.

hosako · ‎03-14-2018

I read http://druid.io/docs/latest/ingestion/command-line-hadoop-indexer.html and tried the following command: java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -classpath /usr/hdp/current/druid-overlord/conf/_common:/usr/hdp/current/druid-overlord/lib/*:/etc/hadoop/conf io.druid.cli.Main index hadoop ./hadoop_index_spec.json But this job fails with below: 2018-03-14T07:37:06,132 INFO [main] io.druid.indexer.JobHelper - Deleting path[/tmp/druid/mmcellh/2018-03-14T071308.731Z_55fbb15cd4d4454885d909c870837f93] 2018-03-14T07:37:06,150 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!! java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151] at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:117) [druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] at io.druid.cli.Main.main(Main.java:108) [druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.IndexGeneratorJob] failed! at io.druid.indexer.JobHelper.runJobs(JobHelper.java:389) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:95) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:131) ~[druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] at io.druid.cli.Main.main(Main.java:108) ~[druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] ... 6 more And the yarn application log shows "xxxx is not a valid DFS filename": 2018-03-14T07:31:41,369 ERROR [main] io.druid.indexer.JobHelper - Exception in retry loop java.lang.IllegalArgumentException: Pathname /tmp/data/index/output/mmcellh/2014-02-11T10:00:00.000Z_2014-02-11T11:00:00.000Z/2018-03-14T07:13:08.731Z/0/index.zip.3 from hdfs://sandbox-hdp.hortonworks.com:8020/tmp/data/index/output/mmcellh/2014-02-11T10:00:00.000Z_2014-02-11T11:00:00.000Z/2018-03-14T07:13:08.731Z/0/index.zip.3 is not a valid DFS filename. at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:217) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:480) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:476) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:491) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:417) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:930) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?] at io.druid.indexer.JobHelper$4.push(JobHelper.java:415) [druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235] ... https://github.com/druid-io/druid/pull/1121 looks similar but this should have been fixed in HDP 2.6.3. So I'm wondering if the classpath I'm using is correct.

hosako · ‎03-13-2018

This issue may happen when Hive Metastore's 'DBS' table contains a location which doesn't have port. For example, 'hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db' I think above is a valid location path, but when HS2 is restarted from Ambari, Ambari replaces not only this 'DBS' location, but also all 'SDS' locations with, for example, like below: old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A So that next time hiveserver2 is restarted, you don't see this behaviour, but you still need to correct SDS location.

hosako · ‎02-15-2018

HI @Slim I'm seeing "Connected to Druid but could not retrieve datasource information" when I create a table. Could you have any idea where I should check?

hosako · ‎01-25-2018

Hi @Kuldeep Kulkarni Does this tutorial still work with Ambari 2.6?

Online	Offline
Last Visited	‎05-16-2018 07:44 AM

Member Since	‎09-24-2015 11:58 PM
Last Visited	‎05-16-2018 07:44 AM
Posts	144
Kudos received	70

Cloudera Community

Cloudera Community

Re: How to GET Get Service Definition APIs in HDP ...

Re: How can I enable Kerberos debug logging for Sp...

Re: Install HUE on a separate Node

Re: In HDP 2.3.2 tez.container.max.java.heap.fract...

Re: How to delete/remove storage policy ?

Re: With Ambari 2.6.1, how can I change "Max Reten...

With Ambari 2.6.1, how can I change "Max Retention...

Re: Error while bootstrapping: Cannot run program ...

Re: What would be the right command to start Druid...

Re: What would be the right command to start Druid...

Re: What would be the right command to start Druid...

What would be the right command to start Druid Had...

Re: getting issue after run the command in Hive

Re: Hive druid handler "java.lang.NoClassDefFoundE...

Re: Automate HDP installation using Ambari Bluepri...