Member since
09-24-2015
144
Posts
72
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1311 | 08-15-2017 08:15 AM | |
6118 | 01-24-2017 06:58 AM | |
1595 | 08-03-2016 06:45 AM | |
2900 | 06-01-2016 10:08 PM | |
2487 | 04-07-2016 10:30 AM |
03-26-2018
09:44 PM
Hi @Olivér Szabó, I see the new dates in solrconfig.xml in the Zookeeper, but newly added docs still used previous days, so that I posted this question. After unloading ranger_audit core multiple times, it eventually worked. But I was wondering if unload/reload is required? If yes, why Ambari doesn't do that?
... View more
03-26-2018
05:17 AM
Looks like updating "Max Retention Days" from Ambari Web UI does not update not only existing docs (which is expected) but also newly added docs. I unloaded the ranger audit core multiple times and restarted Ranger to recreate the collection, then it eventually worked. Is this expected behaviour? If so, why Ambari doesn't unload (or reload if that should work)? (a bug?)
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
-
Apache Solr
03-26-2018
04:49 AM
1 Kudo
As the fix is "prop_value = prop_value.replace("/usr/lib/python2.6/site-packages", "/usr/lib/ambari-server/lib")", how about creating a symlink? ln -s /usr/lib/ambari-server/lib /usr/lib/python2.6/site-packages
... View more
03-22-2018
03:18 AM
Thanks a lot! java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -Ddruid.storage.storageDirectory=hdfs://`hostname -f`:8020/tmp/data/index/output -Ddruid.storage.type=hdfs -classpath /usr/hdp/current/druid-overlord/extensions/druid-hdfs-storage/*:/usr/hdp/current/druid-overlord/lib/*:/usr/hdp/current/druid-overlord/conf/_common:/etc/hadoop/conf/ io.druid.cli.Main index hadoop ./hadoop_index_spec.json Above worked. Mine is sandbox so using `hostname -f`.
... View more
03-19-2018
02:51 AM
The core-site.xml under /etc/hadoop/conf shows: <property>
<name>fs.defaultFS</name>
<value>hdfs://sandbox-hdp.hortonworks.com:8020</value>
<final>true</final>
</property> So... I guess my config is OK? Do I need to add "druid.indexer.fork.property.druid.indexer.task.hadoopWorkingPath" in some property file and add this in the -cp?
... View more
03-15-2018
11:28 PM
Thank you, @Nishant Bangarwa I sent those by email.
... View more
03-14-2018
08:05 AM
I read http://druid.io/docs/latest/ingestion/command-line-hadoop-indexer.html and tried the following command: java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -classpath /usr/hdp/current/druid-overlord/conf/_common:/usr/hdp/current/druid-overlord/lib/*:/etc/hadoop/conf io.druid.cli.Main index hadoop ./hadoop_index_spec.json But this job fails with below: 2018-03-14T07:37:06,132 INFO [main] io.druid.indexer.JobHelper - Deleting path[/tmp/druid/mmcellh/2018-03-14T071308.731Z_55fbb15cd4d4454885d909c870837f93]
2018-03-14T07:37:06,150 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!!
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:117) [druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
at io.druid.cli.Main.main(Main.java:108) [druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.IndexGeneratorJob] failed!
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:389) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:95) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:131) ~[druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
at io.druid.cli.Main.main(Main.java:108) ~[druid-services-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
... 6 more
And the yarn application log shows "xxxx is not a valid DFS filename": 2018-03-14T07:31:41,369 ERROR [main] io.druid.indexer.JobHelper - Exception in retry loop
java.lang.IllegalArgumentException: Pathname /tmp/data/index/output/mmcellh/2014-02-11T10:00:00.000Z_2014-02-11T11:00:00.000Z/2018-03-14T07:13:08.731Z/0/index.zip.3 from hdfs://sandbox-hdp.hortonworks.com:8020/tmp/data/index/output/mmcellh/2014-02-11T10:00:00.000Z_2014-02-11T11:00:00.000Z/2018-03-14T07:13:08.731Z/0/index.zip.3 is not a valid DFS filename.
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:217) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:480) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:476) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:491) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:417) ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:930) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891) ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at io.druid.indexer.JobHelper$4.push(JobHelper.java:415) [druid-indexing-hadoop-0.10.1.2.6.3.0-235.jar:0.10.1.2.6.3.0-235]
... https://github.com/druid-io/druid/pull/1121 looks similar but this should have been fixed in HDP 2.6.3. So I'm wondering if the classpath I'm using is correct.
... View more
Labels:
- Labels:
-
Apache Hadoop
03-13-2018
04:20 AM
1 Kudo
This issue may happen when Hive Metastore's 'DBS' table contains a location which doesn't have port. For example, 'hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db' I think above is a valid location path, but when HS2 is restarted from Ambari, Ambari replaces not only this 'DBS' location, but also all 'SDS' locations with, for example, like below: old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A
So that next time hiveserver2 is restarted, you don't see this behaviour, but you still need to correct SDS location.
... View more
02-15-2018
06:48 AM
HI @Slim I'm seeing "Connected to Druid but could not retrieve datasource information" when I create a table. Could you have any idea where I should check?
... View more
01-25-2018
06:40 AM
Hi @Kuldeep Kulkarni Does this tutorial still work with Ambari 2.6?
... View more