Member since
03-16-2018
2
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5504 | 03-21-2018 01:29 PM |
03-21-2018
01:29 PM
I'm running this index job via the command line using the jars as described here: http://druid.io/docs/latest/ingestion/command-line-hadoop-indexer.html Have determined Druid 0.12.0 has something weird going on in conjunction with the druid-parquet-extensions as the fs.defaultFs set in the conf/druid/_common/common.runtime.properties is seemingly not respected at some point (don't exactly have a ton of time to trace through their open source project). So here is what I have done as a successful workaround, hopefully this will be helpful java -Xmx512m -Ddruid.storage.storageDirectory=hdfs://{my_namenode_ip}:{my_namename_port}/{my_segments_path} -Ddruid.storage.type=hdfs -Dfile.encoding=UTF-8 -classpath extensions/druid-parquet-extensions/*:extensions/druid-avro-extensions:extensions/druid-hdfs-storage:lib/*:conf/druid/_common:{HADOOP_PATH}{HADOOP_JAR} io.druid.cli.Main index hadoop {DRUID_INDEXER_DATA}
... View more
03-16-2018
12:04 PM
@Hajime Having the same problem while testing an update to 0.12.0. Ran into your thread, thought i'd share a link that is seemingly related from awhile ago.. https://groups.google.com/forum/#!topic/druid-development/8u5orNnQlwE "Druid checks the default file system for replacing ":" with "_" and making a valid DFS file path,
What is the value of fs.defaultFS set in hadoop config files ?
can you try pointing this to hdfs filesystem, If its not already doing that ?"
... View more