Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

Loading solr configs in Cloudera SolrCloud

avatar
New Contributor

We try to import our data into SolrCloud using MapReduce Batch Indexing. We face a problem at the reduce phase, that solr.xml cannot be found. We create a 'twitter' collection but looking at the logs, after it failed to load in 'solr.xml', it uses the default one and tries to create 'collection1' (failed) and 'core1' (success) SolrCore. I'm not sure if we need to create our own 'solr.xml' and where to put it (we try to put it at several places but it seems not to load in). Below is the log:

 

2022 [main] INFO  org.apache.solr.hadoop.HeartBeater  - Heart beat reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
2025 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Using this unpacked directory as solr home: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2025 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Creating embedded Solr server with solrHomeDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip, fs: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1828461666_1, ugi=nguyen (auth:SIMPLE)]], outputShardDir: hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2029 [Thread-64] INFO  org.apache.solr.hadoop.HeartBeater  - HeartBeat thread running
2030 [Thread-64] INFO  org.apache.solr.hadoop.HeartBeater  - Issuing heart beat for 1 threads
2083 [main] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2259 [main] INFO  org.apache.solr.hadoop.SolrRecordWriter  - Constructed instance information solr.home /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip (/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip), instance dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/, conf dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/conf/, writing index to solr.data.dir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data, with permdir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2266 [main] INFO  org.apache.solr.core.ConfigSolr  - Loading container configuration from /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml
2267 [main] INFO  org.apache.solr.core.ConfigSolr  - /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml does not exist, using default configuration
2505 [main] INFO  org.apache.solr.core.CoreContainer  - New CoreContainer 696103669
2505 [main] INFO  org.apache.solr.core.CoreContainer  - Loading cores into CoreContainer [instanceDir=/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/]
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting socketTimeout to: 0
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting urlScheme to: http://
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting connTimeout to: 0
2515 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maxConnectionsPerHost to: 20
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting corePoolSize to: 0
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maximumPoolSize to: 2147483647
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting maxThreadIdleTime to: 5
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting sizeOfQueue to: -1
2516 [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  - Setting fairnessPolicy to: false
2527 [main] INFO  org.apache.solr.client.solrj.impl.HttpClientUtil  - Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2648 [main] INFO  org.apache.solr.logging.LogWatcher  - Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2676 [coreLoadExecutor-3-thread-1] INFO  org.apache.solr.core.CoreContainer  - Creating SolrCore 'collection1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1
2677 [coreLoadExecutor-3-thread-1] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/'
2691 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - Failed to load file /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/solrconfig.xml
2693 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - Unable to create core: collection1
org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
    at org.apache.solr.core.Config.<init>(Config.java:116)
    at org.apache.solr.core.Config.<init>(Config.java:86)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
    ... 11 more
2695 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer  - null:org.apache.solr.common.SolrException: Unable to create core: collection1
    at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1158)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:670)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
    ... 10 more
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
    at org.apache.solr.core.Config.<init>(Config.java:116)
    at org.apache.solr.core.Config.<init>(Config.java:86)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
    ... 11 more

2697 [main] INFO  org.apache.solr.core.CoreContainer  - Creating SolrCore 'core1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2697 [main] INFO  org.apache.solr.core.SolrResourceLoader  - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2751 [main] INFO  org.apache.solr.core.SolrConfig  - Adding specified lib dirs to ClassLoader
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/extraction/lib).
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2752 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/clustering/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/langid/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/velocity/lib).
2753 [main] WARN  org.apache.solr.core.SolrResourceLoader  - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2785 [main] INFO  org.apache.solr.update.SolrIndexConfig  - IndexWriter infoStream solr logging is enabled
2790 [main] INFO  org.apache.solr.core.SolrConfig  - Using Lucene MatchVersion: LUCENE_44
2869 [main] INFO  org.apache.solr.core.Config  - Loaded SolrConfig: solrconfig.xml
2879 [main] INFO  org.apache.solr.schema.IndexSchema  - Reading Solr Schema from schema.xml
2937 [main] INFO  org.apache.solr.schema.IndexSchema  - [core1] Schema name=twitter
3352 [main] INFO  org.apache.solr.schema.IndexSchema  - unique key field: id
3471 [main] INFO  org.apache.solr.schema.FileExchangeRateProvider  - Reloading exchange rates from file currency.xml
3478 [main] INFO  org.apache.solr.schema.FileExchangeRateProvider  - Reloading exchange rates from file currency.xml
3635 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Solr Kerberos Authentication disabled
3636 [main] INFO  org.apache.solr.core.JmxMonitoredMap  - No JMX servers found, not exposing Solr information with JMX.
3652 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3686 [main] INFO  org.apache.solr.core.CachingDirectoryFactory  - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3711 [main] WARN  org.apache.solr.core.SolrCore  - [core1] Solr index directory 'hdfs:/master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index' doesn't exist. Creating new index...
3719 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3719 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Number of slabs of block cache [1] with direct memory allocation set to [true]
3720 [main] INFO  org.apache.solr.core.HdfsDirectoryFactory  - Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
3721 [main] INFO  org.apache.solr.store.blockcache.BufferStore  - Initializing the 1024 buffers with [8192] buffers.
3740 [main] INFO  org.apache.solr.store.blockcache.BufferStore  - Initializing the 8192 buffers with [8192] buffers.
3891 [main] INFO  org.apache.solr.core.CachingDirectoryFactory  - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3988 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: init: current segments file is "null"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@65b01d5d
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: now checkpoint "" [0 segments ; isCommit = false]
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IFD][main]: 0 msec to checkpoint
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IW][main]: init: create=true
3992 [main] INFO  org.apache.solr.update.LoggingInfoStream  - [IW][main]: 
dir=NRTCachingDirectory(org.apache.solr.store.hdfs.HdfsDirectory@17e5a6d8 lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@7f117668; maxCacheMB=192.0 maxMergeSizeMB=16.0)

 

Who agreed with this topic