MapReduceIndexerTool with Kerberos

Alessio — Fri, 16 Sep 2022 09:28:41 GMT

Hi,

I'm trying to index some logs with solr.
I defined the schema.xml and the morphline.conf file. I tried to run this on a Cluster without Kerberos and it works fine (CDH 5.4.0).

When I launch this on a Kerberized Cluster (CDH 5.4.0), I cannot index the logs.

I followed the Cloudera documentation

I defined the jaas.conf equal to:

Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 keyTab="/tmp/krb5cc_0"
 storeKey=true
 useTicketCache=false
 principal="user/fully.qualified.domain.name@REALM";
};

or (if I already used 'kinit')

Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=false
 useTicketCache=true
 principal="user/fully.qualified.domain.name@REALM";
 };

but when I launch the following command (from Cloudera documentation)

hadoop  jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar org.apache.solr.hadoop.MapReduceIndexerTool -D 'mapred.child.java.opts=-Xmx2048m' -Djava.security.auth.login.config=/home/user/clouderaSearch/test_collection/jaas.conf --log4j /opt/cloudera/parcels/CDH/share/doc/search-1.0.0+cdh5.4.0+0/examples/solr-nrt/log4j.properties --morphline-file /home/user/clouderaSearch/test_collection/test_morphline.conf --output-dir hdfs://nameservice1:8020/data/test/output_solr --verbose --go-live --zk-host  zookeeper_host:2181/solr --collection test_collection hdfs://nameservice1:8020/data/test/incoming

I get this log:

0    [main] INFO  org.apache.solr.common.cloud.SolrZkClient  - Using default ZkCredentialsProvider
32   [main] INFO  org.apache.solr.common.cloud.ConnectionManager  - Waiting for client to connect to ZooKeeper
57   [main-EventThread] INFO  org.apache.solr.common.cloud.ConnectionManager  - Watcher org.apache.solr.common.cloud.ConnectionManager@17fca9f9 name:ZooKeeperConnection Watcher:zookeeper_host:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
57   [main] INFO  org.apache.solr.common.cloud.ConnectionManager  - Client is connected to ZooKeeper
57   [main] INFO  org.apache.solr.common.cloud.SolrZkClient  - Using default ZkACLProvider
72   [main] INFO  org.apache.solr.common.cloud.ZkStateReader  - Updating cluster state from ZooKeeper...
564  [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Cluster reports 40 mapper slots
1040 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Cluster reports 8 reduce slots
1040 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Using these parameters: numFiles: 1, mappers: 320, realMappers: 1, reducers: 2, shards: 2, fanout: 2147483647, maxSegments: 1
1040 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Randomizing list of 1 input files to spread indexing load more evenly among mappers
1200 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Done. Randomizing list of 1 input files took 5.3011456E7 secs
1217 [main] INFO  org.apache.solr.common.cloud.SolrZkClient  - Using default ZkCredentialsProvider
1217 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  - Waiting for client to connect to ZooKeeper
1230 [main-EventThread] INFO  org.apache.solr.common.cloud.ConnectionManager  - Watcher org.apache.solr.common.cloud.ConnectionManager@74f6a607 name:ZooKeeperConnection Watcher:zookeeper_host:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
1231 [main] INFO  org.apache.solr.common.cloud.ConnectionManager  - Client is connected to ZooKeeper
1231 [main] INFO  org.apache.solr.common.cloud.SolrZkClient  - Using default ZkACLProvider
1231 [main] INFO  org.apache.solr.hadoop.ZooKeeperInspector  - Load collection config from:/collections/test_collection
1250 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/admin-extra.menu-top.html
1260 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/currency.xml
1261 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/protwords.txt
1262 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/mapping-FoldToASCII.txt
1264 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/solrconfig.xml.secure
1265 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/_schema_analysis_synonyms_english.json
1266 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/_rest_managed.json
1267 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/solrconfig.xml
1269 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/_schema_analysis_stopwords_english.json
1271 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/contractions_ga.txt
1272 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_hi.txt
1273 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_eu.txt
1274 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_es.txt
1275 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/contractions_it.txt
1276 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/userdict_ja.txt
1277 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_sv.txt
1278 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stemdict_nl.txt
1280 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stoptags_ja.txt
1281 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_bg.txt
1282 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_hy.txt
1283 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_el.txt
1284 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/hyphenations_ga.txt
1285 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ar.txt
1286 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_fr.txt
1287 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_pt.txt
1288 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_nl.txt
1289 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_th.txt
1290 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_de.txt
1291 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_lv.txt
1292 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/contractions_ca.txt
1293 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ru.txt
1294 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ca.txt
1295 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_da.txt
1296 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_gl.txt
1297 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_no.txt
1298 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_tr.txt
1299 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_fa.txt
1300 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ckb.txt
1301 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_hu.txt
1302 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ro.txt
1303 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_id.txt
1304 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_cz.txt
1305 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ja.txt
1306 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_fi.txt
1307 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_it.txt
1308 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/contractions_fr.txt
1309 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_ga.txt
1310 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/lang/stopwords_en.txt
1311 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/stopwords.txt
1312 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/spellings.txt
1313 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/mapping-ISOLatin1Accent.txt
1314 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/admin-extra.html
1316 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/xslt/luke.xsl
1317 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/xslt/updateXml.xsl
1318 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/xslt/example_atom.xsl
1319 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/xslt/example.xsl
1320 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/xslt/example_rss.xsl
1321 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/synonyms.txt
1322 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/scripts.conf
1323 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/update-script.js
1326 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/query_spatial.vm
1327 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/mime_type_lists.vm
1328 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/results_list.vm
1329 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/cluster_results.vm
1330 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/hit_grouped.vm
1331 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/layout.vm
1332 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/join_doc.vm
1333 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/debug.vm
1334 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/facet_queries.vm
1335 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/query_group.vm
1336 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/main.css
1337 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/VM_global_library.vm
1338 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/pagination_bottom.vm
1339 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/error.vm
1340 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/facets.vm
1341 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/richtext_doc.vm
1342 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/head.vm
1343 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/query_form.vm
1344 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/cluster.vm
1345 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/hit_plain.vm
1346 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/footer.vm
1347 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/jquery.autocomplete.js
1348 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/browse.vm
1349 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/facet_pivot.vm
1350 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/README.txt
1351 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/did_you_mean.vm
1352 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/jquery.autocomplete.css
1353 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/header.vm
1354 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/hit.vm
1355 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/suggest.vm
1356 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/pagination_top.vm
1358 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/query.vm
1359 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/product_doc.vm
1360 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/tabs.vm
1361 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/facet_ranges.vm
1362 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/velocity/facet_fields.vm
1363 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/elevate.xml
1364 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/admin-extra.menu-bottom.html
1365 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/schema.xml
1368 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/clustering/carrot2/lingo-attributes.xml
1369 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/clustering/carrot2/stc-attributes.xml
1370 [main] INFO  org.apache.solr.cloud.ZkController  - Write file /tmp/1431512151138-0/clustering/carrot2/kmeans-attributes.xml
1565 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Indexing 1 files using 1 real mappers into 2 reducers
1637 [main] ERROR org.apache.hadoop.hdfs.KeyProviderCache  - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
1700 [main] ERROR org.apache.hadoop.hdfs.KeyProviderCache  - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
1701 [main] ERROR org.apache.hadoop.hdfs.KeyProviderCache  - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
2745 [main] ERROR org.apache.hadoop.hdfs.KeyProviderCache  - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
2747 [main] ERROR org.apache.hadoop.hdfs.KeyProviderCache  - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
55781 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Done. Indexing 1 files using 1 real mappers into 2 reducers took 1.80719104E10 secs
55895 [main] INFO  org.apache.solr.hadoop.GoLive  - Live merging of output shards into Solr cluster...
55901 [pool-4-thread-2] INFO  org.apache.solr.hadoop.GoLive  - Live merge hdfs://nameservice1:8020/data/test/output_solr/results/part-00000 into http://SolrServer:8983/solr
55901 [pool-4-thread-1] INFO  org.apache.solr.hadoop.GoLive  - Live merge hdfs://nameservice1:8020/data/test/output_solr/results/part-00000 into http://SolrServer:8983/solr
55901 [pool-4-thread-3] INFO  org.apache.solr.hadoop.GoLive  - Live merge hdfs://nameservice1:8020/data/test/output_solr/results/part-00001 into http://SolrServer:8983/solr
55901 [pool-4-thread-4] INFO  org.apache.solr.hadoop.GoLive  - Live merge hdfs://nameservice1:8020/data/test/output_solr/results/part-00001 into http://solrServer:8983/solr
56208 [main] ERROR org.apache.solr.hadoop.GoLive  - Error sending live merge command
java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html><head><title>Apache Tomcat/6.0.43 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - Authentication required</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Authentication required</u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.43</h3></body></html>
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.solr.hadoop.GoLive.goLive(GoLive.java:126)
        at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:920)
        at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:662)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:649)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html><head><title>Apache Tomcat/6.0.43 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - Authentication required</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Authentication required</u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.43</h3></body></html>
        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:581)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:229)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:225)
        at org.apache.solr.client.solrj.request.CoreAdminRequest.process(CoreAdminRequest.java:510)
        at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:100)
        at org.apache.solr.hadoop.GoLive$1.call(GoLive.java:89)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
56212 [main] INFO  org.apache.solr.hadoop.GoLive  - Live merging of index shards into Solr cluster took 1.05398328E8 secs
56212 [main] INFO  org.apache.solr.hadoop.GoLive  - Live merging failed

The merge phase fails (Live merge hdfs://nameservice1:8020/data/test/output_solr/results/part-00000 into http://SolrServer:8983/solr), and I cannot visit the page http://SolrServer:8983/solr because an authentication is required.

Any idea of why is not working?

Many thanks

Alessio

Re: MapReduceIndexerTool with Kerberos

cconner — Wed, 13 May 2015 14:33:41 GMT

Did you include:

HADOOP_OPTS="-Djava.security.auth.login.config=/path/to/jaas.conf"

Before your "hadoop jar" command? IE:

HADOOP_OPTS="-Djava.security.auth.login.config=/path/to/jaas.conf" hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar org.apache.solr.hadoop.MapReduceIndexerTool -D 'mapred.child.java.opts=-Xmx2048m' -Djava.security.auth.login.config=/home/user/clouderaSearch/test_collection/jaas.conf --log4j /opt/cloudera/parcels/CDH/share/doc/search-1.0.0+cdh5.4.0+0/examples/solr-nrt/log4j.properties --morphline-file /home/user/clouderaSearch/test_collection/test_morphline.conf --output-dir hdfs://nameservice1:8020/data/test/output_solr --verbose --go-live --zk-host  zookeeper_host:2181/solr --collection test_collection hdfs://nameservice1:8020/data/test/incoming

Notice how "HADOOP_OPTS" and "hadoop jar" are on the same line, that is how they need to be.

question Re: MapReduceIndexerTool with Kerberos in Archives of Support Questions (Read Only)

MapReduceIndexerTool with Kerberos

Re: MapReduceIndexerTool with Kerberos

Re: MapReduceIndexerTool with Kerberos