Member since
04-11-2018
55
Posts
1
Kudos Received
0
Solutions
11-01-2018
10:05 AM
Yes, running all commands on NN2 which happens to be a Solr node also. I was able to create the collections using the collections api, but solrctl works for some commands and just not the collections command. Maybe some bug, please check the code. Thanks
... View more
11-01-2018
09:35 AM
[centos@ip-172-31-81-167 data]$ solrctl --trace --zk ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr collection --delete ds-collection + shift 1 + test 5 '!=' 0 + case "$1" in + '[' 5 -gt 1 ']' + SOLR_ZK_ENSEMBLE=ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr + shift 2 + test 3 '!=' 0 + case "$1" in + break + '[' -n '' ']' + '[' -n '' ']' + '[' -n '' ']' + '[' -z ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr ']' + SOLR_ADMIN_ZK_CMD='ZKCLI_JVM_FLAGS=${ZKCLI_JVM_FLAGS} LOG4J_PROPS=${SOLR_CONF_DIR}/log4j.properties ${SOLR_HOME}/bin/zkcli.sh -zkhost $SOLR_ZK_ENSEMBLE 2>&3' + '[' 3 -gt 0 ']' + test 3 '!=' 0 + case "$1" in + '[' --delete = --list ']' + '[' --delete = --list-snapshots ']' + '[' 3 -gt 2 ']' + case "$2" in ++ echo --delete ++ tr '[a-z]-' '[A-Z] ' + COL_ACTION=' DELETE' ++ echo DELETE + eval solr_webapi ''\''/admin/collections?action=DELETE&name=ds-collection'\''' ++ solr_webapi '/admin/collections?action=DELETE&name=ds-collection' ++ '[' -z '' ']' +++ get_solr_protocol ++++ eval 'ZKCLI_JVM_FLAGS=${ZKCLI_JVM_FLAGS}' 'LOG4J_PROPS=${SOLR_CONF_DIR}/log4j.properties' '${SOLR_HOME}/bin/zkcli.sh' -zkhost '$SOLR_ZK_ENSEMBLE' '2>&3' -cmd get /clusterprops.json +++++ ZKCLI_JVM_FLAGS= +++++ LOG4J_PROPS=/etc/solr/conf/log4j.properties +++++ /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/solr//bin/zkcli.sh -zkhost ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr -cmd get /clusterprops.json +++ SOLR_STATE='{"urlScheme":"http"}' +++ echo '{"urlScheme":"http"}' +++ grep -i urlScheme +++ grep -q -i https +++ echo http ++ local SOLR_PROTOCOL=http +++ get_solr_state /live_nodes +++ sed -ne 's#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p' ++++ eval 'ZKCLI_JVM_FLAGS=${ZKCLI_JVM_FLAGS}' 'LOG4J_PROPS=${SOLR_CONF_DIR}/log4j.properties' '${SOLR_HOME}/bin/zkcli.sh' -zkhost '$SOLR_ZK_ENSEMBLE' '2>&3' -cmd ls /live_nodes +++++ ZKCLI_JVM_FLAGS= +++++ LOG4J_PROPS=/etc/solr/conf/log4j.properties +++++ /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/solr//bin/zkcli.sh -zkhost ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr -cmd ls /live_nodes +++ SOLR_STATE='/live_nodes (5) /live_nodes/ip-172-31-81-167.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-90-165.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-82-140.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-92-169.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-93-54.ec2.internal:8983_solr (0)' +++ echo '/live_nodes (5) /live_nodes/ip-172-31-81-167.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-90-165.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-82-140.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-92-169.ec2.internal:8983_solr (0) /live_nodes/ip-172-31-93-54.ec2.internal:8983_solr (0)' +++ grep -v '^/ ' ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://ip-172-31-81-167.ec2.internal:8983/solr ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://ip-172-31-90-165.ec2.internal:8983/solr ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://ip-172-31-82-140.ec2.internal:8983/solr ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://ip-172-31-92-169.ec2.internal:8983/solr ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://ip-172-31-93-54.ec2.internal:8983/solr ++ for node in '`get_solr_state '\''/live_nodes'\'' | sed -ne '\''s#/live_nodes/\(.*:[0-9][0-9]*\).*$#\1#p'\''`' 'localhost:$SOLR_PORT' ++ curl -i --retry 5 -s -L -k --negotiate -u : http://localhost:/solr ++ '[' -n '' ']' ++ die 'Error: can'\''t discover Solr URI. Please specify it explicitly via --solr.' ++ '[' '!' -z 'Error: can'\''t discover Solr URI. Please specify it explicitly via --solr.' ']' ++ echo Error: 'can'\''t' discover Solr URI. Please specify it explicitly via --solr. Error: can't discover Solr URI. Please specify it explicitly via --solr. ## Seems like the collection command is not working in CDH 5.15. Instancedir works fine with zk quoram.
... View more
11-01-2018
09:34 AM
solrctl --zk ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr collection --delete ds-collection --trace Error: can't discover Solr URI. Please specify it explicitly via --solr.
... View more
10-30-2018
08:32 AM
Is this a bug in CDH 5.15? The same happens for me. This works solrctl --zk ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr instancedir --create ds-collection /tmp/ds-collection This gives me Solr url not found error: solrctl --zk ip-172-31-90-165.ec2.internal:2181,ip-172-31-81-167.ec2.internal:2181,ip-172-31-92-169.ec2.internal:2181/solr collection --create ds-collection -s 2 -r 2 The above command used to work in older 5.1x versions. Please advise.
... View more
08-23-2018
06:27 AM
Solution found. In the hosts file of the python client machine, add public IP and private host name This is appropriate for a cloud service like AWS. Python lib works fine now. Thanks for @bgooley help on another thread that resolved this too.
... View more
08-22-2018
08:08 PM
@bgooley Thanks for the tip. That worked. So, putting the public IP and private hostname in hosts file on client did the trick. 🙂
... View more
08-22-2018
04:40 PM
Thanks for the comment. Not sure how to do this private to public mapping. Any help is appreciated. Thanks. This is what the hosts file on client looks like right now: 127.0.0.1 localhost.localdomain localhost 192.168.157.154 edgenode.local.com edgenode 18.215.25.118 ec2-18-215-25-118.compute-1.amazonaws.com ec2-18-215-25-118 <-- Public IP of server
... View more
08-22-2018
11:01 AM
OK, I have fixed this issue by replacing the core-site.xml file IP address with the public one. That allows me to list hadoop dirs on cluster. But read/write operations give errors like the following. Any ideas what config changes are needed on client side files to allow this to work? [root@edgenode bin]# hadoop fs -put hdfs-clientconfig-aws.zip /ds-datalake/misc 18/08/22 13:00:28 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:2008) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1715) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1668) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:790) 18/08/22 13:00:28 WARN hdfs.DFSClient: Abandoning BP-49600184-172.31.26.58-1534798007391:blk_1073745239_4416 18/08/22 13:00:28 WARN hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[172.31.26.58:50010,DS-0c88ebaf-aa0b-407c-8b64-e02a02eeac3c,DISK] 18/08/22 13:00:28 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /ds-datalake/misc/hdfs-clientconfig-aws.zip._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1719) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3505) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) at org.apache.hadoop.ipc.Client.call(Client.java:1504) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:425) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy11.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1860) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1656) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:790) put: File /ds-datalake/misc/hdfs-clientconfig-aws.zip._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
... View more