Thank you for your reply and your advice, @GeKas.   The issue has developed a bit more since my last reply, and has become even more puzzling.  Maybe this is just an old bug that I'm tripping on.   It now appears that the issue only occurs when I write the results to a file AND do not include the -B parameter.  I did not detect this pattern until now but it appears consistent with my prior accounts.   Here is everything working when output to the console: $ impala-shell --print_header -i 10.0.0.1 -d myDB -q 'select phenolist from myTable where startpos = 225609789' Starting Impala Shell without Kerberos authentication Connected to 10.0.0.1:21000 Server version: impalad version 2.1.0-cdh4 RELEASE (build 11a45f84eb1f0d441ebad72cf9d65262f6cc2391) Query: use `myDB` Query: select phenolist from myTable where startpos = 225609789 +---------------------+ | phenolist | +---------------------+ | Pelger-Huët anomaly | +---------------------+ Fetched 1 row(s) in 0.35s If I direct results to a file, I get the error: $ impala-shell .... > tmp Unknown Exception : 'ascii' codec can't encode character u'\xeb' in position 83: ordinal not in range(128) And finally, if I add the -B parameter, things work again: $ impala-shell -B ... > tmp Fetched 1 row(s) in 0.35s ... View more

The fact that this was fixed in v1.4 is what is confusing to me, espcecially because I appear to be on version 2.1.   Also, I am able to run the command that was failing in that forum thread.  That is also a bit confusing.   impala-shell -i 10.0.0.1 -d myDB -q "select 'Ѳ'" Starting Impala Shell without Kerberos authentication Connected to 10.0.0.1:21000 Server version: impalad version 2.1.0-cdh4 RELEASE (build 11a45f84eb1f0d441ebad72cf9d65262f6cc2391) Query: use `myDB` Query: select 'Ѳ' Query submitted at: 2018-03-06 16:52:51 (Coordinator: None) Query progress can be monitored at: None/query_plan?query_id=5040fadc08d0c83e:64b2514f9d270fb5 +-----+ | 'ѳ' | +-----+ | Ѳ | +-----+ Yes, I am really using CDH4.  Do you think that I wouldn't be facing this issue on CDH5?   Thanks a bunch for your help! ... View more

Thank you, Ben.  Incidentally, I finally worked up the courage to click that Deploy Client Configuration button a couple days ago.   It worked perfectly.  ... View more

Since it is starting to look like a problem with the configuration files on the host, I am wondering if 'Deploy Client Configuration' could be useful.  (That is the option in CM under the 'Actions' dropdown.)   Also in that dropdown is the option for 'View Client Configuration URLs'.  Selecting the hdfs from the following menu showed this hdfs-site.xml (below).  It contains the correct ip addresses for the current namenodes.   Is 'Deploy Client Configuration' the right solution to my issue?   <property> <name>dfs.namenode.rpc-address.nameservice1.namenode472</name> <value>ip-10-0-0-246.ec2.internal:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.nameservice1.namenode472</name> <value>ip-10-0-0-246.ec2.internal:8022</value> </property> <property> <name>dfs.namenode.http-address.nameservice1.namenode472</name> <value>ip-10-0-0-246.ec2.internal:50070</value> </property> <property> <name>dfs.namenode.https-address.nameservice1.namenode472</name> <value>ip-10-0-0-246.ec2.internal:50470</value> </property> <property> <name>dfs.namenode.rpc-address.nameservice1.namenode374</name> <value>ip-10-0-0-154.ec2.internal:8020</value> </property> <property> ... View more

I have checked the configuration in hdfs-site.xml.  It appears that the old namenodes are still being listed.   The commands below show the relevant settings in the hdfs-site.xml file found in the most recent NAMENODE directory on one of the hosts.   It shows that 10.0.0.154 and 10.0.0.157 are still listed as namenodes.  However, according to CM, 10.0.0.246 should have replaced 10.0.0.157.   How should I address this issue?   root@ip-10-0-0-157:/home/ubuntu# cloudera_client_config() { > local recent_config=$(ls -d1t /var/run/cloudera-scm-agent/process/*NAMENODE | head -1) > grep -A2 -B1 "dfs.namenode.rpc-address" $nndir/hdfs-site.xml > } root@ip-10-0-0-157:/home/ubuntu# cloudera_client_config <property> <name>dfs.namenode.rpc-address.nameservice1.namenode451</name> <value>ip-10-0-0-157.ec2.internal:8020</value> </property> -- <property> <name>dfs.namenode.rpc-address.nameservice1.namenode374</name> <value>ip-10-0-0-154.ec2.internal:8020</value> </property> ... View more

  10.0.0.157 is definitely done.  The namenode role was removed from that host in the course of resolving this issue.  In its place, I added a new host to the cluster and made it the namenode (10.0.0.246).   Here is the screenshot again.  Let me know if you still cannot see it.  It is visible from this link as well.     ... View more

Below is the same command after enabling the logging as you suggested.   It appears to be trying to contact 10.0.0.157, which was the active namenode prior to the issues which I mentioned in the linked post above.   The screenshot shows that 10.0.0.157 is no longer a namenode.  This is surely part of the problem but I do not know how to resolve this.     ubuntu@ip-10-0-0-154:/backup/20171108$ hdfs dfs -ls / 17/11/14 22:03:06 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops) 17/11/14 22:03:06 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops) 17/11/14 22:03:06 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[GetGroups] , always=false, type=DEFAULT, sampleName=Ops) 17/11/14 22:03:06 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics 17/11/14 22:03:06 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty 17/11/14 22:03:06 DEBUG security.Groups: Creating new Groups object 17/11/14 22:03:06 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/11/14 22:03:06 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library 17/11/14 22:03:06 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution 17/11/14 22:03:06 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping 17/11/14 22:03:07 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 17/11/14 22:03:07 DEBUG security.UserGroupInformation: hadoop login 17/11/14 22:03:07 DEBUG security.UserGroupInformation: hadoop login commit 17/11/14 22:03:07 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: ubuntu 17/11/14 22:03:07 DEBUG security.UserGroupInformation: UGI loginUser:ubuntu (auth:SIMPLE) 17/11/14 22:03:07 DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://nameservice1 17/11/14 22:03:07 DEBUG hdfs.NameNodeProxies: multipleLinearRandomRetry = null 17/11/14 22:03:07 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWritable, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@75be16f5 17/11/14 22:03:07 DEBUG ipc.Client: The ping interval is 60000 ms. 17/11/14 22:03:07 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientNamenodeProtocolPB 17/11/14 22:03:07 DEBUG ipc.Client: Connecting to ip-10-0-0-157.ec2.internal/10.0.0.157:8020 17/11/14 22:03:07 DEBUG ipc.Client: closing ipc connection to ip-10-0-0-157.ec2.internal/10.0.0.157:8020: Connection refused java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:510) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:604) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291) at org.apache.hadoop.ipc.Client.call(Client.java:1209)   ... View more

Thank you very much for your reply, @weichiu.   Yes, you are correct about having an odd number of ZK nodes.  CM also warned me about this, which is part of the reason I wanted to reduce it down to 5 ZK nodes.   Regarding the namenode in standby, that is a very relevant topic for me because I just finished a difficult issue where one of my namenodes failed and I needed to switch to the standby node.   You can see the forum post here.  Everything was working good after this but I guess it is possible that I did something wrong.   I have attached a screenshot of CM showing the active and standby namenodes.  Are you suggesting that I should do a manual failover?   ... View more