Re: How to debug the issue "IPC's epoch X is less ...

cnauroth · ‎12-30-2016

@Joshua Adeleke , there is a statement in the article that "block deletion activity in HDFS is asynchronous". This statement also applies when finalizing an upgrade. Since this processing happens asynchronously, it's difficult to put an accurate wall clock estimate on it. In my experience, I've generally seen the physical on-disk block deletions start happening 2-5 minutes after finalizing an upgrade.

cnauroth · ‎06-29-2016

@Payel Datta, have you tried using NetCat to test if that can bind and listen on these ports, like I suggested in an earlier comment? "nc -l 2888" or "nc -l 3888". If a similar failure happens with NetCat, then that would confirm that you need to investigate further for some kind of networking problem at the hosts, though I don't know exactly what that networking problem would be from the information here.

cnauroth · ‎06-23-2016

Hello @Xiaobing Zhou, This may indicate that either a NameNode or JournalNodes were unresponsive for a period of time. This can lead to a cascading failure, whereby a NameNode HA failover occurs, the other NameNode becomes active, the previous NameNode thinks it is still active, and then QJM rejects that NameNode for not operating within the same "epoch" (logical period of time). This is by design, as QJM is intended to prevent 2 NameNodes from mistakenly acting as active in a split-brain scenario. There are multiple potential reasons for unresponsiveness in the NameNode/JournalNode interaction. Reviewing logs from the NameNodes and JournalNodes would likely reveal more details. There are several common causes to watch for: A long stop-the-world garbage collection pause may surpass the timeout threshold for the call. Garbage collection logging would show what kind of garbage collection activity the process is doing. You might also see log messages about the "JvmPauseMonitor". Consider reviewing the article NameNode Garbage Collection Configuration: Best Practices and Rationale to make sure your cluster's heap and garbage collection settings match best practices. In environments that integrate with LDAP for resolution of users' group memberships, load problems on the LDAP infrastructure can cause delays. In extreme cases, we have seen such timeouts at the JournalNodes cause edit logging calls to fail, which causes a NameNode abort and an HA failover. See Hadoop and LDAP: Usage, Load Patterns and Tuning for a more detailed description and potential mitigation steps. It is possible that there is a failure in network connectivity between the NameNode and the JournalNodes. This tends to be rare, because NameNodes and JournalNodes tend to be colocated on the same host or placed relatively close to one another in the network topology. Still, it is worth investigating that basic network connectivity between all NameNode hosts and all JournalNode hosts is working fine.

cnauroth · ‎06-23-2016

@roy p, if you want to route the data from Flume to WASB instead of HDFS, then I expect you can achieve that by changing the "hdfs:" URI to a "wasb:" URI. The full WASB URI will have an authority component that references an Azure Storage account and a container within that account. You can get the WASB URI by looking at configuration property fs.defaultFS in core-site.xml. If that doesn't work, then I recommend creating a new question specifically asking how to configure Flume to write to a file system different from HDFS. Please also apply the "flume" tag to the question. That will help get attention from Flume experts.

cnauroth · ‎06-22-2016

@roy p, in an HDInsight cluster, the default file system is WASB, which is a Hadoop-compatible file system backed by Azure Storage. The default file system is defined by property fs.defaultFS in core-site.xml. In an HDInsight cluster, you'll see this property set to a "wasb:" URI. When running Hadoop FileSystem Shell commands, if the path is not a qualified URI naming the scheme of the file system, then it assumes that you want the default file system. Thus, running "hadoop fs -ls /" shows results from the WASB file system as persisted in Azure Storage. HDInsight clusters also run a local instance of HDFS as a supplementary, non-default file system. For a file system that is not the default, the shell commands may reference paths in that file system by qualifying the URI with the scheme. Thus, running "hadoop fs -ls hdfs://mycluster/" shows results from the local HDFS file system, even though WASB is the default file system in an HDInsight cluster. Since the two commands reference paths on two different file systems, each containing its own set of files, the final results displayed are different.

cnauroth · ‎06-17-2016

@manichinnari555, I'm glad to hear this helped. I believe setting at table creation time should be sufficient.

cnauroth · ‎06-17-2016

@manichinnari555, I noticed that the table was stored as Parquet. HIVE-11401 is a known bug in Hive filtering on a partition column in Parquet. There is no immediate plan to bring this patch into HDP, but a known workaround is to disable predicate pushdown by setting property hive.optimize.index.filter to false.

cnauroth · ‎06-17-2016

@Payel Datta, I would not expect those permissions on the myid file to be a problem. Even though root is the owner, the permissions still allow read access to everyone. The ZooKeeper process only needs read access to that file. Have you tried any network troubleshooting with tools like NetCat, like I suggested in the last comment?

cnauroth · ‎06-17-2016

@Eric Periard, no, you are not going crazy. 🙂 You're correct that JIRA issue AMBARI-15235 is related. That's a change that helps on the display side. AMBARI-17603 is another patch that gets more at the root cause of the problem by optimizing the JMX query.

cnauroth · ‎06-16-2016

Summary HDFS Rolling Upgrade facilitates software upgrade of independent individual components in an HDFS cluster. During the upgrade window, HDFS will not physically delete blocks. Normal block deletion resumes after the administrator finalizes the upgrade. A common source of operational problems is forgetting to finalize an upgrade. If left unaddressed, HDFS will run out of storage capacity. Attempts to delete files will not free space. To avoid this problem, always finalize HDFS rolling upgrades in a timely fashion. This information applies to both Ambari Rolling & Express Upgrade. Rolling Upgrade Block Handling The high-level workflow of a rolling upgrade for the administrator is: Initiate rolling upgrade. Perform software upgrade on individual nodes. Run typical workloads and validate new software works. If validation is successful, finalize the upgrade. If validation discovers a problem, revert to the prior software via one of 2 options: Rollback - Restore prior software and restore cluster data to its pre-upgrade state. Downgrade - Restore prior software, but preserve data changes that occurred during the upgrade window. The Apache Hadoop documentation on HDFS Rolling Upgrade covers the specific commands in more detail. To satisfy the requirements of Rollback, HDFS will not delete blocks during a rolling upgrade window, which is the time between initiating the rolling upgrade and finalizing it. During this window, DataNodes handle block deletions by moving the blocks to a special directory named "trash" instead of physically deleting them. While the blocks reside in trash, they are not visible to clients performing reads. Thus, the files are logically deleted, but the blocks still consume physical space on the DataNode volumes. If the administrator chooses to rollback, the DataNodes restore these blocks from the trash directory to restore the cluster's data to its pre-upgrade state. After the upgrade is finalized, normal block deletion processing resumes. Blocks previously saved to trash will be physically deleted. New deletion activity will result in a physical delete, not moving the block to trash. Block deletion is asynchronous, so there may be propagation delays between the user deleting a file and the space being freed as reported by tools like "hdfs dfsadmin -report". Impact on HDFS Space Utilization An important consequence of this behavior is that during a rolling upgrade window, HDFS space utilization will rise continuously. Attempting to free space by deleting files will be ineffective, because the blocks will be moved to the trash directory instead of physically deleted. Please also note that this behavior applies not only to files that existed before the upgrade, but also new files created during the upgrade window. All deletes are handled by moving the blocks to trash. An administrator might notice that even after deleting a large amount of files, various tools continue to report high space consumption. This includes "hdfs dfsadmin -report", JMX metrics (which are consumed by Apache Ambari) and the NameNode web UI. If a cluster shows these symptoms, check if a rolling upgrade has not been finalized. There are multiple ways to check this. The "hdfs dfsadmin -rollingUpgrade query" command will report "Proceed with rolling upgrade", and the "Finalize Time" will be unspecified. > hdfs dfsadmin -rollingUpgrade query QUERY rolling upgrade ... Proceed with rolling upgrade: Block Pool ID: BP-1273075337-10.22.2.98-1466102062415 Start Time: Thu Jun 16 14:55:09 PDT 2016 (=1466114109053) Finalize Time: <NOT FINALIZED> The NameNode web UI will display a banner at the top stating "Rolling upgrade started". JMX metrics also expose "RollingUpgradeStatus", which will have a "finalizeTime" of 0 if the upgrade has not been finalized. > curl 'http://10.22.2.98:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo' ... "RollingUpgradeStatus" : { "blockPoolId" : "BP-1273075337-10.22.2.98-1466102062415", "createdRollbackImages" : true, "finalizeTime" : 0, "startTime" : 1466114109053 }, ... DataNode Disk Layout This section explores the layout on disk for DataNodes that have logically deleted blocks during a rolling upgrade window. The following discussion uses a small testing cluster containing only one file. This shows a typical disk layout on a DataNode volume hosting exactly one block replica: data/dfs/data/current ├── BP-1273075337-10.22.2.98-1466102062415 │ ├── current │ │ ├── VERSION │ │ ├── dfsUsed │ │ ├── finalized │ │ │ └── subdir0 │ │ │ └── subdir0 │ │ │ ├── blk_1073741825 │ │ │ └── blk_1073741825_1001.meta │ │ └── rbw │ ├── scanner.cursor │ └── tmp └── VERSION The block file and its corresponding metadata file are in the "finalized" directory. If this file were deleted during a rolling upgrade window, then the block file and its corresponding metadata file would move to the trash directory: data/dfs/data/current ├── BP-1273075337-10.22.2.98-1466102062415 │ ├── RollingUpgradeInProgress │ ├── current │ │ ├── VERSION │ │ ├── dfsUsed │ │ ├── finalized │ │ │ └── subdir0 │ │ │ └── subdir0 │ │ └── rbw │ ├── scanner.cursor │ ├── tmp │ └── trash │ └── finalized │ └── subdir0 │ └── subdir0 │ ├── blk_1073741825 │ └── blk_1073741825_1001.meta └── VERSION As a reminder, block deletion activity in HDFS is asynchronous. It may take several minutes after running the "hdfs dfs -rm" command before the block moves from finalized to trash. One way to determine extra space consumption by logically deleted files is to run a "du" command on the trash directory. > du -hs data/dfs/data/current/BP-1273075337-10.22.2.98-1466102062415/trash 8.0K data/dfs/data/current/BP-1273075337-10.22.2.98-1466102062415/trash Assuming relatively even data distribution across nodes in the cluster, if this shows that a significant proportion of the volume's capacity is consumed by the trash directory, then that is a sign that the unfinalized rolling upgrade is the source of the space consumption. Conclusion Finalize those upgrades!

cnauroth · ‎06-16-2016

@Eric Periard, there is a known issue right now in the way Ambari determines HA status for the NameNodes. Ambari uses a JMX query to each NameNode. The current implementation of that query fetches more data than is strictly necessary for checking HA status, and this can cause delays in processing that query. The symptom of this is that the Ambari UI will misreport the active/standby status of the NameNodes as you described. The problem is intermittent, so a browser refresh is likely to show correct behavior. There is a fix in development now for Ambari to use a lighter-weight JMX query that won't be to prone to this problem. This does not indicate a problem with the health of HDFS. As you noted, users are still able to read and write files. The problem is limited to the reporting of HA status displayed in Ambari.

cnauroth · ‎06-16-2016

@Zack Riesland, the S3N file system will buffer to a local disk area first before flushing data to the S3 bucket. I suspect that depending on the amount of concurrent copy activity happening on the node (number of DistCp mapper tasks actively copying to S3N concurrently), you might hit the limit of available disk space for that buffering. The directory used by S3N for this buffering is configurable via property fs.s3.buffer.dir in core-site.xml. See below for the full specification of that property and its default value. I recommend reviewing this in your cluster to make sure that it's configured to point to a large enough volume to support the workload. You can specify a comma-separated list of multiple paths too if you want to use multiple disks. <property> <name>fs.s3.buffer.dir</name> <value>${hadoop.tmp.dir}/s3</value> <description>Determines where on the local filesystem the s3:/s3n: filesystem should store files before sending them to S3 (or after retrieving them from S3). </description> </property>

cnauroth · ‎06-16-2016

@Payel Datta, thank you for sharing the full stack trace. I expect this will turn out to be some kind of misconfiguration, either of the host network settings or of ZooKeeper's connection configuration. On the ZooKeeper side, I recommend reviewing the zoo.cfg files and the myid files. On each host, the myid file must match up correctly with the address settings in zoo.cfg. For example, on the node with myid=1, look in zoo.cfg for the server.1 settings. Make sure those settings have the correct matching host or IP address. Perhaps the addresses in zoo.cfg do not match correctly with the network interface on the host. If the settings in zoo.cfg refer to a hostname/IP address for which the host does not have a listening network interface, then the bind won't be able to succeed. On the networking side, you might try using basic tools like NetCat to see if it's possible to set up a listening server bound to port 3888. If that succeeds, then it's likely not a host OS networking problem. If it fails though, then that's worth further investigation on the networking side, independent of ZooKeeper.

cnauroth · ‎06-15-2016

@Zack Riesland, yes, there is a -bandwidth option. For full documentation of the available command line options, refer to the Apache documentation on DistCp.

cnauroth · ‎06-15-2016

@Zack Riesland, your understanding of DistCp is correct. It performs a raw byte-by-byte copy from the source to the destination. If that data is compressed ORC at the source, then that's what it will be at the destination too. According to AWS blog posts, Elastic MapReduce does support use of ORC. This is not a scenario I have tested myself though. I'd recommend a quick prototyping end-to-end test to make sure it meets your requirements: DistCp a small ORC data set to S3, and then see if you can query it successfully from EMR.

cnauroth · ‎06-15-2016

@Payel Datta, you won't need to declare the leader explicitly. The ZooKeeper ensemble negotiates a leader node automatically by itself. Do you have more details on that bind exception? Is it possible that something else on the host is already using that port?

cnauroth · ‎06-15-2016

@Zack Riesland, have you considered trying DistCp to copy the raw files from a source hdfs: URI to a destination s3n: or s3a: URI? It's possible this would be able to move the data more quickly than the Hive insert into/select from. If it's still important to have Hive metadata referencing the table at the s3n: or s3a: location, then you could handle that by creating an external table after completion of the DistCp.

cnauroth · ‎06-15-2016

@Thees Gieselmann, here are a few follow-ups. Note that by setting HADOOP_ROOT_LOGGER, you would set this for multiple Hadoop processes, not just the NameNode. I just wanted to make sure that was your goal. The duplication of arguments on the command line is a known bug, which will be fixed in Apache Hadoop 3. You can refer to configs at the path /etc/hadoop/conf. There are symlinks in place there that will redirect to the correct configuration directory based on the currently active HDP version, such as 2.3.4.0-3485. Unfortunately, I don't have any further experience with logstash to address your remaining question. Passing the logger configuration looks right at this point, so perhaps it's time to investigate what logstash itself provides for troubleshooting.

cnauroth · ‎06-14-2016

@Dennis Fridlyand, I'm glad to hear this helped!

cnauroth · ‎06-14-2016

@Dennis Fridlyand, thank you for sharing the mapper and reducer code. I think I've spotted a bug in the reducer that I can help with. protected void map(LongWritable key, BytesWritable value, Mapper<LongWritable, BytesWritable, Text, Text>.Context context) throws IOException, InterruptedException { final String json = new String(value.getBytes(), "UTF-8"); Here, value is an instance of BytesWritable. A BytesWritable is a wrapper over an underlying byte array. That underlying buffer may be reused multiple times to represent different records. The actual length of the data within the buffer that is considered valid is tracked separately using a numeric size field. For more details, please see the source code for BytesWritable. By calling value.getBytes(), the mapper code is accessing the raw underlying buffer. This buffer might still contain trailing data from a previous record. Only the data in the buffer up to the length returned by value.getLength() is truly valid. The recommendation is to switch to using the copyBytes() method, which contains additional logic to copy only the currently valid bytes of the underlying buffer. I recommend making the following change in the mapper code. final String json = new String(value.copyBytes(), "UTF-8"); Would you please try that?

cnauroth · ‎06-14-2016

@Thees Gieselmann, Hadoop daemons may override the setting of hadoop.root.logger at process launch time by passing a -Dhadoop.root.logger argument. hadoop.root.logger=INFO,console,logstash If you have the above setting in log4j.properties, then it acts as the default if the process launch does not pass -Dhadoop.root.logger. However, if the argument is passed, then that argument acts as an override, and the value specified in log4j.properties is ignored. One way to check if this is happening is to look at the process table, such as by running this: ps auxwww | grep NameNode | grep 'hadoop.root.logger' Then, look at the full command line of the NameNode process. If it does not include your logstash appender, then this is the likely explanation for what you are seeing. To change the arguments passed at launch of the NameNode, edit hadoop-env.sh and find the variable HADOOP_NAMENODE_OPTS. Within the value for that environment variable, you can add the setting for -Dhadoop.root.logger=INFO,console,logstash. If you add your logstash appender there and restart the NameNode, then it will pass that argument value down to the new process launch, and I expect it will activate your logstash appender. Also, the more common setting in HDP deployments is INFO,RFA, so it might be more appropriate for you to set the value to -Dhadoop.root.logger=INFO,RFA,logstash.

cnauroth · ‎06-14-2016

@Payel Datta, these exceptions indicate that for this member of the ZooKeeper ensemble, it cannot connect to port 3888 for the other 2 members of the ensemble. 2016-06-14 06:38:38,664 - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@383] - Cannot open channel to 2 at election address zookeeper-2/54.253.26.67:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:795) 2016-06-14 06:38:38,665 - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@383] - Cannot open channel to 3 at election address zookeeper-3/54.66.23.197:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:795) Port 3888 is used for ZooKeeper's leader election protocol. Without a successful connection, the ensemble cannot successfully elect a leader. Note that the message occurs for the connection to both of the other hosts: zookeeper-2/54.253.26.67 and zookeeper-3/54.66.23.197. This warning indicates that client connections were rejected, because the ZooKeeper ensemble is not fully initialized. This is expected behavior if an ensemble cannot elect a leader and complete its initialization. 2016-06-14 06:38:40,477 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running I recommend reviewing ZooKeeper logs from all 3 nodes in the ensemble to try to find root cause. If netstat reports that there is nothing listening on port 3888 on all 3 nodes, then try looking earlier in the logs to see if there was possibly a bind error when ZooKeeper tried to use port 3888. If nothing is easily found in the logs, try restarting all 3 ZooKeeper processes to get a fresh run. That might make it easier to see what is happening when it tries to bind to port 3888.

cnauroth · ‎06-14-2016

@Dennis Fridlyand, would you please also share the code for the ArchiveMergeMapper and ArchiveMergeReducer classes?

cnauroth · ‎06-14-2016

@subacini balakrishnan, HTTP calls to both the NameNode and the DataNode will utilize SSL. Since it utilizes SSL for the data transfer performed with the DataNode, the bytes in transit are encrypted and cannot be read by a man-in-the-middle attacker. The way this works is that the HTTP client first initiates a call to the NameNode using either the "http" or "https" scheme. For a file read or write operation, the NameNode will select an appropriate DataNode and send an HTTP 302 redirect response back to the client telling it to reconnect to that DataNode to complete its request. When the NameNode performs this redirect, it detects the scheme of the incoming call that was sent to it and preserves that scheme in the Location header of the HTTP 302 redirect response. Thus, for a request originating at the NameNode via "http", the redirection will point to an "http" URL on a DataNode, and for a request originating at the NameNode via "https", the redirection will point to an "https" URL on a DataNode.

cnauroth · ‎06-14-2016

@Alexander Yau, the error shown here is caused by a mismatch between the value class configured for the job at job submission time and what the reducer is attempting to write for the job output. The exception text indicates it expects IntWritable, but instead received an instance of MapWritable. java.io.IOException: wrong value class: class org.apache.hadoop.io.MapWritable is not class org.apache.hadoop.io.IntWritable At job submission time, the output class is set to IntWritable. job.setOutputValueClass(IntWritable.class); However, the reducer class parameterizes the output value type to MapWritable. public static class IntSumReducer extends Reducer<Text, IntWritable, Text, MapWritable> { Likewise, the reducer logic writes a MapWritable instance to the context. private MapWritable result = new MapWritable(); ... result.put(myurl, new IntWritable(sum)); context.write(mykey, result); To fix this error, you'll need to set up the job submission and the reducer to use the same output value class. Judging from the description you gave for what you're trying to achieve with this job, it sounds like you want MapWritable for the outputs. Therefore, I recommend testing again with the line of code from the job submission changed to this: job.setOutputValueClass(MapWritable.class);

cnauroth · ‎06-13-2016

@Tom Ellis, you mentioned finding the SaslRpcClient class. That's a very important piece. This is the class that handles SASL authentication for any client-server interaction that uses Hadoop's common RPC framework. The core Hadoop daemons in HDFS and YARN, such as NameNode and ResourceManager, make use of this RPC framework. Many other services throughout the Hadoop ecosystem also use this RPC framework. Clients of those servers will use the SaslRpcClient class as the entry point for SASL negotiation. This is typically performed on connection establishment to a server, such as the first time a Hadoop process attempts an RPC to the NameNode or the ResourceManager. The exact service to use is negotiated between client and server at the beginning of the connection establishment, during the negotiation code that you mentioned finding. The service value will be different per Hadoop daemon, driven by the shortened principal name, e.g. "nn". However, you won't find anything in the Hadoop source code that explicitly references the TGS. Instead, the Hadoop code delegates to the GSS API provided by the JDK for the low-level implementation of the Kerberos protocol, including handling of the TGS. If you're interested in digging into that, the code is visible in the OpenJDK project. Here is a link to the relevant Java package in the OpenJDK 7 tree: http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/f51368baecd9/src/share/classes/sun/security/jgss/krb5 Some of the most relevant classes there would be Krb5InitCredential and Krb5Context.

cnauroth · ‎06-13-2016

@Sumit Nigam, answering your individual questions: 1. No, I expect SmartSense would not be able to analyze an HBase instance deployed via Slider at this time. SmartSense works by constructing a model of the cluster, including its configuration files, and then running a set of rules against those configuration files to generate recommendations. In the case of Slider, the HBase configuration files would reside inside its Slider Application Package, not the typical file system location. I don't believe SmartSense currently is equipped to inspect Slider application packages. @sheetal or @Paul Codding, could you please confirm (or deny) this? 2. SmartSense operates by running a set of rules against captured information of a cluster's configuration, including HDP component configuration files and host OS configuration. It does not perform an exhaustive capture of all logs in the cluster and execute rules against those logs. Some of the cases you described likely would be better served by runtime operational monitoring in Ambari. 3. SmartSense is capable of identifying and recommending use of secure mode and additional security best practices. There is also a rule that checks configured open file limits and makes recommendations if limits are not within an acceptable range.

cnauroth · ‎06-13-2016

@Artem Ervits, the risk of executing as the yarn user relates to several statements from the Apache Hadoop documentation on Secure Mode. Specifically, the section on the NodeManager states the following: For maximum security, this executor sets up restricted permissions and user/group ownership of local files and directories used by the containers such as the shared objects, jars, intermediate files, log files etc. Particularly note that, because of this, except the application owner and NodeManager, no other user can access any of the local files/directories including those localized as part of the distributed cache. Therefore, by executing YARN containers as user "yarn", which is the same as the user running the NodeManager, the container process can get full access to localized file content. This would open a risk of users writing arbitrary application code that scans the local disk looking for localized files that potentially contain sensitive data, or even changing the contents of user-submitted executables to mount a code injection attack. It would also be possible to access files owned by the yarn user on HDFS.

cnauroth · ‎06-13-2016

@ScipioTheYounger, I expect this is similar to another question you asked. https://community.hortonworks.com/questions/35574/switch-namenode-ha-zookeeper-access-from-no-securi.html I'll repeat the same information here for simplicity. change ha.zookeeper.acl in core-site.xml to this: <property> <name>ha.zookeeper.acl</name> <value>sasl:nn:rwcda</value> </property> Then, you'd want to run the following to reformat ZooKeeper for NameNode HA, which would reinitialize the znode used by NameNode HA to coordinate automatic failover. hdfs zkfc -formatZK -force The tricky part, as you noticed, is getting that command to authenticate with SASL. The ZooKeeper and SASL guide in the Apache documentation discusses implementation and configuration of SASL in ZooKeeper in detail. For this particular command, you can use this procedure. First, create a JAAS configuration file at /etc/hadoop/conf/hdfs_jaas.conf: Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/nn.service.keytab" principal="nn/<HOST>@EXAMPLE.COM"; }; Note that the will be different depending on the NameNode hostnames in your environment. Likewise, you'll need to change EXAMPLE.COM to the correct Kerberos realm. Next, edit /etc/hadoop/conf/hadoop-env.sh, and add the following line to enable SASL when running the zkfc command. export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=zookeeper -Djava.security.auth.login.config=/etc/hadoop/conf/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client ${HADOOP_ZKFC_OPTS}" Then, run the "hdfs zkfc -formatZK -force" command.

cnauroth · ‎06-13-2016

@Manoj Dhake , the error you mentioned comes from a Hadoop class named Path. This class is used widely throughout the Hadoop ecosystem to represent a path to a particular file or directory in a file system. Conceptually, it is similar to a URI with some additional validation and normalizing logic specific to the Hadoop ecosystem's expectations. As stated in the exception, it is illegal to attempt to make a Path from an empty string. This looks like something in the call sequence either failed to obtain correct information to create a Path, or possibly dropped information. Is there a longer stack trace that accompanies the error? If so, then I recommend looking at that stack trace in more detail. Depending on the classes and methods referenced in the stack trace, I expect it will pinpoint whether the issue is happening in Hive or Oozie.

Online	Offline
Last Visited	‎01-13-2017 05:20 PM

Date Registered	‎09-29-2015 10:51 PM
Last Visited	‎01-13-2017 05:20 PM
Total Messages Posted	123
Total Kudos Received	215

Re: How to debug the issue "IPC's epoch X is less ...

Re: Why hdfs://mycluster/ different from /

Re: querying a partition table

Re: NameNode HA Ambari Display Issue

Re: Tips for optimizing export to S3(n) ?

Re: HDFS Space Consumption During Rolling Upgrade

Re: Hi, am unable to install HDP via Ambari. Zooke...

Re: How to debug the issue "IPC's epoch X is less ...

Re: Why hdfs://mycluster/ different from /

Re: Why hdfs://mycluster/ different from /

Re: querying a partition table

Re: querying a partition table

Re: Hi, am unable to install HDP via Ambari. Zooke...

Re: NameNode HA Ambari Display Issue

HDFS Space Consumption During Rolling Upgrade

Re: NameNode HA Ambari Display Issue

Re: Tips for optimizing export to S3(n) ?

Re: Hi, am unable to install HDP via Ambari. Zooke...

Re: Tips for optimizing export to S3(n) ?

Re: Tips for optimizing export to S3(n) ?

Re: Hi, am unable to install HDP via Ambari. Zooke...

Re: Tips for optimizing export to S3(n) ?

Re: Configure log4j appender in HDFS to send metri...

Re: Read sequence file in MapReduce

Re: Read sequence file in MapReduce

Re: Configure log4j appender in HDFS to send metri...

Re: Hi, am unable to install HDP via Ambari. Zooke...

Re: Read sequence file in MapReduce

Re: SSL Configurations for HSFTP Sources

Re: class org.apache.hadoop.io.MapWritable is not ...

Re: How does the client know which service ticket ...

Re: SmartSense based troubleshooting for YARN cont...

Re: what are security implications of whitelisting...

Re: Zookeeper kerberos authentication

Re: IllegalArgumentException: Can not create a Pat...