I am trying to use distcp to copy data across two cluster. Please look at the below error, any help? hadoop distcp <source FS>/tmp <target FS>/tmp/distcp-test 18/08/05 06:58:40 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://hazr0217a3f2a44.cloud.wal-mart.com:8020/tmp], targetPath=hdfs://hazr02268026510.cloud.wal-mart.com:8020/tmp/distcp-test, targetPathExists=true, filtersFile='null', verboseLog=false} 18/08/05 06:58:40 INFO client.RMProxy: Connecting to ResourceManager at <ResourceManager> 18/08/05 06:58:40 INFO client.AHSProxy: Connecting to Application History server at <History server> ^C18/08/05 06:58:55 ERROR hdfs.DFSClient: Failed to close inode 19745 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hdfs/.staging/_distcp363233272/fileList.seq (inode 19745): File does not exist. Holder DFSClient_NONMAPREDUCE_1862896603_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3693) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3781) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3748) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:912) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:549) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554) at org.apache.hadoop.ipc.Client.call(Client.java:1498) at org.apache.hadoop.ipc.Client.call(Client.java:1398) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:503) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185) at com.sun.proxy.$Proxy11.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2496) at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2472) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2437) at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:949) at org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:981) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ... View more

We are using Hortonworks ODBC driver to connect hive from C# application. Here is a sample code to fire query on Hive. static void Main(string[] args) { string connectionString = "DSN=AzureHDP"; string query = "SELECT * FROM schema_5218.demandquantity"; int queryTimeout = 100; try { using (var connection = new OdbcConnection(connectionString)) { Console.WriteLine("Opening Hive connection."); connection.Open(); using (var cmd = connection.CreateCommand()) { cmd.CommandTimeout = queryTimeout; cmd.CommandText = query; Console.WriteLine("Executing Hive query."); cmd.ExecuteNonQuery(); Console.WriteLine("Hive query executed successfully."); Console.ReadKey(false); } } } catch (Exception e) { Console.WriteLine($"Encountered error during Hive query execution. [{e.Message}]"); Console.ReadKey(false); throw; } } Issue is that if the network connection drops after the query has been fired on Hive(before query returns) the resulting exception results in application crash. Reason is that AccessViolcationException is thrown which are not sent to application catch block. So this exception becomes an un-handled exception causing the application to crash. Is this due to any bug in the ODBC driver? Is there any workaround or fix awaited? There are certain settings which can be done to catch such exception but those are not recommended by Microsoft. Due this this reason I am not very keep to use these settings. Those settings are [HandleProcessCorruptedStateExceptions] & [SecurityCritical] and legacyCorruptedStateExceptionsPolicy. Attaching actual application code screenshot. You can that the exception is thrown at the close of the connection using clause and the control does not go to the catch block. Hortonworks Driver Version: 2.01.10.1014 .Net Version: 4.7 C# version: 7.3 ... View more

I am trying to deploy HDP cluster on Azure which uses Azure Data Lake storage. I followed the instructions here for Ambari and cluster setup. I followed the instruction here for ADLS access. When is try to start Hive I get the following error. <code>Exception in thread "main" java.lang.RuntimeException: java.io.IOException: The ownership on the staging directory adl://home/tmp/hive/root/_tez_session_dir/c96088dd-444e-42e3-9293-251656b01b17 is not as expected. It is owned by <ADLS Service Principal>. The directory must be owned by the submitter root or by root at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) I am able to access ADLS via the HDFS commands Permission on /tmp folder > hdfs dfs -ls /tmp Found 1 items drwxrwxrwt+ - <ADLS Service Principal> 0 2018-01-20 21:29 /tmp/hive TIA <br> ... View more