i am trying to remove the hive external data in blob storage using below command and data got delete but i am getting error what is is error mean..can you help me to understand the error and soulution. hdfs dfs -rm -R wasbs://data@datadev.blob.core.windows.net/backup_dwh/ -rm: Fatal internal error java.lang.NullPointerException at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:448) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:2707) at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:1340) at org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:166) at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:109) at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:153) at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:118) at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:297) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:356) ... View more

Hi Team, while i deleting the hdfs data in blob which is having hive paritioned data , i am getting below error but data was deleted successfully..can you please us to understand why i am receiving this error ? hadoop fs -rm -r wasbs://dev001data@datadev.blob.core.windows.net/prodcp/fact_tm_all/ 18/07/09 13:43:34 WARN azure.AzureFileSystemThreadPoolExecutor: Disabling threads for Rename operation as thread count 0 is <= 1 18/07/09 13:47:04 INFO azure.AzureFileSystemThreadPoolExecutor: Time taken for Rename operation is: 210770 ms with threads: 0 -rm: Fatal internal error java.lang.NullPointerException at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:448) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:2707) at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:1340) at org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:166) at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:109) at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:153) at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:118) at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:297) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:356) ... View more

Hi Team, i am facing this issue recursively. Currently i am working on HIVE tables and facing issue with hive partitions ,we have script to drop partitions if exist based on dynamic values and add the new partitions based on new data comes. i have uploaded two files with same partitions with different data in gap of 1 hours, both files are processed successfully and logs showing that partitions are adding drooped properly, but while checking the data in table, data was not there. After some time i have load same files same which i have uploaded last and partitions are properly placed and i could able to see the data in hive. Please help us here how to resolve this issue ? what might be root cause of this issue, we are facing this issue recursively. ... View more

Hi Team, i am using HDP 3.6(hdp 2.6) i have bash script like below after execution this script it return value to next process but even though the script run successfully and executing all stages in script completed successfully it is returning 1, because of this next process (hql script having some business log steps ) is not taking place and if i remove the return statement the next process going smoothly. this problem came after the Hadoop cluster up gradation(HDP 3.4 to 3.6 ) which having Ubuntu 14 and new cluster have Ubuntu 16. rm ~/sqoop/"$TABLE"/* rmdir ~/sqoop/"$TABLE" return $? Can you please help us to understand what is issue and what happening if remove the return command here and what is impact in production if remove the Return statement? ... View more

Hi..Team, I am using HDI 3.4 ..Today i am facing HIVE meta store issue and unable open the HIVE cli , while i manually start the hive meta store .it is not starting and giving below error resource_management.core.exceptions.Fail: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/conf.server ; /usr/hdp/current/hive-metastore/bin/schematool -initSchema -dbType azuredb -userName fsfewrwffsfsfgwerwregdfgw -passWord [PROTECTED] -verbose' returned 1. WARNING: Use "yarn jar" to launch YARN applications. Metastore connection URL: jdbc:sqlserver://zbbprodsqlserver.database.secure.windows.net;database=zbbprodHiveOozie;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=300 Metastore Connection Driver : com.microsoft.sqlserver.jdbc.SQLServerDriver Metastore connection User: fsfewrwffsfsfgwerwregdfgw Exception in thread "main" java.lang.ExceptionInInitializerError at javax.crypto.KeyAgreement.getInstance(KeyAgreement.java:179) at sun.security.ssl.JsseJce.getKeyAgreement(JsseJce.java:287) at sun.security.ssl.JsseJce.isEcAvailable(JsseJce.java:199) at sun.security.ssl.CipherSuite$KeyExchange.isAvailable(CipherSuite.java:378) at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:194) at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:340) at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:298) at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:593) at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:557) at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:109) at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1616) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1401) at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1068) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:904) at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:451) at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1014) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:77) at org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:121) at org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:169) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:272) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:258) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:508) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:94) ... 30 more Caused by: java.lang.SecurityException: Cannot locate policy or framework files! at javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:317) at javax.crypto.JceSecurity.access$000(JceSecurity.java:50) at javax.crypto.JceSecurity$1.run(JceSecurity.java:86) at java.security.AccessController.doPrivileged(Native Method) at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:83) ... View more

Hi i have a data for 2015 to 2018 year i am using hive query to get the data based on below condition but i am getting meta exception Query : select ct_code,date_code from dim_table where substr(date_code,1,4)>=2015 and substr(date_code,1,4)<=2018; FAILED: SemanticException MetaException(message:Exception thrown when executing query) If i give select with out where conditon: select * from dim_table it is working fine. or when i split above query to parts based on years it also working fine. select ct_code,date_code from dim_table where substr(date_code,1,4)>=2015 and substr(date_code,1,4)<=2017 ; select ct_code,date_code from dim_table where substr(date_code,1,4)=2018; But when i give only where substr(date_code,1,4)>=2015 it also not working Can you please help me to understand that why i am getting this issue. ... View more

HI , I am using pig script in tez, in pig script i am storing the output file into some hdfs directory..previously the output of pig script generate like part-r-00000..but now pig script generate output file like part-v000-o000-r-00000..can you please help me to understand what is difference on these two..why it has change ... View more

Hi Team, i am currently using HDI 3.4 cluster for PROD and Dev i got request to copy the all prod hive table from Prod HIVE TO DEV , i am using blob storage to store the data . can you help me how to do in blob? ... View more

job-status.pngHi team, I am using hive with tez in HDP and i am running hive query,it was completed 99.27% and it is stuck in reducer phase, attached the screenshot. hive-log.png job-status.pn mem-space.png same job was done in 30min in last run, please suggest why it is happen and solution for this issue? i have below memory space in linux :swap total used free shared buffers cached Mem: 258041 254721 3320 0 3449 237158 -/+ buffers/cache: 14114 243927 Swap: 49151 660 48491 Please suggest ... View more

Hi , I am new to HD insight cluster..i want to test the connectivity of hive database using Microsoft ODBC driver I have below details given data souce name: ABC HIVE host: current namenode hostname port : 443 database: default hive server type: hive server 2 mechanism: windows azure HDinsight servive username: password: Can you please help me what is username and password here? and any other details if need to change like port number(1000) ... View more