Thanks, I checked that information. ... View more

Logs are hiveserver2.log and hivemetastore.log. I'm gonna ask you some questions. First, check whether existing directory of hive table or not in HDFS. As far as I know, do not use param "--hive-overwrite" with "--delete-target-dir". Just remove "--hive-overwrite", and re-run your query. The sqoop job cannot re-create hive table's directory in HDFS normally after removed existing table directory. I think, it is a bug.. Second, check the type matching of columns between Oracle Table and Hive Table. * had to be cast to a less precise type in Hive <- this issue is related with supporting hive datatypes from Oracle, MySQL, PostgreSQL..etc. Such as "Oracle Table : Timestamp - Hive Table - String. ... View more

Can you attach sqoop log and hive service logs? ... View more

No. 840GB, that means a single node has almost 120GB RAM, and it's not ideal way to maintain system. Because each nodes need some free memory for other services such os applications or agents which are using by ambari and etc. Just start 90GB to 100GB, then you can slightly change for that. ... View more

Here's quite a simple solution for these issue cases. issue 1. In case of NameNode/Secondary NameNode, if NameNode Server is fully damaged to disk which is installed NameNode metadata, Secondary NameNode is not convert to NameNode, and you can't fully recover NameNode metadata. issue 2. In case of NameNode/Secondary NameNode, if NameNode service is down, then you'll be unable to execute hadoop MR job or Yarn application or access HDFS Filesystem. Issue 3. In case of NameNode HA(Active/StandBy NameNode), if Active NameNode Server is fully damaged to disk which is installed NameNode metadata, the StandBy NameNode is switched Active NameNode within 30 to 40 sec. And you'll have some time for recovering damaged former NameNode Server. Issue4. In case of NameNode HA(Active/StandBy NameNode), if NameNode service is down, then StandBy NameNode is switched Active NameNode within 30 to 40 sec, and you can use and MR job or YARN application or access HDFS filesystem. ... View more

You should let me know your system memory of your hadoop cluster. If you have three nodes for datanode and nodemanager with 128GB RAM per node, then you can set All YARN containers memory and Min/Max Container Memory from Ambari Web. That depends on the available system memory, preferentially I recommend to set these memory options such as 1024MB or 2048 MB for Min Container Size, 4GB or 8GB or higher for Max Container Size. And All YARN Containers Memory is 90GB ~ 100GB. Of course All YARN Container Memory it depends on the all datanode's available memory for nodemanager. ... View more

Try these configs. hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> -- <value>file:/home/sameer/mydata/hdfs/namenode</value> -> Wrong <value>/data/hdfs/namenode</value> // Do not use "/home" directory path for hdfs service. That because /home/sameer directory permission is 700, and it'll cause permission issues. // And normally many services which are deploying in Linux or Ubuntu are using local file option with "file:///SOME_PATH" </property> <property> <name>dfs.datanode.data.dir</name> -- <value>file:/home/sameer/mydata/hdfs/datanode</value> -> Wrong <value>/data/hdfs/datanode</value> </property> </configuration> core-site.xml <configuration> <property> <name>fs.default.name </name> -- <value> hdfs://localhost:9000 </value> -> Wrong <value> hdfs://NAMENODE_SERVER_FQDN:9000 </value> // fs.default.name is deprecated since that hadoop v2 as I know. // And I'm not recommend using localhost value for hostname. Just use FQDN even if it's a single node. </property> </configuration> yarn-site.xml <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <- Add <name>yarn.web-proxy.address</name> <value>YARN_SERVER_FQDN:8089</value> // Installed resourcemanager's fqdn in server. </property> <property> <- Add <name>yarn.resourcemanager.address</name> <value>YARN_SERVER_FQDN:8032</value> // Installed resourcemanager's fqdn in server. </property> </configuration> mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <- Add this config for jobhistoryserver <value>NAMENODE_SERVER_FQDN:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <- Add this config for jobhistoryserver <value>NAMENODE_SERVER_FQDN:19888</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <- Add this config for jobhistoryserver <value>/mr-history/tmp</value> // It is temporarily directory in HDFS for MR Jobs. </property> <property> <name>mapreduce.jobhistory.done-dir</name> <- Add this config for jobhistoryserver <value>/mr-history/done</value> // It is finished MR Jobs directory in HDFS. </property> </configuration> ... View more

Did you upgrade os from base version with "yum upgrade" command? ... View more

Try this option. .option("quote", ".option("quote", "\u0000") ... View more