Can you also post application logs for application_1501335737720_0013? ... View more

Can you post the Ambari console log for the operation and application logs for application_1501335737720_0014? ... View more

What is your Sqoop command looks like? Are you using a custom DB query to pull the data and is that syntactically correct? ... View more

Can you check and post what HIVE_AUX_JARS_PATH is set to in /etc/hive/conf/hive-env.sh? ... View more

Running 2 Metastore/HS2 with different config is possible. If your cluster is Ambari managed, you could potentially achieve that by using Ambari Config Groups. ... View more

Are you interacting with an ACID table in non-ACID mode? This can happen, you could try restarting the Hive services after enabling ACID (setting hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager) and then try the query. ... View more

As you mentioned that you want 4 instances of HS2, 2 in binary HA and 2 in http HA, you can can acheive that by registering similar transport protocols with the same zookeeper namespace. So, in case of the two binary HA instances, you can define the configs like this hive.server2.zookeeper.namespace=hiveserver2-binary and for the other two http HA instances hive.server2.zookeeper.namespace=hiveserver2-http Then in the connection URL JDBC clients would provide the appropriate zkNamespace jdbc:hive2://$ZK_URL/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-<protocol> ... View more

Typically beeline -u "<jdbc_url>" -n <username> -w <password>, note the jdbc_url is quoted. Are you sure you did that? ... View more

You need to catch the appropriate exception as the error indicates. Here is an example: try { Class.forName(driverName); } catch (ClassNotFoundException ex) { ex.printStackTrace(); } ... View more

Other than what @Ishan mentioned, multiple HiveServer2 and Metastore are recommended generally in a High Availability setup. This way when one of the services goes down the other service is available for the clients to connect to. Typically edge nodes are client facing nodes, so the client configs and tools are installed on them rather than services. ... View more

Your driverName is set incorrectly, it should be "org.apache.hive.jdbc.HiveDriver". ... View more

Is this a fresh instance? Can you run the following command on the Hive node and post the output? /usr/hdp/current/hive-server2-hive2/bin/schematool -dbType <dbtype> -initSchema -verbose This will definitely throw more light on what is the issue here. ... View more

There are two problems here with the DDL: Use of delimiter with ORC. ORC is in itself a format so you don't have to provide a delimiter. You would define an external table with a location, also ACID tables cannot be external tables, see here. The proper definition for this table would be: CREATE TABLE FIREWALL_LOGS( time STRING, ip STRING, country STRING, status INT ) CLUSTERED BY (time) into 25 buckets STORED AS ORC TBLPROPERTIES("transactional"="true"); ... View more

The most upto date document to follow for configuring would be this one: https://community.hortonworks.com/articles/101181/rowcolumn-level-security-in-sql-for-apache-spark-2.html ... View more

Hive uses the MySQL jdbc connector for issuing queries to MySQL. You can turn on logging through the jdbc connection url (append &profileSQL=true&logger=com.mysql.jdbc.log.StandardLogger) you are using to connect to mysql db in hive-site.xml. To turn it on the MySQL server you need to set the following in the my.cnf. general_log = 1 general_log_file = /path/to/query.log See more information here. ... View more

For using Hive sink the format on the table would be ORC. ... View more

If the goal is to be able to read this data from Hive, then you can use Flume Hive Sink to stream data into Hive table directly. ... View more

desc <tablename> shows the table definition (including column names with the datatype) while select * from <tablename> shows the data in the table. ... View more

You can fetch the stats from within hive. See the following thread: https://community.hortonworks.com/questions/103263/cannot-generate-stats-for-partitioned-hive-table.htm You can use the following command: -- In Hive1 for unpartitioned table SHOW tblproperties <tablename>; -- In Hive1 for partitioned table DESC formatted <tablename> partition (<partitionspec>); -- If using Hive2 on HDP-2.6, following will work on both partitioned and unpartitioned table DESC formatted <tablename>; ... View more

Can you also post namenode logs? ... View more

Did you verify the data by viewing this data using beeline on the cluster? Does it show up correctly lining up under the expected columns when you do a select * from <tablename>. You can run desc [extended|formatted] <tablename> to view the metadata for the table. ... View more

Can you check the following property settings on your cluster? <property> <name>yarn.scheduler.capacity.queue-mappings-override.enable</name> <value>false</value> <description> If a queue mapping is present and override is set to true, it will override the queue value specified by the user. This can be used by administrators to place jobs in queues that are different than the one specified by the user. The default is false - user can specify to a non-default queue. </description> </property> What you desire is true for this parameter. ... View more

Hortonworks doesn't provide a .NET plugin but does provide ODBC driver to connect to HiveServer2 service in Hive. Also look at the following thread for more ideas: https://community.hortonworks.com/questions/345/connecting-to-hadoophive-from-net.html ... View more

Overall information is available across partitions only if you are on HDP-2.6 and using Hive2 which i see you are currently not from your description (1.2.1.2.5). If you are using hive 2 the information would be something like: beeline> desc formatted `test_table`; | Table Parameters: | | | | | COLUMN_STATS_ACCURATE | {\"BASIC_STATS\":\"true\"} | | | numFiles | 3 | | | numRows | 5 | | | rawDataSize | 21 | | | totalSize | 26 | | | transient_lastDdlTime | 1495086467 | In case of Hive 1 unfortunately you will have to run the command per partition and aggregate manually. ... View more

General guidance would be to choose extra large nodes for master nodes and regular for worker nodes. Look at the following articles for detailed guidance: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_cluster-planning/content/ch_hardware-recommendations_chapter.html https://hortonworks.com/blog/deploying-hadoop-cluster-amazon-ec2-hortonworks/ https://community.hortonworks.com/articles/14512/ambari-on-ec2.html You can find non-Ambari manual installation instructions here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/ch_getting_ready_chapter.html ... View more

@Jay SenSharma is right about the logging issue. But that is not the cause of your alert. Basically the metastore alert is failing while running the "show databases" command against the Hive Metastore. You should check whether the server mentioned in the thrift url of hive.metastore.uris is running. Also check the metastore log on that specific host for detailed error. Also on that node check the hive cli log under /tmp/<executing user>/hive.log, where <executing user> might be ambari-qa. ... View more

Hortonworks distribution relies on the Netezza connector provided in the default distribution. SQOOP-846 provided a connector for Netezza in Sqoop-1.4.4. ... View more