Solved

Hi, I'm having a time out issue when trying to perform and insert operation into HIVE using the ODBC driver (v2.6.1.1001 (64 bit) Windows) as a linked server in SQL Server. When I perfrom a SELECT query using OPENQUERY function, the driver will wait till execution is complete no matter how many minutes it take. However, when I perform an INSERT query, the driver always return a timeout message when the execution time exceeds 30 seconds. I set all timeout parameters to 0, however, the issue persists. Is there a way to make INSERTS taking longer than 30 seconds succeed? From HUE, the inserts work properly and succeed after around 1:30 minutes. Appreciate your insights.------------- Below is a sample trace log of the driver when executing the insert statement:May 05 16:57:03.394 TRACE 9468 HardyTCLIServiceThreadSafeClient::ExecuteStatement: +++++ profile leave +++++May 05 16:57:03.394 TRACE 9468 HardyTCLIServiceClient::CheckCancel: +++++ enter +++++

Hello HW community,I have a problem with my HiveMetastore & Hiveserver2 dont start at all due to the database connexion with the jdbc driver.i proprelly executed the "https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-administration/content/using_hive_with_mysql.html " i have the 3 hive users on mysql, i created the hive database and linked the mysql-connector-java.jar to my ambari-server. I tried some solution kind of replace the "localhost" in the Database URL by the host name or the ip address too, but same error.PS: all the others services are turning.My error is :stderr: Traceback (most recent call last):  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 203, in <module>    HiveMetastore().execute()  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute    method(env)  File "/var/lib/ambari-agent/cache/common

Dear all,I have a flow in NiFi as the photo shows.I'm getting data from a URL and trying to put it in Hbase in a table that I made through hbase shell.here is my puthbasejson config :hbase-site.xml is already existing in the path that I gave to the hbase controller.I do not know what is wrong.I have sandbox HDP that I activated HDF on it using CDA in virtualbox and I have allocated around 12G to my VM and I allocated virtual memory to my system from my disk in my computer. My computer is 16G. Hbase, HDFS, NIFI are all active in ambari.Thanks,Best regards,

Hi All. We have an issue with cloudera navigator UI on  6.2 manual installation ..Background.. Installed the navigator audit and metadat server roles on a cluster backed by external postgresql database CDH is using auto_TLS Navigator metadata and audit roles are status green on the cloudera manager UI   However when i browse to the navigator UI we get a status 503 service unavailable message.   we have confirmed the navigator role is up and running  [root@cmhdgateway01 hue]# lsof -i:7187COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEjava 12766 cloudera-scm 532u IPv4 814278 0t0 TCP *:7187 (LISTEN)python2.7 17534 hue 31u IPv4 955050 0t0 TCP cmhdgateway01.cso.ie:50144->cmhdgateway01.cso.ie:7187 (CLOSE_WAIT)[root@cmhdgateway01 hue]#   The following is captured on the startuplogs of navigator 2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.specification.version=1.82019-05-09 12:14:42,970 INFO com.cloudera.nav.ac

Hi Everyone I use Hortonwork 3.1.1 on Centos 7, everything start nomarlly after install, but yesterday service Yarn and Mapreduce stop, i try to restart but after few second it automatically stop. Please help me ! Here is Log on /var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log2019-04-27 15:57:23,150 INFO resourcemanager.RMAppManager$ApplicationSummary: appId=application_1554293667897_0131,name=JavaHBaseDistributedScan demo_kafka,user=hbase,queue=default,state=FINISHED,trackingUrl=http://bigdata-01.am.local:8088/proxy/application_1554293667897_0131/,appMasterHost=N/A,submitTime=1555412153616,startTime=1555412153617,finishTime=1555412160200,finalStatus=SUCCEEDED,memorySeconds=18035,vcoreSeconds=10,preemptedMemorySeconds=18035,preemptedVcoreSeconds=10,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=SPARK,resourceSeconds=18035 MB-seconds\, 10 vcore-seconds,preemptedResourceSeconds=18035 MB-seconds\, 10 vcore-seconds

Note:  I purposefully made the title of this post the same as a 2015 post.  This is not a duplicate, the information in the old post seems to no longer be valid. Hi, I want to check which versions of every hadoop component (i.e. which version of Hive, HDFS, HBase etc.) come in newest CDH, but I can't seem to find it. I looked into release notes, but no luck, is it written down anywhere maybe? The link (http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh_package_tarball.html) provided here (https://community.cloudera.com/t5/CDH-Manual-Installation/Where-to-find-which-versions-of-components-are-in-newest-CDH/m-p/24091/highlight/true#M627) no longer works and I cannot find this information for either CDH6 or CDS.   Can someone provide updated links? Topics:CDH ManifestCDH InventoryCDH ArchitectureCDS ManifestCDS InventoryCDS Architecture Thanks.

I was waiting for the new changes to the certifications by Hortonworks to be completed in order register for HCA. But now I do not see the option to purchase the HCA certification. It says 'Currently Unavailable' even after the 31 December 2018 which was supposed to be the date after which the exam was supposed to be active again. Why is this and when will this be made available, if at all it is going to be. Or Is this going to be removed from the certifications altogether? The other certifications are available for purchase. And do any of the other certifications require HCA as a pre-requisite? ThanksNanda

In HDP Sandbox 2.6.4 I imported mysql to hdfs but when I tried to import from mysql to hive with the following command:[maria_dev@sandbox-hdp ~]$ sqoop import --connect jdbc:mysql://sandbox-hdp.hortonworks.com/azhadoop --username root --password hadoop --query 'select * from iris_mysql WHERE $CONDITIONS' --m 1 --hive-import --hive-table azhadoop.iris_hive --target-dir /tmp/hive_tempBut I got this error:19/04/27 14:22:19 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@4b8ee4de is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@4b8ee4de is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that y

I'm facing this issue when try to using GPU on YARN:Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested resource type=[yarn.io/gpu] < 0 or greater than maximum allowed allocation. Requested resource=<memory:3072, vCores:1, yarn.io/gpu: 1>, maximum allowed allocation=<memory:9216, vCores:9>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:9216, vCores:9, yarn.io/gpu: 9223372036854775807>I already enabled GPU on my cluster but some how, it still showing that the (without yarn.io/gpu) maximum allowed allocation=<memory:9216, vCores:9>

We are testing our first cluster and appear to have failing Spark jobs due to lack of disk space while the job is running. Filesystem                             Size  Used Avail Use% Mounted on/dev/mapper/centos_usnj1cldrmst1-root   50G   49G  1.5G  98% /devtmpfs                                24G     0   24G   0% /devtmpfs                                   24G     0   24G   0% /dev/shmtmpfs   &n

Hi Cloudera Community ,  How i can change  the compression codec of hive at runtime. I'm reading some table on avro format compressed with snappy and i'm triying to write a similiar table compressed on snappy but the result is compressed on "deflate", after try with multiple options the resulting files were compressed with the same codec.  Can you help me to identify my issue on the following sentences, or what can i do to define the compression codec of hive at runtime. "set hive.exec.compress.output=true;SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;SET mapred.output.compression.type=BLOCK;SET hive.exec.dynamic.partition.mode=nonstrict;  CREATE external table IF NOT EXISTS tableX partitioned by (year Int)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerO

Hi, i am new to hadoop, just start exploring in Zeppelin Hive. I had tried to create temporarily table as simple as below%jdbc(hive)CREATE TEMPORARY TABLE tmp1 (c1 string);Then I logout and re-login, I still able to view the table. How to close the HIVE session properly?

Hello,   had someone a Solr (Clouder Search) configuration running that index all of the Cloudera logs? Is this possible? I would like to have a better log analyzing and alerting than manual checking in Cloudera Manager...   Regards Oliver

I double that, but I check here anyway. Our system is on CDH-5.15.2. The resource manager and job scheduler is YARN:$ yarn versionHadoop 2.6.0-cdh5.15.2Subversion http://github.com/cloudera/hadoop -r c97bcbf0cba923467d45f5519b1953f436c64f12Compiled by jenkins on 2018-11-13T13:53ZCompiled with protoc 2.5.0From source with checksum 9d2d5b887383c7d4b811372f867c6440This command was run using /opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/jars/hadoop-common-2.6.0-cdh5.15.2.jar There are two users, they want isolated environment to run some experiments. I wonder if we can reserve two nodes for their use.  Any suggestions and discussion are very welcome.  Best Regards,Vincent

Hive partition – I have set 10 partition in a table, how can I reduce it to 6 partition without dropping table from HDFS?

Hi, I am trying to figure out how to run a sample Apache Hive query using Apache Oozie, but I'm not sure how to set it up or what the best approach is. Could someone guide me on how to do this?

How can I configure and setup a load balancer for my HUE and OOZIE services?

We have a data flow wherein standardized transaction logs are generated as xml files in a particular directory in VM. We want to publish these logs to a Kafka topic. Following NiFi/MiNiFi flow is what we are using:ListFile ---> FetchFile (completion strategy - Move File) ---> PublishKafkaEach transaction translates to single xml file, at peak hours, over a million files are getting generated per hour, i.e. approximately 300 files per second. Our goal is achieve the above flow using MiNiFi.ListFile uses two listing strategies, tracking timestamps and tracking entities. Initially data flow was created with by selecting tracking timestamps option. Here we observed that several files were not picked by NiFi / MiNiFi. The files that were picked were eventually moved to a different directory but some files were not picked. To put a number to this, 2% - 5% of files were not picked (thus not published to Kafka topic). The behavior was observed in both NiFi and corresponding YAML file in

Good Afternoon,I am creating an apache nifi flow in which I want to flow files from Oracle DB to IBM/DB2, I have completed that part of the flow. I need help with configuring a way to reconcile failed files, I want to know exactly which files failed and the error which caused it. I'm considering using a PUTHDFS to put the failed files into a HIVE TABLE that way I can capture the flow file error. I would then re-enter the failure files back into the flow via GETHDFS.So, I'm curious is this possible? Also is there a mechanism in NiFi to identify the singular file that fails in a batch?

Hello,I am new in Sandbox HDP, I am trying to connect to Hive using JDBC, I used the JDBC URL given by Ambari, and here is my code:private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";         public static void main(String[] args) throws SQLException {        try {            Class.forName(driverName);        } catch (ClassNotFoundException e) {            e.printStackTrace();            System.out.println(e.getMessage());        }        Connection con = DriverManager.getConnection(            "jdbc:hive2://sandbox-hdp.hortonworks.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2");}However, I got this error:Exception in thread "main" java.sql.SQLException: Invalid URL: jdbc:hi

  Dear all,   I got the following error in my HDFS datanodes. However , kinit with the keytab files that generated by the CM is fine and java security policy has been applied.   Is there anyway to diagnose such a problem?   For more information: no LDAP is set in the cluster nodes. cloudera manager and CDH version 5.13.2.   javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757) at java.security.AccessCont

Hi everyone, please help!I'm using ListSFTP processor to get files and process them. In the first execution the processor take the files normally, but when you start the processor the second time it doesn't. I know this is the correct behavior for this processor. Is there any way to start and initialize the processor via NIFI API in such a way that It takes the files again? I need this approach because I need to do testing. Thanks!

Hi everyone,    Hoping someone can help.    I'm getting this error when impalad connects:    RPC error: Client for <hostname>:23000 hits an unexpected exception authorize: cannot authorize peer, type: N6apache6thrift9transport13TSSLException   I get a similar error when SSL is disabled.    the connect/disconnect cycle repeats about every 1/2 second forever.   The log does include a message that the cert was successfully loaded so I don't think it's a problem with that, but I'm open to any suggestions.   Is the type: noted above literally referring to a connection to an HBase thrift server?    Anyone know more about this specific error message?    Thanks!    

I have created a NIFI template. After that I have made few changes in template, now im not understanding hwo to save the template with changes included.

Accessing hive metastore from pyspark on a kerberized cluster throws an error ....Py4JJavaError: An error occurred while calling o27.sql.: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException; at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106) at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114) at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102) at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141) at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136) at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55) at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$