Solved

It seems hiveserver2 can't connect to hive metastore,here is my log. 2020-01-31T15:47:46,804 INFO [main]: metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current connections: 02020-01-31T15:47:46,804 INFO [main]: metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI thrift://novalinq-server:90832020-01-31T15:47:46,805 INFO [main]: metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current connections: 12020-01-31T15:47:46,806 INFO [main]: metastore.HiveMetaStoreClient (:()) - Connected to metastore.2020-01-31T15:47:46,814 WARN [main]: metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection. Attempting to reconnect (8 of 24) after 5s. My jbdc is:jdbc:hive2://novalinq-server:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 And the last error hiveserver2 gives me is: Connection failed on host novalinq-server:10000 (Traceback (most recent call last): File "/

Hi All, I have directory structure like below on Unix file system,/DIR1/DIR2/DIR3/DIR4/DIR5   I have files under the directories DIR3 & DIR5. However I want to read files only from DIR5.I tried with various regex but could not get the right one . Could you please help me with the Regex to be used in ListFile processor to read files from DIR5 only.   Thanks in Advance!  

Hello all,   Problem statement: We have 3 layers in our project,for each layer we have  nifi flow. Currently, the layers or nifi flow for each layer is getting executed one after another (sequentially).   Execution flow: layer1->layer2->layer3->  so on.   The trigger point or input parameters to the flow is nothing but number of source files. Which will be processed further in every layer one by one.   I am looking for such solution ,where my next nifi flow for next layer should not get triggered until completion of previous one for previous layer and for all files.   And if current layer got succeeded for all the files then only next one will kick off.   Please suggest me your thought process on the same. I will be very appreciated on your response.   Thank you in advance.    

Hello Team,   Consider i have a client which is negotiating a timeout from the ZK server.   I have observed that 0x36e18413b196cfc  is the session id and i could see that ZK is terminating the connection after 23,419 seconds ??? Could anyone suggest why ??      2020-01-30 01:03:23,762 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x36e18413b196cfc at /xx.xx.xx.xx:379522020-01-30 01:03:23,762 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x36e18413b196cfc2020-01-30 01:03:23,762 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@617] - Established session 0x36e18413b196cfc with negotiated timeout 10000 for client /xx.xx.xx.xx:37952EndOfStreamException: Unable to read additional data from client sessionid 0x36e18413b196cfc, likely client has closed socket2020-01-30 07:33:42,962 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@103

Hi,I have a CDH 6.1 cluster which has < 100 nodes(free Cloudera Express) . I'm thinking about extending the cluster to > 100 nodes.I've found this link:https://www.cloudera.com/content/dam/www/marketing/resources/datasheets/cloudera-enterprise-datasheet.pdf.landing.html Unfortunately it seems like the enterprise license is not only for CDH but for all Cloudera products(CDP, Data Hub, HDP Enterprise). What's the cost per node of a CDH Cloudera Express cluster? How is it related to this pricing?:https://www.cloudera.com/products/pricing.htmlRegards

Hi all, Anyone can help me to Access HDFS directory from  windows machine ? I want to backup folder and file from hdfs to my Windows 10 Machine. I am now using HDP 2.7.4.0 Thank You.

Hello everyone, I'm running HUE version 4. I can create workflow and schedule it as well just fine, but only if the account is in "SUPERUSER" mode. i.e. under user profile, if the user has all the role permissions but no superuser is checked under user profile, no Scheduler (Workflow, Scheduling, Bundle) is shown at all in the menu. I want to grant a standard user the workflow/scheduling creating option but I don't want it to be a superuser, can I accomplish this somehow? Please guide me, thanks.   

Hi , We recently upgrade our DEV and PROD environments from HDF 3.3 to HDF 3.4.1.1 after following upgrade instructions on a HDF-only cluster. After the upgrade, schema registry is giving the following exceptions in both environments for clients using the confluent API:ERROR [2019-07-30 11:17:42.014] [dw-94 - POST /api/v1/confluent/subjects/EgressLiveAllCXXX1-value/versions] c.h.r.s.w.ConfluentSchemaRegistryCompatibleResource - Encountered error while adding subject [EgressLiveAllCowBehavior1-value]java.lang.RuntimeException: Failed to obtain write lock : schema_metadata_info-EgressLiveAllCXXXX1-value in 120 sec at com.hortonworks.registries.schemaregistry.DefaultSchemaRegistry.lockSchemaMetadata(DefaultSchemaRegistry.java:520) at com.hortonworks.registries.schemaregistry.DefaultSchemaRegistry.addSchemaVersion(DefaultSchemaRegistry.java:489) at com.hortonworks.registries.schemaregistry.webservice.ConfluentSchemaRegistryCompatibleResource.lambda$registerSchemaVers

I use the Impala Shell to create the Kudu table, but it need to insert TBLPROPERTIES ('kudu.master_addresses' = '...'); in the CREATE TABLE commands, and I tried to replace the kudu.master_addresses parameter values as follows:quickstart.cloudera:7051quickstart.cloudera:21000localhost:705110.0.2.15:7051But I still get a RPC connection error message. How do I modify the parameter value or take other solutions?(“Virtual machine working environment”, “Impala Shell commands and error messages” are shown as follows) Virtual machine working environment: Virtual Machine: Cloudera Quickstart Virtual Machine (Using Oracle VM Virtualbox, download URL: https://www.cloudera.com/downloads/quickstart_vms/5-13.html)(1) Version: 5.13.0-0(2) Number of processors: 2(3) Controller: IDE Controller(4) Memory space: 5120 MB(5) Hard drive: 64.00 GB storage, solid state drive(6) OS: Red Hat, 64-bit(7) MySQL 5.1.73(8) HBase 1.2.0-cdh5.13.0(9) Hadoop 2.6.0-cdh5.13.0(10) Impala 2.10.0-cdh5.13.0 Impala

Dear community I have installed a hadoop cluster on 8 servers using Ambari Hortonworks.I am able to access webhdfs using the ip address and the default port 50070 without authentication. How can I secure Webhdfs? P.S I did not enable using kerberos in Ambari > Enable kerberos , should I do it? Any suggestion will be appreciatedThanksAsma

hi all   how to verify if specific service required restart?   for example - HDFS service   in case HDFS service required restart then ambari will show this as     is it possible by rest api to understand if service required restart or not?    for example for - YARN / HIVE , etc

Hi, I configured HDFS HA on our HDP 3.1 cluster and since then the Spark2 Thrift server fails to start. Our Spark setup still uses the Hive metastore (MySql database) I did the HA configuration process according to the HDP / Cloudera documentation through Ambari and also ran   hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://<nameservice-id>/apps/hive/warehouse hdfs://<namenode>:8020/apps/hive/warehouse   and   hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://<nameservice-id>/apps/spark/warehouse hdfs://<namenode>:8020/apps/spark/warehouse    I suspect that the new namenode-id is still missing in some configuration entry somewhere. The file /var/log/spark2/spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1...log contains the following exceptions:   20/01/31 11:27:13 INFO HiveClientImpl: Warehouse location for Hive client (version 3.0.0) is /apps/

Hey ! I had this issue with Apache metron Enrichment can't find enrichment_list table in HBase. Below is the full exception from Ambari: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_master.py", line 118, in <module> Enrichment().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_master.py", line 73, in start commands.load_enrichment_coprocessor() File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_commands.py", line 235, in load_enrichment_coprocessor Execute(command, user=self.__params.metron_user, tries=1, logoutput=True) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-pack

Hello everyone,    I'm a newbie to the world of Apache metron. And I'm facing an issue with Apache storm topology. At first i wanted to test a simple syslog RFC-5424 to start my journey with metron. First, I've created a kafka topic: syslog5424 and then created syslog5424.json parser file in zookeeper, and another one for indexing. Afterwards i reloaded the zookper configuration with this command: ./zk_load_configs.sh -i ../config/zookeeper -m PUSH -z mdr-1:2181 Then i started the parser topology through the management UI, and then i followed the instruction for defining an index in elastic search with a template. this is my reference documentation that i'm following by changing the squid into syslog: https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry   And in Storm UI I keep getting this error msg, without showing any data in Kibana:   2020-01-26 17:11:05.949 o.a.k.c.p.KafkaProducer T

i have XML files about 1-2 GB i want it to convert into CSV format using apache nifi.so that i can process CSV for joining the two csv.

Hi All,  I am tyring to execute update query through putSQL processor. I want to update number of processed file count column of one of the transactional table in mysql.  The issue is that whenever  multiple flowfiles coming as input to putSQL processor at a time,the processed file count is getting incremented and updated only once. Ideally,the behaviour should be like increment the counter one by one for each flowfile and then update the processed file count column in MySQL table.   NiFi flow: Fetch processed file count using select query->updateattribute : processed file count+1->putSQL:update query.   Thanks in advance Please share you response on it.

\ZooKeeper Server Process Connection failed: Expected response imok, Actual response ruok is not executed because it is not in the whitelist. to XXXXXXXXXXX:2181 when i start zookeeper on ambari its successfully stated but service is stopped immediately. its showing above error /

We have ambari cluster we have issue with ambari server example ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting. ambari server create the PID and service is up but server not listening on port 8080 we check the firewall and iptables but this services are down we check the PostgreSQL service ( is running well ) systemctl status postgresql.service● postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/postgresql.service;

Hi   All,   I have used three process groups in nifi (pg1,pg2,pg3). There is an output port called as pg1_Out coming out from pg1. This port is connected to pg2_in and pg3_in input ports.   pg1_out------>pg2_in   |   |   | pg3_in   Problem : when i trigger these process groups then flowfiles are passing to only pg3_in but not to pg2_in. Ideally , the flow files should get passed to both input port. I do not understand why am i seeing such behaviour in nifi.   Does anyone has faced such problem before? Please respose to me your thoughts on this. I would be really appreciated to your help.    

when i was fire command  ->  sudo ambari-server start show error   Using python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logsudo ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start.........Unable to determine server PID. Retrying.........Unable to determine server PID. Retrying.........Unable to

Ambari UI is running slow. Currently, the cluster size is 6 nodes 

Caused by: java.sql.SQLException: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:134) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper$1.run(HiveConnectionWrapper.java:78) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper$1.run(HiveConnectionWrapper.java:75) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper.connect(HiveConnectionWrapper.java:75) ... 98 more Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to rea

Hello, I am just started to play with CDH and I really love the VM with QuickStarts for CDH 5.13 since everything is ready download. However, I noticed that some components are out of date and I woud like to test external import feature from HUE. Could I ask you to provide the quickest way to upgrade the HUE on QuickStarts for CDH 5.13 VM to the newest version?   Best regards

We want to use NIFI as workflow in CDH and we didn't see any support for NIFI in CDH documents. Do you have a NIFI support on Enterprise CDH? Do you have any plan to add NIFI in CDH?  

I have a query where as per the requirement the column name should be 'abc xyz?'. I am using tilted quotes to accommodate the space and special characters in between column name but i am getting the below error with '?':   Parameters cannot be used with normal Statement objects, use PreparedStatements instead.   I am using SQL Workbench for connecting to Hive.