Q&A

Hi , We recently upgrade our DEV and PROD environments from HDF 3.3 to HDF 3.4.1.1 after following upgrade instructions on a HDF-only cluster. After the upgrade, schema registry is giving the following exceptions in both environments for clients using the confluent API:ERROR [2019-07-30 11:17:42.014] [dw-94 - POST /api/v1/confluent/subjects/EgressLiveAllCXXX1-value/versions] c.h.r.s.w.ConfluentSchemaRegistryCompatibleResource - Encountered error while adding subject [EgressLiveAllCowBehavior1-value]java.lang.RuntimeException: Failed to obtain write lock : schema_metadata_info-EgressLiveAllCXXXX1-value in 120 sec at com.hortonworks.registries.schemaregistry.DefaultSchemaRegistry.lockSchemaMetadata(DefaultSchemaRegistry.java:520) at com.hortonworks.registries.schemaregistry.DefaultSchemaRegistry.addSchemaVersion(DefaultSchemaRegistry.java:489) at com.hortonworks.registries.schemaregistry.webservice.ConfluentSchemaRegistryCompatibleResource.lambda$registerSchemaVers

I use the Impala Shell to create the Kudu table, but it need to insert TBLPROPERTIES ('kudu.master_addresses' = '...'); in the CREATE TABLE commands, and I tried to replace the kudu.master_addresses parameter values as follows:quickstart.cloudera:7051quickstart.cloudera:21000localhost:705110.0.2.15:7051But I still get a RPC connection error message. How do I modify the parameter value or take other solutions?(“Virtual machine working environment”, “Impala Shell commands and error messages” are shown as follows) Virtual machine working environment: Virtual Machine: Cloudera Quickstart Virtual Machine (Using Oracle VM Virtualbox, download URL: https://www.cloudera.com/downloads/quickstart_vms/5-13.html)(1) Version: 5.13.0-0(2) Number of processors: 2(3) Controller: IDE Controller(4) Memory space: 5120 MB(5) Hard drive: 64.00 GB storage, solid state drive(6) OS: Red Hat, 64-bit(7) MySQL 5.1.73(8) HBase 1.2.0-cdh5.13.0(9) Hadoop 2.6.0-cdh5.13.0(10) Impala 2.10.0-cdh5.13.0 Impala

Dear community I have installed a hadoop cluster on 8 servers using Ambari Hortonworks.I am able to access webhdfs using the ip address and the default port 50070 without authentication. How can I secure Webhdfs? P.S I did not enable using kerberos in Ambari > Enable kerberos , should I do it? Any suggestion will be appreciatedThanksAsma

hi all   how to verify if specific service required restart?   for example - HDFS service   in case HDFS service required restart then ambari will show this as     is it possible by rest api to understand if service required restart or not?    for example for - YARN / HIVE , etc

Hi, I configured HDFS HA on our HDP 3.1 cluster and since then the Spark2 Thrift server fails to start. Our Spark setup still uses the Hive metastore (MySql database) I did the HA configuration process according to the HDP / Cloudera documentation through Ambari and also ran   hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://<nameservice-id>/apps/hive/warehouse hdfs://<namenode>:8020/apps/hive/warehouse   and   hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://<nameservice-id>/apps/spark/warehouse hdfs://<namenode>:8020/apps/spark/warehouse    I suspect that the new namenode-id is still missing in some configuration entry somewhere. The file /var/log/spark2/spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1...log contains the following exceptions:   20/01/31 11:27:13 INFO HiveClientImpl: Warehouse location for Hive client (version 3.0.0) is /apps/

Hey ! I had this issue with Apache metron Enrichment can't find enrichment_list table in HBase. Below is the full exception from Ambari: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_master.py", line 118, in <module> Enrichment().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_master.py", line 73, in start commands.load_enrichment_coprocessor() File "/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/enrichment_commands.py", line 235, in load_enrichment_coprocessor Execute(command, user=self.__params.metron_user, tries=1, logoutput=True) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-pack

Hello everyone,    I'm a newbie to the world of Apache metron. And I'm facing an issue with Apache storm topology. At first i wanted to test a simple syslog RFC-5424 to start my journey with metron. First, I've created a kafka topic: syslog5424 and then created syslog5424.json parser file in zookeeper, and another one for indexing. Afterwards i reloaded the zookper configuration with this command: ./zk_load_configs.sh -i ../config/zookeeper -m PUSH -z mdr-1:2181 Then i started the parser topology through the management UI, and then i followed the instruction for defining an index in elastic search with a template. this is my reference documentation that i'm following by changing the squid into syslog: https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry   And in Storm UI I keep getting this error msg, without showing any data in Kibana:   2020-01-26 17:11:05.949 o.a.k.c.p.KafkaProducer T

i have XML files about 1-2 GB i want it to convert into CSV format using apache nifi.so that i can process CSV for joining the two csv.

Hi All,  I am tyring to execute update query through putSQL processor. I want to update number of processed file count column of one of the transactional table in mysql.  The issue is that whenever  multiple flowfiles coming as input to putSQL processor at a time,the processed file count is getting incremented and updated only once. Ideally,the behaviour should be like increment the counter one by one for each flowfile and then update the processed file count column in MySQL table.   NiFi flow: Fetch processed file count using select query->updateattribute : processed file count+1->putSQL:update query.   Thanks in advance Please share you response on it.

\ZooKeeper Server Process Connection failed: Expected response imok, Actual response ruok is not executed because it is not in the whitelist. to XXXXXXXXXXX:2181 when i start zookeeper on ambari its successfully stated but service is stopped immediately. its showing above error /

We have ambari cluster we have issue with ambari server example ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting. ambari server create the PID and service is up but server not listening on port 8080 we check the firewall and iptables but this services are down we check the PostgreSQL service ( is running well ) systemctl status postgresql.service● postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/postgresql.service;

Hi   All,   I have used three process groups in nifi (pg1,pg2,pg3). There is an output port called as pg1_Out coming out from pg1. This port is connected to pg2_in and pg3_in input ports.   pg1_out------>pg2_in   |   |   | pg3_in   Problem : when i trigger these process groups then flowfiles are passing to only pg3_in but not to pg2_in. Ideally , the flow files should get passed to both input port. I do not understand why am i seeing such behaviour in nifi.   Does anyone has faced such problem before? Please respose to me your thoughts on this. I would be really appreciated to your help.    

when i was fire command  ->  sudo ambari-server start show error   Using python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logsudo ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start.........Unable to determine server PID. Retrying.........Unable to determine server PID. Retrying.........Unable to

Ambari UI is running slow. Currently, the cluster size is 6 nodes 

Caused by: java.sql.SQLException: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:134) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper$1.run(HiveConnectionWrapper.java:78) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper$1.run(HiveConnectionWrapper.java:75) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.ambari.view.hive20.internal.HiveConnectionWrapper.connect(HiveConnectionWrapper.java:75) ... 98 more Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to rea

Hello, I am just started to play with CDH and I really love the VM with QuickStarts for CDH 5.13 since everything is ready download. However, I noticed that some components are out of date and I woud like to test external import feature from HUE. Could I ask you to provide the quickest way to upgrade the HUE on QuickStarts for CDH 5.13 VM to the newest version?   Best regards

We want to use NIFI as workflow in CDH and we didn't see any support for NIFI in CDH documents. Do you have a NIFI support on Enterprise CDH? Do you have any plan to add NIFI in CDH?  

I have a query where as per the requirement the column name should be 'abc xyz?'. I am using tilted quotes to accommodate the space and special characters in between column name but i am getting the below error with '?':   Parameters cannot be used with normal Statement objects, use PreparedStatements instead.   I am using SQL Workbench for connecting to Hive. 

I use Windows 10 Pro. With Docker, I try to start HDP-Sandbox 2.6.5.docker container start sandbox-hdpworks fine but when I try to start sandbox-proxy:docker container start sandbox-proxyI get the following error:Error response from daemon: driver failed programming external connectivity on endpoint sandbox-proxy (26aa3064fa2cf790e2f529ed1c865944013893b8f14259ef5109407b5b95e08c): Error starting userland proxy: Bind for 0.0.0.0:50111: unexpected error Permission denied Error: failed to start containers: sandbox-proxy

  - /grid/1/- /grid/2/- /grid/3/- /grid/4/- /grid/5/ Attached image where I don't know how to add those routes if someone can help me. Greetings

we can verify the HDFS version on hadoop cluster   as the following:   su hdfs hdfs version     but since hdfs client also installed on all data-nodes machines   we need to verify also the hdfs client version   any cli / command / binaries files , that can captured the hdfs client version ?

How to add user in kerberized cluster? I have AD integration with kerberos? Do we need to add user on OS level ? @Shelton

Hello,   I want to ask what is the proper way to define the terms in Gettwitter processor terms to filter. I'm trying to retrieve tweets that include the word bt by just writing bt in terms to filter however i'm getting tweets that include the two letters bt in a word such as bt*** or ****bt or ***bt***.   Here is a screenshot of the processor  and this is a sample tweet retrieved by the processor        I just need tweets with the two letters word bt. I tried to use single quote like 'bt' and double quote "bt" but i'm not getting any data     Thank you

Hi, Is it possible to attempt CCA 175 only with python?

how to read csv have comma in cell and new line character as well . like i have somme columns with description (hi,how are you).  i am trying to read with spark in scala . please guide