Q&A

hi allwe prepare to upgrade our cluster we have HDP version - 2.6.4 and ambari - 2.6.1we want to upgrade to HDP 2.6.5 and ambari 2.6.2.2what are the hortonworks document that explain this upgrade ?

Hi, We have just built a new kafka cluster(3.1.0-1.3.1.0.p0.35) and integrated it with kerberos. Kerberos is integrated with AD. We are able to produce and consume with kafka principle which is local but with AD users, it fails with below error on console. 19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to CLIENT_COMPLETE19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to COMPLETE19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Initiating API versions fetch from node -1.19/06/09 08:14:22 DEBUG network.Selector: [Producer clientId=console-producer] Connection with server2.kafka4.corp/180.20.92.23 disconnectedjava.io.EOFExceptionat org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChanne

Hello,After many days of POC and some tuning I am getting some better results however the performance seems less then what I was expecting. I understand the need of better hardware as Impala is meant for bigdata however I also need to know its performance on some regular systems. For this I created a parquet partitioned table and filled it with 500million data, I feel query is taking twice the time it should take.I had set PARQUET_FILE_SIZE = 512MB Details of my table -tbl_parq_512m +------------------------------+-------------------------------------------------------------------+----------------------+| name | type | comment |+------------------------------+-------------------------------------------------------------------+----------------------+| # col_name | data_type | comment ||

For Hadoop 3.1 i am using HWC Connector to connect to Hive and execute the queries.Below is the code snippet used for testing.spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jarimport com.hortonworks.hwc.HiveWarehouseSessionimport com.hortonworks.hwc.HiveWarehouseSession._val hive = HiveWarehouseSession.session(spark).build()val query = sc.textFile("/user/temp/hive.hql").collect().mkString.split(";").foreach(qry => hive.executeQuery(qry))query.show()This is throwing error as given below.<console>:30: error: value show is not a member of UnitIs this not the proper way to run a hql file using Spark.Please suggest.

hi allI want to hear from hortonworks guys about the following ideawe want to build the following architecture1. ambari cluster that include ( HDP 3.1 )3-5 kafka machines 3 zookeper servers that are dedicated only for the kafka machines2. ambari cluster that include all other application ( HDP 3.1 )HDFS - serviceYARN - service MAPREDUCE - serviceHIVE - serviceSPARK3 Zookeepers that dedicated to hdfs , yarn , etcdata nodes machines so both cluster should talk with each other like one big clusteris this possible ?

hi allWe are using the hortonworks kits for our hadoop cluster ( HDP - 2.6.4 - HDP 3.1 )I just wondering if any of kafka versions in HDP 2.X / 3.X must have a zookeeper servers??Ormaybe some kafka version can works without zookeeper??

hi allhow to download the zookeeper server pkgs by wget from hortonworks repository ( HDP 2.6.4 )

hi allwe download the HDP packages from HDP 2.6.4 version and we want to installed only a new 3 zookeeper servers on new machines ( without ambari )how to exclude the zookeeper packages from the HDP 2.6.4 kitso we can install only the zookeeper servers on the new machines

Hi All   Cloud deployment for the Hadoop stack on public Cloud provides cost/performance benefits for the temporary/batch workloads. But there is a need to keep master/management nodes only permanently up and running to maintain the meta-data for Navigator and all the other configurations made, etc..   What is the best practice to have a temporary cluster on the Public Cloud without caveats (for eg: not to loose lineage in the Navigator, the cluster has to remain permanent and how to create a minimum permanent cluster with data nodes added/removed daily?)   Thanks CK

The scenario:Working within Azure... I was able to deploy a cluster via an Altus Director Configuration FileThe problem is that Altus is having some trouble connecting to the Cloudera Manager... When I tried to later add nodes, via Altus Web-UI, I get the following error:  The questions:1) Is this a bug or did I miss something in the configuration file? 2) As a work-around, can I add additional nodes via a new configuration file?   Thank you

we have HDP version 2.6.4 in our cluster cluster include 285 data node machines and 3 kafka machinesfor now zookeeper servers are installed on the masters machines and zookeepers are not independent machinesbut since this is very important production cluster we think to separate the zookeeper from the masters machines and install the zookeeper servers on separate servers to be independent machinesam I right here?

While restarting SmartSense I am getting the below attached error. Please look into it:-SmartSense.txt

hi allthe following link is a great info to know versions that supportedhttps://supportmatrix.hortonworks.com/ but how to know the kafka version of each HDP version?secondlets take the HDP version - 3.1can we compare the kafka version that contain in HDP 3.1 to kafka confluent version ?one of the problems is that kafka confluent latest version is much higher then the kafka version in HDP 3.1why this Gap ?

NULL

Due to Amabri "DB configs consistency check failed" .. I have deleted all the service configs of HBASE from ambari DB as suggested in one of the articles https://community.hortonworks.com/questions/69240/ambar-server-start-failes-with-db-configs-consiste.html but after that .. I am able to start ambari server without errors but unable to see hbase service in Amabri UI

Hi, I need some help with JOLT-syntax.JSON-Input{  "nummer": "22",  "id": "177",  "table": [    {      "zn": 1,      "stfflbisart": 10    },    {      "zn": 2,      "stfflbisart": 50    }  ]}JOLT-Specification[{    "operation": "shift",    "spec": {      "id": "ID",      "nummer": "Nummer",      "table": {        "*": {          "zn": "ArtikelPreise_Pos.[#2].ZeileNr",          "stfflbisart": "ArtikelPreise_Pos.[#2].StaffelBis"        }      }    }}, {    "operation": "default",    "spec": {      "Default_Kopf": "${VAR_KD}"    }    }]JSON-Output{  "ID" : "177",  "Nummer" : "22",  "ArtikelPreise_Pos" : [ {    "Zeil

Hi All,I'm trying to upgrade Ambari server and agents from 2.7.1 to 2.7.3 using below link.https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade/content/upgrade_ambari.html But it is failing in "yum upgrade ambari-server". I have tried cleaning up repository & cache but still facing same issue.    [root@host ~]# yum upgrade ambari-server    Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager    No packages marked for update[root@host conf]# yum repolist | grep ambariambari-2.7.3.0                                      ambari Version - amba     13[root@host conf]#Below is content of ambari.repo[root@host yum.repos.d]# cat ambari.repo#VERSION_NUMBER=2.7.3.0-139[ambari-2.7.3.0]#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsonname=ambari Version - ambari-2.7.3.0baseurl=http://public-

2019-06-27 03:39:42,688 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 546, in <module> CheckHost().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 207, in actionexecute raise Fail(error_message) resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties. com.mysql.jdbc.exception

hi allcan you upgrade HDP 2.6.4 to 3.1 and stay with ambari 2.6.1?

is it safe to remove the /tmp/hive/hive folder? ( from hdfs )ashdfs dfs -rm -r /tmp/hive/hivethe reason for that because under /tmp/hive/hive we have thousand of files and we cant delete therm hdfs dfs -ls /tmp/hive/Found 7 itemsdrwx------   - admin     hdfs          0 2019-03-05 12:00 /tmp/hive/admindrwx------   - drt   hdfs          0 2019-06-16 14:02 /tmp/hive/drtdrwx------   - ambari-qa hdfs          0 2019-06-16 15:11 /tmp/hive/ambari-qadrwx------   - anonymous hdfs          0 2019-06-16 08:57 /tmp/hive/anonymousdrwx------   - hdfs      hdfs          0 2019-06-13 08:42 /tmp/hive/hdfsdrwx------   - hive      hdfs          0 2019-06-13 10:58 /tmp/hive/hivedrwx------   - root      hdfs          0 2018-07-17 23:37

Hi there! I'm try to deploy Apache Ranger ranger-tagsync module without Apache Atlas.I'm enable option "TAG_SOURCE_FILE_ENABLED = true" but can't understand the way of creating needed tags.json file.In my plan, my own deamon will simply generate this file for tagging hive-columns giving tag-based policies opportunity masking this columns for usergroups.Mostly i'am follow next article:https://community.hortonworks.com/articles/135895/how-to-mask-columns-in-hive-with-atlas-and-ranger.htmlbut it's a problem to create working file-prototype.I'am need worked examples of this file or Atlas REST output with same actions.Thank you.

I am new to hortonworks . I have created a shell script with the Sqoop import command . I want to execute the .sh script to run the sqoop import command .I have tried the following :1. I have uploaded the .sh script in a folder /home/scripts/sqoop_test.sh2. The permission : -rwxr-xr-x 1 root root 215 Jun 25 18:44 sqoop_test.sh3. Then I tried to run the file : ./sqoop_test.sh4. I am getting the error : Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-xMy query is :1. Do I need to upload the .sh script in a folder within HDFS2. How do I setup the permission for the script so that it runs successfully3. How to execute .sh file from within hdfs so that I do not get the permission denied errorThanks

Hi,I have NiFi 1.7.1 running as a service on Ubuntu 18.04.The service is running under root.The instance is set up to gather modbus data on a network using a custom processor, and send the data via S2S to another instance on a separate machine. The instance is also gathering other data such as nifi resources and sending them via S2S as well.My issue is that for some reason, when there is no user logged in to ubuntu, the flow of modbus data stops. Either via the ubuntu GUI or SSH. when I log back in, the flow starts back up. I cannot check if the service is running directly as I would need to log in for that, but the non modbus data continues to flow via S2S so I think the service is running.There doesn't seem to be anything obvious in the nifi-app.log, or bootstrap logs.Has anybody encountered unusual behaviour when the OS is not logged in?Thank you

Hello, I've set a cluster with CDH 6.20, 1 master 4 slaves, with the parcels way. Everything is working quite good, except the connection between Hue and S3.I've followed the instructions from this guide The hosts are running on Ubuntu 18.04, Python version is 2.7.15rc1S When I try to navigate my buckets in Hue filebrowser, I get this error in the GUI : "Unknown error occurred". On the Hue server logs, here is the complete error :Internal Server Error: /filebrowser/view=S3A:/Traceback (most recent call last):File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/exception.py", line 41, in innerresponse = get_response(request)File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_responseresponse = self._get_response(request)File "/opt/cloudera/parce

Hi , we are in the process of creating HDP 2.6 cluster where in RHEL OS will be integrated with AD for authentication. We will using AD as the KDC. My question if we create a local UNIX user called HIVEUSER and use any BI tool to connect to HIVE using this user, will the local user be able to get authenticated and access Hive tables in kerberized cluster?or the HIVEUSER should be in AD?