Solved

hi allwhat is the normal gap between machines in ambari cluster in seconds?for example ( the following example described gap of 20 seconds )on master01Fri Jul 5 07:37:23 UTC 2019but on kafka machine we haveFri Jul 5 07:37:43 UTC 2019

hi allwe are in the middle of rolling upgradewe install the HDP 2.6.5 pkgs from ambarinow we perform rolling upgraderolling upgrade window say that spark thrift servers are down and in maintenance modethis is right we stop the spark thrift services since we are not use the SPARK2_THRIFTSERVER , and not want to do spark upgradeso only spark clients and spark history are up and running but rolling upgrade cant continue because ( according to the rolling upgrade message we must to start the spark thrift services and disable the maintenance mode )so how to continue from this stage?Reason: The following Service Components should be in a started state. Please invoke a service Stop and full Start and try again. SPARK2: SPARK2_THRIFTSERVER (in INSTALLED on host master01.sys674.com)Failed on: SPARK2

hi allwe notice about no option to un-install the installation of the rpm's ( when doing upgrade from 2.6.4 to 2.6.5 )why?

My requirement is to upgrade HDF (not HDP) . Basically my environment have two separate clusters - HDP and HDFI want to upgrade only HDF to 3.4.0 Version from HDF 3.0.2 . Ambari version currently installed is 2.6.1 and HDF version is 3.0.2 . Can you please help me to know if I can directly upgrade to 3.4.0 from 3.0.2 ?I tried looking into this but no luckhttps://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.4.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.htmlCan i directly perform HDF upgrade to 3.4.0 from 3.0.2.5 ? Or does it requires first to upgrade to some version first and then upgrade it to higher version hdf 3.4.0 ?

hi allwe are very confuse because un-consistence recommendation in hortonworks documentsfrom the link: ( https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/upgrading_hdp_stack.html )we can see that: "It is strongly recommended that you first upgrade to Ambari 2.6.2 "NoteBe sure to review the available HDP upgrade scenarios below. It is strongly recommended that you first upgrade to Ambari 2.6.2 before upgrading HDP unless otherwise noted. Ambari 2.6.2 supports patch and maintenance upgrades to HDP 2.6.5.x stack. After upgrading Ambari, be sure the cluster is operating normally and service checks are passing prior to attempting an HDP upgrade.Ambari 2.6.2 does not support IOP-HDP migration. Customers migrating from IOP should use Ambari 2.6.1 to migrate IOP to HDP 2.6.4, then upgrade Ambari to 2.6.2 and use it to upgrade HDP 2.6.4 to HDP 2.6.5.but from ( https://supportmatrix.hortonworks.com/ )we can see that HDP - 2.6.5 can be installed on ambari 2.6.1

I am trying to exchange a partition from staging db after merging the incremental data with the existing one as below:1. Created staging table with partition: CREATE TABLE stg.customers_testcontrol_staging(customer_id bigint,customer_name string, customer_number string,status string,attribute_category string,attribute1 string, attribute2 string, attribute3 string, attribute4 string, attribute5 string) PARTITIONED BY (source_name string) ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' Location('/apps/hive/warehouse/stg.db/customers_testcontrol_staging'2. Inserted data into above table after merging with the base table data  INSERT OVERWRITE TABLE finstg.customers_testcontrol_staging PARTITION (source_name) SELECT t1.* FROM (SELECT * FROM base.customers where source_name='ORACLE' UNION ALL SELECT * FROM external.customers_incre

"File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/conf.server ; /usr/hdp/current/hive-server2-hive2/bin/schematool -initSchema -dbType mysql -userName hive -passWord [PROTECTED] -verbose' returned 1. Your account has expired; please contact your system administrator"

If i use an extract text processor to extract a key and value from a text, then i always get 2 attributes with the same value.For example:If i use test: ([\w .]+) as my expression, then when i process a file with content "test: me" then the single data capture produces two attributes which as far as i can see are exactly the same.myattribute : "me"myattribute.1 : "me"Is there any way i can configure this to get just 1 attribute? Or is there a better way to extract key= value data from a file content?eg. attribute1:500, attribute2:600Many thanks in advance.

hi allwe prepare to upgrade our clusterwe have HDP version - 2.6.4 and ambari - 2.6.1we want to upgrade to the latest HDP 3.1 and latest ambari 2.7.2what are the hortonworks document that explain this upgrade ?is it safe to jump from old HDP and ambari version to the latest ? dose hortonwoks tested this scenario ? ( upgrade HDP 2.6.4 to HDP 3.1 )

I'm trying to upgrade HDP 3.0 to 3.1 but after Ambari upgrade HDP services are not listed in dashboard, instead cluster installation wizard come up.[root@host ~]# curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://host/api/v1/clusters/"{  "href" : "http://host/api/v1/clusters/",  "items" : [ ]}[root@host~]#while I can see services running in background[root@host ~]# jps27457 HMaster28711 TimelineReaderServer25675 jar30637 ZeppelinServer29427 Main6195 HRegionServer2324 AmbariServer31860 Main26869 ResourceManager29013 Main20150 Jps23895 SupportToolServer26391 Main10041 RemoteInterpreterServer4219 NameNode25212 QuorumPeerMain3679 DFSZKFailoverController

Hi guys,I have installed and configured cloudera manager, i decided to clean eveyrithing and install it again, and is failing when distributing parcels

hi allwe prepare to upgrade our cluster we have HDP version - 2.6.4 and ambari - 2.6.1we want to upgrade to HDP 2.6.5 and ambari 2.6.2.2what are the hortonworks document that explain this upgrade ?

Hi, We have just built a new kafka cluster(3.1.0-1.3.1.0.p0.35) and integrated it with kerberos. Kerberos is integrated with AD. We are able to produce and consume with kafka principle which is local but with AD users, it fails with below error on console. 19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to CLIENT_COMPLETE19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to COMPLETE19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Initiating API versions fetch from node -1.19/06/09 08:14:22 DEBUG network.Selector: [Producer clientId=console-producer] Connection with server2.kafka4.corp/180.20.92.23 disconnectedjava.io.EOFExceptionat org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChanne

Hello,After many days of POC and some tuning I am getting some better results however the performance seems less then what I was expecting. I understand the need of better hardware as Impala is meant for bigdata however I also need to know its performance on some regular systems. For this I created a parquet partitioned table and filled it with 500million data, I feel query is taking twice the time it should take.I had set PARQUET_FILE_SIZE = 512MB Details of my table -tbl_parq_512m +------------------------------+-------------------------------------------------------------------+----------------------+| name | type | comment |+------------------------------+-------------------------------------------------------------------+----------------------+| # col_name | data_type | comment ||

For Hadoop 3.1 i am using HWC Connector to connect to Hive and execute the queries.Below is the code snippet used for testing.spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jarimport com.hortonworks.hwc.HiveWarehouseSessionimport com.hortonworks.hwc.HiveWarehouseSession._val hive = HiveWarehouseSession.session(spark).build()val query = sc.textFile("/user/temp/hive.hql").collect().mkString.split(";").foreach(qry => hive.executeQuery(qry))query.show()This is throwing error as given below.<console>:30: error: value show is not a member of UnitIs this not the proper way to run a hql file using Spark.Please suggest.

hi allI want to hear from hortonworks guys about the following ideawe want to build the following architecture1. ambari cluster that include ( HDP 3.1 )3-5 kafka machines 3 zookeper servers that are dedicated only for the kafka machines2. ambari cluster that include all other application ( HDP 3.1 )HDFS - serviceYARN - service MAPREDUCE - serviceHIVE - serviceSPARK3 Zookeepers that dedicated to hdfs , yarn , etcdata nodes machines so both cluster should talk with each other like one big clusteris this possible ?

hi allWe are using the hortonworks kits for our hadoop cluster ( HDP - 2.6.4 - HDP 3.1 )I just wondering if any of kafka versions in HDP 2.X / 3.X must have a zookeeper servers??Ormaybe some kafka version can works without zookeeper??

hi allhow to download the zookeeper server pkgs by wget from hortonworks repository ( HDP 2.6.4 )

hi allwe download the HDP packages from HDP 2.6.4 version and we want to installed only a new 3 zookeeper servers on new machines ( without ambari )how to exclude the zookeeper packages from the HDP 2.6.4 kitso we can install only the zookeeper servers on the new machines

Hi All   Cloud deployment for the Hadoop stack on public Cloud provides cost/performance benefits for the temporary/batch workloads. But there is a need to keep master/management nodes only permanently up and running to maintain the meta-data for Navigator and all the other configurations made, etc..   What is the best practice to have a temporary cluster on the Public Cloud without caveats (for eg: not to loose lineage in the Navigator, the cluster has to remain permanent and how to create a minimum permanent cluster with data nodes added/removed daily?)   Thanks CK

The scenario:Working within Azure... I was able to deploy a cluster via an Altus Director Configuration FileThe problem is that Altus is having some trouble connecting to the Cloudera Manager... When I tried to later add nodes, via Altus Web-UI, I get the following error:  The questions:1) Is this a bug or did I miss something in the configuration file? 2) As a work-around, can I add additional nodes via a new configuration file?   Thank you

we have HDP version 2.6.4 in our cluster cluster include 285 data node machines and 3 kafka machinesfor now zookeeper servers are installed on the masters machines and zookeepers are not independent machinesbut since this is very important production cluster we think to separate the zookeeper from the masters machines and install the zookeeper servers on separate servers to be independent machinesam I right here?

While restarting SmartSense I am getting the below attached error. Please look into it:-SmartSense.txt

hi allthe following link is a great info to know versions that supportedhttps://supportmatrix.hortonworks.com/ but how to know the kafka version of each HDP version?secondlets take the HDP version - 3.1can we compare the kafka version that contain in HDP 3.1 to kafka confluent version ?one of the problems is that kafka confluent latest version is much higher then the kafka version in HDP 3.1why this Gap ?

NULL