Q&A

Hi I've tried to implement firewall using the firewalld of my rhel7 server which is also our datanode, what rules and ports do I need to implement here so that the cloudera services is still functioning and will not block there connection? firewall-cmd --list-allpublictarget: defaulticmp-block-inversion: nointerfaces:sources:services: dhcpv6-client sshports:protocols:masquerade: noforward-ports:source-ports:icmp-blocks:rich rules:

How to find why "Reserved Space for Replicas" constantly increasing and how to control/set a space size allocated for the "Reserved Space for Replicas" in Hadoop? We found that the "Reserved Space for Replicas" not relate to "Non DFS" space and it can increase constantly up to rebooting Data Node-s. We didn't find 😞 how to limit space allocated for the "Reserved Space for Replicas" 😞 We thought that dfs.datanode.du.reserved can control "Reserved Space for Replicas" but it is not!

As all know when is need to print the kafka broker id’s we can use the following cli zookeeper-shell.sh zoo_server1:2181 <<< "ls /brokers/ids" this cli print the following   WatchedEvent state:SyncConnected type:None path:null[1018, 1017, 1016]   Its means that we have kafka with id’s 101810171016 But our kafka names are   Kafka_confluent01Kafka_confluent02Kafka_confluent03 So how to know which kafka broker id ( 1018 , 1017 , 1016 ) is belong to the real host ( Kafka_confluent01 / Kafka_confluent02 / Kafka_confluent03 )  

Hi Team,   Using Ambari; 2.7.3 HDP: 3.0   I am trying to install hive service with existing postgresql but it is failing with below ERROR and Exception. Can anyone advice me as where did i go wrong?   Created user 'hive' with password as 'admin' created databases 'hive' GRANT ALL PRIVILEGES ON DATABASE hive TO hive Downloaded jar file and ran setup ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar   Below is the ERROR message   Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module>HiveMetastore().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 61, in startcreate_metastore_schema() # execute without config lockFile "/var/lib/ambari-agent/ca

I would like to configure a list of urls so that my nifi InvokeHttp processor iterates sequentially over them by a schedule (e.g. once per day). Instead of hard-coding 'n' number of InvokeHttp processors each with their own url, I would like to have a local configuration file (or nifi artifact) list multiple urls and iterate these to trigger in sequence one generic InvokeHttps processor instance for each url. What is the best way to do this?

@stevenmatison hi to everyone ...i want to pass the attribute in http invoked processor but the attribute which contain the date and time like 2020-06-01T09:47:59.946Z format....i want to subtract the present time with 10 min.-------this expression i have seen in this community ${now():toNumber():minus(???):format('???')} what i have to put in the place of question mark for 10 min minus ...60000?? and forformat  2020-06-01T09:47:59.946Z(+%FT%T.%3NZ  if i use this i'll get error in format )  ..... and what processor i have to use...currently i m using updateAttribute for date format    thanks in advance 

.. [INFO] ------------------------------------------------------------------------[INFO] Building metron-config 0.7.1[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ metron-config ---[INFO] Deleting /root/metron/metron-interface/metron-config/node (includes = [], excludes = [])[INFO] Deleting /root/metron/metron-interface/metron-config/node_modules (includes = [], excludes = [])[INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-config ---[INFO] [INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (default) @ metron-config ---[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/root/metron/metron-interface/metron-config/target/jacoco.exec[INFO] [INFO] --- frontend-maven-plugin:1.3:install-node-and-npm (install node an

Hi guys,I have been using AWS machine for practice.My Question is :How i Create & Recover Snapshot ? Please explain.Thanks,Mudassar Hussain

0I m using Spark 2.4.5, Hive 3.1.2, Hadoop 3.2.1. While running hive in spark i got the following exception,Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x; This is my source code,package com.spark.hiveconnectimport java.io.Fileimport org.apache.spark.sql.{Row, SaveMode, SparkSession}object sourceToHIve { case class Record(key: Int, value: String) def main(args: Array[String]){ val warehouseLocation = new File("spark-warehouse").getAbsolutePath val spark = SparkSession .builder() .appName("Spark Hive Example").master("local") .config("spark.sql.warehouse.dir", warehouseLocation) .enableHiveSupport() .getOrCreate() import spark.implicits._ import spark.sql sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive") sql("LOAD DATA LOCAL INPATH '/usr/local/spark3/examples/src/main/resources/k

Hi,   I am facing an issue: hive table created in data analytics studio is not listed under tables (refer below screen print, tables listed under "Search Tables" text box) But when doing select that table is shown in results (refer below screen print, under results) Could you please let me know what is the issue here and how it can be corrected. More information - I am using HDP 3.0 sandbox with CDA enabled.

I have installed HDP sandbox 3.0  and installed NIFI in the same cluster using HDF 3.4 mpack.The problem i am facing is, NIFI  won't open from my host browser even though it is running."listening to Bootstrap on port 37693" also no errors whatsoever from nifi I can curl to NIFI from within hdp (using a shell) but not from my host (windows).Error from curl:* Trying 127.0.0.1...* Connected to localhost (127.0.0.1) port 9090 (#0)> GET /nifi HTTP/1.1> Host: localhost:9090> User-Agent: curl/7.55.1> Accept: */** Recv failure: Connection was aborted* Closing connection 0curl: (56) Recv failure: Connection was aborted All other services are running fine. A few things i have tested but didn't work:Tried changing nifi.web.http.host to 0.0.0.0 or 127.0.0.1.Tried changing port from 9090 to some other but problem persist.(just to mention port is forwarding as well from VirtualBox)Disabled my windows firewall just to be sure it is not blocking that port.I have tr

How do I download cloudera virtualBox image? I remember a year back there were clear download buttons. What changed and is cloudera not free to use now after the merger? If there is a download link available please direct me there as it is hard to find a way through the new download page on cloudera. 

How to connect to NIFI GetMongo using x509 connection with certificates ?

Is there a way to clear the state of a processor using another processor or groovy scripting (in an ExecuteStateProcessor) ?The goal is to reset the state of a ScrollElasticSearchHttp processor to be able to run it more than once.Important. I don't want do it manually using the UI ("state management" then "clear state", as it is explained in https://community.hortonworks.com/questions/64405/nifi-clear-state.html)I want this operation be done after a scroll execution.Refer the nifi doc : https://help.syncfusion.com/data-integration/processors/scrollelasticsearchhttp...Scrolls through an Elasticsearch query using the specified connection properties. This processor is intended to be run on the primary node, and is designed for scrolling through huge result sets, as in the case of a reindex. The state must be cleared before another query can be run. ...But how do that ?Thanks by advance.

I am enabling HDFS data at rest encryption in CDH 6.3 and while adding the KMS service, i noticed that /var/lib/kms-keytrustee/keytrustee/.keytrustee/ is actually doesn't exist on all the KMS hosts when trying to synchronise the KMS hosts private keys. Has anyone come across such issue and what is the way forward for this?

hello everyone ....please help me out ...i m working on one system that system received alarm from client ...we generally received 10000 alarm ......we want to configure our web with nifi for some other update in alarm ...for this we have used invoked http processor that get all the alarm from web app and then we used split json processor that split one alarm in one flowfile then we used evaluate jsonpath that extract id attribute which will be use in detectduplicate processor ....actually the invoked http processor give repeated alarm but we want only one copy of one alarm ...every 10 min the http processor get alarm from web app but those alarm should not be duplicate ....is it possible to do this .....if you have any other option for this work please suggest me.and in detectduplicate ----CacheEntryIndentifier - ${id}age of duration - 15 min distributed cache service - DistributedMapCacheClientService and one more thing i have used  DistributedMapCacheClientServic

I tried to create a folder using the admin user, at the time of creating it I get the following error: Permission denied: user = admin, access = WRITE, inode = "/ user": hdfs: hdfs: drwxr-xr-x When checking the permissions in Ambari> Users> admin, it tells me that I have all the permissions activated. How can i solve this problem?

Ranger Group level authorization for NIFI is not working whereas user-level it's working do we need to enable or integrate underlying servers to AD/LDAP in order to work?

@stevenmatisonis it possible to apply filter condition on invoke HTTP processor that give all the alarm from web app in every 5 min but will not fetch the duplicate alarm that already fetched before ? or else is it possible when the alarm will come in to my web app then workflow in NIFI will trigger and gives recent alarm which came just in 10 min ...i have tried with DetectDuplicate processor ...but not getting any result ..i think i made mistake in configuration in detecduplicate processor.. please direct me ....and if you have any other route for this please provide me the whole way(configuration an all) ..i'm new in nifi ..don't know much more about that  detectduplicate processorwhen i clicked in arrow in side of RedisDistributedMapCacheClintService ...this below config has open(RedisConnectionPoolService)..and in connection string option i have put web address with port number is it wrong ..or else we have to put nifi localhost address with port no.??  

Hi all,    I have reinstalled ambari and trying to setup a three-node cluster. However, I keep getting the same error on the last step about multiple ambari-hdp-repos-1-2 and now 51. I am running centos7 and the install is being done offline through local repositories.    I know this has been covered in another thread but the answer is not very clear to me. My HDP, HDP-UTILS and SOLR repo urls are correct. A   Any help please??? Thank you!     2020-04-15 15:50:40,562 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=None -> 2.62020-04-15 15:50:40,566 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf2020-04-15 15:50:40,567 - Group['livy'] {}2020-04-15 15:50:40,609 - Adding group Group['livy']2020-04-15 15:50:42,177 - Group['spark'] {}2020-04-15 15:50:42,177 - Adding group Group['spark']2020-04-15 15:50:42,200 - Group['solr'] {}2020-04-15 15:50:42,201 - Adding group Group['solr']2020-04-15 15:50:42,229 -

Hello, I want to create a Case in the Cloudera support Portal. Even after finishing registeration, it redirects me this page belowCan someone help me please?

Hello All,     My Ranger DB is corrupted and I have no backup to restore it. Can I drop the corrupt db and recreate a new db for Ranger. This is for my Kerberized cluster in my test environment.   Any advice.

 I am working on a simple code to get a row from HBase which is not working through HBase Java API but the same row is retrievable from the shell.Example row Ids: F\x0A@z\x11\x0D\xB4\xD3\x05"S273320039#999279720=\x0A@z\x1AI\xA4\xD8h\x12\x09754427158#999598625F\x0A@z\x11\x0D\xA6@\x0C\x04\xDA490229991#999279958M\x0A@z\x13\x092\x89a\x12\x111059947933#999223281\x00\x0A@E\xFFn[\x18\x9F\xD4-1447846881#5241968320Result from one of the RowId through shell:   hbase(main):003:0> get 'OSMNodes_z3_geometry_ingestionTimestamp_v6',"\x00\x0A@E\xFFn[\x18\x9F\xD4-1447846881#5241968320"COLUMN CELL d: timestamp=1588532276719, value=\x03\x00\x06\x02\x00\x12\x00-\x005\x00=\x00R\x00Y\x01x\x00\x00\x0 0\x00\x01\x08\x03\xC0V\xE9\x9AU\x81;\xD5\xBF\xF5\xA7FE^\xAE\xE2\x7F\xF8\x00\x00\x00\x00\x00\x00\

Hello All, I had a DE575 exam scheduled for Feb 2020 but, due to the platform unavailability, I could not do it.I was expecting a mail from cloudera when the platform is live again. But I havent received any communication from Cloudera ever since. Recently I heard that the Spark(CCA 175) exam is available in the upgraded platform, but no news on DE575.Anyway, out of curiosity I checked in examslocal to see  if it allows me to reschedule the DE575 and it did. I just wanted to make sure that the exam (DE575) is really available .Does anyone have any clue. Thanks a lot!

Hi Team, Can any one please help me,how to download CDP Data Center Trial Download in Windows?My Doubt is --> Is it possible to download in windows and can we work on windows? Regards,Siva.