Solved

Hello, I am using Sandbox HDP 3.0, my Zeppelin UI is not running. I have included the content of the log file, have everyone see it and help me fix it. Thanks very much!   File Log   INFO [2019-10-16 13:05:24,804] ({main} ZeppelinConfiguration.java[create]:121) - Load configuration from file:/etc/zeppelin/3.0.1.0-187/0/zeppelin-site.xmlINFO [2019-10-16 13:05:25,089] ({main} ZeppelinConfiguration.java[create]:129) - Server Host: 0.0.0.0INFO [2019-10-16 13:05:25,089] ({main} ZeppelinConfiguration.java[create]:131) - Server Port: 9995INFO [2019-10-16 13:05:25,090] ({main} ZeppelinConfiguration.java[create]:135) - Context Path: /INFO [2019-10-16 13:05:25,095] ({main} ZeppelinConfiguration.java[create]:136) - Zeppelin Version: 0.8.0INFO [2019-10-16 13:05:25,137] ({main} Log.java[initialized]:193) - Logging initialized @3285ms to org.eclipse.jetty.util.log.Slf4jLogINFO [2019-10-16 13:05:25,452] ({main} ZeppelinServer.java[setupWebAppContext]:387) - ZeppelinServer Webapp path: /usr/

hi everyone ,We want to put data in web app using API endpoint https://localhost:8000/api/event/${id}/attributes this is the API endpoint that i have used in invoke HTTP processor in Remote URL field with PUT method. we want to create new attribute with value and this attribute is a dictionary type so we have to pass the data in the dictionary format for this we have used updateattribute processor and attributetojson processor. updateAttribute processor snapAttributetojson processor snapand the below is out of attributetoJson processor (flowfile){"note":"note from nifi","attributes":"{\n \"workflow\":\"process\"\n }","id":"1ecfbcbb-38bf-49e9-a77e-a88w"} the error msg in invoke http processor { "code": 500, "errors": [ "Traceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/flask/app.py\", line 1949, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.6/site-packages/No value set as per my

Hi! I have a DBCPConnectionPool on NiFi 1.11.3 with this configuration:Normally it works ok, but sometimes the query get stuck on a PutSQL, no errors for minutes and then a timeout (very basic queries)I was trying some configurations on the pool but it seems I didn't quite understand the best way to troubleshoot this, I was thinking in each processor add a retry+counter+error. But don't know where to set the time out for queries from start of it and no per connection. Maybe a way is to not keep connections open until the query starts, how should I set the pool to generate the connection on request from a processor and terminate it before execution.I think it's a Athena problem with pools because I have no problem with another databases like postgres and mysql.Any ideas?? Thank you! 

        updateAttribute processor config and this is the output of AttributetoJson processor ....i want tags value in the list like ["UNIX01"] 

Hi, There was an issue of replication of Hive metadata to DAS; I was trying to resolve it by resetting tables by taking reference of "https://docs.cloudera.com/HDPDocuments/DAS/DAS-1.2.1/troubleshooting/content/das_replication_failure_in_event_processor.html". After executing SQL commands successfully; I was trying to start DAS. Under which "Data Analytics Studio Event Processor" was started successfully and running but "Data Analytics Studio Webapp" was started successfully but not running. Below is the configuration detailHDP - 3.0.1DAS - 1.0.2.0.0 Steps I used;1) Stops DAS from Ambari2) taken backup of /var/log/das log files and created empty files of event-processor.log, das-webapp.log files.3) Executed reset tables SQL commands to das database in postgres 4) Started DAS from Ambari. Other observations - 1) After start of DAS nothing is written in event-processor.log, das-webapp.log files.2) But can see das.db_replication_info was getting refreshed with id

Hello guys,I'm trying to run full import from a table in SQL Server. But sqoop is interpreting the data type incorrectly. Column in SQL Server: ErrorParams (nvarchar(max))Sqoop translates it to ErrorParams varchar, which generates an error. create table `dataengsandbox`.`task` (`id` INT,`recordid` int,`state` int,`prevstate` int,`instanceid` int,`parentjobid` int,`closed` boolean,`requeuecount` int,`autorequeuecount` int,`modifytime` string,`submittime` string,`starttime` string,`endtime` string,`createtime` string,`timestamp` binary,`pendingreason` int,`failurereason` int,`exitcode` int,`failednodeid` int,`errorcode` int,`errorparams` varchar,`output` string,`executionfailureretrycount` int)stored as parquet I would expect it to translate it to ErrorParams varchar(64) instead, or another value.FAILED: ParseException line 22:22 mismatched input ',' expecting ( near 'varchar' in primitive type specification Is there a way to make sqoop cast it correctly? Sqoop j

I have got a difficulty with PutEmail processor. It doesn't send (attach) messages more than 1 Kb.Is it a bug?

Hi Experts: Following the instruction as in https://docs.cloudera.com/documentation/enterprise/6/latest/topics/install_cm_cdh.html , we set up a test CDH 6.3.3 cluster. We also enabled Kerberos for security. The Hive, Impala, HBase command line clients all make connections and are functioning basically. Most parts of Hue works too, except the HBase Browser throws error - "Api Error: TSocket read 0 bytes". Please see below the Hue access.log and HBase ThriftServer log. What to check further to resolve the issue? Thank you!   /var/log/hue/access.log --------------------------- [02/Jun/2020 07:16:11 -0700] INFO 173.2.217.185 xe46 - "POST /hbase/api/getClusters HTTP/1.1" returned in 11ms[02/Jun/2020 07:16:11 -0700] INFO 173.2.217.185 xe46 - "POST /notebook/api/autocomplete/default HTTP/1.1" returned in 152ms[02/Jun/2020 07:16:12 -0700] INFO 173.2.217.185 xe46 - "POST /hbase/api/getTableList/HBase HTTP/1.1" returned in 96ms[02/Jun/2020 07:16:12 -0700] ERROR 173.2.217.185 xe46 - "P

Hi I've tried to implement firewall using the firewalld of my rhel7 server which is also our datanode, what rules and ports do I need to implement here so that the cloudera services is still functioning and will not block there connection? firewall-cmd --list-allpublictarget: defaulticmp-block-inversion: nointerfaces:sources:services: dhcpv6-client sshports:protocols:masquerade: noforward-ports:source-ports:icmp-blocks:rich rules:

How to find why "Reserved Space for Replicas" constantly increasing and how to control/set a space size allocated for the "Reserved Space for Replicas" in Hadoop? We found that the "Reserved Space for Replicas" not relate to "Non DFS" space and it can increase constantly up to rebooting Data Node-s. We didn't find 😞 how to limit space allocated for the "Reserved Space for Replicas" 😞 We thought that dfs.datanode.du.reserved can control "Reserved Space for Replicas" but it is not!

As all know when is need to print the kafka broker id’s we can use the following cli zookeeper-shell.sh zoo_server1:2181 <<< "ls /brokers/ids" this cli print the following   WatchedEvent state:SyncConnected type:None path:null[1018, 1017, 1016]   Its means that we have kafka with id’s 101810171016 But our kafka names are   Kafka_confluent01Kafka_confluent02Kafka_confluent03 So how to know which kafka broker id ( 1018 , 1017 , 1016 ) is belong to the real host ( Kafka_confluent01 / Kafka_confluent02 / Kafka_confluent03 )  

Hi Team,   Using Ambari; 2.7.3 HDP: 3.0   I am trying to install hive service with existing postgresql but it is failing with below ERROR and Exception. Can anyone advice me as where did i go wrong?   Created user 'hive' with password as 'admin' created databases 'hive' GRANT ALL PRIVILEGES ON DATABASE hive TO hive Downloaded jar file and ran setup ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar   Below is the ERROR message   Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module>HiveMetastore().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 61, in startcreate_metastore_schema() # execute without config lockFile "/var/lib/ambari-agent/ca

I would like to configure a list of urls so that my nifi InvokeHttp processor iterates sequentially over them by a schedule (e.g. once per day). Instead of hard-coding 'n' number of InvokeHttp processors each with their own url, I would like to have a local configuration file (or nifi artifact) list multiple urls and iterate these to trigger in sequence one generic InvokeHttps processor instance for each url. What is the best way to do this?

@stevenmatison hi to everyone ...i want to pass the attribute in http invoked processor but the attribute which contain the date and time like 2020-06-01T09:47:59.946Z format....i want to subtract the present time with 10 min.-------this expression i have seen in this community ${now():toNumber():minus(???):format('???')} what i have to put in the place of question mark for 10 min minus ...60000?? and forformat  2020-06-01T09:47:59.946Z(+%FT%T.%3NZ  if i use this i'll get error in format )  ..... and what processor i have to use...currently i m using updateAttribute for date format    thanks in advance 

.. [INFO] ------------------------------------------------------------------------[INFO] Building metron-config 0.7.1[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ metron-config ---[INFO] Deleting /root/metron/metron-interface/metron-config/node (includes = [], excludes = [])[INFO] Deleting /root/metron/metron-interface/metron-config/node_modules (includes = [], excludes = [])[INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ metron-config ---[INFO] [INFO] --- jacoco-maven-plugin:0.8.3:prepare-agent (default) @ metron-config ---[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.3/org.jacoco.agent-0.8.3-runtime.jar=destfile=/root/metron/metron-interface/metron-config/target/jacoco.exec[INFO] [INFO] --- frontend-maven-plugin:1.3:install-node-and-npm (install node an

Hi guys,I have been using AWS machine for practice.My Question is :How i Create & Recover Snapshot ? Please explain.Thanks,Mudassar Hussain

0I m using Spark 2.4.5, Hive 3.1.2, Hadoop 3.2.1. While running hive in spark i got the following exception,Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x; This is my source code,package com.spark.hiveconnectimport java.io.Fileimport org.apache.spark.sql.{Row, SaveMode, SparkSession}object sourceToHIve { case class Record(key: Int, value: String) def main(args: Array[String]){ val warehouseLocation = new File("spark-warehouse").getAbsolutePath val spark = SparkSession .builder() .appName("Spark Hive Example").master("local") .config("spark.sql.warehouse.dir", warehouseLocation) .enableHiveSupport() .getOrCreate() import spark.implicits._ import spark.sql sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive") sql("LOAD DATA LOCAL INPATH '/usr/local/spark3/examples/src/main/resources/k

Hi,   I am facing an issue: hive table created in data analytics studio is not listed under tables (refer below screen print, tables listed under "Search Tables" text box) But when doing select that table is shown in results (refer below screen print, under results) Could you please let me know what is the issue here and how it can be corrected. More information - I am using HDP 3.0 sandbox with CDA enabled.

I have installed HDP sandbox 3.0  and installed NIFI in the same cluster using HDF 3.4 mpack.The problem i am facing is, NIFI  won't open from my host browser even though it is running."listening to Bootstrap on port 37693" also no errors whatsoever from nifi I can curl to NIFI from within hdp (using a shell) but not from my host (windows).Error from curl:* Trying 127.0.0.1...* Connected to localhost (127.0.0.1) port 9090 (#0)> GET /nifi HTTP/1.1> Host: localhost:9090> User-Agent: curl/7.55.1> Accept: */** Recv failure: Connection was aborted* Closing connection 0curl: (56) Recv failure: Connection was aborted All other services are running fine. A few things i have tested but didn't work:Tried changing nifi.web.http.host to 0.0.0.0 or 127.0.0.1.Tried changing port from 9090 to some other but problem persist.(just to mention port is forwarding as well from VirtualBox)Disabled my windows firewall just to be sure it is not blocking that port.I have tr

How do I download cloudera virtualBox image? I remember a year back there were clear download buttons. What changed and is cloudera not free to use now after the merger? If there is a download link available please direct me there as it is hard to find a way through the new download page on cloudera. 

How to connect to NIFI GetMongo using x509 connection with certificates ?

Is there a way to clear the state of a processor using another processor or groovy scripting (in an ExecuteStateProcessor) ?The goal is to reset the state of a ScrollElasticSearchHttp processor to be able to run it more than once.Important. I don't want do it manually using the UI ("state management" then "clear state", as it is explained in https://community.hortonworks.com/questions/64405/nifi-clear-state.html)I want this operation be done after a scroll execution.Refer the nifi doc : https://help.syncfusion.com/data-integration/processors/scrollelasticsearchhttp...Scrolls through an Elasticsearch query using the specified connection properties. This processor is intended to be run on the primary node, and is designed for scrolling through huge result sets, as in the case of a reindex. The state must be cleared before another query can be run. ...But how do that ?Thanks by advance.

I am enabling HDFS data at rest encryption in CDH 6.3 and while adding the KMS service, i noticed that /var/lib/kms-keytrustee/keytrustee/.keytrustee/ is actually doesn't exist on all the KMS hosts when trying to synchronise the KMS hosts private keys. Has anyone come across such issue and what is the way forward for this?

hello everyone ....please help me out ...i m working on one system that system received alarm from client ...we generally received 10000 alarm ......we want to configure our web with nifi for some other update in alarm ...for this we have used invoked http processor that get all the alarm from web app and then we used split json processor that split one alarm in one flowfile then we used evaluate jsonpath that extract id attribute which will be use in detectduplicate processor ....actually the invoked http processor give repeated alarm but we want only one copy of one alarm ...every 10 min the http processor get alarm from web app but those alarm should not be duplicate ....is it possible to do this .....if you have any other option for this work please suggest me.and in detectduplicate ----CacheEntryIndentifier - ${id}age of duration - 15 min distributed cache service - DistributedMapCacheClientService and one more thing i have used  DistributedMapCacheClientServic

I tried to create a folder using the admin user, at the time of creating it I get the following error: Permission denied: user = admin, access = WRITE, inode = "/ user": hdfs: hdfs: drwxr-xr-x When checking the permissions in Ambari> Users> admin, it tells me that I have all the permissions activated. How can i solve this problem?