Q&A

I am able to Export and Import controller services from one instance to another. Processors which are referencing to a controller service are not being mapped/selected. Looking for better possible way.   Thanks in Advance

I get the following error when connecting to Hive.     Exception in thread "main" java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: Invalid status 21. at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) at com.cloudera.hiveserver2.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source) at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.hiveserver2.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)   &n

Hi,   I'm trying to get data from an SFTP server. The recommended pattern from articles I have read suggest ListFTP to Remote Process Group, Input Port to FetchSFTP. I'm accessing Nifi on a local instance (localhost:9090/nifi).   After adding a RPG to the canvas as well as an input port, I get the error "Connection refused on the RPG" and "'http://localhost:9090/nifi/' does not have any input ports." when I try connect ListFTP to the Remote Process Group.  

We are planning to upgrade our Cloudera cluster from 5.8.4 to 6.3.3, but our OS is in IBM AIX, just want to know if it is still supported by the new version of Cloudera?

Hi, Need help on below query. We have a scenario where for any column defined with datatype as string in Hadoop, NULL value is loaded as blank. However, any column other than string datatype (INT,DOUBLE etc) NULL values are loaded as NULL in Hadoop.    Column name          Data Type service1end               string service1start              string service2end               string service2start              string firstlinemaintcost    double       Is this a default behavior of Hadoop/hive?

I'm wondering is it possible to create NiFi workflows with the existing processors using REST Api or CLI instead of creating workflows in GUI. Is REST Api can be used to create workflows or only to modify and schedule workflows. If we can create workflows using REST api or CLI, any articles or blogs to look into it?Let's say I want to create a workflow which simply takes file form local and put it to hdfs. This can be possible by drag and drop the processors in GUI but Is it possible to do the same using REST API or CLI or any other thing which we can do programatically?  It would be a great help if you guide me regarding this.Thanks a lot.

Dear All, I have ran into a very troublesome issue which is creating headache for the last 1 week. I installed and HDF Cluster(v3.4.1.1) with the services Nifi,Kafka,Metrics and Smart Sense. Everything was working perfectly until I decided to enable Kerberos(MIT) in the cluster. During the keberization, everything went fine till the last step.(Start and Test services). At that point, the Zookeeper server start got failed and thereafter no services are starting. The following is the error I got from the zookeeper logs.Any help is appreciated.   2020-03-06 18:38:51,489 - INFO [main:QuorumPeerMain@127] - Starting quorum peer2020-03-06 18:38:51,552 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly:javax.security.auth.login.LoginException: Message stream modified (41)at org.apache.zookeeper.server.ServerCnxnFactory.config

I am trying to install cloudera on machines with fips enabled and because of this the install keeps failing. Specifically because the version of open ssl with fips enabled doesn't allow for md5. My question is there a specific process for installing a cloudera cluster on a machine with fips enabled?

Hi Cloudera Community, Our client is planning to upgrade their CDH and Cloudera Manager from version 5.8.4 to 6.3.3, may I please if do you have a recommended upgrade path for this? Thanks in advance!

https://www.cloudera.com/products/pricing.html   Where is cloudera's promise and committment to open-source ? How are we supposed to use the CDP stack on an on-premise solution ? When is the release which supports on-premise deployments going to be available ?

Hi, I’m facing a problem that I could not solve. Can somebody help me?The details are below. Thanks! Cloudera CDH 6.2.1 HUE bundleHue 4.3.0 - cdh6.2.1 Problem:Can not open workflow editor, or scheduler editorLog: [28/Feb/2020 09:42:16 +0100] access ERROR 172.26.151.2 admin - “POST /desktop/log_js_error HTTP/1.1”-- JS ERROR: {“msg”:“Uncaught ReferenceError: Unable to process binding “click: function(){return openShareModal }”\nMessage: openShareModal is not defined”,“url”:“http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js",“line”:52,“column”:14162,“stack”:"ReferenceError: Unable to process binding “click: function(){return openShareModal }”\nMessage: openShareModal is not defined\n at click (eval at t (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:52:10372), :3:58)\n at a (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:51:1216)\n at init (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:

How can I use ReplaceText processor to function like dos2unix command, i.e. change \r\n to \n My file comes in with ^M at the end of each line In the ReplaceText process, I have a Search Value of \r\n That is correctly finding the Windows linefeeds However, for Replacement value, I have tried \n also $1\n   They each put the character n where I want the line feed to be. Thanks!

Hi,    I'm currently using merge_content processor as merge avro files from two sources like kafka_consumer processor and fetchHDFS file. While converting avro file into one with merge content processor yesterday around 680MB trying to convert but processor drop the file and join with new files and i can't able to recover that data also, because content_repository backup i limit. Can you please help me out for this use_case processor how much size can be good or is there any setting needs to modifiy in nifi.properties. 

Hey Guys,   Need to establish links between dependent flows within the same NiFi cluster / canvas - e.g. one flow finishes ETL, triggering another flow to begin its ETL on the result of the first's.   From what I gather this could be achieved in 3 ways: 1. Direct connections between flows - undesirable in our production environment and data governance and a spider's web of dependencies 2. Remote processor groups (RPGs) - if can send / receive from same NiFi Cluster  3. HTTP   Could you please advise on options 2 & 3 as a solution for this?  RPGs seem simpler than handling HTTP requests / responses yet require an input port on the top level of the flow, which would then need to be directly connected into each flow (being another version of option 1).   Thanks

I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302 and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303 and file abcd.out.gz.20200302 under /data/20200302.   How can i achieve this in NiFi. 

We use NiFi mainly for file transfer, processing these files etc. The requirement is to pull reports from NiFi which has the statistics such as how many files were processed from NiFi, how many of them are successfully transferred to destination, how many are failed and waiting in different queues.The NiFi rest api documentation(https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) has the API's to pull this information, but i did not find any info on how to use these API's. Is the API's exposed out of the box or should it be enabled somehow.Please share your suggestions on this.If there is any other way to extract the data statistics from NiFi please share them as well.

Hi,    I have an ExecuteStreamCommand processor which executes a Python script. This takes a long time to execute ~ 5 mins. So I increased the number of concurrent tasks from 1, 4 and 8 but this had. no impact on the performance. I have an 8 core Intel i9 Mac machine with 32 GB RAM. I read that typically the number of concurrent tasks is roughly equal to  2 or 4 times the cores. Could you let me know why there is improvement? How can I improve the performance?   Thanks Ganesh

Hi,   I am trying to create an additional HDFS useruser. So far, I have: Created a new user  Checked the value of Superuser Group (dfs.permissions.supergroup, dfs.permissions.superusergroup). Since we are using unix shell based groups mapping, I created a unix group on all hosts with a name matching the value of Superuser Group Added the desired superuser to the new group on all hosts. I then tested and the new user cannot access all folders on HDFS.   A little investigation shows that neither dfs.permissions.superusergroup nor dfs.permissions.supergroup are present in either hdfs-site.xml or core-site.xml.   How do I make CM modify the configuration file so the superuser group is deployed.   Are there any other steps I am missing?   We are using CDH 5.16.   Thanks, Tom

Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:   If I try and run any of the examples ...they fail. [hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4WARNING: Use "yarn jar" to launch YARN applications.Number of Maps = 4Samples per Map example.com= 4Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Starting Job20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_000920/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 420/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:420/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address

Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//__ / .__/\_,_/_/ /_/\_\ version 2.3.0.2.6.5.0-292/_/Using Python version 2.7.14 (default, Dec 7 2017 17:05:42)SparkSession available as 'spark'.>>>>>> df=spark.sql('select * from sws_dev.vw_dlx_rpr_ordr_dtl_base limit 1').show()[Stage 0:=====================> (18 + 28) / 46]20/03/03 07:01:08 ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file /tmp/blockmgr-c5bcbbe3-8da0-44a0-8025-1b183c81d532/03/temp_shuffle_280c5065-f954-4ec8-b3d0-7c1f5c18b581java.io.FileNotFoundException: /tmp/blockmgr-c5bcbbe3-8da0-44a0-8025-1b183c81d532/03/temp_shuffle_280c5065-f954-4ec8-b3d0-7c1f5c18b581 (Too many open files)at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270)at java.io.FileOutputStream.<init>(FileOutputStream.java:213)at org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(D

I have triggered dataflow and its running everyday. For 1st run, ListFile is able to list files and pass it on to next starge FetchFile. But when i touch file in my input directory, ListFile is unable to list those files. 

Hi,   Which property in Cloudera manager should i update to toggle between the cluster and standalone mode for the nifi cluster? Also, the location of the nifi.properties in the CFM.... Its not under /var/lib/nifi/

We are loading data from oracle table to Hive using NiFi. At the time of loading we will need to do some reference table lookup.(e.g Customer data has Country_code = IN in Oracle, while loading Customer data in HDFS,lookup using Country_code=In will be looked up in reference table and new column with value Country_detail = INDIA will be populated in Customer data).   Could you please guide me how it can be done?

Dear all.   I created one hive temp tables as below - but how can we identified that tables is temp tables or not.       CREATE temporary TABLE IF NOT EXISTS employee ( eid int, name String,salary String, destination String)COMMENT ‘Employee details’ROW FORMAT DELIMITEDFIELDS TERMINATED BY ‘\t’LINES TERMINATED BY ‘\n’STORED AS TEXTFILE;     i used below command to described but not getting any info-     describe formatted employee;col_name data_type comment1 # col_name data_type comment 2 NULL NULL3 eid int 4 name string 5 salary string 6 destination string 7 NULL NULL8 # Detailed Table Information NULL NULL9 Database: h7 NULL10 OwnerType: USER NULL11 Owner: **** NULL12 CreateTime: Tue Mar 03 08:50:28 EST 2020 NULL13 LastAccessTime: UNKNOWN NULL14 Retention: 0 NULL15 Locati

Hi,I'm kinda a beginner in using REST API and i want to create multiple entities at the same time.Watching documentation, you can do it with v2/ENTITY/BULK, however i don't understand how the json file have to be formatted.When I create ONE entity, I do it with v2/ENTITY i just do it like this:curl -s -i -X POST -H 'Content-Type: application/json' 'Accept: application/json' -u admin:$MDP_ATLAS "$ATLAS_BASE_URL/v2/entity" -d '{"entities":{ "typeName": "Referenceable", "attributes": {"qualifiedName":"some.qualified.name1"}}}' But with BULK the doc is saying to format the file like this:{ "entities" : [ { "classifications" : [ { "entityGuid" : "...", "entityStatus" : "ACTIVE", "propagate" : true, "removePropagationsOnEntityDelete" : true, "validityPeriods" : [ { }, { } ], "attributes" : { "property1" : { }, "property2" : { } }, "typeName" : "..." }, { "entityGuid" : "...", "entityStatus" : "ACTIVE", "propagat