Solved

I have this flowchart in nifi, where json files (ConsumerKafka) are consumed and then pass them in mongodb   the first Json file is:   {"_id": "Eems_20200312", "Name": "Eems", "Date": "2020-03-12 23:14:02", "Volatility ": "5,328", "Value risk": "12,395"}{"_id": " Esprinet_20200312", "Name": " Esprinet ", "Date": "2020-03-12 23:14:02", "Volatility ": "5,654", "Value risk": "12,765"}{"_id": " Eurotech_20200312", "Name": " Eurotech ", "Date": "2020-03-12 23:14:02", "Volatility ": "5,534", "Value risk": "12,832"}   the second Json file is:   {"_id": "Eems_20200312", "Name": "Eems", "Last": "0,066", "%": "-9,04", "Hour": "17:26", "Min": "0,061", "Max": "0,069", "Open": "0,068", "Date": "2020-03-12 23:14:02"}{"_id": "Esprinet_20200312", "Name": "Esprinet", "Last": "3,495", "%": "-14,86", "Hour": "17:35", "Min": "3,47", "Max": "3,89", "Open": "3,89", "Date": "2020-03-12 23:14:02"}{"_id": "Eurotech_20200312", "Name": "Eurotech", "Last": "5,05", "%": "-12,17", "Hour": "17:36

Below is the text and I need to write the regular expression so that I can extract attribute from this field.  2017-04-27 09:07:20.517342 10.69.1.138 -> 10.69.1.100 <4>Apr 27 03:37:13 kernel: [ 78.294310] TS|4294745280|00:14:5a:03:29:33|RSSI_BCN|34   timestamp:2017-04-27 09:07:20.517342  source_ip:10.69.1.138 dest_ip:10.69.1.100 mac:00:14:5a:03:29:33 obm:RSSI_BCN dbm:34 How can I write the regex sothat I can collect each of the field from the text?

Hi Team,   We have 12 nodes cluster hosted on premise in 2 different regions. The question is how can we keep the data in 2 data centers in sync and what will be the latency.   Appreciate your help.   Thanks & Regards  

Hi All,   When Kerberos and TLS is enabled on the CDH cluster, role logs are not reported in Cloudera Manager under the respective services. CM reports 'No log messages at the specified URL' for each of the following: Role Log File Stdout Stderr Interestingly, the log files mentioned on the log pages (on CM) exist on the underlined host and contain all & correct log entries; They just don't get reported at CM service log pages. When tried downloading full log files, it throws error: HTTP ERROR 403 Problem accessing /cmf/process/335/logs. Reason: Unexpected end of file from serverThe server declined access to the page or resource.   Powered by Jetty:// 9.4.14.v20181114   Same error is observed when attempted fetching logs using CM API.   Environment: CDH 6.2 OS Redhat 7.7  

Dear community,   After downgrading CDH 6.1 from Enterprise to Express edition, is it possible to upgrade again to Enterprise using same license file used in Enterprise edition?   Our problem is that we want to add additional data nodes to cluster, which will result in exceeding valid licensed node count. So we are thinking to downgrade to Express, add nodes, and wait for procurement for licenses 🙂  

Hello, I am at below screen where I have to give host names.   http://localhost:7180/cmf/express-wizard/hosts   I have installed VirtualBox on Mac OS and using Ubuntu 16.   Have installed below cloudera-manager. wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin   I am confused what should i give in hostname in the current screen (http://localhost:7180/cmf/express-wizard/hosts) I tried 127.0.0.1 / localhost/ and ip addres which i got from ifconfiug but none of them is working Your help will be highly appreciated !!!!

I have a column in a table and value is (1;13004211,13004211_02_13004212,4000000003378605589,1105,2000,)need output like   Column 1 Column 2 Column 3 Column 4  Column 5 13004211 133004211_02_13004212 4000000003378605589 1105 2000   Note: I don't  want the first 1;

I am able to Export and Import controller services from one instance to another. Processors which are referencing to a controller service are not being mapped/selected. Looking for better possible way.   Thanks in Advance

I get the following error when connecting to Hive.     Exception in thread "main" java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: Invalid status 21. at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) at com.cloudera.hiveserver2.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source) at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.hiveserver2.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)   &n

Hi,   I'm trying to get data from an SFTP server. The recommended pattern from articles I have read suggest ListFTP to Remote Process Group, Input Port to FetchSFTP. I'm accessing Nifi on a local instance (localhost:9090/nifi).   After adding a RPG to the canvas as well as an input port, I get the error "Connection refused on the RPG" and "'http://localhost:9090/nifi/' does not have any input ports." when I try connect ListFTP to the Remote Process Group.  

We are planning to upgrade our Cloudera cluster from 5.8.4 to 6.3.3, but our OS is in IBM AIX, just want to know if it is still supported by the new version of Cloudera?

Hi, Need help on below query. We have a scenario where for any column defined with datatype as string in Hadoop, NULL value is loaded as blank. However, any column other than string datatype (INT,DOUBLE etc) NULL values are loaded as NULL in Hadoop.    Column name          Data Type service1end               string service1start              string service2end               string service2start              string firstlinemaintcost    double       Is this a default behavior of Hadoop/hive?

I'm wondering is it possible to create NiFi workflows with the existing processors using REST Api or CLI instead of creating workflows in GUI. Is REST Api can be used to create workflows or only to modify and schedule workflows. If we can create workflows using REST api or CLI, any articles or blogs to look into it?Let's say I want to create a workflow which simply takes file form local and put it to hdfs. This can be possible by drag and drop the processors in GUI but Is it possible to do the same using REST API or CLI or any other thing which we can do programatically?  It would be a great help if you guide me regarding this.Thanks a lot.

Dear All, I have ran into a very troublesome issue which is creating headache for the last 1 week. I installed and HDF Cluster(v3.4.1.1) with the services Nifi,Kafka,Metrics and Smart Sense. Everything was working perfectly until I decided to enable Kerberos(MIT) in the cluster. During the keberization, everything went fine till the last step.(Start and Test services). At that point, the Zookeeper server start got failed and thereafter no services are starting. The following is the error I got from the zookeeper logs.Any help is appreciated.   2020-03-06 18:38:51,489 - INFO [main:QuorumPeerMain@127] - Starting quorum peer2020-03-06 18:38:51,552 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly:javax.security.auth.login.LoginException: Message stream modified (41)at org.apache.zookeeper.server.ServerCnxnFactory.config

I am trying to install cloudera on machines with fips enabled and because of this the install keeps failing. Specifically because the version of open ssl with fips enabled doesn't allow for md5. My question is there a specific process for installing a cloudera cluster on a machine with fips enabled?

Hi Cloudera Community, Our client is planning to upgrade their CDH and Cloudera Manager from version 5.8.4 to 6.3.3, may I please if do you have a recommended upgrade path for this? Thanks in advance!

https://www.cloudera.com/products/pricing.html   Where is cloudera's promise and committment to open-source ? How are we supposed to use the CDP stack on an on-premise solution ? When is the release which supports on-premise deployments going to be available ?

Hi, I’m facing a problem that I could not solve. Can somebody help me?The details are below. Thanks! Cloudera CDH 6.2.1 HUE bundleHue 4.3.0 - cdh6.2.1 Problem:Can not open workflow editor, or scheduler editorLog: [28/Feb/2020 09:42:16 +0100] access ERROR 172.26.151.2 admin - “POST /desktop/log_js_error HTTP/1.1”-- JS ERROR: {“msg”:“Uncaught ReferenceError: Unable to process binding “click: function(){return openShareModal }”\nMessage: openShareModal is not defined”,“url”:“http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js",“line”:52,“column”:14162,“stack”:"ReferenceError: Unable to process binding “click: function(){return openShareModal }”\nMessage: openShareModal is not defined\n at click (eval at t (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:52:10372), :3:58)\n at a (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:51:1216)\n at init (http://172.26.151.122:8888/static/desktop/js/hue-bundle.651d3f665f75.js:

How can I use ReplaceText processor to function like dos2unix command, i.e. change \r\n to \n My file comes in with ^M at the end of each line In the ReplaceText process, I have a Search Value of \r\n That is correctly finding the Windows linefeeds However, for Replacement value, I have tried \n also $1\n   They each put the character n where I want the line feed to be. Thanks!

Hi,    I'm currently using merge_content processor as merge avro files from two sources like kafka_consumer processor and fetchHDFS file. While converting avro file into one with merge content processor yesterday around 680MB trying to convert but processor drop the file and join with new files and i can't able to recover that data also, because content_repository backup i limit. Can you please help me out for this use_case processor how much size can be good or is there any setting needs to modifiy in nifi.properties. 

Hey Guys,   Need to establish links between dependent flows within the same NiFi cluster / canvas - e.g. one flow finishes ETL, triggering another flow to begin its ETL on the result of the first's.   From what I gather this could be achieved in 3 ways: 1. Direct connections between flows - undesirable in our production environment and data governance and a spider's web of dependencies 2. Remote processor groups (RPGs) - if can send / receive from same NiFi Cluster  3. HTTP   Could you please advise on options 2 & 3 as a solution for this?  RPGs seem simpler than handling HTTP requests / responses yet require an input port on the top level of the flow, which would then need to be directly connected into each flow (being another version of option 1).   Thanks

I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302 and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303 and file abcd.out.gz.20200302 under /data/20200302.   How can i achieve this in NiFi. 

We use NiFi mainly for file transfer, processing these files etc. The requirement is to pull reports from NiFi which has the statistics such as how many files were processed from NiFi, how many of them are successfully transferred to destination, how many are failed and waiting in different queues.The NiFi rest api documentation(https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) has the API's to pull this information, but i did not find any info on how to use these API's. Is the API's exposed out of the box or should it be enabled somehow.Please share your suggestions on this.If there is any other way to extract the data statistics from NiFi please share them as well.

Hi,    I have an ExecuteStreamCommand processor which executes a Python script. This takes a long time to execute ~ 5 mins. So I increased the number of concurrent tasks from 1, 4 and 8 but this had. no impact on the performance. I have an 8 core Intel i9 Mac machine with 32 GB RAM. I read that typically the number of concurrent tasks is roughly equal to  2 or 4 times the cores. Could you let me know why there is improvement? How can I improve the performance?   Thanks Ganesh

Hi,   I am trying to create an additional HDFS useruser. So far, I have: Created a new user  Checked the value of Superuser Group (dfs.permissions.supergroup, dfs.permissions.superusergroup). Since we are using unix shell based groups mapping, I created a unix group on all hosts with a name matching the value of Superuser Group Added the desired superuser to the new group on all hosts. I then tested and the new user cannot access all folders on HDFS.   A little investigation shows that neither dfs.permissions.superusergroup nor dfs.permissions.supergroup are present in either hdfs-site.xml or core-site.xml.   How do I make CM modify the configuration file so the superuser group is deployed.   Are there any other steps I am missing?   We are using CDH 5.16.   Thanks, Tom