Solved

Hi, i can't compress so many files.txt into a Parquet format to save in HDFS.How can i do that? @ApacheNifi 

I used the PutParquet processor with CSVReader to compress files.txt in parquet format and moving them to HDFS.Checking from the Browse Directory of Hadoop however the files keep .txt Is it normal? Are they saved in parquet format?Thank you! @ApacheNifi Configuration of PutParquet processor: Hadoop:

We made a mistake while configuring the storage of our worker nodes and we ended up adding a data mount point of a really small disk to the yarn.nodemanager.local-dirs property. This causes some of our very data-intensive spark jobs to fail as this disk runs out of space pretty quickly as these jobs generate a lot of staging data. Right now the value of this property is the following: /u01/hadoop/yarn/nm,/u02/hadoop/yarn/nm,/u03/hadoop/yarn/nm,/u04/hadoop/yarn/nm We just want to remove /u04/hadoop/yarn/nm from the property, all the other directories are mounted on disks with enough capacity.So, is it safe to just remove this directory from the yarn.nodemanager.local-dirs property? Or do we have to back up the data in /u04/hadoop/yarn/nm and create another data mount point using a disk with enough capacity and put that backup there?

hican I have the parcels located on an FS so the installation will include a copy instead of downloading from a remote URL?I believe it will be faster solutionthis is for testing cluster environmentsbetter still - can I copy the parcels to the /opt/cloudera/parcels myself - while I'm preparing the VM - will cdp know that the file is already there or do I need to update also the psql metadata db?thanksdoron

i have added on my cluster 1 node that is empty, and I want to add the keytrustee service in it but I encounter this when setting up : The Key Trustee Server service should not be added to a cluster with existing services.   does the key trustee server needs a separate new cluster or just a new empty node where it will be installed is okay?

Hi All, I have a scenario where I will get a number of records from csv file, the task is to read the csv and split each record as one file and save the each file and sheet name with the record name of the 1st column(exclude 1st row as header).   1. What I have done is  read the csv file using Getfile,  2. then used splittext processor, to split each record as one csv file by setting property Header line count to 1. 3. then need to extract the 1st record at column1, use that record(2nd row and 2nd column) value as file name and sheet name for the each individual file   Original csv file: after split the there should two files, one with the name ab123.csv and c35ks.csv and also sheet name also should be changed. ID Description status ab123 Eldon Base for stackable storage shelf, platinum   ab123.csv   ID Description status c35ks 1.7 Cubic Foot Compact "Cube" Office Refrigerators   c35ks.csv   How to get above out puts after the wo

Hi All,  I have nifi flow fthat fetches one json file and using the extractText processor, i want to store entire flowfile content(assume small json file) into one flowfile attribute. But (.*) is not extracting whole json and instead extracting only { . Please help with Java regular expression to extract full json afnd also share if any Java regular expression documentation which i can refer for Nifi extracttext processor.  I having json file which contains format like {"filename1":"insert into filename1","filename2":"insert into filename2","filename3":"insert into filename3"} I will retrieve the filenames in generic way/dynamic using JsonPath(${filename})  in updateattribute . Please assist to extract whole json content into one flow file attribute .  Thanks @Nifi@NifiDev

First let me say I've searched the board and I've seen the responses and I have a dilemma. Before my cluster establishes I can connect to my servers leveraging Initial Admin and the p12 that I've created. I've verified that all of my nodes are in my authorizations.xml file and reference my users.xml accurately as /proxy resources If I'm not attached to a node prior to joining the cluster the error appears like thisAfter clustering is complete attempt to connect yields the following response in the nifi-users.log2021-01-06 20:38:38,206 WARN [NiFi Web Server-157] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.I don't know where to go from here

Hi Everyone. I already have a Data Lake and Data Hub running in my CDP environment but I’m having issues with registering the ‘Classic Clusters’.After clicking ‘Get Started’ the next screen is basically a blank page? Attached are screenshots for reference. Please help to provide your inputs. Appreciate your help 

Hi, I am using the ExecuteSQL processor in NiFi to write records to PostgreSQL.The problem I have is when running on multi-thread (for example 10), the processor gets stuck and runs too slow. This doesn't happen in a single thread. However, single thread is not efficient so doesn't work for me.After getting slow, I stop the processor, terminate it, and start it again. For almost 5-10 seconds everything is ok and then the same problem.I also have a similar problem with the InvokeHTTP processor. Thank you in advance.

Hello . I am trying to read the mysql bin log data from mysql database and i am getting the error as"Could not connect to Distributed Map Cache server to get table information"when i added the Distributed Map Cache Client value in CaptureChangeMysql Aapache nifi processor .screenshots are attached for the reference  OS :Ubuntu (AWS AC2)NIFI : 1.8.0Mysql : 5.7.*i   

does CDP still needs rack topology/awareness for 10 + nodes cluster? if yes may I know the steps on doing this please. thanks

Hi eveyone, Ambari :- 2.7.4.0HDP :- 3.1.4.0 I want to know whether postgresql in addition to MYSQL ( I will be using MYSQL in advanced database configuration while setting up Ambari server as my choice ) is needed for Ambari ( 2.7.4 ) installation or not. If it is needed should I install it on both Ambari Server host and also Ambari Agent hosts or only Ambari Server host. On which hosts is it required? Please help!@KuldeepK @jsensharma @MattWho @Shelton     

Hello! I'm trying to install Ambari Server as instructed at https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/download_the_ambari_repo_amznlnx2.html I tried the command (replacing username with my username and respective password), but I get the "Authentication Failed" for both commands below.  sudo wget -nv https://username:password@archive.cloudera.com/p/ambari/amazonlinux2/2.x/updates/2.7.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo   sudo wget -nv https://archive.cloudera.com/p/ambari/amazonlinux2/2.x/updates/2.7.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo  I'm using Amazon Linux 2 AMI  I also tried on my local machine (Mac OS) and got the same error message. How should I proceed? Thank you!

Following latest Cloudera announcement about moving to a subscription model.https://www.cloudera.com/downloads/paywall-expansion.html Does that mean that if you have a cluster running 5.x without a subscription, you can no longer access the cluster or Cloudera manager ? or might this still work if you have already downloaded the parcels using Cloudera Manager?

Following latest Cloudera announcement about moving to a subscription model.https://www.cloudera.com/downloads/paywall-expansion.html Does that mean that if you have a cluster running 5.x without a subscription, you can no longer access the cluster or Cloudera manager ? or might this still work if you have already downloaded the parcels using Cloudera Manager?

 After making changes  asssh-copy-id -i /home/oozie/.ssh/id_rsa.pub cloudera@example.local.com getting permission issue in oozie workflow oozi-W@ssh-34f1] Error starting action [ssh-34f1]. ErrorType [NON_TRANSIENT], ErrorCode [AUTH_FAILED], Message [AUTH_FAILED: Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 abc.com  mkdir -p oozie-oozi/0000000-201209133111712-oozie-oozi-W/ssh-34f1--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive).

I'm leveraging CDF and Nifi to orchestrate copying data from a relational database to Hive v3 on a daily basis.  This is using CDF and CDP (on-prem).  I have logic working for creating the tables via an AvroToORC conversion and then using the hive.ddl attribute to create the table.  I have two primary issues: 1) There are no unique columns or primary keys in the source system.  I have a Nifi processor that runs "delete from database.tablename" on Hive.  This clears out the data and leaves the table structure.  On large tables this can take some time.  The reason I have to do this is the PutHive3Streaming processor is not able to recognize duplicates and thus will continually append to the database and over-inflate it with duplicate records.  Are there other options for not needing to drop all the entries but still inserting the data? 2) From a performance standpoint, PutHive3Streaming is working but it's quite slow.  I've compared

Hi ,I have to log Nifi processor success or errors status in shell script to monitor the flow . Please help with solution . I have tried using Updateattribute and log the message in a filesystem and could not complete it . Also i can not access nfi-app.log file into Edge node. Would like to do through REST API calls or some other solution .  Thanks

I have a scenario where list of files are coming from previous processor, where for each file, I have to create json file with attributes of the flowfile. In AttributesToJSON processor configuration there is option to extract pipeline attributes and can create json files/object, if we set Include Core Attributes to true, it will read some of the file properties and forms the json file. the out for the above case in my scenario is …{"fragment.size":"125"file.group:"root",file.lastModifiedTime:"2020-12-22T15:09:13+0000",fragment.identifier:"ee5770ea-8406-400a-a2fd-2362bd706fe0",fragment.index:"1",file.creationTime:"2020-12-22T15:09:13+0000",file.lastAccessTime:"2020-12-22T17:34:22+0000",segment.original.filename:"Sample-Spreadsheet-10000-rows.csv",file.owner:"root",fragment.count:"2",file.permissions:"rw-r--r--",text.line.count:"1"}} But the files has other properties, like absolute.path, filename, uuid are missing in the above json file. My requirement is, get the absolute.path

Hi, I have sensitive parameter defined in process group's parameter context. The documentation says that this parameter can be assigned to a sensitive property only. So I want to know how to create custom sensitive property specific to my nifi processor. I did see few posts where they are asking to use InvokeScriptedProcessor, but I did not find any example. Any suggestions here would be of great help. Thanks, 

After Setup for the Tez Shuffle Handler following instructions on : https://tez.apache.org/shuffle-handler.html I'm getting below error in query execution:Vertex failed, vertexName=Map 2, vertexId=vertex_1608273679503_0002_2_01, diagnostics=[Task failed, taskId=task_1608273679503_0002_2_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1608273679503_0002_2_01_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.Acces

Hello Team, We are using Impala 3.2.0 there are some queries which were working 3-4 weeks back and now they have suddenly started throwing "IllegalStateException: null" exception. In last 3-4 nothing has changed on our Impala cluster. Error logs also does not suggest the cause of the error.  Can anybody help us as to why this issue arising.Query which failing looks something like this WITH banner_test AS (WITH layout_id AS ( SELECT <some columns> FROM table_v1 WHERE ( layout_component_id = <some_id>) ), layouts AS (SELECT <some columns>, coalesce() AS col1, <some columns> FROM table_v2 lch WHERE lch.some_id IN (SELECT some_id FROM table_v3 WHERE some_field_v1 IN (SELECT some_field from layout_id))

Hello All, I have an HDP 3.1 cluster that is neither kerberized nor it has ranger installed. Now, I have a requirement to Kerberize it and also enable Role Based Access Control using Ranger. 1. Should I first go with Kerberization and then installation of Ranger or vice versa?2. Any cautions to be taken with both of these approaches? 3. Is there a requirement/recommended for implementing TLS/SSL for all the services while Kerberizing the cluster? Thank You,Megh Vidani

Hi,     Im new to Apache Nifi and i'm looking on how to filter the CSV data using specific column.      I'm able to convert to JOSN without filtering, but SplitText and RouteonAttribute processors are not helping to filter the data. Below is my input CSV. Input:number,name,resourceState,location,manufacturer111897,lok,INSTALLED,HYD,ABC115677,redd,RETIRED,BLR,ABC1108448,eswar,PROP_INITIAL,CLT,ABC1116740,wqwq,INITIAL,AA,ABC Filtering should be based on resourceState column to consider INSTALLED and RETIRED data. So, converted JSON should have only 2 rows like below. Expected JSON Output:[{"number": "111897","name": "lok","resourceState": "INSTALLED","location": "HYD","manufacturer": "ABC"},{"number": "115677","name": "redd","resourceState": "RETIRED","location": "BLR","manufacturer": "ABC"}] Please help me on the CSV filtering part. Thanks in Advance.