Solved

I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi.Appreciate if you could guide me on this.

I have different input files for different sources. Can I create a common Nifi flow which processes all the different kind of files? Each files will be having different types of columns and different types of output table to be loaded. For example, File 1 will be having column A, Column B to be loaded to Table AB, File 2 will be having column C, Column D, Column E to be loaded to Table CDE. Can I achieve this in a single flow or should I create different flows for different types of files?

Hello.I'm using an ExecuteSQL Processor to extract data from Oracle DB. The query has multiple joins and returns a large number of fields.The problem is that ExecuteSQL Processor returns a single flow file (avro format) that is huge. I want to split the flow file (based in a number of rows, for example) and then merge them at the proper moment.I have read about QueryDatabaseTable and GenerateFetchTable which are processors that can split the output flow files but looks like these processors aren't able to perform joins.. Does anyone knows a workaround?

I have a scenario where the columns in a CSV are consistently changing, I want to validate this CSV irrespective of the changes Currently I am validating using a static schema, by placing a static schema in Schema Text property of Validate Record processor. But everytime there is a change in file format(a new column is added or removed) I have to update the schema in Schema text property of processor.Is there a way I can make this dynamic ?Currently I am using NIFI 1.11.4(I couldnt find InferSchema processor or JSON to Avro converter)

Hello, I get the below error message in my NiFi logs when I tried to write a file to HDFS, but when i try to write a file with in the hadoop cluster it works fine, but from NiFi it fails with the below message.My Nifi service is started by root and, when I have a local NiFi instance i am able to write the file to HDFS, where as from the NiFi cluster am unable to do so, any help would be highly appreciated, I am trying another solution, if that works then I will post it over here.  2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.c.s.StandardProcessScheduler Starting LogMessage[id=b4bc6d2c-0173-1000-0000-00002905a41b]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.controller.StandardProcessorNode Starting LogMessage[id=b4bc6d2c-0173-1000-0000-00002905a41b]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056] o.a.n.c.s.StandardProcessScheduler Starting LogMessage[id=b4bd264b-0173-1000-0000-000018f91304]2020-08-10 13:41:59,519 INFO [NiFi Web Server-32056

Hi All,   Good Afternoon.   We have CDH 5.14 and Impala 2.11 in one of our Cloudera cluster. We have a requirement to upgrade Impala to 3.2 in the same cluster.   Please suggest us the best way to achieve this. Can we just upgrade Impala without upgrading CDH.   Regards, Banshi.

Hello,I tested ListenHTTP on a single node under some load generated by jmeter. I observed that for the one or two client threads the ListenHTTP works fine, however,having 10 client threads leads to 60% request refused by 503,Service Unavailable. Is there any parameter, which helps to come accross the issue?Also the component does not indicate any problems in the GUI. Best regardsJaro

Hello,   I was using Cloudera Enterprise 5.16 (Trial) which is now downgraded to Cloudera Express, since then I am facing issues starting Reports Manager. I am using the default embedded database.   CM / CDH - 5.16.2     2020-09-06 14:14:29,706 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.2020-09-06 14:14:29,711 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 4100.2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 41002020-09-06 14:14:29,725 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication.2020-09-06 14:14:29,761 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: nonejava.net.ConnectException: Connection refused at java.net.PlainSocketIm

Hi Cloudera Community, I am currently using spark1.6 version on cdh5.16.1. Now I have a requirement to upgrade spark from 1.6 to 2.3. So, I want to know which cdh version meets my requirement.

Hi,I want to convert json data to avro.I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:{"name": "person","namespace": "nifi","type": "record","fields": [{"name": "firstname", "type": "string" },{ "name": "age", "type": "int" }]}But i am getting this as my output : prathik*arun,where  is a red dotI am new to Apache Nifi.

Hi, I am running into a problem with deploying a custom controller service to NiFi (I am using the latest version of NIfi 1.12 as of writing this). Now what I am trying to do is to create a custom controller service that would allow me to access a Virtuoso DB instance (similarly to DBCPConnectionPool). This controller relies on 2 external jar files (provided by Virtuoso) that I have placed inside a /libs folder under the module that implements the interface of my service. I have pointed to these jars as dependencies in the pom.xml of that same module and maven is able to import them. Furthermore, I am able to successfully compile and run a JUnit test of my implementation of the service which consists of simply getting a connection and checking if it is open.The problem arises when I compile my service into a NAR and then deploy that service in NiFi. NiFi is able to detect it but when I try to enable it, it throws a NoClassDefFoundError that is related to these external j

Searched for this and have not found the solution or I'm having the stupids. From an API I get a JSON file that I then use the EvaluateJsonPath processor to add an attribute to the flow file. One of these attributes is "date_posted" which is in epoch, seconds since 1st Jan 1970. I need to convert this into a date format so using the UpdateAttribute processor. I have tried these${date_posted:multiply(1000):format("yyyy-MM-dd HH:mm:ss","GMT")}${date_posted:toNumber():multiply(1000):format("yyyy-MM-dd HH:mm:ss","GMT")} and even for testing just these:${date_posted:toNumber()}${date_postedmultiply(1000)}${date_posted:toNumber():multiply(1000)} Yet afterwards my date_posted attribute ends up with an empty string:What am I missing?

Hi guys, This is basic question but not able to solve it...I created a scheduled workflow in Oozie to run a Shell script witch runs some impala-shell cmd. Everything worked fine since I added a where clause for one of them. I'm wondering if it's possible to pass a where clause on a query impala-shell ? Here is my basic cmd in shell script :impala-shell -B -q 'select col1,col2,Col3 from MyDB.MyTable where col1 IN ('argument1','argument2','argument3','argument4','argument5') order by col1,col2' --delimited --output_delimiter='|' -o ${DIR_OUT}/MyTable_$(date +"%Y%m%d-%H%M%S").csv --print_header 2>${DIR_LOG}/MyTable_$(date +"%Y%m%d-%H%M%S").log; It works fine without "where col1 IN ('argument1','argument2','argument3','argument4','argument5')" Is there a way to do it without using a file ? (with -f option ???) Thanks for your adviseAlex

CFM 1.0.1Could someone please explain to me why, after reconfiguring to disable SSL, when I try to access the webUI it still is attempting to connect on port 8443? Is this a zookeeper issue? How can I fix this?I had another cluster previously that start out as SSL disabled, but after configuring it to have a cert via nifi toolkit CA, and getting it WORKING with LDAP auth, it would try to request things from port 8080 http, then after vote resolved it will kill the NiFi node.So here I am now. Was unable to get SSL verified while following the 5 year old guide courtesy of @MattWho (you are awesome, i really appreciate all the info you've shared) I followed that guide to a T but the server reporting the cert was not verified.Now I'm trying to just connect to the nifi node via HTTP, so i disabled all related SSL things in the config, yet: Cannot replicate request to Node nifi.domain.net:8443 because the node is not connectedIt still tries to connect to port 8443. Can someone

Hello, when importing data from DB2 via Sqoop to Hive the stored data in /warehouse/tablespace/managed/hive/databasename/tablename/ is steadily growing.For every import (with --hive-import and --hive-overwrite set) there is a new folder: "base_000000n" created. Thus the parent folder is steadily growing. Any way to delete the old folders before importing new data with Sqoop? regards

Hello, I have CDH 5.16.2 with HBase 1.2.0 and a new Cluster with CDH 6.2.0 with HBase 2.1.0. I would like to use CopyTable to copy data into the new cluster. The containers give an error:ERROR [main] org.apache.hadoop.hbase.client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Is this a pre-hbase-1.0.0 or asynchbase client? Client is invoking getClosestRowBefore removed in hbase-2.0.0 replaced by reverse Scan. at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2452) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.Rp

Hi all   Getting following error when trying to connect via ODBC driver version 2.6.4.1004 (64 bit) :-   [Cloudera][DriverSupport] (1160) Cannot enable SSL for the connection when connecting to a server that has not enabled SSL. If the server has SSL enabled, please check if it has been configured to use a SSL protocol version that is lower than what is allowed for the connection. The minimum SSL protocol version allowed for the connection is: TLS 1.2.   Please suggest.

I am unable to set below parameter when running hive query on spark engine using hive jdbc.   Cannot modify hive.spark.client.server.connect.timeout at runtime. It is in the list of parameters that can't be modified at runtime or is prefixed by a restricted variable.   set hive.spark.client.server.connect.timeout=90000 set hive.spark.client.connect.timeout=900000;   Both parameter is giving error in CDH6.3.3 but it was working in CDH5.16.1. Kindly let us know the option to avoid timeout error.

Hi, As  I learnt that the flow file attributes are held in memory in the nifi process, is it a good idea to clean up the attributes that have been defined earlier and delete them using UpdateAttribute processor if we know that we wouldn't use them anymore ? Or does deleting doesn't matter at all with regards to the memory usage? Please clarify.

I want to host an API in Nifi using HandleHTTP processor. And i wanted to pass some dummy json data. I would like to get some information/resources about how to do it.I am new to Nifi. Thanks in Advance.

Hello guys, has anyone taken the exam since registration of CCA 175 re-opened? If you tried could you tell something about the “new” exam? Is schema of exam different? Thanks in advance Cheers

Hi I've checked this procedure in cloudera on how to validate the encryption of my KMS and HDFS Create a zone and link to the key.su hdfs hdfs crypto -createZone -keyName mykey1 -path /tmp/zone1 Create a file, put it in your zone and ensure the file can be decrypted.su echo "Hello World" > /tmp/helloWorld.txthadoop fs -put /tmp/helloWorld.txt /tmp/zone1hadoop fs -cat /tmp/zone1/helloWorld.txt rm /tmp/helloWorld.txt Just want to ask, i have an existing hdfs directory there that I want to encrypt, my questions are below:1. Can I encrypt and existing hdfs directory using this command hdfs crypto -createZone -keyName mykey1 -path /tmp/zone1?2. if I encrypt the hdfs directory, does the encryption implemented on its sub directories and files under it?  

please suggest if anyone has successfully integrated with AD. I'm trying to use below configs and it's not working  ldapRealm= org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.url = ldap://ad.abc.com:389ldapRealm.contextFactory.authenticationMechanism = simpleldapRealm.contextFactory.systemUsername = svc_abcldapRealm.contextFactory.systemPassword = passwdddd#ldapRealm.searchBase = DC=abc,DC=abc,DC=COMldapRealm.userSearchBase = OU=IDM,DC=abc,DC=abc,DC=comldapRealm.userSearchScope = subtreeldapRealm.userSearchAttributeName = sAMAccountNameldapRealm.userSearchFilter = (&(objectclass=person)(sAMAccountName={0}))ldapRealm.authorizationEnabled = trueldapRealm.groupSearchBase = OU=abcGroups,DC=abc,DC=abc,DC=comldapRealm.groupObjectClass = groupldapRealm.memberAttribute= memberldapRealm.groupSearchScope = subtreeldapRealm.groupSearchFilter = (&(objectclass=group)(member={0}))ldapRealm.memberAttributeValueTemplate= $CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=co

Hi,I am currently facing issue with NiFi 1.7.0 (2 nodes),My flows had been running fine since months. Couple of weeks ago we did a downgrading of the nifi machines. We went from CPU 16 RAM 128G to CPU 4 RAM 32G.After the dowgrading we changed parameters to make sure that nifi take into consideration the new size of the machinesInitial values:Initial memory allocation: 80GMax memory allocation: 112gNew values:Initial memory allocation: 20GMax memory allocation: 26Gwithin nifi we did also change in NIFI Genaral SettingsfromMaximum timer driven thread count: 64Maximum event driven thread count: 16toMaximum timer driven thread count: 2Maximum event driven thread count: 4Now some time when we try to load we get an enexpected error occured with2020-08-18 10:57:51,942 ERROR [NiFi logging handler] org.apache.nifi.StdErr OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000780000000, 218103808, 0) failed; error='Cannot allocate memory' (errno=12)2020-08-18 10:57:51,956 INFO [

I would like to know if there were any perverse effects on using partitioned tables with a single bucket clause. This is to generate only one file in the partition. I use this because the compaction major process doesn't work very good. .I am using this solution to avoid having multiple small files in the partition. In fact, the compaction mechanism (Major) does not always allow only one file in the partition.