Q&A

Hello allI have a slightly annoying problem in NiFi, I have some JSON records that have a field 'date' nested under 'sources' that I want to turn to Epoch time. Using updateRecord I am able to do this using the following config:As you can see where this field is present it works, however sometimes this field is not present, in these cases the record value is blank:To do deal with these blank field I did some searching online and found the following 'isBlank' could be used in updateRecord property to deal with blank fields:The outcome of this fixes the blank field issue... but, doesn't convert the present field values to epoch, see below:Does anyone know how I can get these two actions to perform at the same time? Thanks in advance. Andy 

Hello and thank you for reading  I Have created a randomRead test and it took 40 minutes while randomWrite took 5 minutes is that normal ?  below is the output for Write and read operations  Write test output hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite 1   2022-10-26 13:32:53,776 INFO [main] hbase.PerformanceEvaluation: RandomWriteTest test run options={"addColumns":true,"autoFlush":false,"blockEncoding":"NONE","blockSize":65536,"bloomType":"ROW","bufferSize":2097152,"caching":30,"cmdName":"randomWrite","columns":1,"compression":"NONE","cycles":1,"families":1,"filterAll":false,"flushCommits":true,"inMemoryCF":false,"inMemoryCompaction":"NONE","measureAfter":0,"multiGet":0,"noOfTags":1,"nomapred":true,"numClientThreads":1,"oneCon":false,"perClientRunRows":1048576,"period":104857,"presplitRegions":0,"randomSleep":0,"replicas":1,"reportLatency":false,"sampleRate":1.0,"size":1.0,"splitPolicy":null,"

I have to do some custom preprocessing tasks on a huge data file (~200GB). currently, its works as below way.select * from tablepreprocessing line by linereturn a new single flow fileso I decided to convert the above approach to the below way.get the row count from the user (let's assume the user gives 1000)execute select * query as resultSetread the results line by line (rs.next())when the line count reaches 1000 return the flow file and continues to other linesSo my approach is as below onTrigger public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { logger = getLogger(); FlowFile flowFile = session.get(); if (flowFile == null) { return; } try { final Long rowLimit = context.getProperty(ProcessorUtils.MAX_RECORD).evaluateAttributeExpressions(flowFile).asLong(); Connection conn = DriverManager.getConnection( // db connection properties

Hi guys, Anyone know scripts to remove hdfs file more than 1 years? Thanks,Mazz

I have a requirement to notify once the files on the destination is processed. The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage. Each state has a different count of files. Once the files are sent to cloud storage, notify the end user using email for each state. I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state.  

Hi everyone, I'm geting the next error in NiFi: InvokeHTTP[] Routing to Failure due to exception: IOException thrown from InvokeHTTP[]: IOException -> okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR: org.apache.nifi.processor.exception.ProcessException: IOException thrown from InvokeHTTP[]:okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR I already teste it in Postman and works fine! Any help to try to fix it? Thanks   

Hi Team, Using QueryRecord to filter the incoming array of objects where each object has an array field.The field can be an empty array or it contains some objects in it but it is not null. Tried a lot of SQL queries but didn't succeed. Also, I am trying to consider Record based approach as I want to work on multiple flow files instead of single ones In Actual flow wanted to use ForkRecord on this array but it outputs an empty array if any object contains an empty array in the path so had to use QueryRecord before ForkRecord to filter those empty arrays. Not sure if it's the bug/expected behavior in ForkRecord Flow: GenerateFlowFile:Custom Text: [ { "fields": [ { "field": "field1" }, { "field": "field1" } ] }, { "fields": [] }, { "fields": [ { "field": "field2" }, { "field": "field2" } ] }, { "fields": [ { "field": "field3" }, { "field": "field3" } ] }] Expected output: To filter out the 2nd object whose fie

I am using the importPoliciesFromFile API call but see it deletes and creates the policies. Will this cause any impact on authorization if someone hits the api at the same time or is this API synchronous? By synchronous I mean, does it stop all other code execution and user interaction until the call returns?

Our vulnerability scanning found these two vulnerabilities on our CDP Private Cloud 7.1.7-SP1, CVE-2022-22970 and CVE-2022-22971.  There are several versions of spring-core in the parcel, none of which are the recommended version:./jars/spring-core-4.3.29.RELEASE.jar./jars/spring-core-5.2.18.RELEASE.jar./jars/spring-core-5.3.10.jar./jars/spring-core-5.3.12.jar./jars/spring-core-5.3.13.jar./jars/spring-core-5.3.4.jarIs CDP vulnerable to these vulnerabilities? https://tanzu.vmware.com/security/cve-2022-22970https://tanzu.vmware.com/security/cve-2022-22971 

Hola compañeros,  Requiero de una ayuda y es que he tratado de conectarme a Impala desde Tableau, pero no me permite. Revise la solución recomendada https://kb.tableau.com/articles/Issue/error-code-b19090e0-impala-connector-not-connecting-via-http-transport?lang=es&_ga=2.9066673.509334502.1666737880-398800890.1659558110 por Tableau pero no me funciona, ¿Qué más podría revisar?   Les agradezco si me pueden ayudar :).

Hi Team, We have total 46 nodes cluster.9(2 Master & 7 slave )  nodes on Baremetal and 37 nodes on VCP(VM's).  Can we set any limit/quota on CDH side by restrict using maximum bandwidth from N/W switches??

I have 2 process that I need to execute on parallelo and I have a 3rd process what I wanto to start when firsts boths process will been finished, so my question is: There is a some box processor that I can use in order to implement this logic or somebody has a some idea to develop it. Thank a lot and I am waiting for your answersBest regards

I am a beginner. I submitted the HQL(create table bds_hive.test_ft19 (name string)) using spark3-shell and experienced three permission errors. These errors indicate that I do not have permissions for the corresponding paths on HDFS. According to the error message, I configured the corresponding path permission in Ranger, but it did not work. In addition, there is no denied record in the audit log. I hope guys can help me analyze and provide solutions. Hadoop version:3.1.1.7.1.7.1000-141Hive version:3.1.3000.7.1.7.1000-141Spark version:3.2.1.3.2.7171000.1-1 

When I am going to install hive and hive on tez following issue is reflecting again and again in my cluster "hive metastore canary failed to create catalog" can anyone please have a solution for it.  

Hello everyone, I have a NiFi cluster (1.12.1) with 3 secure nodes with SSL and 3 more external zookeeper nodes, I need to update this guy to version (1.17.0). I think about uploading a new environment with 1.17.0 and migrating the flow files? Would work?

we have set up 4 node cluster and all other 3 nodes start without any exceptions. and in NIFI uiit shows our down node is in connecting state.but our down NIFI node is not running.its logs as below.2022-10-17 19:04:51,860 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 114432022-10-17 19:04:51,950 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinatorjava.lang.IllegalStateException: Already closed or has not been startedat com.google.common.base.Preconditions.checkState(Preconditions.java:173)at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:165)at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1120)at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:349)at org.apache.nifi.we

Hi, I am new to NiIf and need your help please.I have a HandleHttpRequest processor and want to get the URL parameters into JSON.The URL looks similar to this:http://mydomain.de/getsomedata?key1=value1&key2=value2 but may can also be: http://mydomain.de/getsomedata?myVar1=myValue1&myVar2=value2&lang=DESo No. of parameter and parameter names and values can change of course.Now I need a processor that extracts any of key/value pairs from http.query.string into:{  params: {    "key1":"value1",    "key2":"value2"  }} or for 2nd example:{  params: {    "myVar1":"value1",    "myVar2":"value2",    "lang":"DE"  }} How can I achieve that? Is there any processor I can use or should I use ExecuteScript processor?

Hi there,I am trying to load simple CSV to mySQL but i have some difficulties with NULL and empty string values.This is my table for example:create table test1 (id int not null, name varchar(255) not null,age int  null);and my data looks like this in the mySQL:*the blank values are not NULL they were inserted like:insert into test(id,name) values (2,'');Now when I export this into CSV and try to load this CSV back to mySQL it fails to understand that those empty values are just blank and not NULLs.This is my NiFi configuration: I followed the template from another question I found here and removed some parts that I find like I don't need (or maybe I need??) = removed the  AvroSchemaRegistry and instead used Use String Fields From Header.I tried many things from exporting values with double quotes,single quotes but still didn't manage to figure out how to load empty/blank/'' values instead of NULLs.Please help 🙂

HI Team,System Env Details: I am trying setup NIFI on AWS3 node clusterNIFI version:  1.16.0authorisers.xml :       <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">xxx@xxx.com</property> <property name="Initial User Identity 2">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_1%%</property> <property name="Initial User Identity 3">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_2%%</property> <property name="Initial User Identity 4">C=US, ST=xxxxx, O=xxxx, OU=xxxx, CN=%%cn_user_identity_3%%</property> </userGroupProvider> <accessPolicyProvider>

Hello all...i am very new to big data. I currently have downloaded HDP (sandbox) 2.6.5. I also have downloaded HDF 3.1.1 where I practice. Is there anyway i can download HDF services on HDP 2.6.5? if not is there any way to download only Nifi on hdp2.6.5? It will also be very nice to have any answer to download airflow schduler instead of using bulit-in workflow manager?...i try to look and many documentations that was recommended here but no avail.Thank you

Hi all We have Ambari HDP cluster ( HDP version - 2.6.4 ) , with 420 workers linux machines ( when each worker include data node and node manager service ) Unfortunately Ambari DB is damaged , and we not have Ambari DB dump , so we cant recover Ambari DB  , so actually we not have Ambari and Ambari GUI But HDFS disks on workers machines include HDFS data , and name node is still working with all data  as ( journal/hdfsha/current/ ) and ( namenode/current )So HDFS works without Ambari  So regarding what I said until now - it is possible install new Ambari cluster from scratch , and then add existing working HDFS data to the cluster ?Dose hortonworks / cloudera have procedure for this process ?

Hi,I want to execute a flow in two NiFi nodes and I'm fetching data from Cassandra running in the primary node only, but then I have a DistributeLoad with "round robin" strategy to distribute the data for the two nodes.The rest of the processors are with Execution on "All Nodes", is the DistributeLoad redundant? Meaning, if I don't use the DistributeLoad but still use the Execution on "All Nodes" for the rest of the processors, I can still have parallel processing jobs across the nodes? Or are the FlowFiles only processed in the current node?Thanks 

Hi Team,I am creating year date time folder thru groovy ExecuteScript processor and using command execute using creating sh file and writing some content in that.. and passing that sh file to downstream ExecuteStreamCommand processor. Its failing with permission error on sh file.How to give full permission to the file which we create. def commmnd1 = "mkdir -p /shared_path/data/poc/test/1/script"def commmnd3 = "mkdir -p /shared_path/data/poc/test/1/output"def process1 = commmnd1.execute();def process3 = commmnd3.execute();def filePath = "${script_dir}/abcd.sh"def file2 = new File(filePath)file2.write "#!/bin/bash\n";and one more, ExecuteStreamCommand executing sh file and it generating some files in dynamically created by previous ExecuteScript processor and these are files does not have full permission. How to give full permission which we create folder and files dynamically. Please suggest. Thanks in Advance. 

Hello all Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the value of another attribute called 'hostname'. For instance if:#1: hostname = Mickey Mouse#2: hostname = James Bond I want to use a 'contains' statement (or similar ) to change the output of new attribute 'hive_database' property:hive_databasevalue:${hostname:contains('Mickey'):<OUTPUT>('cartoon')} <OR>${hostname:contains('Mickey'):<OUTPUT>('movie')} So that #1 output would be: property= hive_database value= cartoon The parts I'm stuck on is the OUTPUT and OR parts above, do you know if this would be possible using the update attribute processor? I am trying to avoid having to use routeOnAttribute to break out multiple different 'hosname' values to send data to the relevant 'hive_database'  Hope this makes sense, unable to share screenshots etc as sensitive vales. Any help greatly appreciated. 

Hello Community,I have a configuration where ListenSysylog is listenig and provides the received data to InvokeHTTP. Also I use a funnel to redirect Http failures to the InvokeHttp once again. The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts droping incoming data. I do not want to lost any incoming data and in the same time I want to be sure that data will be send. Do you have a similar problem? Is there any way to limit proccessors?Thanks