Q&A

I have two hive tables and I want to manually create a lineage relationship between them using the Atlas API.I'm trying to run this POST request:```curl --location 'https://[url]/atlas/api/atlas/v2/entity' \--header 'Content-Type: application/json' \--header 'Authorization: ••••••' \--data-raw '{"entity": {"typeName": "hive_process","qualifiedName": "my_etl_process@cluster","name": "my_etl_process","description": "ETL process from input_table to output_table","attributes": {"inputs": [{"guid": "9768381b-8783-49c3-850d-39bf1f14b73f"}],"outputs": [{"guid": "998de7ba-2254-418d-b405-656eba428643"}]}}}'```I'm getting this 404 response:```{"errorCode": "ATLAS-404-00-007","errorMessage": "Invalid instance creation/updation parameters passed : hive_process.qualifiedName: mandatory attribute value missing in type Referenceable"}``` Any suggestions for how to manually create lineage using the API would be appreciated.

Is it a best practice the team make use of the nifi to build Rest API endpoints ?How nifi handle multiples http connections ?I see nifi as a data transform pipeline tool instead of a drag and drop webui to build Rest API endpoint.Best Regards.

Good Day!We are trying to insert the records into HBase tables using the NiFi processor - PutHBaseRecord. But we are facing the below error. Please help us with your expertise. Thanks!Processor Used: PutHBaseRecord Error Message11:09:31 GST  ERRORPutHBaseRecord[id=ee32a0cc-0190-1567-1233-12334fd02c9e] Failed to put records to HBase.: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 5 actions: FailedServerException: 5 times, servers with issues

Templates are deprecated from 2.XI have existing nifi process groups in the form of exported .json filesBelow is the requirement for REST API:1. Create process group using exported .json file if not exist2. Update existing process groupI am able to uplod to existing process group using below curl:curl --location 'https://localhost:8443/nifi-api/process-groups/1e373804-0191-1000-e950-856de82e267d/process-groups/upload' \--header 'Cookie: __Secure-Request-Token=2dae7036-f4c3-4115-9865-6c9d7984dfa8' \--header 'Authorization: Bearer esometoken3M' \--form 'positionX="11.0"' \--form 'clientId="11"' \--form 'disconnectNode="false"' \--form 'groupName="empty"' \--form 'positionY="557.0"' \--form 'file=@"/home/gwl/Downloads/existing-flow.json"' But when i try to upload completely new process group using below curl:curl --location 'https://localhost:8443/nifi-api/process-groups/01911000-3804-1e37-10a0-827d946513c6/process-groups/upload' \--header 'Cookie: __Secure-Request-Token=2dae7036-f4c

HiI'm wondering why the users are not mapped to the AD group as shown below.These are my configurations in Ranger. * COMMON CONFIGSLDAP/AD URL : ldap://myADIp:389Bind User: cn=admin,ou=ou1,dc=dc1,dc=dc2,dc=dc3Incremental Sync: YesEnable LDAP STARTTLS: No * USER CONFIGSUsername Attribute: sAMAccountNameUser Object Class: userUser Search Base: ou=ou1,dc=dc1,dc=dc2,dc=dc3User Search Filter: (empty)User Search Scope: subUser Group Name Attribute: memberof,ismemberofGroup User Map Sync: Yes * GROUP CONFIGSGroup Member Attribute: memberGroup Name Attribute: cnGroup Object Class: groupGroup Search Base: ou=ou1,dc=dc1,dc=dc2,dc=dc3Group Search Filter: (empty)Enable Group Search First: NoSync nested Groups: No  I would appreciate if you answer to me about this.Thanks. 

 I have ConsumeJMS Processor in Nifi, which i have setup using JndiJmsConnectionFactoryHandler. I want to execute this nifi processor with a custom keystore. like how we do when we run java class as follow$JAVA_HOME/bin/java -Djavax.net.ssl.trustStore=keystore -Djavax.net.ssl.keystore=keystore -Djavax.net.ssl.trustStorePassword=**** -Djavax.net.ssl.keyStorePassword=**** -classpath ___ com.example.ApplicationI have checked the truststore we have at the root nifi directory. which is configured in bootstrap configuration file we don't want to touch it we just want to run consumeJMS Processor with a some specific keystore and truststore.

Hello,I wanted to understand what happens to the process and flow files which is currently running when the nifi gets restarted? Please share the details for the better understanding of nifi.

Hi Team,I want to achive the below mentioned transformation in Nifi. using any processor. Please help me to get this done.sample input:{"date": "35 days 11:13:10.88","key1": "value1","keyToBeMapped1": "hostname.com","key2": "value2","key3": "value3","key4": "value4","keyToBeMapped2": "High Paging Rate","key5": "PAGING","keyToBeMapped3": "A high paging activity has been detected on host abc.lab.com. This could mean that too many processes are being run","Entity OID": "keyToBeMapped1","Parameter": "keyToBeMapped2","Description": "keyToBeMapped3"}Expected Output:{"date": "35 days 11:13:10.88","key1": "value1","keyToBeMapped1": "hostname.com","key2": "value2","key3": "value3","key4": "value4","keyToBeMapped2": "High Paging Rate","key5": "PAGING","keyToBeMapped3": "A high paging activity has been detected on host abc.lab.com. This could mean that too many processes are being run","Entity OID": "hostname.com","Parameter": "High Paging Rate","Description": "A high paging activity has been det

We're trying to install CDP 7.1.7 on Isilon with OneFS 8.2.2 We found several documents that we tried to follow: - https://infohub.delltechnologies.com/section-assets/h18730-dell-emc-powerscale-onefs-cdp-private-base-install-guide- https://dl.dell.com/content/docu92689_PowerScale_OneFS_HDFS_Reference_Guide.pdf?language=en_US&source=Coveo- https://infohub.delltechnologies.com/l/powerscale-onefs-user-mapping-mapping-identities-across-authentication-providers/mapping-rules-cannot-contain-user-principal-name And more....We also used the isilon hadoop tools and found a few bugs which we provided Pull Requests for (https://github.com/Isilon/isilon_hadoop_tools/pull/105 & https://github.com/Isilon/isilon_hadoop_tools/pull/106 ) so it seems as if thas hasn't been used in a while. We were able to finish the basic setup but we do have issues creating the proper User Mapping Rules.The docs say:"You cannot use a user principal name in a user mapping rule. A user princip

Hi All, How to fetch all files in a directory based on one specific file arrive in that directory Ex: when i've file with extension .trg is there i've to fetch all files (irrespective of name and extension of these files) in that directorylist there are .txt, .xml and . xlsx files are there in \\server\share\test\all these files should be fetched whenever there is file with extension .trg arrive  I've tried this scenario using  ListSMB-->wait/notify--> Fetchsmb  but didn't working are there any solutions suggested by anyone? 

Hi there,Now for something totally different 🙂 we need to encrypt and decrypt some files for a flow we are working on. Initially gpg was configured on a linux server and we want to move this functionality to nifi.On the linux server we are using this version of gpg:gpg (GnuPG) 2.0.22libgcrypt 1.5.3We encrypt a test file and get a file with a size of 656 bytes. This file can be decrypted by the next system that needs to process this file.We configure nifi as follow:When reading the same file in the flow, all goes well and we send the output to the next system. Which unfortunately fails to decrypt the file. They get an invalid key length error. So I checked the file size from nifi and looks like 394 bytes only.The keyring files used in public key service is an export from the linux system. Which should work fine. Passphrase and all are the same.So we were wondering which gpg is being used by nifi??? Could it be a version compatibility issue? or something else.Any tips would be very help

Dear Team,I am trying to create an AVRO schema where a field can accept both string data(Expected Format 2) and a dynamic array (Expected Format 1).AVRO schema has event/processFlow/characteristic []/value which is supposed to accept both dynamic array and string. For the “object_avro”, we are passing “object_json” as input. However as per "object_avro it is accepting only Expected Format 1 JSON block.Kindly provide us a solution where an AVRO schema can accept both Expected Format 1 and Expected Format 2 JSON's.Avro Schema:{"doc": "Sample schema to help you get started","fields": [{"name": "event","type": {"fields": [{"name": "processFlow","type": {"fields": [{"name": "id","type": "string"},{"name": "characteristic","type": {"items": {"fields": [{"default": null,"name": "id","type": ["null","string"]},{"default": null,"name": "name","type": ["null","string"]},{"default": null,"name": "valueType","type": ["null","string"]},{"name": "value","type": {"items": {"type": "map","values": ["s

Hello,I'm trying to acces Spark History UI through Apache Knox proxy but I get this error the following error:Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://<spark-host>:18080/api/v1/applications?limit=2147483647&status=completed. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 302.I can' t find something related to CORS in Spark do you know how I could fix that ?Thanks for your answer

I have configure knox in host knox111.abc.hadoop.dc.corp.com and namenodes in nn01.abc.hadoop.dc.corp.com, nn02.abc.hadoop.dc.corp.com.The knoxsso redirection works fine when directly configuring NN with individual Knox host but when using VIP that is configured as knox-hadoop-def.corp.com the redirection to NN doesn't work.I have configured below whitelist setting but still redirection fails. VIP is configured with CA certs and individual knox hosts are configured with self-signed.gateway.dispatch.whitelist = .*knoxsso.redirect.whitelist.regex = .* knoxsso.redirect.whitelist.regex = corp.comAny help is appreciated!

@mburgess --I am using JsonQueryElasticSearch  to fetch data from elastic .I am using query to fetch only  one record using a particular event_id the processor reads the same data and duplicate records are stored in Oracle DB

Hi everyone, I would like some clarification. Thank you.When we enabled Kerberos, we encountered some errors during the "start clusters" phase in the command details, specifically when starting HBase.Subsequently, we examined the HBase log: cat /var/log/hbase/hbase-cmf-hbase-REGIONSERVER-cdp717w2.am.ocp.poc.log.out2024-05-02 23:14:41,118 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:16020-0x30046d0bb0d003e, quorum=cdp717m1.am.ocp.poc:2181,cdp717mo.am.ocp.poc:2181,cdp717w0.am.ocp.poc:2181, baseZNode=/hbase Unable to get data of znode /hbase/runningorg.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:120) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:35

I am trying to automate the cloudera services installation process how can we manually install services like hue,hive,ranger using package install

I used NiFi from HDF sandbox (PC A) to copy files to HDFS of HDP sandbox (PC B). NiFi can read files list from HDFS without any error. The flowchart is shown below:However, NiFi can not get files or put files on HDFS, the errors are shown below:GetHDFS processor error:PutHDFS processor error:Can anyone help me? Thank you very much.

Error while trying to upgrade nifi-registry from 1.25.0 to 2.0.0M1-------------------------------------------------------------------------------------------------------------------Error : ERROR [main] o.springframework.boot.SpringApplication Application run failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessResource' defined in file [/opt/nifi-registry/nifi-registry-current/work/jetty/nifi-registry-web-api-2.0.0-M1.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessResource.class]: Unsatisfied dependency expressed through constructor parameter 5: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception with message: Identity Provider configuration XML was prote

I have 5 nodes of NiFi cluster. using 3000 processors. which contains 150+ pipelines(which have some simple use cases like reading the date from RDBMS or from file directory, applying some transformation, and writing it to Kudu table or HDFS. and in some use cases we are making http calls by using InvokeHTTP and JOLTTransformation processor). we are using PutKudu processor in many pipelines. some of the custom processors also we used(like ExecuteGroovyScript, GetSMPP).PFB the cluster details :Each node has 24 GB ram.2 nodes have 16 core, another 2 nodes have 32 core and the remaining node have 48 cores.  Almost all the times, I am not seeing much files in the queue and active threads are also like less than 50 in each node. but still we find the NiFi UI slow. and each node consuming more than 80% of jvm heap space regularly. CPU utilization are also going beyond 80%Can someone review my configuration or guide me here to resolve this issue. 

I'm working on a data processing flow using Apache NiFi, and I have a scenario that I'd like some guidance on.I have an Execute SQL script that retrieves data from a table in a SQL Server. Among the data returned, I have source and target folder paths.My goal is to loop through each of these source folder paths. For each folder, if there are any files present, I need to:Insert a record into the database to log the file's details.Move the file from the source folder to the corresponding target folder.I'm currently using NiFi, and I'm wondering what processors and strategies I should consider to achieve this workflow efficiently. Any insights, recommendations, or examples of similar workflows would be greatly appreciated.Thank you in advance for your help!

We have a requirement to collect all the data that we receive for one day in one text file. We get the data at different time intervals, We need to collect all day on a text file with the filename as currentdate.txt.Let's say, on 24th July 2024 we received some data like "This is data 1" in the morning then we need to create a new file with the name "24072024.txt" and add the data to the file. Later in the afternoon, we received data like "This is data2".  then this data needs to be added to the text file - "24072024.txt".  Also if the file does not exist then we need to create the new file with that date. So please help us with the NiFi flow how to append the text in the file if it exists and then how to handle the condition to create a new text file if it does not exist.

We're doing a bit of research here and trying to understand if LLMs are only available by accessing them through git and huggingface or whether there are some in built LLMs accessible using CML. Thank you

Is it possible to conenct and query Nessie catalog using Hue editors/connectors?

I am doing an ETL project that is taking data from CSV files and loading into a postgresql database. The files can contain up to 1M rows, and I need to process many files.Using NiFI version 1.15.2.The flow that I have built can be simplified into "Fetch CSV" -> "Convert to Avro" -> "PutDatabaseRecord".But I am having an issue where any timestamp columns that have microsecond precision are being truncated to seconds when inserted into the database. The project's requirements are for no data loss, so losing the microseconds will fail the requirement.I have tried using both inferred and explicit schemas in the controller services, but both methods truncate the timestamps. I created a test flow and database to explore the issue. The database has columns for storing the timestamp in string, timestamp and as an epoch number. My schema for the controller service{"name": "timestampTest","namespace": "nifi","type": "record","fields": [{"name": "bom_stn_num", "type": "s