Q&A

I am trying to pull the records from the Hbase based on the row key in pyspark using the following details  but the records which is returning showing all the records (i,e without filter), can you please help me to resolving the issue 

hi cloudera, I need to use Spark on a host that is not part of the Cloudera cluster to run Spark jobs on the Cloudera cluster.Is it possible to use it this way? If yes, how to configure?what I've already tried:1. Download "https://www.apache.org/dyn/closer.lua/spark/spark-3.3.4/spark-3.3.4-bin-hadoop3.tgz"2. Copy the "conf" files from the Cloudera cluster and send them to the new Spark directory3. exported the variables "HADOOP_CONF_DIR" and "SPARK_CONF_DIR" and "SPARK_HOME" using the new spark directory "spark-3.3.4-bin-hadoop3" with the files4. When trying to run spark-shell as an example, nothing happens, it hangs as shown below:Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 3.3.4/_/Using Scala version 2.13.8 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.16.1)Type in expressions to have them evaluated.Typ

Hi - I've got everything working with NIFI 2.0.0 M2 when the the python command line is commented out but I'd like to get it up and running with the python command line uncommented.nifi.properties file linenifi.python.command=/usr/bin/python3.11 nifi-app.log info:Caused by: java.lang.IllegalStateException: Failed to communicate with Python Controllerat org.apache.nifi.controller.FlowController.<init>(FlowController.java:581)at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:411)at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:83)at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)... 95 common frames omittedCaused by: java.io.IOException: Failed to create Python Environment ./work/python/controller: process existed with code 1at org.apache.nifi.py4j.PythonProcess.setupEnvironment(PythonProcess.java:223)at org.ap

From Hue security browser,  i have a RBAC role (admin), which was given all permission on all databases.I have this Database A which consists of table1,2,3I want to deny this admin role from dropping Database A -> Table1 and Table2.Can you show me how this can be done?

Hi Team,I am using Nifi (3-node cluster) version 1.23.2 in my environment. My doubt, is there any case if Nifi processor (any processor within a process group) stops suddenly due to load/any other issue? If that happens, then what will be the solution? Does it lead to data loss?  

Hi Team, hoping for the best that someone will help me out from this issue ASAP.I am working with nifi in local machine from couple of months  to develop some jobs.now as part of developing one job I need to read remote file in nifi by using minifi,as like tailfile processor how it reads nifi log files continuously I have to read remote log files.in the process of finding solution  I seen that by using minifi we can do that.so firstly I installed minifi remote server because I have to read remote log files by providing local machine path of outputfolder those log files will place in that path.After installing minifi in remote server.unzippped it and tried to start the services by using ./minifi sh start .now I could see services are started and next firstly tried to take and input port in nifi UI and connected that with putfile processor by providing output path and provided connection for input and putfile .nextly taken a processor group and inside that processor group

We try to use PutHDFS process ,automatically the owner of file which created by PutHDFS in Cloudera  will be same domain user which NIFI running.The OS of NIFI is window 10 and the server of Hive is Cloudera docker.how we can update owner of file from domain user to Cloudera user.

Good evening, I am having a lot of trouble getting an instance of NiFi running with Mutual TLS. I have different errors depending on whether the effort involves version 1.xx or version 2.xx. I'm hoping to get some help. This is all being attempted via Docker.My YAML:```version: "3.9"services:  nifi:    image: apache/nifi:1.24.0    # hostname: nifi    container_name: nifi    ports:      # - 8443:8443      - "9449:9449"    environment:      - NIFI_WEB_HTTPS_PORT=9449      - NIFI_WEB_HTTPS_HOST=0.0.0.0      - NIFI_WEB_PROXY_HOST=localhost,localhost:9449,192.168.1.56,192.168.1.56:9449      - AUTH=tls      - KEYSTORE_PATH=/opt/nifi/nifi-current/certs/jks/nifi-ks.jks      - KEYSTORE_TYPE=JKS      - KEYSTORE_PASSWORD=randomPassword      - TRUSTSTORE_PATH=/opt/nifi/nifi-current/certs/jks/trus

I am currently running Hbase CDH 5.5 on Centos 7. I need to upgrade to RHEL8. Does CDH need to be upgraded as well ? What happens to the existing data ?

Hello ,I'm using NIFI to replicate 250 million messages between Kafka topics.The problem is that NIFI replicates messages in a non-sequential order, resulting in the destination topic storing messages differently than the source topic.for examplesource topic - partition 0offset:5 key:a value:v1offset:6 key:a value:v2offset:7 key:a value:v3destination topic - partition 0offset:5 key:a value:v2offset:6 key:a value:v1offset:7 key:a value:v3The topics are configured with a cleanup policy: compact.I'm using ConsumeKafka and PublishKafka processors to replicate topics.      

I am running count(*) from table (Apache Phoenix 5.1, HBase2), it results in high number of requests in HBase1,4k requests for table with 7k rowsIn previous version (Hortonworks, Hbase 1.1, Phoenix 4.7)  count query was not resulting such high number of requests in HBase.tried (major) compaction and updating Phoenix statistics, not helped.Is this expected or miss some configuration? obviously it makes fullscan even when there is primary key for the tableHBase UI x Count query

I have this JSON:{ "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby **bleep**", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J.R.R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ]}I want a list/record set of categories. CSV, plain text, doesn't matter, but let's say CSV:category"refereces""fiction""fiction"I have tried many things, too many to repeat them all here. But basically, I have a GenerateFlowFile where the JSON is hard-coded, then a QueryReport where the reader is a JsonPathReader where I have properties for all fields in the JSON:store $.storebook $.store.book[*]category $.store.book[*].categoryetc.Just to see what'

I created a template to convert data from querying Postgres database and then convert it into xml. I want to run it by using command line arguments. Are there any way to do this in NiFi? Thank you

I have set following:nifi.flowfile.repository.implementation: org.apache.nifi.controller.repository.VolatileFlowFileRepositorynifi.provenance.repository.implementation: org.apache.nifi.provenance.VolatileProvenanceRepositoryWith this, I expect disk io operations to go down(checked with nmon). But its just opposite and disk writes have gone up multiple times.Why so?

Hi,I am using 3 node NiFi (Version: 1.23.2) cluster. If a node goes down, it is known that the flow can be directed to another node, but what about the data that is queued in the failed node? Is there anyway to replicate the data from failed node to other node? or will have to wait until the node comes back up? Please explain how to handle this scenario.

Hi everyone,I'm trying to figure out what causes this Impala warning that is generated when external tables are created:Impala does not have READ_WRITE access to path 'hdfs://xxxxxxxxxxx'At first, we thought it was a permissions issue with Ranger, but after numerous attempts, we still get the same warning.We also checked the permissions on the HDFS paths, but there are folders with permissions of 775, 750, and 755, so there doesn't seem to be a correlation between the warning and POSIX permissions.Could it be an issue with the user and/or group? In many paths, the owner and group of the folder are hdfs:hdfs. Should the owner be changed to impala?Unfortunately, I haven't found any helpful documentation on this topic that would allow me to eliminate the warning. Many users think that the tables are actually not being created and that there is no way to write to them.

Hi,I have an issue with the split/merge of a flowfile containing data in GeoJSON format. The input records have a "geometry" field. Out of the 100,000 elements, 99,999 are polygons (schema ARRAY[ARRAY[ARRAY[DOUBLE]]]), and 1 is a multipolygon (ARRAY[ARRAY[ARRAY[ARRAY[DOUBLE]]]]). The split appears to work fine, but the merge fails with the following reason: MergeRecord[id=018d1000-b9c2-12b6-b7e9-c2b9feebd172] Failed to write MapRecord[{idu=17142000BE0307, geometry=MapRecord[{type=MultiPolygon, coordinates=[Ljava.lang.Object;@49cbb2a3}], statut_foncier=Privé}] with reader schema ["idu" : "STRING", "geometry" : "RECORD", "statut_foncier" : "STRING"] and writer schema ["idu" : "STRING", "geometry" : "RECORD", "statut_foncier" : "STRING"] as a JSON Object due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Ljava.lang.Object;@201dfc5d] of type class [Ljava.lang.Object; to Double for field coordinates: org.apache.nifi.serialization.rec

I have a mysql destination table with column type datetime.my nifi flow is like convertJsontoSQL -> putsql (into destination table) The action is delete for putsql.after conversion of jsontosql i am getting a value '2023-03-28 15:54:28' which is a valid timestamp.but the flow is failing with below error: duetoThevalue of the sqlargs. 14.value is '2023-03-28 15:54:28',which cannot be converted to a timestamp; routing to failure: java.sql.SQLDataException:The value of the sql.args. 14.value is '2023-03-28 15:54:28',which cannot be converted to a timestamp- Caused by: java. text. ParseException: Unparseable date: "2023-03-28 15:54:28" I can able to use the same "2023-03-28 15:54:28" for manually inserting into a sample datetime column.and can able to delete a record with where clause as this value.I could not understand why Nifi is still not able to convert it to timestamp. Any suggestions are appreciated.

Hi All,we try to fetch the results from database according to the content of cvs file.after converting the csv to json format by SpileRecord  then we try to  get the content of Json record  by  EvaluateJsonPath  then we  try to get value of attribute by  UpdateAttribute  but the  FF_Content attribute into inEvaluateJsonPath not recognized . listFile->FetchFiles->SpileRecord->EvaluateJsonPath ->UpdateAttribute ->ExecuteSQL   

Facing above exception when connecting to Phoenix :Facing above exception when connecting to Phoenix failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]] Using below Phoenix jars with version :phoenix-5.0.0-cdh6.2.0-client.jarphoenix5-spark-6.0.0.7.1.8.44-1.jarphoenix-spark-5.0.0-cdh6.2.0.jarScala : 2.4.0Spark : 2.11

Hi We have a nifi cluster of 3 nodes on which we are implementing SSL. All the configuration changes are done but when we access NIFI UI on our browser It throws this error - System ErrorThe request contained an invalid host header [127.0.0.1:<port>] in the request [/nifi/]. Check for request manipulation or third-party intercept.Valid host headers are [empty] or:Our port forwarding is like this - ssh -L  18443:<node-1-ip-adress>:8443 -i ~example.comssh -L  28443:<node-2-ip-adress>:8443 -i ~example.comssh -L  38443:<node-3-ip-adress>:8443 -i ~example.comThen our ssh tunelling via putty is like this - L28443    127.0.0.1:18443L48443    127.0.0.1:38443L58444    127.0.0.1:58443Any ideas how to resolve it ?  

the platform is : CDH 5.14.4after setting federation, the hdfs entrypoint is: viewfs://cluster6/ , the back namenode entrypoint is : hdfs://nameservice/ and hdfs://nameservice2/add new warehouse directory is  /user2/hive/warehouse and managed by nameservice2.but i found the strange things is  if i create database using hdfs://nameservice2, sentry can sync ACL with hdfs, but if i used viewfs://cluster6/ , sentry can not sync ACL.for example:create database test location '/user2/hive/warehouse/test.db'    can not sync ACLbut create database test location 'hdfs://nameservice2/user2/hive/warehouse/test.db' is ok.who knows why ?

Dear all, I have a query regarding the insertion of a flowfile into a MySQL Database. Specifically, I would like to know how to retrieve the auto-incremented ID from the database and store it as a flowfile attribute.Consider the following example flowfile (representing one record, with many more records in reality):My goal is to insert this data into 'table1,' which has an auto-incremented ID field (let's assume that after insertion, the ID will be 3001). Could you please guide me on how to retrieve this auto-incremented ID and assign it to a flowfile attribute?

Hi there, I need some help regarding using a flow.xml.gz from 1.13 to 1.24. First our nifi setup: 2 node cluster running on ec2 instances in aws where we run nifi in docker containers. We have succesfully upgraded from 1.13 to 1.24 after making a backup of the flow.xml.gz file.We then stop the nifi containers and put back the flow.xml.gz from version 1.13. Then startup nifi. Unfortunately we get the following error:Jan 19 07:20:58 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,216 INFO [main] org.eclipse.jetty.server.Server Started @78240msJan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,304 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:58,326 INFO [main] o.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8888Jan 19 07:20:59 nifi-node-1 docker[27934]: 2024-01-19 07:20:59,036 WARN [main] o.a.nifi.fingerprint.FingerprintFactory Schema validation error pa

After having lost all my test work I decided it is time to investigate how to persist data in NiFi when running it in a docker image. I use the official image. This is what I managed to gather so far: services: nifi: container_name: nifi image: apache/nifi:latest ports: - "8443:8443" - "16543:16543" volumes: - "C:/nifi/data:/data" - "C:/nifi/state:/opt/nifi/nifi-current/state" - "C:/nifi/database:/opt/nifi/nifi-current/database_repository" - "C:/nifi/flowfile:/opt/nifi/nifi-current/flowfile_repository" - "C:/nifi/content:/opt/nifi/nifi-current/content_repository" - "C:/nifi/provenance:/opt/nifi/nifi-current/provenance_repository" When stopping/staring the container everything is OK (login credentials and content)When removing the container, the credentials are persisted but the content is gone. I presume that there is yet another directory that I should mount in order to have the data saved there. Which one is it?&nbsp