Q&A

Hi everyone.We have an NiFi cluster consisting of 3 nodes. After the failure of the disk subsystem on one of the nodes, it was in the ReadOnly state for a long time. After resolving the issue and after restarting cluster we are getting the following error on problematic node:2022-09-16 15:31:07,255 ERROR [Load-Balanced Client Thread-4] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer xxxxxx:9443java.io.EOFException: Expected StandardFlowFileRecord[uuid=6ce9e262-b20b-4372-a3b9-43c2c00e8caa,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663256223724-231072817, container=default, section=49], offset=387990, length=2203],offset=0,name=04190e1f-fdca-4352-a796-6b6c9ce41baa,size=2203] to contain 2203 bytes but the content repository only had 1130 bytes for it at org.apache.nifi.controller.queue.clustered.ContentRepositoryFlowFileAccess$1.ensureNotTruncated(ContentRepositoryFlowFileAccess.java:83) at org.apache.nifi.controller.queue.clustered.Con

I have about 25-30 XML message types and each message type has its own XSD.  I need to validate each message against their respective XSD.  When using the ValidateXML processor is there any way to dynamically assign the appropriate XSD to a flow file based on on attribute value?  I don't see the purpose/benefit of using so-called variables when said variables aren't even variable--they are STATIC! Why does this processor ONLY use variable_registry variables and not attribute values like every other processor in NiFi?    

  Hi all,I want to add 15 minutes to a date-time value using the transform json processor, but with my configuration I get the same value as before. I expect for the end field to have 15 minutes more than the start field. Does anyone have any idea? [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30:00", "out" : 0}] My jolt Specification [ { "operation": "modify-overwrite-beta", "spec": { "*":{ "end": "=${end:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}" } } }] The result is the same  [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30

Hi. Could somebody help me please with composite group provider settings in authorizers.I'm not sure what should I put in "Initial admin identity" and "Initial user identity 1".1) When I write just username (like nifi-admin) I cannot login to NiFi since I don't undertand where I can get a password.2) When I write a full LDAP name (like uid=freeipa,cn=users,cn=accounts,dc=foo,dc=bar) I successfully log in but I cannot access a NiFi interface.3) When I use name of user who already exists in LDAP, I get an error, that user presents in both file and LDAP provider settings. This case is configured below in my authorizers.xml file:<authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1"

Hi all.We have a cloud HDFS cluster.We have DataNode with different size '/data' (from 1TB to 3TB). Node #1, which has 3TB, is 500Mb full, node #2, which has 1TB, is 900Mb full. Can you please tell me why this is happening and how to fix it?The value of rebalancing threshold = 5.

Hey Everyone,i would like insert a string into another string with the query Record Processor with the sql function STUFF but i run into an error. Anyone have an Idea how i can adjust the sql statemant or how i can this string manipulation with the nifi expression language I want to insert a colon in a column with the time of dayfrom thatto the001500:15 This is my input Data from an json file [ { "store_id" : 1011, "date" : 20220915, "start_time" : "0015", "start_end" : "0015", "out" : 0}, { "store_id" : 1011, "date" : 20220915, "start_time" : "0030", "start_end" : "0030", "out" : 0}] This is my Sql expression SELECT "store_id", "date", STUFF("start_time",2,0,':') as "start", "start_end", "out"FROM FLOWFILE This is the error message i got QueryRecord[id=46658fcd-0183-1000-0000-000015e8db65] Unable to query StandardFlowFileRecord[uuid=641e1c71-ff9b-45dc-a948-e527eff450f6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id

Hi Guys ,i have a problem with Nifi login   please can´t help me ?this problem start when open the web interface of Nifi later config ldap login  .Present the next message has been attached imagen . for fix the problem -> select log out  is there any other solution ?Regards! 

Hey everyone, i get a csv file as record from an api call and don't need all columns and want to convert the datetime columns to a time format. For example from 15.09.2022 00:15:00 to 00:15 This is my csv recordInterval startInterval stopCamera serial numberCounter namePedestrians coming inPedestrians going out15.09.2022 00:00:0015.09.2022 00:15:00000000111123400  This is my setting in the Query record ProcessorQuery SELECT #{TodayAxisV2Param} as "date", "Interval start" as "start time", -- Expected FORMAT(HH:mm) "Interval stop" as "end time", -- Expected FORMAT(HH:mm) "Pedestrians going out" as "out"FROM FLOWFILE I expect this result for the time fields date start timeend timeout2022091500:0000:150 Does anyone have an idea how I can do this with the Query Record Processor?

hello cloudera community, we are trying to create a keytab with the main one: "HTTP/hostname@DOMAIN.LOCAL" with the command: ktpass -princ HTTP/hostname@DOMAIN.LOCAL -mapuser livy-http -crypto ALL -ptype KRB5_NT_PRINCIPAL -pass password2022 -target domain.local -out c:\temp\livy-http.keytab but I try to validate the ticket with this keytab returns the error: Exception: krb_error 24 Pre-authentication information was invalid (24) Pre-authentication information was invalid KrbException: Pre-authentication information was invalid (24)at sun.security.krb5.KrbAsRep.<init>(Unknown Source)at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)at sun.security.krb5.internal.tools.Kinit.<init>(Unknown Source)at sun.security.krb5.internal.tools.Kinit.main(Unknown Source)Caused by: KrbException: Identifier doesn't match expected value (906)at sun.security.krb5.internal.KDCRep.init(Unknown So

Hello everyone. I am trying to train a ML model on a cluster by using a docker image and spark-submit with yarn.I already tried to follow this process before on a training cluster I made and I succeeded. But when I run it this time, yarn prompts that one of the mounts is invalid.Of course I tried with Kerborise and without, both didn't work and both runs didn't imply any problems related to connection so we are good from this side.  This is what I tried:spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=image:v1 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data01/yarn/nm/:/data01/yarn/nm/:ro,/data02/yarn/nm/:/data02/yarn/nm/:ro,/data03/yarn/nm/:/data03/yarn/nm/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \--conf s

hello to all,I have many connections in time_wait on the ipc port 1019 of the datanode:More than 600 time_wait and about 250 established.Is that normal?I’m afraid that’s why index write closed on solr errors (the index is on hdfs).The servers are downloaded and the datanode does not saturate the jvm heap.I couldn’t find any max connection configuration for port 1019Any ideas? Environment:HDP 3.1.5.0-152 with HDFS 3.1.1 Thanks in advance

Hi,  I'd like to catch any key that match a certain reg ex in jolt. The purpose is to clean and order keys into the proper name. An example worth thoushand words: possible inputs:{"my_id":234535,"my company sa": "whatever value","my comp RH": "whatever value",...}{"my_id":234536,"my_comp_sa" : "whatever value","DRH" : "whatever value"...} output:{"my_id":234535,"my_company_sa": "whatever value","my_company_rh" : "whatever value"}

Hi community,hope someone can help me on below. My Flow File content is something like this:  [ {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071253","new_appointment_date" : "202301081253",}, {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071252","new_appointment_date" : "",} ]  I need to add a Query Record processor in order to grab only one JSON record for each Flow File.The condition is: Get a single result with highest value of situation_date or new_appointment_date. Just to explain better, the logic behind should be the following:Grab the max of situation_date, the max of new_appointment_date and compare each other, then take the highest of the comparison and get entire JSON record. In MYSQL or T-SQL it could be easier, but if i understand well Query Record is based on Apache Calci

Hi everyone,i had a .csv file converted to json with the convert csv to json processor and configured the csv reader. In my csv file i have names with german umlauts. I have use character set UTF 8 or ISO-8859-1 but the output is not the right for german umlauts.  the csv file has the following character set from the source system This are my setting of the csv reader controller serviceDoes anyone have an idea for the correct setting.Thanks

Hello Nifi Community, We have integrated our Nifi 1.16.2 with LDAP AD server.We have created an Initial Local Admin (nifi_ldap) and used "composite-configurable-user-group-provider" as user group provider. We also restricted to one particular group of LDAP server (namely "EDH_ML"). But none of the users of this group ("EDH_ML") is able to access the Nifi and getting "Insufficient Permission Error". Could someone can help us to resolve this error? -- Sharing nifi screenshot and configuration settings/logs Nifi Users Nifi Login Error Nifi User Policies Authorizer.xml<userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">cn=Service Account\, nifi_ldap,ou=Service Accounts,o

we are using ambari 2.6.2.2 with hdp 2.6.5 the permissions are being managed by the ranger, but checking the ranger settings, it's only managing the /apps/hive/warehouse directory on the hdfs the problem we are having is in the /area/data/pll directory on the hdfs, this directory has acl rwx permission for the user using beeline to create an external table is returning the error: Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [service_user] does not have [ALL] privilege on [hdfs://service1/area/data/pll] (state=42000,code=40000) but using hive cli does not return error, table is created successfully. What could be causing this problem?

Hello, I am running Flink 1.14.4 with 3 jobs running in it. JVM: openjdk version "1.8.0_312". All jobs are running without any issues (at least not obviously noticed) however, I see the below exception in the log. Im not able to understand the reason for the exception. Not sure what is the issue. Appreciate any help on this. Exception: java.nio.file.InvalidPathException: Nul character not allowed (full stack trace below) 2022-08-15 16:35:15.350 [flink-rest-server-netty-worker-thread-5] WARN org.apache.flink.runtime.rest.FileUploadHandler - File upload failed.java.nio.file.InvalidPathException: Nul character not allowed: bash' '-c' #cmdlinux})).(#p new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).(#process #p.start()).(#ros (@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).(@org.apache.commons.io.IOUtils@copy(#process.getInputStream() #ros)).(#ros.flush())}b at sun.nio.fs.UnixPath.checkNotNul(UnixPath.java

Hi Everyone, i would like to add an attribute to the following json file using the jolt transfomr processor. Unfortunately my json file remains unchanged with the following transformation Original Json File     { "logs": [], "ranges": [ { "day": "2022-08-31T22:00:00.000Z", "manager": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 }, "crew": { "costByRemunerationId": [ { "remunerationId": "salary", "remunerationUnit": 2, "compensation": 76.66666666666667, "worktimeInH": 0, "durationInDays": 0, "quantity": 0, "ancillaryCost": 15.854666666666667 } ], "total": 76.66666666666667, "totalHours": 0, "ancillaryCost": 15.854666666666667 }, "trainee": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 },

Hy everyone, I need help to extract a part from a url with the extratct text processor. /organisations/60910c2f18572ebe8aa7a677/detailedLaborcosts? (Note the ID may change)I expect this result 60910c2f18572ebe8aa7a677 Thanks for suggestions

Hi. I have just started on a project, where we want to set up a landscape of nifi instanses, connected to nifi registry. And we are planning on making a dev. environment a test and a prod.The question started, when we made versioned nifi flows with versioned nifi flows inside, for code reuse and easier maintenance. But that cannot be moved easily between nifi registry instanses? Do we connect all of the nifi instanses to one registry or what is the best practice? Or what are we doing wrong?

I want to filter my rest api with date range. The rest api has dateFrom and dateTo parameters. I want to set dateFrom to 5 days ago and dateTo to now. How do i set the parameters in NiFi ?My api looks like the below URL:https://api.aa/reports/api/order_report/?format=json&dateFrom=&dateTo=

All hosts have the same issue : Installation failed. Failed to copy installation files. Details  /tmp/scm_prepare_node.9aRx6niZFailed to download package. Status code: 404  I am using the free trial version.  Log error:2021-04-23 01:03:08,424 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Invalid parcel in manifest : CDH-7.1.6-1.cd$2021-04-23 01:03:11,178 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: htt$2021-04-23 01:03:12,739 INFO ParcelUpdateService:com.cloudera.cmf.paywall.PaywallHelper: (2 skipped) License incomplete; unable to use it for authentication $

Hi, All !   I am having the same problem as below. Drop empty Impala partitions - Stack Overflow Impala external table partitions still show up in stats with row count 0 after deleting the data in HDFS and altering (like ALTER TABLE table RECOVER PARTITIONS) refreshing (REFRESH table) and invalidation of metadata. Trying to drop partitions one by one works, but there are tens of partitions which should be removed and it would be quite tedious. Dropping and recreating the table would also be an option but that way all the statistics would be dropped together with the table. Is there any kind of other options in impala to get this done? Is this only a workaround using Hive below? Found a workaround through HIVE. By issuing MSCK REPAIR TABLE tablename SYNC PARTITIONS then refreshing the table in impala, the empty partitions disappear. Also, I could not find an Issue in Impala JIRA.If anyone knows of an Issue in Impala JIRA, please let me know.   My environment is below.

Hello,Is there a way to retrieve all HDFS groups and respective users, in Cloudera, using Java or Scala/Spark code?In other words, I’m looking for Java/Scala code equivalent to "hdfs groups <user1>, <user2>, ...".Ideally, the code would run in Zeppelin.Many thanks. 

Hi all, I've been struggling to deploy HDP 2.6.5 via docker.I downloaded and ran the deploy script but kept receiving this error for this line of command: + docker exec -t sandbox-hdp sh -c rm -rf /var/run/postgresql/*; systemctl restart postgresql;Failed to get D-Bus connection: No such file or directory Calling systemctl thru the container CLI gives the same error. I used this script to replace the container's /bin/systemctl but was unable to start postgresql ( systemctl start -v postgresql yieldsWARNING:systemctl:No PIDFile for forking '/usr/lib/systemd/system/postgresql.service'WARNING:systemctl:forking start not active)which resulted in unsuccessful start of ambari-server and localhost:1080 kept giving 502 bad gateway. Any idea how to solve this? Thanks!