Solved

Hello, please help me.env  jdk8 + jce  HDP 3.1.5, ambari 3.7.5Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown:---------------------- ambari-server.log --------------------2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 42022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 33282022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 10242022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationCluster

I used nifi 1.13.2 cluster of 2 nodes.Node1 is rhel7, node2 is redos, embedded Zookeeper.The cluster works correctly (nodes see each other, the coordinator is selected.)Certificates on nodes are self-signed.Faced with the problem of authorization in the UI.LDAP authorization does not work (without cluster the LDAP authorization worked on both nodes).I am trying to connect to node1.When I enter my username and password, I return back to the authorization page.The log on the node1:2022-09-19 05:36:14,892 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node1.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)2022-09-19 05:36:14,895 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <user name>2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>2022-09-19 05:36:14,895 DEBUG [Ni

I would like to know how is the best way to create surrogate key or SK in Impala. If there is no UDF function SURROGATE_KEY(). How could I do?

Hi everyone, I have a problem because I dont know how I can access are all variables (see image) from a script groovy.Maybe someone want to share some idea in order to solution my problemRegards. 

Hello,I am trying to extract value from a key/value pair json record. I am trying to extract the value of "image" key.  I used an ExtractText processor but there was no match. Although, there was a match when i used an online regex extractor(https://onlinetexttools.com/extract-regex-matches-from-text). Regex i used in the extracttext processor: (?<=\"image\"\s:\s")[A-Z-a-z-0-9\-\:\/\.\_]+ My json record:{ "id" : "03ee73b8-a553-4575-8dfa-d0da4e7939e9", "url" : "https://ll.thespacedevs.com/2.0.0/launch/03ee73b8-a553-4575-8dfa-d0da4e7939e9/", "launch_library_id" : null, "slug" : "falcon-9-block-5-galaxy-33-34", "name" : "Falcon 9 Block 5 | Galaxy 33 & 34", "status" : { "id" : 2, "name" : "TBD" }, "net" : "2022-10-05T23:07:00Z", "window_end" : "2022-10-06T00:14:00Z", "window_start" : "2022-10-05T23:07:00Z", "inhold" : false, "tbdtime" : false, "tbddate" : false, "probability" : null, "holdreason" : "", "failreason" : "", "hashtag" : null,

If I have a flow file with the following JSON how can I1) Evaluate "addresses" to determine if it is of type Array or type Map2) If type Map then convert "addresses" into an array using native NiFi capabilities (i.e. no string parsing)?  { "name": "John Doe", "addresses": { "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }}   This is what I need it to look like:  { "name": "John Doe", "addresses": [{ "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }]}   I appreciate the input and support!  Thank you.

Hi there, does Cloudera support Iceberg in on-prem deployments? If not, by when it's expected to be available?

how to know why sparkcontext was shut down ??ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut down

Hi, I have lots of generated hprof files in /tmp. 1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid98619.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid97929.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid96759.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid92257.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid9035.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87683.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87640.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid86040.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid82228.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77347.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77317.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid75484.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid72638.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid71811.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid7086.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid66069.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65902.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65186.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid61886.hprof1.3G /tmp/m

How to uninstall NiFi from Linux 7.9? The process org.apache.nifi.registry.NiFiRegistry is up and running on the server. We need to remove the product completely. I am not sure how it was installed.We do not use d2iq dcos framework.  

Hello everyoneI am using PutSql processor and I need to execute a Procedure from the Oracle database, I has used the follow code in the SQL Statement:declareinputvar varchar2(20) := "1";outputvar varchar2(20) ;beginMyPKG.MyStProcedure( inputvar, outputvar);end;My questions are: 1) How can I recieve the input parameter and pass to it my code? and 2) How can I send the output parameter to next queue o some attribute ?

Hi All, I have been trying to using group by with having clause in the ExecuteSQL processor but it doesn't seem to be returning the count. The SQL query looks like : select id_num, count(*) from xyz where id_num<>0group by id_numhaving count(*)>1 I put this in the SQL select query. I just get an empty array. How do I use group by and having in the select query in Nifi? A normal query without these clauses works perfectly but I need a count to check for duplicates in the database. Thank you in Advance

hello cloudera community, we are having a problem accessing the hive cli of a certain user and the spark-shell too when executing the query "show databases" in hive cli returns the error: "FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset " when executing the "show databases" query in spark-shell returns the error: "org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException;" "WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException" when we use beeline and run the query "show databases" it works without problem could you help us with this problem? we are using cl

Hi there, In Cloudera manager, after installing the parcels of CDH, and CFM.added the Nifi service by installing nifi csd files in csd repo.and zookeeper is also added as dependency. After starting the NIFI service, NIFI node status is Down all the time.And couldn't find any folders like log, conf in the parcel-repo to check the error logs. Nifi node status is unknown health. couldn't start the service. We have hosted the services on a single host.Please, suggest a way to identify the issue. Below is the directory structure of CFM parcel and nifi files. 

Hi, I want to ask a few questions about having multiple Hive Mestastore instances and how to correctly configure the compaction initiator and cleaner threads on these. The point of having multiple Hive Metastore instances is because we have many concurrent connections and, as we have seen on the official documentation (https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-hive.html), it is not recommended to have more than 16-24GB of Java Heap due to the impact of Java garbage collection on active processing by the service. 16-24GB of Java Heap is the recommended amount of memory to manage 41-80 concurrent connections (we have around 50 on each Metastore). So, what i want to ask is what is the correct configuration for the compactor initiator and cleaner threads (hive.compactor.initiator.on) if we have multiple Hive Metastore instances. Can it be enabled on every instance? Should it be only enabled on 1 instance? Our environment is ru

Hi folks,I have a problem on my CDP Datalake causing failure on master node.Apparently some services fail due to lack of free memory space.   Everything seems to work correctly but I don't know what can happen if the free space reduces further.I have tried these solutions, however, with no effect if not momentary:- lowering some alarm thresholds (from 10% to 7% of free space)- removing some log files from the machine in /var/log and /tmp folders @shehbazk Can you give me some advice?And I would also like to understand if it is possible and how to increase the capacity higher than the current 236 GB. @pandav  

Hi,  I am trying to secure NiFi using LDAP configuration. I followed blog by mintops and pvillard articles for reference. I am running NiFi on windows (not in cluster configuration). I am able to get to the login screen, but then I am getting an error on logging in ( The supplied username and password are invalid)            I followed most instructions as mentioned in the blog by mintops. Can someone help me in the direction, on what might be the problem?PS: NiFi version and toolkit versions -1.8.0.I haven't created certificates, just the configs needed to update in Keystore and truststore passwords by the toolkit.The LDAP configurations were already being used in another program, so that is also not the issue. login-identity-provider.xmlauthorizers.xml:        nifi.properties              &nb

I am working on a proof of concept situation with a Nifi Cluster on Openshift. As part of the PoC I want to enable authentication (currently have a working cluster with no auth), so I am wondering if it possible to do it with the Single User method as it is built in and should be the easiest to setup. I have been unable to find any documentation outlining this process or possibility, so if there is any outlining this setup, assuming it is possible, please direct me towards it.

I have a Handle HTTP Request Processor to receive incoming requests , which have various workflows connected to it, I want to apply authentication on this processor for incoming requests, I have figured out that  I need to set "Need Authentication" and configure a secure socket layer  Context service, But I have no idea how secure socket layer works,what are key stores and trust stores ? how to generate them and how to use them, Do the client needs to send certificates ? how do I verify it ? I intend to test it by sending request to this processor using Postman.Any help is appreciated, Thank You.

Hi everyone.We have an NiFi cluster consisting of 3 nodes. After the failure of the disk subsystem on one of the nodes, it was in the ReadOnly state for a long time. After resolving the issue and after restarting cluster we are getting the following error on problematic node:2022-09-16 15:31:07,255 ERROR [Load-Balanced Client Thread-4] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer xxxxxx:9443java.io.EOFException: Expected StandardFlowFileRecord[uuid=6ce9e262-b20b-4372-a3b9-43c2c00e8caa,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663256223724-231072817, container=default, section=49], offset=387990, length=2203],offset=0,name=04190e1f-fdca-4352-a796-6b6c9ce41baa,size=2203] to contain 2203 bytes but the content repository only had 1130 bytes for it at org.apache.nifi.controller.queue.clustered.ContentRepositoryFlowFileAccess$1.ensureNotTruncated(ContentRepositoryFlowFileAccess.java:83) at org.apache.nifi.controller.queue.clustered.Con

I have about 25-30 XML message types and each message type has its own XSD.  I need to validate each message against their respective XSD.  When using the ValidateXML processor is there any way to dynamically assign the appropriate XSD to a flow file based on on attribute value?  I don't see the purpose/benefit of using so-called variables when said variables aren't even variable--they are STATIC! Why does this processor ONLY use variable_registry variables and not attribute values like every other processor in NiFi?    

  Hi all,I want to add 15 minutes to a date-time value using the transform json processor, but with my configuration I get the same value as before. I expect for the end field to have 15 minutes more than the start field. Does anyone have any idea? [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30:00", "out" : 0}] My jolt Specification [ { "operation": "modify-overwrite-beta", "spec": { "*":{ "end": "=${end:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}" } } }] The result is the same  [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30

Hi. Could somebody help me please with composite group provider settings in authorizers.I'm not sure what should I put in "Initial admin identity" and "Initial user identity 1".1) When I write just username (like nifi-admin) I cannot login to NiFi since I don't undertand where I can get a password.2) When I write a full LDAP name (like uid=freeipa,cn=users,cn=accounts,dc=foo,dc=bar) I successfully log in but I cannot access a NiFi interface.3) When I use name of user who already exists in LDAP, I get an error, that user presents in both file and LDAP provider settings. This case is configured below in my authorizers.xml file:<authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1"

Hi all.We have a cloud HDFS cluster.We have DataNode with different size '/data' (from 1TB to 3TB). Node #1, which has 3TB, is 500Mb full, node #2, which has 1TB, is 900Mb full. Can you please tell me why this is happening and how to fix it?The value of rebalancing threshold = 5.

Hey Everyone,i would like insert a string into another string with the query Record Processor with the sql function STUFF but i run into an error. Anyone have an Idea how i can adjust the sql statemant or how i can this string manipulation with the nifi expression language I want to insert a colon in a column with the time of dayfrom thatto the001500:15 This is my input Data from an json file [ { "store_id" : 1011, "date" : 20220915, "start_time" : "0015", "start_end" : "0015", "out" : 0}, { "store_id" : 1011, "date" : 20220915, "start_time" : "0030", "start_end" : "0030", "out" : 0}] This is my Sql expression SELECT "store_id", "date", STUFF("start_time",2,0,':') as "start", "start_end", "out"FROM FLOWFILE This is the error message i got QueryRecord[id=46658fcd-0183-1000-0000-000015e8db65] Unable to query StandardFlowFileRecord[uuid=641e1c71-ff9b-45dc-a948-e527eff450f6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id