Find answers, ask questions, and share your expertise
Recently active
Hi everyone,I've trying to monitoring nifi api. In our nifi we have configure oidc login and it works fine.As monitoring, we using zabbix.The problem in zabbix is when try to check api page (https://nificluster.info/nifi-api/flow/cluster/summary) and I receive http 401 error. In the web scenario monitoring, I have configured 2 steps: connect to https://nificluster.info/nifi/ with login and password. This steps it's ok, the required string - "log out" - and required status code 200 works. In this step I've configured viariables username, password for logging and follow redirects flag enable.connect to api https://nificluster.info/nifi-api/flow/cluster/summary without any parameters except requird string with number of cluster nodes and required status code 200.Who could help me resolve this 401 error on the api nifi?
Hello Everyone, We are triggering our Nifi processors (version 1.16.2 , LDAP integrated ) through Curl within cron job. Curl is invoking the processor using access token (as we moved from http call to https) Generating access token : curl https://$hostip:$port/nifi-api/access/token -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username="${ldapusername}"&password="${ldappassword}"' '--compressed --insecure Curl invoking Nifi processor using acces token stored in variable $nifi_token : curl -i -H 'Content-Type: application/json' -H 'Authorization:Bearer '$nifi_token -XPUT -d '{"id":"'${processorid }'","state":"STOPPED"}' https://$hostip:$port/nifi-api/flow/process-groups/$processorid '--insecure' But it is failing for token expiration : WWW-Authenticate: Bearer error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Expired JWT", error_uri="https://tools.ietf.org/html/rfc67
Is there a size limit to the tar file generated by MergeContent?ERRORMergeContent[id=8a06a034-0183-1000-b8b5-c045959a56e9] Failed to process bundle of 2 files due to entry size '14743306287' is too big ( > 8589934591 ).; rolling back sessions: java.lang.RuntimeException: entry size '14743306287' is too big ( > 8589934591 ).
Hi, I want to save the out put of the file after replace the text.when i put xml file for consumed NIFI replace the text and move xml file into processed folder,NIFI didn't change into original xml file,i just want to know during process what is the code which NIFI use.
hello cloudera community, we are having problems using livy with job spark to read hive by jupyter notebook when we run a simple query, for example: "spark.sql("show show databases").show()" returns the error below"org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient" Could you help us with this setup? ps: we are using cdh 5.16.x
Hello, please help me.env jdk8 + jce HDP 3.1.5, ambari 3.7.5Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown:---------------------- ambari-server.log --------------------2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 42022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 33282022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 10242022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 30722022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationCluster
I used nifi 1.13.2 cluster of 2 nodes.Node1 is rhel7, node2 is redos, embedded Zookeeper.The cluster works correctly (nodes see each other, the coordinator is selected.)Certificates on nodes are self-signed.Faced with the problem of authorization in the UI.LDAP authorization does not work (without cluster the LDAP authorization worked on both nodes).I am trying to connect to node1.When I enter my username and password, I return back to the authorization page.The log on the node1:2022-09-19 05:36:14,892 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node1.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)2022-09-19 05:36:14,895 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <user name>2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>2022-09-19 05:36:14,895 DEBUG [Ni
I would like to know how is the best way to create surrogate key or SK in Impala. If there is no UDF function SURROGATE_KEY(). How could I do?
Hi everyone, I have a problem because I dont know how I can access are all variables (see image) from a script groovy.Maybe someone want to share some idea in order to solution my problemRegards.
Hello,I am trying to extract value from a key/value pair json record. I am trying to extract the value of "image" key. I used an ExtractText processor but there was no match. Although, there was a match when i used an online regex extractor(https://onlinetexttools.com/extract-regex-matches-from-text). Regex i used in the extracttext processor: (?<=\"image\"\s:\s")[A-Z-a-z-0-9\-\:\/\.\_]+ My json record:{ "id" : "03ee73b8-a553-4575-8dfa-d0da4e7939e9", "url" : "https://ll.thespacedevs.com/2.0.0/launch/03ee73b8-a553-4575-8dfa-d0da4e7939e9/", "launch_library_id" : null, "slug" : "falcon-9-block-5-galaxy-33-34", "name" : "Falcon 9 Block 5 | Galaxy 33 & 34", "status" : { "id" : 2, "name" : "TBD" }, "net" : "2022-10-05T23:07:00Z", "window_end" : "2022-10-06T00:14:00Z", "window_start" : "2022-10-05T23:07:00Z", "inhold" : false, "tbdtime" : false, "tbddate" : false, "probability" : null, "holdreason" : "", "failreason" : "", "hashtag" : null,
If I have a flow file with the following JSON how can I1) Evaluate "addresses" to determine if it is of type Array or type Map2) If type Map then convert "addresses" into an array using native NiFi capabilities (i.e. no string parsing)? { "name": "John Doe", "addresses": { "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }} This is what I need it to look like: { "name": "John Doe", "addresses": [{ "work": { "number": "123", "street": "5th Avenue", "city": "New York", "state": "NY", "zip": "10020" } }]} I appreciate the input and support! Thank you.
Hi there, does Cloudera support Iceberg in on-prem deployments? If not, by when it's expected to be available?
Hi Team, We are running some Hive, and Impala queries and getting the below error for the job. It was running fine earlier, but now it's failing. We tried multiple ways, like re-created the tables and then tested with the REFRESH command for the table. Whenever the job executes then automatically permission is changed with the user. It was like impala: hiveCould you please suggest how we can w take it forward? Note: We recently upgraded the cluster to CDP.
Hello I am trying to make a data flow through Nifi which are listFile -> fetchFile -> data transformation -> putKudu However, I want to somehow pause the fetchFile processor or hold on to the flowfiles received from the local server until a previous flowfile completely goes through the downstream, after putKudu processor, then release the next flowfile one by one.I looked at Wait and Notify processors but could not find a way to notify the next flowfile in the queue to be released. Is it possible to apply this logic in the Nifi without using custom processor? Any help is appreciated.
Hello, a new CVE appears on Apache Spark. Does it impact every versions of Spark ?https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-33891Thanks in advance for your help.
how to know why sparkcontext was shut down ??ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut down
Hi, I have lots of generated hprof files in /tmp. 1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid98619.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid97929.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid96759.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid92257.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid9035.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87683.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid87640.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid86040.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid82228.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77347.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid77317.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid75484.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid72638.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid71811.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid7086.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid66069.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65902.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid65186.hprof1.3G /tmp/mgmt_mgmt-REPORTSMANAGER_pid61886.hprof1.3G /tmp/m
How to uninstall NiFi from Linux 7.9? The process org.apache.nifi.registry.NiFiRegistry is up and running on the server. We need to remove the product completely. I am not sure how it was installed.We do not use d2iq dcos framework.
Hello everyoneI am using PutSql processor and I need to execute a Procedure from the Oracle database, I has used the follow code in the SQL Statement:declareinputvar varchar2(20) := "1";outputvar varchar2(20) ;beginMyPKG.MyStProcedure( inputvar, outputvar);end;My questions are: 1) How can I recieve the input parameter and pass to it my code? and 2) How can I send the output parameter to next queue o some attribute ?
I need to send an attachment by email through Nifi, but I can't get the PDF file because there are several files from several people within the same directory, I have a BASE64_ENCODE field in the Oracle table that is the same as the generated PDF file. How can I pass this information from this field through Nifi's PutEmail processor?
Hi All, I have been trying to using group by with having clause in the ExecuteSQL processor but it doesn't seem to be returning the count. The SQL query looks like : select id_num, count(*) from xyz where id_num<>0group by id_numhaving count(*)>1 I put this in the SQL select query. I just get an empty array. How do I use group by and having in the select query in Nifi? A normal query without these clauses works perfectly but I need a count to check for duplicates in the database. Thank you in Advance
Hi, I am trying to secure NiFi authentication using LDAP configuration. ). I am able to get to the login screen, but then I am getting an error on logging in ( The supplied username and password are invalid) even though the username and password are correct.Can someone help me in the direction, on what might be the problem?
hello cloudera community, we are having a problem accessing the hive cli of a certain user and the spark-shell too when executing the query "show databases" in hive cli returns the error: "FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset " when executing the "show databases" query in spark-shell returns the error: "org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException;" "WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException" when we use beeline and run the query "show databases" it works without problem could you help us with this problem? we are using cl
Hi there, In Cloudera manager, after installing the parcels of CDH, and CFM.added the Nifi service by installing nifi csd files in csd repo.and zookeeper is also added as dependency. After starting the NIFI service, NIFI node status is Down all the time.And couldn't find any folders like log, conf in the parcel-repo to check the error logs. Nifi node status is unknown health. couldn't start the service. We have hosted the services on a single host.Please, suggest a way to identify the issue. Below is the directory structure of CFM parcel and nifi files.
Hi I am using the cloudera hortonworks sandbox docker image, and have followed this tutorial to run Jupyter notebooks: https://community.cloudera.com/t5/Support-Questions/Installing-Jupyter-on-sandbox/td-p/201683 This works. The notebook is started using the python kernal. The error is encountered when attempting to create the spark context: FileNotFoundError: [Errno 2] No such file or directory: '/usr/hdp/current/spark-client/./bin/spark-submit' FileNotFoundError Traceback (most recent call last)<ipython-input-4-fbb9eeb69493> in <module>----> 1 spark = SparkSession.builder.master("local").appName("myApp").getOrCreate()/usr/local/lib/python3.6/site-packages/pyspark/sql/session.py in getOrCreate(self) 226 sparkConf.set(key, value) 227 # This SparkContext may be an existing one.--> 228 sc = SparkContext.getOrCreate(sparkConf) 229
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.