Find answers, ask questions, and share your expertise
Recently active
Hi Team, I want to setup kerberos client in my Mac Laptop having MacOS Monterey (version 12.6.5). I have put the krb5.conf file at below paths./etc/krb5.conf/Library/Preferences/edu.mit.kerberosBut when I try to run kinit, i get gelow error.--kinit -kt /Users/banshidhar_sahoo/Desktop/POC_KEYTAB/test.headless.keytab test@EXAMPLE.COMkinit: krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 0 KDCs-- I have also set the ENV Variable as below:KRB5_CONFIG=/etc/krb5.conf But getting same error while doing kinit. Can you please suggest how to point to krb5.conf so that it can reach out to the correct kdc server. Regards,Banshi.
I have two processes1. consumerkafkarecord --mergerecord --puthdfs2. consumerkafkarecord --mergecontent--puthdfswhen I use process 1, I have files in ndfs on the output readable spark, database, python libraries without problems, but the files are not larger than 200mb, all different sizes, although 500mb is set, but it is not filledwhen I use process 2 with the same parameters in mb and the number of lines, the files I get are exactly 500mb, but these files do not open, not by spark, not by any database, not by python librariesquestion why?I also want a large file always 500mb and so that it can be read without problems as in process 1mergecontent settingsMerge Strategy Bin-Packing AlgorithmMerge Format Binary ConcatenationAttribute Strategy Keep Only Common AttributesCorrelation Attribute Name No value setMinimum Number of Entries 10000Maximum Number of Entries 1000000Minimum Group Size 100 MBMaximum Group Size 500 MBMax Bin Age No v
How do I restrict the access to an endpoint provided by HandleHttpRequest to specific users? I did setup successfully a HandleHttpRequest-HandleHttpResponse chain, with Client Authentication = "Need Authentication". I provided a key- and truststore and users with a client certificate trusted by CA cert in the truststore can access the endpoint. However, I do not want everybody in my company to access this endpoint, I want to restrict this to a small group of users (actually to one single server). For testing purposes I created an empty truststore and imported into this truststore only my client certificate. However, now I cannot access the endpoint anymore (ERR_BAD_SSL_CLIENT_AUTH_CERT). How do I configure the JKS truststore to only trust one specific client? Or how do I configure HandleHttpRequest processor? If this is not possible, are there any other possibilities?Could I somehow use a Nifi-REST-API-Token?Could I somehow authenticate with the help
hello cloudera community, we check in the graph "total_kudu_on_disk_size_across_kudu_replicas" and there are tables with 500GB with that, we need to know what is the recommended size for a kudu table?
Nifi provenance data is not visible and remains at 80% in searching for provenance events after upgrading from 1.13.0 to 1.15.3 Below are the error logs : 2022-04-05 23:51:22,905 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Eventsorg.apache.lucene.store.AlreadyClosedException: this IndexWriter is closedat org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at jav
Hi, does anyone knows whether there are any restrictions on using external shuffle services on CDP?For instance, are the following external shuffle services supported?:- uber rss- apache uniffle
Hi team, We were in process of migrating the flow from HDF to CDP. We migrated few and while testing the flow they are working fine on new home we encountered an issue. We were working on flow in which we encountered and issue with processor 'InferAvroSchema 1.18.0.2.2.6.0-260' where as procssor 'InferAvroSchema 1.9.0.3.4.0.22-1' which was working fine with the data. While working with new processor we encountered an issue which is listed below: ERROR b11042ff-c573-1d90-0000-0000338c6dea rit-dev-nifi-nifi0.regn-cld.fxd6-2a7z.cloudera.site:8443InferAvroSchema[id=b11042ff-c573-1d90-0000-0000338c6dea] Processing halted: yielding [1 sec]: java.lang.NoSuchMethodError: org.apache.avro.Schema$Field.<init>(Ljava/lang/String;Lorg/apache/avro/Schema;Ljava/lang/String;Lorg/codehaus/jackson/JsonNode;)V Nifi Verson:
While attempting to migrate data from MSSQL to Postgres with nearly a thousand tables, I am currently retrieving the list of tables (LDT) and splitting it into individual tables using splitAvro. Now, I'm seeking a solution to efficiently map these tables to their corresponding names in the target system and transfer the data to the tables that share the same names in both the source and target databases. Additionally, I would like to know if Nifi offers a feature to convert and migrate schemas between different databases.Thank you.
Hi Team, I trying to configure Apache Nifi LDAP authentication, Here is my nifi.properties. -----nifi.security.autoreload.enabled=falsenifi.security.autoreload.interval=10 secsnifi.security.keystore=/var/tmp/mks/certs/keystore.jksnifi.security.keystoreType=PKCS12nifi.security.keystorePasswd=************nifi.security.keyPasswd=nifi.security.truststore=/var/tmp/mks/certs/keystore.jksnifi.security.truststoreType=JKSnifi.security.truststorePasswd=**************nifi.security.user.authorizer=managed-authorizernifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.provider=ldap-providernifi.security.user.jws.key.rotation.period=PT1Hnifi.security.ocsp.responder.url=nifi.security.ocsp.responder.certificate=------ Here is my "login-identity-providers.xml" -------><provider><identifier>ldap-provider</identifier><class>org.apache.nifi.ldap.LdapProvider</class><property name="Authentication Strategy">LDAPS&
Hi All, I am using putdatabaserecord to store a value into DB.Now i have a flow based on failure link to (this processor), that this failed record will be stored in disc for troubleshooting.I can store the failed record in disc by using putfile processor.Now i wanted it more informative during debugging, as the attribute "putdatabaserecord.error" contains the error reason, i want to append the flowfile content with this error reason. I tried to acheive this using below flow..putsql/putdbrecord -> failure-> updateattribute (this is to give the flowfile a realistic name)-> attributesToJson (destination flowfileattribute/content -> putfilebut in attributestoJson , if the destination is kept as flowfileattribute, the flowfile dont get this attribute or if it is kept as flowfile content, original flowfile content is replaced...My requirement is to append this single attribute to the existing flowfile content. Any help or suggestion is greatly appreciated.
I have an InvokeHTTP processor Apache NiFi, I need to send a curl request with this contentcurl https://reqbin.com/echo/post/json-X POST-H 'Content-Type: application/json'-d '{ "login": "my_login", "password":"my_password"}'I can't figure out what to do -d
Hi Community,hope someone can help me to build this transform: My Input:{"Orders": [{"Headers": {"UniqShipment": "0"},"Goods": [{"GoodsTypeID": 3,"GoodsTypeName": "ALTRI MOBILI LEGNO","GoodsDetails": [{"Packs": 1}, {"Packs": 1}, {"Packs": 1}]}],"References": [{"TypeReference": "DT","ValueReference": "006611","DateReference": "2023-04-14"}, {"TypeReference": "CM","ValueReference": "JOHN"}, {"TypeReference": "OC","ValueReference": "V1250-0010"}, {"TypeReference": "RG","ValueReference": "TRCA_CATR726240_CA_006611_14_04_2023.json"}]}]}My Desired Output:{ "header": { "UniqShipment": "0" }, "rows": { "GoodsTypeName": ["ALTRI MOBILI LEGNO", "ALTRI MOBILI LEGNO", "ALTRI MOBILI LEGNO"], "Packs": [1, 1, 1], "ValueReference": ["V1250-0010", "V1250-0010", "V1250-0010"] }}My Transformation so far:[ { "operation": "shift", "spec": { "Orders": { "*": { "Headers": "header", "Goods": { "*": { "GoodsDetails": { "*": {
When parsing a json array with a json tree reader (in this case using convert record), I am presented with this error, this testing was done in Nifi Version 1.21.0. However this error never occurs in Version 1.18.0. An example of json that fails in 1.21.0 that works in 1.18.0 [ {"a": [ {"b": [{}]}]}, {"a": [ {"b": []}]}] Any help would be greatly appreciated.
Does Apache NiFi supports MQTT 5? If it does, how do i get the Response Topic?
I have a deployed model un Cloudera Machines Learning. The model is trained once a month, which I do through a pipeline (Jobs). It must be deployed after that again to take the updated model. Can the deployment be done in a way other than manual?
I would like to get records count(select count(1) from table where col1=‘abc’) from a sql table. Could someone please help me with that ?
I am not able to change my company name in my profile, as below shows.I also noticed that the system is now showing that "Partner (No Longer Active)".not sure how to handle above two issue.
There is a Quorum Kafka cluster with parameters. listeners=PLAINTEXT://:9092,CONTROLLER://:9093, SASL_PLAINTEXT://:9094security.inter.broker.protocol= SASL_PLAINTEXTsasl.mechanism.inter.broker.protocol=PLAINsasl.enabled.mechanisms=PLAINallow.everyone.if.no.acl.found=truesuper.users=User:adminadvertised.listeners=SASL_PLAINTEXT://:9092, PLAINTEXT://:9093 Here you can see that port 9094 is set to SASL_PLAINTEXT.In nifi, I can write to a topic using the PublishKafka_2_6 processor, using port 9094.But it is impossible to read from the same topic and port 9094. The processor does not show any errors. I used different versions of the consumekafka processor. No results.I am able to read from the topic on the insecure port 9092. I am also able to read on the SASL_PLAINTEXT port through other tools such as datagrip. But for some reason nifi does not want to read through this port. help me please.nifi version 1.16.3
Hey there,I need some help with configuring EvaluateJsonpath because I'm having trouble with it. Everything else seems to be working fine, but I always get stuck at this part. I've tried looking for solutions online, but none of them seem to work or they're just outdated. It's possible that I messed up something in the other function blocks, but they still work if I remove the EvaluateJsonpath. Any ideas? This is the json file:{ "random": 3171504720, "als": 29024, "mic": 47334, "gyro": { "x": -26, "y": 8, "z": 1 }, "accel": { "x": -1829, "y": 447, "z": 16802 }}
Hi All, I have Nifi running on EKS cluser with 3 nodes. Every now and then, flow.xml.gz gets corrupted in one of the node and it fails to start back up again. Same things happens with other nodes sooner or later and ultimately NiFi crashes. The error from nodes startup logs - schema validation Error parsing Flow configuration at line <line_no>, col <col no > : cvc-complex-type:2.4.d : Invalid content was found starting with element 'property'. No child element is expected at this point.It has become a pain to maintain Nifi because of this issue. Please suggest a solution.
HelloI have the following :Tomcat server (local)Nifi (local)customer csv fileIn my flow I am getting the file content and transforming it to a json body (the body I need to send it in the API post).The problem is to use the invokeHTTP processor to send the api (onboard) I need to get a token via another API (auth) and add it in the header of the main api (onboard). Auth API : Sends a json to the tomcat : { "user_name": "User", "password": "Password"} And get a json with a token valid for 2 min { "id": 1, "user_name": "user", "token": "eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJzdXBlcmFkbWluIiwidGV...", "roles": [ "USER", "ADMIN" ], "homePage": null, "priorite": 0} The problem is when I pass the customers json to the auth API it will take that json as the body for that API and I lose my customer json.Is there another way to get the token from the auth API and use it in the onboard API header withou
We have several CDSW jobs set up where we call a python script with a main() function which is called via if __name__ == "__main__": main() Our problem is that these jobs are erronously shown as "Success" even if the python code throws an Exception. How can we set up our CDSW such that these jobs are correctly shown as "Failure"?
I am using Nifi parameter context to avoid hardcoding multiple variables.I used toolkit to create parameter context and used params inside my flow.Now I want to update some values of this existing parameter context.. 1. to update some values or 2. to add new parameters.I could not find a way using toolkit in the documentation. I see an answer here https://community.cloudera.com/t5/Support-Questions/Update-parameter-contexts-dynamically-using-REST-API/m-p/347697 But i am looking for any solution based on toolkit or other easier approach, so that it can be easily handled while deployments to next environments or while updating it in production environments. Please help me with your approach or suggestions.
I am trying to update parameter context within Nifi which is allocated to several process groups. We would like to update the value of one parameter within the parameter context. Please provide some sample payload to do to update parameter contexts either PUT or POST method
Hi, I am using hive 3.1.2 with hadoop 3.3.5. I manually trigger minor compaction but it's stuck in initiated state. I can't find any documentation on how to configure for workers. I read somewhere it is automatical when you set hive.txn.manager to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager and hive.support.concurrency to true. I have also set below parameters but it is still stuck in initiated state. $ hive -e 'show compactions' 2>/dev/nullCompactionId Database Table Partition Type State Worker Start Time Duration(ms) HadoopJobId1 default employee_trans --- MINOR initiated --- --- --- --- hive.compactor.abortedtxn.threshold=200hive.compactor.check.interval=100shive.compactor.cleaner.run.interval=500mshive.compactor.compact.insert.only=truehive.compactor.delta.num.threshold=2hive.compactor.delta.pct.threshold=0.1hive.compactor.history.reaper.interval=2mhive.compactor.history.retention.attempted=2hive.compactor.history.retention.failed=3hive.compactor.hist
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.