Find answers, ask questions, and share your expertise
Recently active
Hi,I have some schedulers in OOZIE through hue and some of them some times fail. However when I run them manually after, they end successfully. Is there any way to put retry policy in my WFs? Here is the error that I am taking:Exit code of the Shell command 1<<< Invocation of Shell command completed <<<java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apa
HELLO, IM TRYing to refresh automatically via ssis package a cube ssas tabular which is connected to hive via ODBC cloudera latest driver. When I´m manually processing my cube It works without any problems , but when I´m using SSIS to automate the process I have this error message (It worked in ssis fine just this 4 days I have this problem) Analysis Services Execute DDL Task] Error: The JSON DDL request failed with the following error: Failed to save modifications to the server. Error returned: 'OLE DB or ODBC error: [DataSource.Error] ODBC: ERROR [HY000] [Cloudera][ThriftExtension] (14) Unexpected response from server during a HTTP connection: SSL_read: errno = 9. ERROR [HY000] [Cloudera][ThriftExtension] (14) Unexpected response from server during a HTTP connection: SSL_read: errno = 9.. OLE DB or ODBC error: The command has been canceled.. OLE DB or ODBC error: The command has been canceled.. OLE DB or ODBC error: The command has been canceled
I am learning the ExecuteScript processor. I want to understand, how to use python script to fetch a file from local, similar to what the GetFile processor does. Thanks
sql("show databases").show() sql("show tables").show()error
Hello Team, we are doing CDC by pushing data to Kafka and another pipeline will be reading data from Kafka. Whenever we restart the second pipeline (read from Kafka to Kudu), I notice there are thousands of records coming. I would like to know how Kafka keeps the checkpoints? Is there any setting to change it? Thanks,Roshan
Hello! So I am trying to figure out how to work with the EnforceOrder processor.I have 200 flowfiles and every flowfile has attribute partition_number. Many flowfiles has the same filename. I wanna use EnforceOrder to merge them accroding to filename and partition_number attributebut non of the flowfiles ever go to success relationship queue.Here's my settings:What am I doing wrong?
I want to insert splitted records from splitjson processor in database table using nifi.For eg: Below is splitjson processor output. I want to push all the flow files in database table having same fragment.identifier.table columns are mentioned below:identity_id|json|person_refwhere identity_id should be auto sequence number json would flow-file contentperson_ref is the foreign key having int data typeso table will have 13 entries and identity_id column will be unique numbers in sequence, json column will have different flow file contents and person_ref column should be same for all 13 entries lets say 111.
I have a pyspark dataframe with names likeN. PlainfieldNorth PlainfieldWest Home LandNEWYORKnewyorkSo. PlainfieldS. PlaindieldSome of them contain dots and spaces between initials and some do not. How can they be converted to: n Plainfieldnorth plainfieldwest homelandnewyorknewyorkso plainfields plainfield(with no dots and spaces between initials and 1 space between initials and name)I tried using the following but it only replaces dots and doesn't remove spaces between initials:names_modified = names.withColumn("name_clean", regexp_replace("name", r"\.",""))After removing the whitespaces and dots is there any way get the distinct values.like this.north plainfieldwest homelandnewyorkso plainfield
I'm trying to housekeep our hdfs filesystem and found alot of very old files that are owned by spark user. I don't remember creating a spark user that would generate these files, hence when I invoke an hdfs dfs -rm -R files...it's giving me a file ownership/permission error. Is there any work around I can bypass this? just like sudo in unix? Thank you.
Hi,I have 3-node Nifi cluster.Please, i would like to ask if each node of the cluster can output different results (i.e no duplicates) from one another.SCENARIO:Let say i have 10 records on a DB.If node1 output the first record, i want node2 or node3 not to output first record again. Thank you.
Hi, Does anyone know if Cloudera changed the URL to validate the Cloudera Certificate license number? I used to see the information from the URL 'https://certification.cloudera.com/verify/' and is not working now. Best,Sruthi Kumar
I have the issue whereby I receive a file into NiFi on a single node but wish to copy this onto multiple nodes.Has anyone done this before and what would be the process to do so?
Hi,Please, kindly help.Use Case:Minifi should send logs to a Secured 3 Nodes Nifi Cluster; Method:I am trying to generate client certificate on one of the Nifi nodes(https://masternode:9443)to use for minifi configuration file.I ran this command:./tls-toolkit.sh client -c masternode -D 'CN=nifiadmin, OU=NIFI' -C /tmp/minifi/ -t nifinifinifinifi ERROR:tls-toolkit.sh: JAVA_HOME not set; results may vary[main] INFO org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient - Requesting new certificate from masternode:9443[main] INFO org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPerformer - Requesting certificate with dn CN=nifiadmin,OU=NIFI from masternode:9443Service client error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Thank you.
HelloWe just install a new CDP cluster. AutoTLS is configured. When we want to kerberise the cluster with the wizard we have this error ldap_add: Constraint violation (19)additional info: 000021C7: AtrErr: DSID-03200EA2, #1:0: 000021C7: DSID-03200EA2, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 90303 (servicePrincipalName) this error appear during the creation of the object HTTP/master_name. How can i resolve this issu?
When try to distribute Parcel file, I got errors like below. << cloudera-scm-agent/cloudera-flood.log >>[27/Jun/2022 09:44:18 +0000] 23024 Thread-2 server INFO stats_alert CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcel: [1000] 0 0 0 0 120 0 0 80 0 0[27/Jun/2022 09:44:30 +0000] 23024 MainThread server INFO Adding torrent: file:///data/opt/cloudera/parcel-cache/CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcel.torrent[27/Jun/2022 09:44:30 +0000] 23024 MainThread rpc ERROR Failed handling RPC: AddTorrent Already present torrent: CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcelTraceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/util/rpc.py", line 176, in executeRPCreturn fn(*args)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/server.py", line 278, in addTorrenttorrent = self.depot.importTorrent(torrentUrl)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/depot.py", line 101, in importTorrentraise TorrentError("
After configuring my Dockerfile with:ENV NIFI_WEB_HTTPS_HOST=0.0.0.0ENV NIFI_WEB_HTTPS_PORT=8443 I was able to run the container and access UI. If I increase replica, my cluster informations registries two address nodes 0.0.0.0:8443 and any changes I try to make I receive the error: Node 0.0.0.0:8443 is unable to fulfill this request due to: Transaction 00671bfc-20b8-4257-8590-8936e7265fde is already in progress. If I remove NIFI_WEB_HTTPS_HOST property, I cannot access UI, but log say that Jetty is running with container name, not 0.0.0.0. Is it possible to run nifi in a container without 0.0.0.0?
Hello Cloudera Team,I have a requirement read hdfs file from kerberized cluster in spark local mode when i try this i am facing Client cannot authenticate via:[TOKEN, KERBEROS]. This was the API i implemented , public static void main(String[] args) {SparkConf conf = new SparkConf().setAppName("spark-ml").setMaster("local[1]").set("hadoop.security.authentication", "KERBEROS").set("spark.hadoop.fs.defaultFS","hdfs://server35:8020").set("hadoop.rpc.protection", "privacy").set("hadoop.security.authorization", "true").set("spark.history.kerberos.enabled", "true").set("spark.kerberos.keytab", "D:\\resources\\ruleuser.keytab").set("spar.kerberos.principal", "ruleuser/server35@HADOOP.COM").set("principal", "ruleuser/server35@HADOOP.COM").set("keytab","D:\\resources\\ruleuser.keytab").set("spark.files", "D:\\resources\\core-site.xml,D:\\resources\\hdfs-site.xml").set("class", "Main2");SparkContext context = new SparkContext(conf);context.addFile("D:\\resources\\", true);SparkSe
Hi,Please, I have a very large data to fetch incrementally( i need to store state) from an Oracle Database.And my query contains multiple JOIN statements.Please, what is the best processor to use for this scenerio because from the information i gathered i was told QueryDatabaseTableRecord proc cannot handle JOIN statements.Thanks.
I am using CM5.16 and using custom repositories but my issue is I am not able to see the Backup menu which is usually on the toolbar. Do i have to enable it?
I'm writing some pyspark code where I have a dataframe that I want to write to a hive table.I'm using a command like this. dataframe.write.mode("overwrite").saveAsTable(“bh_test”) Everything I've read online indicates that this should, by default, create a managed table. However, no matter what I try, it always creates an external table. Is there a configuration setting somewhere that overrides the default behavior?
I'm configuring LDAP access (AD) on Nifi 1.0.1.0 being managed by Cloudera Manager, but when trying to access nifi with AD credentials, the error is returned: "Unknown user with identity 'user.bind'. Contact the system administrator." In Nifi's "nifi.initial.admin.identity" settings it looks like this: "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" What could be wrong for not being able to access?
Please, kindly help.I am trying to do Minifi - Nifi s2s communication.So, i was trying to generate a flow file(to use for minifi) on one of Secured Nifi cluster nodes.- i dropped an input port on the root canvas & connected it to putfile processor- i created a process group. I dropped tailfile processor & Remote Process Group processor inside it.- I configured RPG with the url of the root canvasI don't actually know what i am doing wrong because all my configurations seems to be good.# Site to Site propertiesnifi.remote.input.host=masternodenifi.remote.input.secure=truenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs Below are the the pics.
Cluster has three journal nodes all three were in bad health and name nodes went down.Since then I am trying to bring up the name nodes.I am getting this error- 2022-06-25 08:32:24,365 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1165msGC pool 'G1 Young Generation' had collection(s): count=2 time=1574msGC pool 'G1 Old Generation' had collection(s): count=1 time=218271ms2022-06-25 08:32:24,474 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log: 2434229/2578837 transactions completed. (94%)2022-06-25 08:32:24,764 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Error replaying edit log at offset 554705629. Expected transaction ID was 60366342312Recent opcode offsets: 554704754 554705115 554705361 554705629java.io.IOException: Premature EOF from inputStreamat org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:203)at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPr
I believe I found a bug in the NiFi LookupRecord processor.When there is no match for a key in the DatabaseRecordLookupService it returns an error: The lookup processor config:The input JSON has the following structure: [ { "id": 1, "name": "Bitcoin", "symbol": "BTC" }, { "id": 2, "name": "Apecoin", "symbol": "Ape" }]
Hi, moved Ni-Fi 1.16.3 folder from one destination to another and now getting error for ExecuteScript processor: 'Script Engine' validated against 'ECMAScript' is invalid because Given value not found in allowed set 'Clojure, Groovy, Lua, Python, ruby'. Any idea what's wrong? Thanks a lot. Ludvik
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.