Q&A

I'm trying to housekeep our hdfs filesystem and found alot of very old files that are owned by spark user. I don't remember creating a spark user that would generate these files, hence when I invoke an hdfs dfs -rm -R files...it's giving me a file ownership/permission error. Is there any work around I can bypass this? just like sudo in unix? Thank you. 

Hi,I have 3-node Nifi cluster.Please, i would like to ask if each node of the cluster can output different results (i.e no duplicates) from one another.SCENARIO:Let say i have 10 records on a DB.If node1 output the first record, i want node2 or node3 not to output first record again. Thank you.

Hi,   Does anyone know if Cloudera changed the URL to validate the Cloudera Certificate license number? I used to see the information from the URL 'https://certification.cloudera.com/verify/' and is not working now. Best,Sruthi Kumar

I have the issue whereby I receive a file into NiFi on a single node but wish to copy this onto multiple nodes.Has anyone done this before and what would be the process to do so?

Hi,Please, kindly help.Use Case:Minifi should send logs to a Secured 3 Nodes Nifi Cluster; Method:I am trying to generate client certificate on one of the Nifi nodes(https://masternode:9443)to use for minifi configuration file.I ran this command:./tls-toolkit.sh client -c masternode -D 'CN=nifiadmin, OU=NIFI' -C /tmp/minifi/ -t nifinifinifinifi ERROR:tls-toolkit.sh: JAVA_HOME not set; results may vary[main] INFO org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient - Requesting new certificate from masternode:9443[main] INFO org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPerformer - Requesting certificate with dn CN=nifiadmin,OU=NIFI from masternode:9443Service client error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Thank you.

HelloWe just install a new CDP cluster. AutoTLS is configured. When we want to kerberise the cluster with the wizard we have this error  ldap_add: Constraint violation (19)additional info: 000021C7: AtrErr: DSID-03200EA2, #1:0: 000021C7: DSID-03200EA2, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 90303 (servicePrincipalName) this error appear during the creation of the object HTTP/master_name. How can i resolve this issu? 

When try to distribute Parcel file, I got errors like below.  << cloudera-scm-agent/cloudera-flood.log >>[27/Jun/2022 09:44:18 +0000] 23024 Thread-2 server INFO stats_alert CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcel: [1000] 0 0 0 0 120 0 0 80 0 0[27/Jun/2022 09:44:30 +0000] 23024 MainThread server INFO Adding torrent: file:///data/opt/cloudera/parcel-cache/CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcel.torrent[27/Jun/2022 09:44:30 +0000] 23024 MainThread rpc ERROR Failed handling RPC: AddTorrent Already present torrent: CDH-6.3.4-1.cdh6.3.4.p0.6751098-el7.parcelTraceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/util/rpc.py", line 176, in executeRPCreturn fn(*args)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/server.py", line 278, in addTorrenttorrent = self.depot.importTorrent(torrentUrl)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/flood/depot.py", line 101, in importTorrentraise TorrentError("

After configuring my Dockerfile with:ENV NIFI_WEB_HTTPS_HOST=0.0.0.0ENV NIFI_WEB_HTTPS_PORT=8443 I was able to run the container and access UI. If I increase replica, my cluster informations registries two address nodes 0.0.0.0:8443 and any changes I try to make I receive the error: Node 0.0.0.0:8443 is unable to fulfill this request due to: Transaction 00671bfc-20b8-4257-8590-8936e7265fde is already in progress. If I remove NIFI_WEB_HTTPS_HOST property, I cannot access UI, but log say that Jetty is running with container name, not 0.0.0.0. Is it possible to run nifi in a container without 0.0.0.0?

Hello Cloudera Team,I have a requirement read hdfs file from kerberized cluster in spark local mode when i try this i am facing  Client cannot authenticate via:[TOKEN, KERBEROS]. This was the API i implemented , public static void main(String[] args) {SparkConf conf = new SparkConf().setAppName("spark-ml").setMaster("local[1]").set("hadoop.security.authentication", "KERBEROS").set("spark.hadoop.fs.defaultFS","hdfs://server35:8020").set("hadoop.rpc.protection", "privacy").set("hadoop.security.authorization", "true").set("spark.history.kerberos.enabled", "true").set("spark.kerberos.keytab", "D:\\resources\\ruleuser.keytab").set("spar.kerberos.principal", "ruleuser/server35@HADOOP.COM").set("principal", "ruleuser/server35@HADOOP.COM").set("keytab","D:\\resources\\ruleuser.keytab").set("spark.files", "D:\\resources\\core-site.xml,D:\\resources\\hdfs-site.xml").set("class", "Main2");SparkContext context = new SparkContext(conf);context.addFile("D:\\resources\\", true);SparkSe

Hi,Please, I have a very large data to fetch incrementally( i need to store state) from an Oracle Database.And my query contains multiple JOIN statements.Please, what is the best processor to use for this scenerio because from the information i gathered i was told QueryDatabaseTableRecord proc cannot handle JOIN statements.Thanks.

I am using CM5.16 and using custom repositories but my issue is I am  not able to see the  Backup menu which is usually on the toolbar. Do i have to enable it?

I'm writing some pyspark code where I have a dataframe that I want to write to a hive table.I'm using a command like this. dataframe.write.mode("overwrite").saveAsTable(“bh_test”) Everything I've read online indicates that this should, by default, create  a managed table. However, no matter what I try, it always creates an external table. Is there a configuration setting somewhere that overrides the default behavior?

I'm configuring LDAP access (AD) on Nifi 1.0.1.0 being managed by Cloudera Manager, but when trying to access nifi with AD credentials, the error is returned: "Unknown user with identity 'user.bind'. Contact the system administrator."  In Nifi's "nifi.initial.admin.identity" settings it looks like this: "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local"  What could be wrong for not being able to access?

Please, kindly help.I am trying to do Minifi - Nifi s2s communication.So, i was trying to generate a flow file(to use for minifi) on one of Secured Nifi cluster nodes.- i dropped an input port on the root canvas & connected it to putfile processor- i created a process group. I dropped tailfile processor & Remote Process Group processor inside it.- I configured RPG with the url of the root canvasI don't actually know what i am doing wrong because all my configurations seems to be good.# Site to Site propertiesnifi.remote.input.host=masternodenifi.remote.input.secure=truenifi.remote.input.socket.port=10000nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs Below are the the pics. 

Cluster has three journal nodes all three were in bad health and name nodes went down.Since then I am trying to bring up the name nodes.I am getting this error-  2022-06-25 08:32:24,365 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1165msGC pool 'G1 Young Generation' had collection(s): count=2 time=1574msGC pool 'G1 Old Generation' had collection(s): count=1 time=218271ms2022-06-25 08:32:24,474 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log: 2434229/2578837 transactions completed. (94%)2022-06-25 08:32:24,764 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Error replaying edit log at offset 554705629. Expected transaction ID was 60366342312Recent opcode offsets: 554704754 554705115 554705361 554705629java.io.IOException: Premature EOF from inputStreamat org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:203)at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPr

I believe I found a bug in the NiFi LookupRecord processor.When there is no match for a key in the DatabaseRecordLookupService it returns an error: The lookup processor config:The input JSON has the following structure: [ { "id": 1, "name": "Bitcoin", "symbol": "BTC" }, { "id": 2, "name": "Apecoin", "symbol": "Ape" }]   

Hi, moved Ni-Fi 1.16.3 folder from one destination to another and now getting error for ExecuteScript processor: 'Script Engine' validated against 'ECMAScript' is invalid because Given value not found in allowed set 'Clojure, Groovy, Lua, Python, ruby'. Any idea what's wrong? Thanks a lot. Ludvik

Sensor data coming from ConsumeMQTT processor looks like this{1,1,1,3465689,B4:E6:2D:34:E1:D9,BE:FB:E4:11:91:5A,-42,192.168.0.21,v0.16,15.88,16.35,46.59,95931.16,0,0,0,3.27,0,0,0,0}Expected Output: {    "val 1" : 1,    "val 2" : 1,.......}and so  on I'm preparing this incoming sensor data to be ingested in influxdb kindly help me out which processor is best for this and required configuration Thanks a lot

Hello!I am trying to  read  JSON body from a flowfile and write the length of a JSON array of elements to new flowfile attribute. But somehow I am getting aforementioned error all the time.Here is my script.  from org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackimport json#Define a subclass of StreamCallback for use in session.write()class PyStreamCallback(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): jsn = IOUtils.toString(inputStream, StandardCharsets.UTF_8) array = json.loads(jsn) # type: dict i = 0 while i <= 1: # ищем вложенность второго уровня, чтобы посчитать кол-во элементов (записей в таблице) root_key = list(array.keys())[0] array = array[root_key] i += 1 self.length = str(len(array)) def get_length_of_array(self): return self.length# end classflowfile = session.get()if(flowfile !=

Hello, Hello,I've been trying to setup NiFi with single user authentication but got stuck in a "Access Unknown: Certificate and Token not found" error, and I'm unable to find a solution. Any help would be greatily apreciate.So, I'm trying to deploy in a remote machine with Docker, with authentication provided by NiFi, using a Nginx as reverse proxy.Maybe I have to pass p12 file to NiFi, but I have no idea how to proced.My docker-compose file is as follows version: '3.9'x-base: &base  image: apache/nifi:1.16.2  networks:    - nifix-nifi-environment: &nifi-environment  NIFI_WEB_HTTPS_PORT: 8443  NIFI_CLUSTER_IS_NODE: "true"  NIFI_ZK_CONNECT_STRING: "zookeeper:2181"  NIFI_ELECTION_MAX_WAIT: "30 sec"  NIFI_ELECTION_MAX_CANDIDATES: 2  NIFI_SENSITIVE_PROPS_KEY: "my-random-string"  NIFI_WEB_PROXY_CONTEXT_PATH: "//"  NIFI_CLUSTER_NODE_PROTOCOL_PORT: 8082  NIFI_WEB_PROXY_HOST: "nifi0:8443,nifi0,nifi0.mysite.com,n

HiHow can I make Impala  read only required partitions not all partitionswe are using the results from a sql query RegardsPhilip

NIFI version 1.16.1 nifi-hive3-nar-1.16.2   While loading data to hive tables using PutHive3Streaming, some of the tables loads are getting failed with errors. I tried to change commit size, it didn't help.  out of 50 hive table loads only 5 gets failed with this error and this is repeatable. Record count for these tables are less than 1Million.   2022-06-21 09:53:23,114 ERROR [Timer-Driven Process Thread-21] o.a.n.processors.hive.PutHive3Streaming PutHive3Streaming[id=9c9e3916-2000-1698-a0da-2dc44149819f] Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedorg.apache.nifi.processors.hive.PutHive3Streaming$ShouldRetryException: Aborted transaction cannot be committed: Transaction txnid:10346416 already abortedat org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:512)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:360)at org.apache.hadoop.s

Hello Team, We recently upgraded the CM from version 7.2.X to 7.6.1. Since this is a production cluster we didn't do a restart of Cluster yet. However, we are getting alert with YARN HistoryServer. After checking the logs, we couldn't see any issue from the YARN HistoryServer but we found issues in the Cloudera agent. Please check the logs as shown below.  [16/Jun/2022 07:50:44 +0200] 3249 GM JOBHISTORY throttling_logger ERROR (4 skipped) Error fetching metrics at 'https://xxx.xxx.com:19890/jmx'Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 223, in _collect_and_parse_and_return self._adapter.safety_valve)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/url.py", line 305, in urlopen_with_retry_on_authentication_errors return function() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 245, in _op

Hello Team, We are seeing a frequent entropy issue in our customer cluster as shown below.  /proc/sys/kernel/random/entropy_avail returns 3754. We also installed the rng-tools in all the nodes and the service rngd is also running. After checking the document https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-data-at-rest-encryption-requirements.html#pnavId1 we can see the ExecStart looks as follows ExecStart=/sbin/rngd -f -r /dev/urandom  However, our ExecStart is the default one and looks as follows ExecStart=/sbin/rngd -f  can you please share if updating the ExecStart will solve the issue? Best RegardsSayed Anisul Hoque  

I am using Nifi 1.6.0. I am trying to copy an S3 file into redshift. The json file on S3  looks like this:[{"a":1,"b":2},{"a":3,"b":4}]However this gives an error because of '[' and ']' in the file (https://stackoverflow.com/a/45348425).I need to convert the json from above format to a format which looks like this:{"a":1,"b":2}{"a":3,"b":4}So basically what I am trying to do is to remove '[' and ']' and replace '},' with '}'.The file has over 14 million rows (30 MB in size)How can I achieve this using Nifi?