Q&A

Hello All,Hello Everyone,I installed nifi on a linux server under /opt/nifi_server/nifi. I can connect to nifi using the provided passwords, but when I try to access it remotely, the error "The supplied username and password are not valid" appears. Earlier, it was working fine. I started facing issues when I removed the NIFI folder, re-extracted it to the same location, and started the services.Any assistance would be greatly appreciated. Thanks,Shakib   

We are upgrading our cluster from HDP 2.6.5 to HDP-7.1.70 :During the upgrading process ( about 65 %), it fails to restart the mapreduce/historyserver with the below error.Unable to find a solutionAny idea or help ? raceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 126, in <module>HistoryServer().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 987, in restartself.stop(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 47, in stopimport paramsFile "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/params.py", line 29, in <module>from params_linux import *File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/s

Good morning, afternoon, evening..I am a bit disappointed because I start recently a experience like agent for Cloudera.. well, I start slowly getting my skills done for moving forward.But unfortunately, since few days, I get trouble with the missions. I need to do all the time an deposit for clear up the overdrawn. in the WhatsApp group, some try to reassure me and some don’t have any empathy for my reaction. why all the time, I get this high package?in silver membership, I was wondering if I could be in this situation so often. well, now here we go again.. I am stock and I put all my money in for finish this mission, 39/40. Again!!! Why ????that’s horrible this feeling to lose everything like this. I suppose to get some profit no troubles.  I don’t have any funds and I don’t know what I can do. 

Hi everyone!Can somebody assist me, please?I have a task to consume data from different WebSocket servers (hosted on different IPs). Each WebSocket server has a lot of sensors producing messages, and I need to subscribe and listen to each of the sensors. I need connections to stay alive.This is a simplified communication diagram: As I understood, I can use a ConnectWebSocket processor with JettyWebSocketClient under the hood to create a connection to a particular WebSocket server. The problem I faced with is that I can not dynamically create ClientId in Connect WebSocket processor.That leads to a problem. Let's say, there are 50 sensors producing messages on each of the 3 WebSocket servers I have. I will receive the WebSkocet server's IP address and specific sensor id from upstream processors. I can dynamically change the IP address in JettyWebSocketClient so that I can connect to 3 different servers, but after that, I can not distinguish sensors within ea

Where are the Spark application logs located, and what is the default path of that directory?

Hi everyone,Please find root cause and why yarn resourcemanager not autofailover on my cluster ?I get message ERROR  on fatal event from my yarn-resource.log on yarn rm node (Active) below:2023-02-25 08:04:03,805 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1670355762504_0824_01_000007 Container Transitioned from ACQUIRED to RELEASED2023-02-25 08:29:08,810 WARN org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6669ms for sessionid 0x1000048d78800012023-02-25 08:29:08,810 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6669ms for sessionid 0x1000048d7880001, closing socket connection and attempting reconnect2023-02-25 08:29:08,911 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode...2023-02-25 08:29:08,911 WARN org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService: Lost contact with

sqoop import --connect 'jdbc:sqlserver://X.X.X.X:1433;database=TOTO' --table TOTO --username=toto --password=toto SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]23/03/25 10:33:22 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.8.0-80123/03/25 10:33:22 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.23/03/25 10:33:22 INFO manager.SqlManager: Using default fetchSize of 100023/03/25 10:33:22 INFO tool.CodeGenTool: Beginning code generation23/03/25 10:33:23 E

Hello,I have built a custom Type with no Entities.  However when I try to delete it I get an error, "Given type InformationAssets has references".  When I pull back that Type using the API I can see it has some built-in relationship:"relationshipAttributeDefs": [{"name": "meanings","typeName": "array<AtlasGlossaryTerm>","isOptional": true,"cardinality": "SET","valuesMinCount": -1,"valuesMaxCount": -1,"isUnique": false,"isIndexable": false,"includeInNotification": false,"searchWeight": -1,"relationshipTypeName": "AtlasGlossarySemanticAssignment","isLegacyAttribute": false}]There are no entities under this Type, so there are no Glossary Terms assigned to it.  I'm not sure how this relationship was created in the first place.  I've deleted relationshipDefs before, but this one "AtlasGlossarySemanticAssignment" appears to be some built-in relationship.  Does anyone know how I can remove this in order to delete the Type?Thanks in advance!

Current version: 1.6.0Upgrade version: 1.8.0Applying the cluster, accessing the ui results in an error.(x.x.x.x is hostname, my server ip..)The key was not created as a wildcard.---UI Error---javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/cj+QcD4F2YafhbXSeRmuLznbS24EWVhAvCbU71aVGWo= DN: CN=x,x,x,x, OU=NIFI subjectAltNames: [x.x.x.x]---api access error---https://x.x.x.x:1234/nifi-api/flow/current-userjavax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/!!!#$@!#qtA3h/Nhqm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x.x.x.x]---log Error---2019-04-08 08:26:28,522 WARN [Replicate Request Thread-5] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to x.x.x.x:9443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/IiBnANAzEOL~~~qm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x

So guys,This time I could really use your help with something because I cannot figure this on my own and neither do I know where to look in the source code exactly.My flow is as follows:ConsumeKafka ----> MergeContent (as I have plenty of small files I prefer to merge them in bigger files for further processing) ----> ReplaceText (I have some empty spaces and I want them removed) ---> PartitionRecord. The problems comes here, in PartitionRecord. I have defined two Controller Services, one Record Reader (CSVReader, with a pre-defined working schema) and and Record Writer (ParquetRecordSetWriter, with the same exact schema as in the CSV reader). I have no strange data types, only a couple of FLOATs and around 100 STRINGS. I defined a property called time, which extracts the value from a field in our File. Unfortunately, when executing the flow, I keep on getting the following error message:" PartitionRecord[id=3be1c42e-5fa9-3144-3365-f568bb616028] Processing halted: yi

Hi all, I am practicing the feature in CDSW. When trying to do an experiment,  I got an error in the session as below: -------error-------import pickleimport cdswmodel = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'ModuleNotFoundError Traceback (most recent call last)/tmp/ipykernel_115/2912545845.py in <module>----> 1 model = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'Engine exited with status 1.-------error------- However, in the Build session in Experiment, I see Sklearn is installed in the docker container. -----build-log-----Step 1/5 : FROM docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.8-standard:2022.11.2-b2---> 7ffae291c607Step 2/5 : WORKDIR /home/cdsw---> 22e8b9772338Removing intermediate container 404b7da93746Step 3/5 : COPY sources /home/cdsw---> 90d2c92ea316Removing intermediate container 959005050d3cStep 4/5 : RUN su cdsw -c "mkdi

Hi all, I am exploring the features in my CDP cluster. I added Spark service to the cluster, when I try to study Spark and run pyspark in terminal, I got the following error: Type "help", "copyright", "credits" or "license" for more information.Warning: Ignoring non-Spark config property: hdfsWarning: Ignoring non-Spark config property: ExitCodeExceptionWarning: Ignoring non-Spark config property: atSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).23/03/29 02:47:40 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:43 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:49 ERROR spark.SparkContext: Error initializing SparkContext.java.io.FileNotFoundException: File file:/home/asl/2023-03-28 23:17:30,775 WARN [TGT Renewer for asl@MY.CLOUDERA.LAB] security.UserGroupInformation (UserGroupInformation.java:run(1026)) - Exception encountered

I download Nifi flow definition on a Process Group. Then I search the real id of a processor and get nothing. Why the definition does not keep everything as same as those in a Nifi flow? I need to compare two Nifi flows in two environment, DEV and UAT. But the definition does not keep all processors' identifier. It is so difficult for me to find differences.

I want to remove a node from cluster and inside the cluster all node's need to updated with removed node ,example :   i have a cluster with 4 nodes node -1 , 2 [ Primary ], 3 , 4 i want to remove node -3 from cluster , and it should be updated automatically in [ authorizers.xml ] to remaining working  nodes.  please help its bit urgent 

We are running a spark streaming job which reads data from kafka and writes to RDBMS. We dont want this job to fail easily due to minor fluctuations in cluster health/issue. Write now spark job has configuration to retry 5 attempts before the whole job fails. But all these retries are happening in quick succession , one after another. Is there a way we can put some delay/sleep time between retry attempts for this job?

Does Apache NiFi 1.19.1 or 1.20.0 Have Kotlin in the Software Stack?  And, how can I confirm this? Thank you

Hello, I'm trying to start/stop a processor using the NiFi API in a kerberized NiFi cluster (Cloudera Data Flow) but the PUT request to do it fails in identifying the authenticated user. Other requests like get processor information, get current user information, etc. works fine. I'm running the process in a bash script in the same machine as the NiFi node. I have just one node in this environment with load balancer. Here is the process I follow: 1. Create kerberos token in the system (kinit my.username) 2. Create NiFi token calling the /access/kerberos endpoint curl -k -X POST --negotiate -u : "https://node-ip:8443/nifi-api/access/kerberos"​  3.Store the token in a variable and check I'm authenticated curl -k --header "Authorization: Bearer $token" "https://node-ip:8443/nifi-api/access" {"accessStatus":{"identity":"my.username","status":"ACTIVE","message":"Access Granted: Token authenticated."}}​ 4.Get processor

Hi Team, It would be appreciated if someone please guide me how to set spark memory for spark job, where cluster utilization should take 1%-2% memory only for each spark job. Please share math's logic how to calculate on below cluster node details as - #1 How many working Nodes Cluster we have currently? |>Nodemanagers:166 >Datanodes:159  #2 How many Cores per Node we have currently ? |>64 Cores  #3 How much GB RAM per node. we have currently ? |>503 GB   ==== Wanted to calcuate for spark job ===#1driver-memory#2 executor-memory#3 driver-cores#4 executor-cores#5 num-executor========================Please suggest if any additional parameter help to tune the spark job (execution time and cluster utilization) wise.

Team, My current Apache spark jobs are executing very slow, could you please help me to make it Run Faster. Looking for thumb rule to make spark job it RUN FASTER. Thanks in advance for your kind support always. 

I'm using a tool in which I have to point out the master node (driver node) of the Cloudera Spark Cluster (spark :// <some-spark-master> : 7077). Also as I learned, Spark has "Master Node", "Driver Node" and "Worker Nodes".So I decided to go to the Cloudera Web Manager and checked the Configuration Tab of the Spark service, but all I found are "Gateway instance" and "History Server instance". Where are the "Driver instance" and "Worker instance"? I can't add these two instances in the "Add Role Instances" too  My guess is that it's in Yarn service configuration, but I can't find anything related to "Master", "Driver" or "Worker" either.  So what is the link to "Spark Master" that ends with 7077 (what is the Node)? I can't find it anywhere in the Configuration tab

I'm facing weird issue, not sure why Spark is behaving like this.samplefile.txt:COL1|COL2|COL3|COL4 "1st Data"|"2nd ""\\\\P"" data"|"3rd data"|"4th data" This is my spark code to read data:val df = spark.read.format("csv").option("header","true").option("inferSchema","true").option("delimiter","|").load("\samplefile.xtx")df.show(false)Some how it is combining 2 columns data into one. Spark Scala : 2.4 VersionAny idea why spark is behaving like this.

This error has been reported earlier https://community.cloudera.com/t5/Support-Questions/org-apache-hadoop-util-DiskChecker-DiskErrorException-No/m-p/361337#M238564  I have a 3 node hadoop cluster for doing my research on Medical Side EffectsEach node has Ubuntu 18.04.5 LTS.CDH 6.3.4-1.cdh6.3.4.p0.6751098 Basically I have not been able to run any MR jobs or Hive queries/jobsHive select * works of course because select * does not launch an MR job...However querying a Hive table with just 2 cols and 2 rows fails  hive -e "show create table beatles"CREATE EXTERNAL TABLE `beatles`(`id` int,`name` string)COMMENT 'Beatles Group'ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'WITH SERDEPROPERTIES ('field.delim'=',','line.delim'='\n','serialization.format'=',')STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'hdfs-//hp8300one:8020/data/demo'TBLPROPERTI

Spark jobs failed after running for ~ 600hrs with the below error:  ERROR util.Utils: Uncaught exception in thread shutdown-hook-0java.lang.InterruptedException        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067)        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)        at org.apache.spark.streaming.scheduler.JobScheduler.stop(JobScheduler.scala:133)        at org.apache.spark.streaming.StreamingContext$$anonfun$stop$1.apply$mcV$sp(StreamingContext.scala:686)        at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1314)        at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:685)     

Hi everyone, So I'm trying to write a data quality query that is efficient in counting the number of nulls in particular columns. Currently the query that I use is as below(I actually found a related example here on a related topic) :SELECT CustFirstName, CustLastName,CASE WHEN CustFirstName is null then 1 else 0 end CustFirstNameNullCheck,CASE WHEN CustLastName is null then 1 else 0 end CustLastNameNullCheckFROM SchemaName.DbNameWHERE Date = '2023-03-15' The output that I get is:CustFirstNameCustLastNameCustFirstNameNullCheckCustLastNameNullCheck{null}{null}11 And then hundreds of thousands of single rows of {nulls} that follow(I know, it's a long story). Anyhow, the output that I would LIKE to get, please, is a much simpler layout with just the column(or an alias) and a count of the number of total nulls:CustFirstNameCustLastName500,000 (nulls)500,000 (nulls) I've tried different iterations of SELECT COUNT(*) and just could not get the query right. MapR gives

Hi,I am receiving and HTTP call in NIFI. Which does the below job:Execute the sql and fetch resultConvert the result in to JSON(there are 100 or more number of records)Send back the result as HTTPRESPONSE.Now the issue is that if i want to sent the data part by part( Using pagination mode) . For an example:Receiving the HTTP call . In the call we will be receiving page number. Say it is 5 pages.So if the total record is 100 then first response will be 20 records(100/5).  First response will have 20 records, second response will have another 20. Like that 5 timesHow can i handle this in NIFI ? Please provide your thoughts on the same. Your help will be much appreciated