Find answers, ask questions, and share your expertise
Recently active
Hi Team, It would be appreciated if someone please guide me how to set spark memory for spark job, where cluster utilization should take 1%-2% memory only for each spark job. Please share math's logic how to calculate on below cluster node details as - #1 How many working Nodes Cluster we have currently? |>Nodemanagers:166 >Datanodes:159 #2 How many Cores per Node we have currently ? |>64 Cores #3 How much GB RAM per node. we have currently ? |>503 GB ==== Wanted to calcuate for spark job ===#1driver-memory#2 executor-memory#3 driver-cores#4 executor-cores#5 num-executor========================Please suggest if any additional parameter help to tune the spark job (execution time and cluster utilization) wise.
Team, My current Apache spark jobs are executing very slow, could you please help me to make it Run Faster. Looking for thumb rule to make spark job it RUN FASTER. Thanks in advance for your kind support always.
I'm using a tool in which I have to point out the master node (driver node) of the Cloudera Spark Cluster (spark :// <some-spark-master> : 7077). Also as I learned, Spark has "Master Node", "Driver Node" and "Worker Nodes".So I decided to go to the Cloudera Web Manager and checked the Configuration Tab of the Spark service, but all I found are "Gateway instance" and "History Server instance". Where are the "Driver instance" and "Worker instance"? I can't add these two instances in the "Add Role Instances" too My guess is that it's in Yarn service configuration, but I can't find anything related to "Master", "Driver" or "Worker" either. So what is the link to "Spark Master" that ends with 7077 (what is the Node)? I can't find it anywhere in the Configuration tab
I'm facing weird issue, not sure why Spark is behaving like this.samplefile.txt:COL1|COL2|COL3|COL4 "1st Data"|"2nd ""\\\\P"" data"|"3rd data"|"4th data" This is my spark code to read data:val df = spark.read.format("csv").option("header","true").option("inferSchema","true").option("delimiter","|").load("\samplefile.xtx")df.show(false)Some how it is combining 2 columns data into one. Spark Scala : 2.4 VersionAny idea why spark is behaving like this.
This error has been reported earlier https://community.cloudera.com/t5/Support-Questions/org-apache-hadoop-util-DiskChecker-DiskErrorException-No/m-p/361337#M238564 I have a 3 node hadoop cluster for doing my research on Medical Side EffectsEach node has Ubuntu 18.04.5 LTS.CDH 6.3.4-1.cdh6.3.4.p0.6751098 Basically I have not been able to run any MR jobs or Hive queries/jobsHive select * works of course because select * does not launch an MR job...However querying a Hive table with just 2 cols and 2 rows fails hive -e "show create table beatles"CREATE EXTERNAL TABLE `beatles`(`id` int,`name` string)COMMENT 'Beatles Group'ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'WITH SERDEPROPERTIES ('field.delim'=',','line.delim'='\n','serialization.format'=',')STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'hdfs-//hp8300one:8020/data/demo'TBLPROPERTI
Spark jobs failed after running for ~ 600hrs with the below error: ERROR util.Utils: Uncaught exception in thread shutdown-hook-0java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475) at org.apache.spark.streaming.scheduler.JobScheduler.stop(JobScheduler.scala:133) at org.apache.spark.streaming.StreamingContext$$anonfun$stop$1.apply$mcV$sp(StreamingContext.scala:686) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1314) at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:685)
Hi everyone, So I'm trying to write a data quality query that is efficient in counting the number of nulls in particular columns. Currently the query that I use is as below(I actually found a related example here on a related topic) :SELECT CustFirstName, CustLastName,CASE WHEN CustFirstName is null then 1 else 0 end CustFirstNameNullCheck,CASE WHEN CustLastName is null then 1 else 0 end CustLastNameNullCheckFROM SchemaName.DbNameWHERE Date = '2023-03-15' The output that I get is:CustFirstNameCustLastNameCustFirstNameNullCheckCustLastNameNullCheck{null}{null}11 And then hundreds of thousands of single rows of {nulls} that follow(I know, it's a long story). Anyhow, the output that I would LIKE to get, please, is a much simpler layout with just the column(or an alias) and a count of the number of total nulls:CustFirstNameCustLastName500,000 (nulls)500,000 (nulls) I've tried different iterations of SELECT COUNT(*) and just could not get the query right. MapR gives
Hi,I am receiving and HTTP call in NIFI. Which does the below job:Execute the sql and fetch resultConvert the result in to JSON(there are 100 or more number of records)Send back the result as HTTPRESPONSE.Now the issue is that if i want to sent the data part by part( Using pagination mode) . For an example:Receiving the HTTP call . In the call we will be receiving page number. Say it is 5 pages.So if the total record is 100 then first response will be 20 records(100/5). First response will have 20 records, second response will have another 20. Like that 5 timesHow can i handle this in NIFI ? Please provide your thoughts on the same. Your help will be much appreciated
I trying to ingest a SQL Table to Hive using NiFi. This is easily done by using a QueryRecordTable. However, the records in the SQL table can be modified by hand, so they are not static.Is there any way to update the data in HDFS according to the changes made in the SQL table?
I'm checking whether there are any effects to spark2 by CVE-2022-33891 or not.Is there anyone who can explanin it to me?
Hi, few hours ago zeppelin stopped to run queries and begins to throw errors like WARN org.apache.zeppelin.socket.NotebookServer: Job 20230217-161657_1725299508 is finished, status: ERROR, exception: null, result: %text javax.security.auth.login.LoginException: Unable to obtain password from userIts kerberized AD KDC cluster.ldapsearch and kinit with right principal works fine./etc/hosts are fine too.The rest services on the cluster are green.http://hostname:port/api/credential/ returns {"status":"OK","body":{"userCredentials":{}}}What else can I check?
Guys, have any of you ever had this problem when using the QueryDatabaseTable?I have a datetime field in sql server but when it passes through the QueryDatabaseTable it becomes a varchar and ends up generating an error in Nifi.
Please assist me to find the list of unaccessed or last accessed hive tables/partitions using cloudera navigator, using CDH 6.3.4 version.
Hi,I have query on "Delivery guarantee" parameter in publishKafka processor in NIFI. I have query on below two option supported for the "Delivery guarantee" 1. Best effort2. Guarantee single node delivery I know that the "best effort" does not wait for the ack/response from kafka, but in case of publish failure, when either of the above two option is set, will NIFI try to publish among the available nodes? Like in my setup its the cluster of three kafka nodes. If one of the node is down or if we observe connection timeout issue with one of the node, will NIFI tries to publish with other available nodes? Thanks
Hi ,Am using MSSQL version 12+ , I have connected to DB successfully . I could able to fetch the data eg : Select * from dbname.sys.tables But If i try to query fetch the table records using below processor Getting below error :
I use Impala JDBC to query, I find close() and execute() with synchronized. So if I invoke close() in other thread will wait execute() ,because if execute() is running,close() will blocked waiting for lock. So I should invoke cancal() before close() to avoid block?
I am new to Apache NiFi and need to know how to find a processor or processor group by its integration name, folder, or event name, as well as how to find the specific source and destination file details within any given processor or processor group.
I have step up 3 node (node1, node2, node3) nifi cluster and I am able to access it with http protocol, When I try to make those 3 node secure(https). I am facing "insufficient permissions untrusted proxy" in web UI.After setup 3 node cluster in node1 i followed the steps to make secure connection. Step:1 By using below step I have Generate Keys and Certificates/opt/nifi-toolkit/bin/tls-toolkit.sh standalone -n 'node1' -B ClientPassword -C 'CN=admin,OU=NIFI' -O -o /opt/nifi-1.19.1/conf/certsafter run this I got below files/opt/nifi-1.19.1/conf/certs/├── CN=admin_OU=NIFI.p12├── CN=admin_OU=NIFI.password├── node1│ ├── keystore.jks│ ├── nifi.properties│ └── truststore.jks├── nifi-cert.pem└── nifi-key.keyStep:2I have over write the /opt/nifi-1.19.1/conf/certs/node1/nifi.properties to conf/nifi.propertiesThen Edit the nifi.propertiesnifi.security.user.authorizer=managed-authorizerStep:3Updated Security Properties Key/opt/nifi-1.19.1/bin/nifi.sh set-sensitive-properties-key xyz12345Step
Hello Team, I am trying to send logs to the gaylog server from Nifi for the transfer of each file log.could you please help me with the stepwise procedure to do so?Also i have been trying to copy few files from remote server 1 to 2 and then the same files from 1 will be moved to 3 with the logs of each files in getting pushed to graylog. can you please help me with the step wise procedure to perform this action. i have been struggling from weeks to do above two task.
Hello We have a HDFS cluster configure to work with Kerberos and we want to use nifi processor PutParquet to write file into hadoop cluster.do we need to configure nifi to work with kerberos too in order to connect to Hadoop?what are the steps i need to take in order to do the connection between the two clusters (nifi --> HDFS) Thanks
Hi All, Greetings! Assume that I have a data flow, and the final result is inserting to the SQL database (Microsoft), but the step before! I have the following JSON format for each flow file content as following: { "sbrActivityType" : { "1" : "ACT+A", "3" : "ACT+A", "2" : "ACT+X", "5" : "ACT+X", "4" : "ACT+X" }, "sbrDateTime" : "Sun Mar 19 13:45:15 AST 2023"} Then, my desired output format look like the following:INSERT INTO sbrTestNifiData (sbrActivityType, sbrDateTime) VALUES (?, ?) So, it does not insert correctly to my database table which is looks like the following: CREATE TABLE [dbo].[sbrTestNifiData]( [sbrId] [int] IDENTITY(1,1) NOT NULL primary key, [sbrActivityType] [varchar](50) NULL, [sbrDateTime] [varchar](50) NULL); So, it should inserted to my database table as a five records:sbrIdsbrActivityTypesbrDateTime1ACT+ASun Mar 19 11:56:40 AST 20232ACT+ASun Mar 19 11:56:40 AST 20233ACT+XSun Mar 19 11:56:40 AST 2023
Hi everyone, I'm working to a new feature with an existing nifi cluster to provide a new service to add an interface with serveral kerberized HDP Cluster. I would like to know if a single Nifi cluster can use several realms in the same krb5 file.Reading official documentation, nifi can do it (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#kerberos_properties) : " If necessary the krb5 file can support multiple realms." It seems ok but because at this time I have no way to have several hadoop cluster for testing (to notice : my cluster is working already with one kerberized cluster hadoop ), is anybody can confirm or reject this design: one cluster Nifi with different realms to communicate with multiple kerberized hdp cluster. Thanks for your help and as soon as I have several kerberized cluster hadoop for testing, I will update this article.
Hi, All!I have a problem with NIFI ListSFTP processor. After started processor errored with Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty! ListSFTP processor Version 1.12.0 and the Properties as follows:Error screenshot from UI:
Dear all, I want to run COMPUTE INCREMENTAL STATS with PARTITION option on some partitions of a partitioned table.It is unclear to me from the documentation if DROP INCREMENTAL STATS must be run on those partitions. For my understanding of how INCREMENTAL STATS with the PARTITION option work, it should not be required, but I've found below sentence on this page https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_perf_stats.html#perf_stats "In cases where new files are added to an existing partition, issue a REFRESH statement for the table, followed by a DROP INCREMENTAL STATS and COMPUTE INCREMENTAL STATS sequence for the changed partition.". I'd prefer to avoid running DROP INCREMENTAL STATS if not required. Does anyone know? Many thanks in advance.
Hello,I am trying to solve a strange behavior of NiFi but can't really figure it out.My flow consists of a GenerateTableFetch (which gets executed once per day), a ExecuteSQLRecord (which gets executed once it receives the files from GenerateTableFetch) and finally an PutS3Bucket.Now, both the GenerateTableFetch and the ExecuteSQLRecord are using the same Controller Service, a DBConnectionPool which connects to an Oracle Database.In the previous days I have noticed that I get plenty of sessions from NiFi (which is ok) but these sessions never get killed/terminated and they remain inactive. I am using the following configurations within my DBConnectionPool Service:Max Wait Time: 500 millisMax Total Connections: 51Minimum Idle Connections: 0Max Idle Connections: 5Max Connection Lifetime: -1Time Between Eviction Runs: -1Minimum Evictable Idle Time: 30 minsSoft Minimum Evictable Idle Time: -1Unfortunately I was not able to find any information in the documentation (or maybe I did not fully
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.