Member since
12-21-2017
67
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1070 | 10-15-2018 10:01 AM | |
4043 | 03-26-2018 08:23 AM |
11-01-2022
12:01 AM
Hi @Siddu198 Add this config to your job: set("mapreduce.fileoutputcommitter.algorithm.version","2")
... View more
09-03-2019
11:09 AM
Hi, To view the spark logs of an completed application you can view the logs by running the below command yarn logs -applicationId application_xxxxxxxxxxxxx_yyyyyy -appOwner <userowner> > application_xxxxxxxxxxxxx_yyyyyy.log Thanks AKR
... View more
08-28-2019
04:47 AM
I am facing same issue. Did you find any solution?
... View more
02-01-2019
04:25 PM
We ran into the same issue because we rely on poor mans DNS via local hosts file, as we don't have control over the infrastructure.
To solve this issue of advertising non existent hostnames there are two solutions:
1. create separate configuration groups for each kafka broker and override `listeners` property with explicit IP of the relevant node
2. setup ambari-agent to specify a custom public hostname and use template variable at kafka config to use that property
Solution 2 introduces a fix or another problem, it depends on your setup: By setting up ambari-agent to use a custom public hostname links from ambari to services like HDFS UI, YARN UI, Spark UI, Zeppelin, etc. will use this value.
To setup solution 2: Create public hostname script
Place a file at
/var/lib/ambari-agent/public_hostname.sh and make it executable chmod a+x /var/lib/ambari-agent/public_hostname.sh with following content:
#!/bin/sh
hostname -I | awk '{print $1}'
Change ambari-agent config
at
/etc/ambari-agent/conf/ambari-agent.ini , add property at agent section:
public_hostname_script=/var/lib/ambari-agent/public_hostname.sh
Restart ambari agent
ambari-agent restart
Configure Kafka Broker listener
PLAINTEXT://{{config['agentLevelParams']['public_hostname']}}:6667 Restart Kafka
... View more
11-13-2018
08:49 AM
Thanks @KB And another question: When my spark application writing massive of data to hdfs, it always throws error message like following: No lease on /user/xx/sample_2016/_temporary/0/_temporary/attempt_201604141035_0058_m_019029_0/part-r-19029-1b93e1fa-9284-4f2c-821a-c83795ad27c1.gz.parquet:File does not exist.HolderDFSClient_NONMAPREDUCE_1239207978_115 does not have any open files.
How to solve this problem? I search online and others said it is related to dfs.datanode.max.xcievers
... View more
10-15-2018
10:01 AM
1 Kudo
Solve by using HttpFs. It set a gateway where no need to access data node.
... View more
10-10-2018
06:05 AM
1000 is for spark. You can set common.max_count at a global level. You should not have negative results if you increase the limit. But if your data size if very huge then you may need to tweak the above mentioned params accordingly.
... View more
08-16-2018
07:15 AM
Hi @Jonathan Sneep Fine. thanks. I have added the user and group info to my namenode. So the typical way to adding the new user or group is creating the user and group on namenode, and waiting for usersync to sync the user info to Ranger? So if I don't care the group policy, creating internal user in ranger and specifying them in allow conditions also works? At least it seems work in practice..
... View more
07-27-2018
10:21 AM
@Junfeng Chen I am facing a similar problem , can you please share the steps you performed to resolve it.
... View more
06-12-2018
11:33 PM
hi~ i am facing the same problem....can u show the steps to slove?
... View more