Member since
12-05-2016
52
Posts
9
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
164 | 01-05-2017 05:27 AM |
01-08-2019
06:00 AM
Spark submit: spark-submit \
--master yarn \
--deploy-mode cluster \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf -Dlog4j.configuration=xxx -Djava.util.Arrays.useLegacyMergeSort=true" \
--conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf -Dlog4j.configuration=xxx -Djava.util.Arrays.useLegacyMergeSort=true" \
--conf spark.ui.port=18086 \
--conf spark.executor.memory=${executor_memory} \
--conf spark.executor.instances=${num_executors} \
--conf spark.executor.cores=${executor_cores} \
--conf spark.driver.memory=4g \
--conf spark.driver.maxResultSize=3g \
--conf spark.kafka.broker.ingest=xxx \
--conf spark.kafka.zookeeper.ingest=xxx \
--conf spark.kafka.broker.egest=xxx \
--conf spark.kafka.topic.input=xxx \
--conf spark.kafka.topic.output=xxx \
--conf spark.kafka.input.interval=10 \
--conf spark.kafka.group=xxx \
--conf spark.streaming.kafka.maxRetries=10 \
--conf spark.kafka.security.protocol.ingress=SASL_PLAINTEXT \
--conf spark.kafka.security.protocol.egress=SASL_PLAINTEXT \
--conf spark.fetch.message.max.bytes=104857600 \
--conf spark.hive.enable.stats=true \
--conf spark.streaming.backpressure.enabled=true \
--conf spark.streaming.kafka.maxRatePerPartition=1 \
--conf spark.streaming.receiver.maxRate=10 \
--conf spark.executor.heartbeatInterval=120s \
--conf spark.network.timeout=600s \
--conf spark.yarn.scheduler.heartbeat.interval-ms=1000 \
--conf spark.sql.parquet.compression.codec=snappy \
--conf spark.scheduler.minRegisteredResourcesRatio=1 \
--conf spark.yarn.maxAppAttempts=10 \
--conf spark.yarn.am.attemptFailuresValidityInterval=1h \
--conf spark.yarn.max.executor.failures=$((8 * ${num_executors})) `# Increase max executor failures (Default: max(numExecutors * 2, 3))` \
--conf spark.yarn.executor.failuresValidityInterval=1h \
--conf spark.task.maxFailures=8 \
--conf spark.yarn.submit.waitAppCompletion=false \
--conf spark.yarn.principal=xxx \
--conf spark.yarn.keytab=xxx \
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true \
--queue default \
${APP_HOME}/xxx.jar
... View more
01-08-2019
05:54 AM
Stack trace: WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Secret
Manager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 155456 for spark) can't be found in cache
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 1
55456 for spark) can't be found in cache
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:818)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2165)
at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1438)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:160)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:157)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:157)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:765)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
... View more
01-03-2019
12:52 AM
This is exactly the way we use spark-submit. A token is valid only for 24 hrs. Yarn renew the token every 24 hrs automatically until it reaches the max lifetime (which is 7 days) then the token cannot get renewed anymore and needs to be reissued. Hence, we are restarting the job every 7 days. Restarting the job every 7 days doesn't seem to be the right approach for a production environment!
... View more
12-20-2018
05:40 AM
We are using a headless keytab to run our long-running spark streaming application. The token is renewed automatically every 1 day until it hits the max life limit. The problem is token is expired after max life (7 days) and we need to restart the job. Is there any way we can re-issue the token and pass it to a job that is already running? It doesn't feel right at all to restart the job every 7 days only due to the token issue.
... View more
Labels:
08-21-2018
05:05 AM
I was wondering if Apache Ranger (Including Ranger KMS) in
HDP 2.6.4 fully supports WASB and ADLS. I would like to understand if we move
to use WASB/ADLS instead of HDFS what functionalities will be impacted
regarding Authentication, Authorisation and Auditing.
... View more
09-26-2017
07:57 AM
Hi, I am investigating to have a huge cluster of Nifi instances distributed across the world. I have been told that Metrics Monitor and Metrics Collector are using a very chatty protocol, so Ambari Metrics Collector may face some serious performance issues regarding receiving telemetry information from lots of Metrics Monitor Agents. I have two concerns as follows. 1- How does it scale for thousands of agents? 2- Can I use it over WAN probably by securing that in a two-way SSL way? Regards, Ali
... View more
Labels:
09-03-2017
11:17 AM
Have you checked Nifi throughput using Content Repo in a JBOD mode instead of Raid? Basically, let application decide for the distribution of data.
... View more
08-17-2017
11:22 AM
I am investigating the feasibility of dual home HDP/HDF cluster as follows. It is not an actual dual home because only one of the interfaces will be presented to HDP and HDF for the sake of simplicity. - All the hosts will have 2 NICs with different IP addresses in different subnets. One in "Hadoop Network" another in "Management network". - "Hadoop network" will be used for HDP and HDF and "Management network" will be used for other services. - AD will be used for LDAPs as well as Kerberos KDC. I have some questions regarding this situation. 1- Suppose there are two different hostnames corresponding to different NICs. How can I manage Kerberos? Let's suppose all the hosts have been Kerberized by using management hostname. How can I enable Kerberos for all of the HDP/HDF applications? 2- To make my life much easier all the hosts come with a single hostname across two NICs. Is there any concern regarding Kerberos in this case? Do I need to consider a special case for SSSD configurations for user/group mapping? 3- In the case that all hosts come with a single hostname across two NICs, how can I manage DNS server? Is there any way that I can have one DNS server and manage DNS lookup based on client network? For example, if the lookup comes from the management network DNS provides the response based on Management IP and if it comes from the data network, the response will be provided based on data network IP.
... View more
08-01-2017
07:03 AM
So it is not required to disable Kerberos, upgrade HDP or install new service and enable Kerberos again?
... View more
08-01-2017
06:56 AM
@Manish Kumar Yadav Sorry I got confused. What is the relation of Windows Integrated Authentication with the order of Kerberisation?
... View more
08-01-2017
12:25 AM
@Geoffrey Shelton Okot What about the order of Kerberization? Which one is safer? Kerberizing at the final step or it doesn't matter?
... View more
07-31-2017
08:11 AM
I am trying to understand when the best time would be to enable Kerberos on a production platform to avoid any potential issue? After integrating Web-UI with LDAP or it is safe to Kerberize cluster and then integrate Web-UI like Ambari, Knox, Nifi, Ranger, Zeppelin and Hive with AD and enable fine grained authorization? Does it make any difference from the technical point of view?
What about installing any new service or upgrading different services? Is it safer to disable Kerberos and install new service/upgrade current services and enable Kerberos again, or it is safe to upgrade current services on a Kerberized cluster?
... View more
07-21-2017
12:09 AM
Briliant. Thanks.
... View more
07-19-2017
04:51 AM
I have used a blueprint to provision an HDP cluster in the following conditions: - Create an Ambari-managed HDP cluster in a manual approach (Non-blueprint installation) - Extract blueprint from the created cluster. - Remove all non-generic parts and all tuning part from the extracted blueprint to make it more generic. - Create host-group mapping file to capture all passwords and hostnames. - Create another cluster with the modified blueprint and host-group mapping file. My question is when you are using a manual installation Ambari selects the recommended values for tuning parameters. However, in the blueprint installation, you need to modify them separately or put all parameters in host-group mapping file which make the blueprint maintenance very hard. I was wondering whether there is another way that Ambari can pick all the parameters in the same way that is used in a Non-blueprint installation.
... View more
Labels:
06-24-2017
09:03 AM
Hi, I was wondering how the cross-forest trust between Free-IPA and AD affect the LDAP lookup and user sync. Can I use IPA LDAPs URL for this purpose or since AD will be used as a directory service, we need to use it's URL for all of the service integration and only use Free-IPA as a KDC for Kerberos? What would be the recommended solution regarding using IPA as a KDC and AD as a directory service?
... View more
04-19-2017
07:51 AM
I was wondering what the best practice is for managing Kerberos in Test, Staging and Production platform? Let's say I am going to use Free-IPA server as a KDC and provide the one-way trust between Free-IPA and organization KDC. How can I manage sharing data among those platforms in the case of having different KDC? Should I use a single KDC for all of the platforms to provide the sharing mechanism easier? Consider the following use case: We have got the main traffic is coming to a Shared HDF cluster. The traffic will be split into 3 paths. One for Test, another for Staging and the last one for the production platform. The incoming traffic will be transfered to Nifi instances through site-to-site communication.
... View more
Labels:
04-13-2017
06:47 AM
@Michael Young Thank you very much. Unfortunately, the hardware vendor we have married to does not provide any option like 24x8 TB. I have to choose between 12x(10/8/6TB) or 56x(10/8/6) TB. What do you think? Should I go for 12x10TB for the cold storage?
... View more
04-12-2017
12:51 PM
1 Kudo
Hi, I have a hard time to decide which option would be better for Hot Data Nodes sizing: A) 12 x 10 TB 7.2k, single 12G Raid Controller Pros: Total cost per GB would be less Cons: Less number of data nodes, It takes more to replicate data in the case of node failure B) 24 x 2 TB 7.2k, single 12G Raid Controller Pros: Higher number of data nodes, It takes less to replicate data in the case of node failure Cons: 1 single disk controller might create a bottleneck for 24 disks especially in the case of using single drive raid-0 to take advantage of Raid controller caching, total cost per GB would be higher I was wondering how bad would be to use a data node with a 56x10TB disk with 2 Raid Controller for a cold storage?
... View more
04-12-2017
06:11 AM
Please follow this instruction to deploy HDP on heterogeneous disks for Hot, Warm and Cold tiering in a single node. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hdfs-administration/content/configuring_archival_storage.html
... View more
04-11-2017
02:33 PM
I was wondering whether there is a best practice for providing Hadoop storage tiering in HDP or not? What would be the recommended replication factor for hot warm and cold storages? Which architecture would be recommended? Having separate data nodes for warm and cold storage or using hybrid disks in the entire platform? Is that possible to provide separate data nodes in HDP cluster through Ambari or it needs some customization at the Hadoop layer?
... View more
Labels:
03-13-2017
12:03 PM
Let's say Storm is compute intensive and Kafka IO intensive. In this case, do you suggest to collocate Storm and Kafka on the same node? I have seen Storm-Zookeeper collocation and separate Kafka
... View more
Labels:
02-26-2017
02:50 AM
So can we virtually build a hierarchy of encryption zone in this way?
... View more
02-26-2017
02:35 AM
I was wondering is there any way to have a hierarchy of encryption zones managing by Ranger-KMS? Suppose we have the following directory structure: /userA/userB/ /userA/userC/ I want to configure HDFS in a way that userB and userA use two different encryption zones, but I would like to be able to access userB and userC folders with userA and be able to encrypt/decrypt data owned by userB or userC. Is there any way to handle this situation with Ranger-KMS?
... View more
02-20-2017
02:12 AM
Let's say I have two different clusters authenticated by Kerberos. Let's say I have enabled TDE and managed that with Ranger-KMS. How can I design a solution which can work in this regards? Based on my understanding from the Kerberos point of view I have to provide a centralised KDC for both clusters. How can a user connect to one of the clusters run a spark query with some of the data stored in cluster 1 HDFS some of that in cluster 2 HDFS? How can a user run a hive aggregation query on both clusters? Even if we provide a single KDC for both of the clusters, since the principle for same services in different clusters would be different, it should not be permitted for a single user to run a single job on data from both clusters at the same time. Because the required service ticket for accessing Hive on cluster 1 would be different that the hive on cluster 2. What about Ranger-KMS? Is there any manual way to integrate both Ranger-KMS instances?
... View more
02-19-2017
02:04 AM
What about RCP encryption? What would be the added value of enabling RPC encryption when we have TDE?
... View more
02-18-2017
02:19 PM
1 Kudo
Hi, I was wondering why we need to have encryption in transit while TDE has been provided already? As far as I understand, TDE is an end-to-end encryption mechanism so that HDFS data will be encrypted/decrypted at the client side. Therefore, data will be encrypted while transferring the wire from client to server and server to client. In this case, what would be the added value of using encryption in-transit separately? Regards.
... View more
02-18-2017
08:06 AM
1- An authorised user on Ranger 1 needs to have access to Ranger 2 cluster (Without TDE) 2- An authorised user on Ranger 1 needs to have access to corresponding user encryption zone at Ranger 2 (with TDE)
... View more