Member since
02-09-2017
42
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2517 | 06-13-2017 02:52 AM | |
| 6363 | 02-10-2017 11:33 AM |
11-15-2022
09:13 PM
Hello @balance002 Kindly let us know if the above Post dated 2022-11-08 assisted your team to identify the affected Hfile(s) & take necessary remediation to ensure Table's access is restored for Users. Regards, Smarak
... View more
02-04-2020
04:02 PM
@asmarz,
One of our members posted a reply on how to add users in the thread you posted a similar question to later the same day.
As this is an older thread which was previously marked 'Solved', you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity for you to provide details specific to your environment about what you did in an attempt to add the relevant user accounts that could aid others in providing a more relevant, accurate answer to your question.
... View more
03-29-2019
04:35 AM
In /etc/yum.repos.d, remove all .repo files pointing to the Internet and copy only .repo files from other servers which are already using your local repo. For HDP nodes, initially you need only 2 .repo files, one for the OS, and ambari.repo. When Ambari adds a new node to the cluster it will copy there HDP.repo and HDP-UTILS.repo. Also, have you set your repository URLs in Ambari-> Admin-> Stack and versions-> Versions -> Manage Versions -> [click on your current version] ?
... View more
03-14-2019
01:03 PM
If you have HA in your cluster, make sure the setting of webhdfs_url in Hue is pointing to the ACTIVE NameNode. You can find webhdfs_url in: (xxx is some digitals for you current run of Hue) /run/cloudera-scm-agent/process/xxx-hue-HUE_SERVER/hue.ini And you can find the active NameNode in: (xx is some digitals for you cluster) http://your.cloudera.manager.host:7180/cmf/services/xx/instances Or, maybe you've missed something after the HA, check below https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_hdfs_ha_cdh_components_config.html#concept_rj1_hsq_bp
... View more
06-13-2017
02:52 AM
Hi,@Sergey Soldatov,This question is solved! Thank you very much!
... View more
02-13-2017
08:10 PM
Thank you, you are right, when I create a kadmin user on each linux machine, you can successfully submit the task!
... View more
02-10-2017
11:33 AM
My problem is solved！ Thank you, spark submitted tasks, I do not need certification of the cluster, but the need for the implementation of the spark machine! Therefore, I need to remove these two parameters: spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-client \
--executor-memory 1G \
--num-executors 1 \
--num-executors 2 \
--driver-memory 1g \
--executor-cores 1 \
#--principal kadmin/admin@NGAA.COM \
#--keytab /home/test/sparktest/princpal/sparkjob.keytab \
/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 12
Submit again to succeed! 17/02/10 16:18:33 INFO spark.SparkContext: Running Spark version 1.6.0
17/02/10 16:18:34 INFO spark.SecurityManager: Changing view acls to: root,hdfs
17/02/10 16:18:34 INFO spark.SecurityManager: Changing modify acls to: root,hdfs
17/02/10 16:18:34 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, hdfs); users with modify permissions: Set(root, hdfs)
17/02/10 16:18:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53300.
17/02/10 16:18:35 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/02/10 16:18:35 INFO Remoting: Starting remoting
17/02/10 16:18:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.10.100.53:59243]
17/02/10 16:18:35 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.10.100.53:59243]
17/02/10 16:18:35 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 59243.
17/02/10 16:18:35 INFO spark.SparkEnv: Registering MapOutputTracker
17/02/10 16:18:35 INFO spark.SparkEnv: Registering BlockManagerMaster
17/02/10 16:18:35 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-1521d8d2-ce43-4c6e-8068-af08ed953b77
17/02/10 16:18:35 INFO storage.MemoryStore: MemoryStore started with capacity 530.3 MB
17/02/10 16:18:35 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/02/10 16:18:36 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/02/10 16:18:36 INFO ui.SparkUI: Started SparkUI at http://10.10.100.53:4040
17/02/10 16:18:36 INFO spark.SparkContext: Added JAR file:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar at spark://10.10.100.53:53300/jars/spark-examples.jar with timestamp 1486714716370
17/02/10 16:18:36 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/10.10.100.51:8032
17/02/10 16:18:37 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers
17/02/10 16:18:37 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/02/10 16:18:37 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/02/10 16:18:37 INFO yarn.Client: Setting up container launch context for our AM
17/02/10 16:18:37 INFO yarn.Client: Setting up the launch environment for our AM container
17/02/10 16:18:37 INFO yarn.Client: Preparing resources for our AM container
17/02/10 16:18:38 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://hadoop2:8020/user/hdfs/.sparkStaging/application_1486705141135_0008
17/02/10 16:18:38 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 52 for hdfs on 10.10.100.52:8020
17/02/10 16:18:39 INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
17/02/10 16:18:39 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/02/10 16:18:39 INFO hive.metastore: Connected to metastore.
17/02/10 16:18:39 INFO hive.metastore: Closed a connection to metastore, current connections: 0
17/02/10 16:18:39 INFO yarn.Client: Uploading resource file:/tmp/spark-f6434659-beb9-437c-b233-8667c48702b9/__spark_conf__2828602694267011736.zip -> hdfs://hadoop2:8020/user/hdfs/.sparkStaging/application_1486705141135_0008/__spark_conf__2828602694267011736.zip
17/02/10 16:18:40 INFO spark.SecurityManager: Changing view acls to: root,hdfs
17/02/10 16:18:40 INFO spark.SecurityManager: Changing modify acls to: root,hdfs
17/02/10 16:18:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, hdfs); users with modify permissions: Set(root, hdfs)
17/02/10 16:18:40 INFO yarn.Client: Submitting application 8 to ResourceManager
17/02/10 16:18:40 INFO impl.YarnClientImpl: Submitted application application_1486705141135_0008
17/02/10 16:18:41 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:41 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1486714720230
final status: UNDEFINED
tracking URL: http://hadoop1:8088/proxy/application_1486705141135_0008/
user: hdfs
17/02/10 16:18:42 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:43 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:44 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:45 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:46 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:47 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:48 INFO yarn.Client: Application report for application_1486705141135_0008 (state: ACCEPTED)
17/02/10 16:18:48 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/02/10 16:18:48 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hadoop1, PROXY_URI_BASES -> http://hadoop1:8088/proxy/application_1486705141135_0008), /proxy/application_1486705141135_0008
17/02/10 16:18:48 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/02/10 16:18:49 INFO yarn.Client: Application report for application_1486705141135_0008 (state: RUNNING)
17/02/10 16:18:49 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: 10.10.100.53
ApplicationMaster RPC port: 0
queue: root.users.hdfs
start time: 1486714720230
final status: UNDEFINED
tracking URL: http://hadoop1:8088/proxy/application_1486705141135_0008/
user: hdfs
17/02/10 16:18:49 INFO cluster.YarnClientSchedulerBackend: Application application_1486705141135_0008 has started running.
17/02/10 16:18:49 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 47910.
17/02/10 16:18:49 INFO netty.NettyBlockTransferService: Server created on 47910
17/02/10 16:18:49 INFO storage.BlockManager: external shuffle service port = 7337
17/02/10 16:18:49 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/02/10 16:18:49 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.10.100.53:47910 with 530.3 MB RAM, BlockManagerId(driver, 10.10.100.53, 47910)
17/02/10 16:18:49 INFO storage.BlockManagerMaster: Registered BlockManager
17/02/10 16:18:49 INFO scheduler.EventLoggingListener: Logging events to hdfs://hadoop2:8020/user/spark/applicationHistory/application_1486705141135_0008
17/02/10 16:18:49 WARN spark.SparkContext: Dynamic Allocation and num executors both set, thus dynamic allocation disabled.
17/02/10 16:18:58 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (hadoop4:49020) with ID 1
17/02/10 16:18:58 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop4:48173 with 530.3 MB RAM, BlockManagerId(1, hadoop4, 48173)
17/02/10 16:19:01 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (hadoop2:52352) with ID 2
17/02/10 16:19:01 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/02/10 16:19:01 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop2:39922 with 530.3 MB RAM, BlockManagerId(2, hadoop2, 39922)
17/02/10 16:19:01 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:36
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 12 output partitions
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:36)
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Missing parents: List()
17/02/10 16:19:01 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents
17/02/10 16:19:01 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1904.0 B, free 1904.0 B)
17/02/10 16:19:02 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1202.0 B, free 3.0 KB)
17/02/10 16:19:02 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.10.100.53:47910 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:02 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
17/02/10 16:19:02 INFO scheduler.DAGScheduler: Submitting 12 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32)
17/02/10 16:19:02 INFO cluster.YarnScheduler: Adding task set 0.0 with 12 tasks
17/02/10 16:19:02 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, hadoop2, partition 0,PROCESS_LOCAL, 2034 bytes)
17/02/10 16:19:02 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, hadoop4, partition 1,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:03 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop4:48173 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:04 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop2:39922 (size: 1202.0 B, free: 530.3 MB)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, hadoop4, partition 2,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 2231 ms on hadoop4 (1/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, hadoop2, partition 3,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, hadoop4, partition 4,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2369 ms on hadoop2 (2/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 127 ms on hadoop4 (3/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, hadoop2, partition 5,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 108 ms on hadoop2 (4/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, hadoop4, partition 6,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 114 ms on hadoop4 (5/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, hadoop2, partition 7,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 85 ms on hadoop2 (6/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, hadoop4, partition 8,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 103 ms on hadoop4 (7/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, hadoop2, partition 9,PROCESS_LOCAL, 2036 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 89 ms on hadoop2 (8/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, hadoop4, partition 10,PROCESS_LOCAL, 2039 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, hadoop2, partition 11,PROCESS_LOCAL, 2040 bytes)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 109 ms on hadoop4 (9/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 83 ms on hadoop2 (10/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 90 ms on hadoop4 (11/12)
17/02/10 16:19:04 INFO scheduler.TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 77 ms on hadoop2 (12/12)
17/02/10 16:19:04 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 2.695 s
17/02/10 16:19:04 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/02/10 16:19:04 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 3.293783 s
Pi is roughly 3.1438333333333333
17/02/10 16:19:05 INFO ui.SparkUI: Stopped Spark web UI at http://10.10.100.53:4040
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down
17/02/10 16:19:05 INFO cluster.YarnClientSchedulerBackend: Stopped
17/02/10 16:19:05 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/02/10 16:19:05 INFO storage.MemoryStore: MemoryStore cleared
17/02/10 16:19:05 INFO storage.BlockManager: BlockManager stopped
17/02/10 16:19:05 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/02/10 16:19:05 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/02/10 16:19:05 INFO spark.SparkContext: Successfully stopped SparkContext
17/02/10 16:19:05 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/02/10 16:19:05 INFO util.ShutdownHookManager: Shutdown hook called
17/02/10 16:19:05 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f6434659-beb9-437c-b233-8667c48702b9
17/02/10 16:19:05 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
... View more
02-09-2017
12:28 AM
zhoudshu, Thank you for you. Your answer perfectly solved this problem!
... View more
11-17-2017
01:37 PM
@Robert Levas Hello, I'm facing a similar issue with Hue. I've detailedly explained my question here: https://community.hortonworks.com/questions/147826/failed-to-access-filesystem-root-through-hue-ui.html. The output of $klist with hue user is mentioned in klist.png. Could you please help me to resolve this issue? I'll be really grateful.
... View more