Member since
09-24-2015
527
Posts
136
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2130 | 06-30-2017 03:15 PM | |
3055 | 10-14-2016 10:08 AM | |
8318 | 09-07-2016 06:04 AM | |
10064 | 08-26-2016 11:27 AM | |
1450 | 08-23-2016 02:09 PM |
05-10-2016
02:01 PM
Hi: i cant see the new node {
"href" : "http://xxx:8080/api/v1/hosts",
"items" : [
{
"href" : "http://xxx:8080/api/v1/hosts/a01hop01",
"Hosts" : {
"cluster_name" : "RSI_DE",
"host_name" : "a01hop01"
}
},
{
"href" : "http://xxx:8080/api/v1/hosts/a01hop02",
"Hosts" : {
"cluster_name" : "RSI_DE",
"host_name" : "a01hop02"
}
},
{
"href" : "http://xxx:8080/api/v1/hosts/a01hop03",
"Hosts" : {
"cluster_name" : "RSI_DE",
"host_name" : "a01hop03"
}
}
]
} Also i can do ssh connection correctly the log for the new ambari agent is:
INFO 2016-05-10 15:44:36,284 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,284 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_collector_zookeeper_server_process with UUID 853d41c8-220e-4e49-95c1-3048d0573e7a
INFO 2016-05-10 15:44:36,284 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,284 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_collector_process with UUID d43e1a99-cc8f-4ea0-a06d-06d54c3d94de
INFO 2016-05-10 15:44:36,284 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,284 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_collector_hbase_master_process with UUID 7fbe2a5d-f7e5-4a00-a4bb-8e8bc89bd07e
INFO 2016-05-10 15:44:36,284 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,284 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_collector_hbase_master_cpu with UUID d86eb46e-5073-431a-80da-fb39e0790a20
INFO 2016-05-10 15:44:36,285 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,285 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling zookeeper_server_process with UUID aee44599-813e-4cb0-8479-eb3a75eaabe7
INFO 2016-05-10 15:44:36,285 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,285 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ambari_agent_disk_usage with UUID eb07da21-8c4c-414e-a0c5-ea3297179b1a
INFO 2016-05-10 15:44:36,285 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,285 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_collector_autostart with UUID cf511140-d46d-4be5-83a2-b5dae7a7b051
INFO 2016-05-10 15:44:36,285 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,285 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling journalnode_process with UUID 185ec91b-4be4-4c1a-a053-75f9e20e5b34
INFO 2016-05-10 15:44:36,285 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,286 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling datanode_unmounted_data_dir with UUID f94a35f8-f048-4fa9-bb17-54485a3620a4
INFO 2016-05-10 15:44:36,286 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,286 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling yarn_nodemanager_health with UUID 3752fb53-f1a4-4589-aff2-90ef704b13da
INFO 2016-05-10 15:44:36,286 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,286 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling datanode_process with UUID 412ebaeb-f960-4ff1-ace7-bd6b01aad068
INFO 2016-05-10 15:44:36,286 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,286 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling yarn_nodemanager_webui with UUID 56a16f35-ce74-4ec8-a90b-5555113a33c1
INFO 2016-05-10 15:44:36,286 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,286 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling datanode_webui with UUID 2f15cec8-5d37-435e-bb02-5e4953b40957
INFO 2016-05-10 15:44:36,287 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,287 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling ams_metrics_monitor_process with UUID 8e6b65e3-62bd-4872-9c34-1b16f4662c1a
INFO 2016-05-10 15:44:36,287 scheduler.py:287 - Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO 2016-05-10 15:44:36,287 AlertSchedulerHandler.py:330 - [AlertScheduler] Scheduling datanode_storage with UUID eaf7eee7-f0b9-4c39-bf03-1afcff9e4a60
INFO 2016-05-10 15:44:36,287 AlertSchedulerHandler.py:139 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x2df8050>; currently running: False
INFO 2016-05-10 15:44:38,293 hostname.py:89 - Read public hostname 'a01hop04' using socket.getfqdn()
INFO 2016-05-10 15:44:38,305 logger.py:67 - call['test -w /'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,311 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,311 logger.py:67 - call['test -w /dev'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,317 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,317 logger.py:67 - call['test -w /dev/shm'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,323 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,323 logger.py:67 - call['test -w /run'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,329 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,329 logger.py:67 - call['test -w /sys/fs/cgroup'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,335 logger.py:67 - call returned (1, '')
INFO 2016-05-10 15:44:38,335 logger.py:67 - call['test -w /tmp'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,340 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,340 logger.py:67 - call['test -w /var/log'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,346 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,346 logger.py:67 - call['test -w /usr/hdp'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,351 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,352 logger.py:67 - call['test -w /home'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,357 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,357 logger.py:67 - call['test -w /boot'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,363 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,363 logger.py:67 - call['test -w /rsiiri/syspri2'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,368 logger.py:67 - call returned (1, '')
INFO 2016-05-10 15:44:38,368 logger.py:67 - call['test -w /rsiiri/syspri3'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,374 logger.py:67 - call returned (1, '')
INFO 2016-05-10 15:44:38,374 logger.py:67 - call['test -w /run/user/0'] {'sudo': True, 'timeout': 5}
INFO 2016-05-10 15:44:38,379 logger.py:67 - call returned (0, '')
INFO 2016-05-10 15:44:38,494 Controller.py:145 - Registering with a01hop04 (172.22.3.246) (agent='{"hardwareProfile": {"kernel": "Linux", "domain": "", "physicalprocessorcount": 2, "kernelrelease": "3.10.0-327.el7.x86_64", "uptime_days": "0", "memorytotal": 1884384, "swapfree": "2.00 GB", "memorysize": 1884384, "osfamily": "redhat", "swapsize": "2.00 GB", "processorcount": 2, "netmask": "255.255.240.0", "timezone": "CET", "hardwareisa": "x86_64", "memoryfree": 592136, "operatingsystem": "centos", "kernelmajversion": "3.10", "kernelversion": "3.10.0", "macaddress": "00:50:56:BB:5F:63", "operatingsystemrelease": "7.2.1511", "ipaddress": "172.22.3.246", "hostname": "a01hop04", "uptime_hours": "21", "fqdn": "a01hop04", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "16257780", "used": "36145420", "percent": "69%", "device": "/dev/mapper/centos-root", "mountpoint": "/", "type": "xfs", "size": "52403200"}, {"available": "932032", "used": "0", "percent": "0%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "932032"}, {"available": "942192", "used": "0", "percent": "0%", "device": "tmpfs", "mountpoint": "/dev/shm", "type": "tmpfs", "size": "942192"}, {"available": "933492", "used": "8700", "percent": "1%", "device": "tmpfs", "mountpoint": "/run", "type": "tmpfs", "size": "942192"}, {"available": "26168620", "used": "32980", "percent": "1%", "device": "/dev/mapper/centos-tmp", "mountpoint": "/tmp", "type": "xfs", "size": "26201600"}, {"available": "25656252", "used": "545348", "percent": "3%", "device": "/dev/mapper/centos-var_log", "mountpoint": "/var/log", "type": "xfs", "size": "26201600"}, {"available": "26168672", "used": "32928", "percent": "1%", "device": "/dev/mapper/centos-usr_hdp", "mountpoint": "/usr/hdp", "type": "xfs", "size": "26201600"}, {"available": "73078988", "used": "1333556", "percent": "2%", "device": "/dev/mapper/centos-home", "mountpoint": "/home", "type": "xfs", "size": "74412544"}, {"available": "383204", "used": "125384", "percent": "25%", "device": "/dev/sda1", "mountpoint": "/boot", "type": "xfs", "size": "508588"}, {"available": "188440", "used": "0", "percent": "0%", "device": "tmpfs", "mountpoint": "/run/user/0", "type": "tmpfs", "size": "188440"}], "hardwaremodel": "x86_64", "uptime_seconds": "78040", "interfaces": "eno16780032,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.2.0.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1462887878490, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "umask": "18", "firewallName": "iptables", "stackFoldersAndFiles": [{"type": "directory", "name": "/etc/hadoop"}, {"type": "directory", "name": "/etc/hive"}, {"type": "directory", "name": "/etc/oozie"}, {"type": "directory", "name": "/etc/zookeeper"}, {"type": "directory", "name": "/etc/hive-hcatalog"}, {"type": "directory", "name": "/etc/tez"}, {"type": "directory", "name": "/etc/falcon"}, {"type": "directory", "name": "/etc/hive-webhcat"}, {"type": "directory", "name": "/etc/spark"}, {"type": "directory", "name": "/etc/pig"}, {"type": "directory", "name": "/var/log/hadoop"}, {"type": "directory", "name": "/var/log/hive"}, {"type": "directory", "name": "/var/log/oozie"}, {"type": "directory", "name": "/var/log/zookeeper"}, {"type": "directory", "name": "/var/log/hive-hcatalog"}, {"type": "directory", "name": "/var/log/falcon"}, {"type": "directory", "name": "/var/log/hadoop-yarn"}, {"type": "directory", "name": "/var/log/hadoop-mapreduce"}, {"type": "directory", "name": "/var/log/spark"}, {"type": "directory", "name": "/usr/lib/flume"}, {"type": "directory", "name": "/usr/lib/storm"}, {"type": "directory", "name": "/var/lib/hive"}, {"type": "directory", "name": "/var/lib/oozie"}, {"type": "directory", "name": "/var/lib/hadoop-hdfs"}, {"type": "directory", "name": "/var/lib/hadoop-yarn"}, {"type": "directory", "name": "/var/lib/hadoop-mapreduce"}, {"type": "directory", "name": "/var/lib/spark"}, {"type": "directory", "name": "/hadoop/zookeeper"}, {"type": "directory", "name": "/hadoop/hdfs"}, {"type": "directory", "name": "/hadoop/yarn"}], "existingUsers": [{"status": "Available", "name": "hive", "homeDir": "/home/hive"}, {"status": "Available", "name": "zookeeper", "homeDir": "/home/zookeeper"}, {"status": "Available", "name": "ams", "homeDir": "/home/ams"}, {"status": "Available", "name": "oozie", "homeDir": "/home/oozie"}, {"status": "Available", "name": "ambari-qa", "homeDir": "/home/ambari-qa"}, {"status": "Available", "name": "tez", "homeDir": "/home/tez"}, {"status": "Available", "name": "hdfs", "homeDir": "/home/hdfs"}, {"status": "Available", "name": "yarn", "homeDir": "/home/yarn"}, {"status": "Available", "name": "hcat", "homeDir": "/home/hcat"}, {"status": "Available", "name": "mapred", "homeDir": "/home/mapred"}, {"status": "Available", "name": "falcon", "homeDir": "/var/lib/falcon"}, {"status": "Available", "name": "flume", "homeDir": "/home/flume"}, {"status": "Available", "name": "spark", "homeDir": "/home/spark"}], "firewallRunning": false}, "timestamp": 1462887878406, "hostname": "a01hop04", "responseId": -1, "publicHostname": "a01hop04"}')
INFO 2016-05-10 15:44:38,494 NetUtil.py:60 - Connecting to https://a01hop01:8440/connection_info
INFO 2016-05-10 15:44:38,582 security.py:99 - SSL Connect being called.. connecting to the server
INFO 2016-05-10 15:44:38,667 security.py:60 - SSL connection established. Two-way SSL authentication is turned off on the server.
ERROR 2016-05-10 15:44:38,672 Controller.py:165 - Cannot register host with non compatible agent version, hostname=a01hop04, agentVersion=2.2.0.0, serverVersion=2.2.1.0
INFO 2016-05-10 15:44:38,672 Controller.py:392 - Registration response from a01hop01 was FAILED
ERROR 2016-05-10 15:44:38,673 main.py:315 - Fatal exception occurred:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 312, in <module>
main(heartbeat_stop_callback)
File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 303, in main
ExitHelper.execute_cleanup()
TypeError: unbound method execute_cleanup() must be called with ExitHelper instance as first argument (got nothing instead)
INFO 2016-05-10 15:45:36,347 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:45:36,349 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
INFO 2016-05-10 15:46:36,355 logger.py:67 - Host contains mounts: ['/sys', '/proc', '/dev', '/sys/kernel/security', '/dev/shm', '/dev/pts', '/run', '/sys/fs/cgroup', '/sys/fs/cgroup/systemd', '/sys/fs/pstore', '/sys/fs/cgroup/cpu,cpuacct', '/sys/fs/cgroup/freezer', '/sys/fs/cgroup/cpuset', '/sys/fs/cgroup/blkio', '/sys/fs/cgroup/perf_event', '/sys/fs/cgroup/memory', '/sys/fs/cgroup/hugetlb', '/sys/fs/cgroup/net_cls', '/sys/fs/cgroup/devices', '/sys/kernel/config', '/', '/proc/sys/fs/binfmt_misc', '/dev/hugepages', '/sys/kernel/debug', '/dev/mqueue', '/var/lib/nfs/rpc_pipefs', '/proc/fs/nfsd', '/tmp', '/var/log', '/usr/hdp', '/home', '/boot', '/rsiiri/syspri2', '/rsiiri/syspri3', '/run/user/0', '/proc/sys/fs/binfmt_misc'].
INFO 2016-05-10 15:46:36,357 logger.py:67 - Mount point for directory /hadoop/hdfs/data is /
INFO 2016-05-10 15:46:36,364 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:46:36,367 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
INFO 2016-05-10 15:47:36,354 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:47:36,354 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
INFO 2016-05-10 15:48:36,321 logger.py:67 - Mount point for directory /hadoop/hdfs/data is /
INFO 2016-05-10 15:48:36,341 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:48:36,341 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
WARNING 2016-05-10 15:49:36,322 base_alert.py:140 - [Alert][ams_metrics_collector_hbase_master_cpu] Unable to execute alert. [Alert][ams_metrics_collector_hbase_master_cpu] Unable to extract JSON from JMX response
INFO 2016-05-10 15:49:36,349 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:49:36,351 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
INFO 2016-05-10 15:50:36,330 logger.py:67 - Mount point for directory /hadoop/hdfs/data is /
INFO 2016-05-10 15:50:36,353 logger.py:67 - Pid file /var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid is empty or does not exist
ERROR 2016-05-10 15:50:36,353 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on a01hop03']
This machine is a clone image from the a01hop03, so...maybe i need to remoce some files. ERROR 2016-05-10 15:44:38,672 Controller.py:165 - Cannot register host with non compatible agent version, hostname=a01hop04, agentVersion=2.2.0.0, serverVersion=2.2.1.0
INFO 2016-05-10 15:44:38,672 Controller.py:392 - Registration response from a01hop01 was FAILED
ERROR 2016-05-10 15:44:38,673 main.py:315 - Fatal exception occurred:
Traceback (most recent call last): i will install the new version and ill see. Thanks
... View more
05-10-2016
07:06 AM
hi: Everything is fine, my question is, i need to install the tarball o start the ambari-agent from the host?? [root@a01hop04 .ssh]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@a01hop04 .ssh]# systemctl disable firewalld
[root@a01hop04 .ssh]# systemctl stop firewalld
Failed to stop firewalld.service: Unit firewalld.service not loaded.
[root@a01hop04 .ssh]# systemctl status firewalld
● firewalld.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
... View more
05-09-2016
04:42 PM
Hi: ani logs is there. One question, I need to install somenthing before?? thanks
... View more
05-09-2016
04:32 PM
Hi: I am triying to add new host into the cluster but, long time the website is like that, any suggestions???
... View more
Labels:
- Labels:
-
Apache Ambari
05-05-2016
07:08 PM
Hi: finally its working with this code: Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))
library(SparkR)
#sparkR.stop()
sparkR.stop()
sc <- SparkR::sparkR.init(master = "yarn-client", sparkEnvir = list(spark.driver.memory="4g"))
hiveContext <- sparkRHive.init(sc)
... View more
05-05-2016
05:36 PM
hi: i need to install in on servers the Spark Thrift Server, or just i need to install that on me manager node??? Thanks
... View more
Labels:
- Labels:
-
Apache Spark
05-05-2016
02:37 PM
Hi: I am trying to SparkR but doesnt work well the code is: Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))
library(SparkR)
sc <- SparkR::sparkR.init(master = "yarn-client")
sqlContext <-sparkRSQL.init(sc)
path <-file.path("/RSI/staging/input/log_json/f6327t.json")
info <-read.json(sqlContext, path)
printSchema(info)
and the log is: > sc <- SparkR::sparkR.init(master = "yarn-client")
Launching java with spark-submit command /usr/hdp/current/spark-client//bin/spark-submit sparkr-shell /tmp/RtmpxnCWXx/backend_port502d157a15ac
16/05/05 16:33:22 INFO SparkContext: Running Spark version 1.6.0
16/05/05 16:33:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/05/05 16:33:23 INFO SecurityManager: Changing view acls to: bigotes
16/05/05 16:33:23 INFO SecurityManager: Changing modify acls to: bigotes
16/05/05 16:33:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bigotes); users with modify permissions: Set(bigotes)
16/05/05 16:33:23 INFO Utils: Successfully started service 'sparkDriver' on port 39914.
16/05/05 16:33:23 INFO Slf4jLogger: Slf4jLogger started
16/05/05 16:33:23 INFO Remoting: Starting remoting
16/05/05 16:33:24 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.1.246.19:55278]
16/05/05 16:33:24 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 55278.
16/05/05 16:33:24 INFO SparkEnv: Registering MapOutputTracker
16/05/05 16:33:24 INFO SparkEnv: Registering BlockManagerMaster
16/05/05 16:33:24 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-fc4a72de-f470-4c3c-9692-bcf941a4b674
16/05/05 16:33:24 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
16/05/05 16:33:24 INFO SparkEnv: Registering OutputCommitCoordinator
16/05/05 16:33:24 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/05 16:33:24 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/05/05 16:33:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/05/05 16:33:24 INFO SparkUI: Started SparkUI at http://10.1.246.19:4040
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
16/05/05 16:33:24 INFO TimelineClientImpl: Timeline service address: http://lnxbig06.cajarural.gcr:8188/ws/v1/timeline/
16/05/05 16:33:25 INFO RMProxy: Connecting to ResourceManager at lnxbig05.cajarural.gcr/10.1.246.19:8050
16/05/05 16:33:25 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
16/05/05 16:33:25 INFO Client: Requesting a new application from cluster with 5 NodeManagers
16/05/05 16:33:25 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (40192 MB per container)
16/05/05 16:33:25 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
16/05/05 16:33:25 INFO Client: Setting up container launch context for our AM
16/05/05 16:33:25 INFO Client: Setting up the launch environment for our AM container
16/05/05 16:33:25 INFO Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://lnxbig05.cajarural.gcr:8020/hdp/apps/2.4.0.0-169/spark/spark-hdp-assembly.jar
16/05/05 16:33:25 INFO Client: Preparing resources for our AM container
16/05/05 16:33:25 INFO Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://lnxbig05.cajarural.gcr:8020/hdp/apps/2.4.0.0-169/spark/spark-hdp-assembly.jar
16/05/05 16:33:25 INFO Client: Source and destination file systems are the same. Not copying hdfs://lnxbig05.cajarural.gcr:8020/hdp/apps/2.4.0.0-169/spark/spark-hdp-assembly.jar
16/05/05 16:33:25 INFO Client: Uploading resource file:/tmp/spark-7c7224cd-1fa8-43d6-b049-a85ce21f18e7/__spark_conf__5347166147727015442.zip -> hdfs://lnxbig05.cajarural.gcr:8020/user/bigotes/.sparkStaging/application_1461739406783_0151/__spark_conf__5347166147727015442.zip
16/05/05 16:33:26 INFO SecurityManager: Changing view acls to: bigotes
16/05/05 16:33:26 INFO SecurityManager: Changing modify acls to: bigotes
16/05/05 16:33:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bigotes); users with modify permissions: Set(bigotes)
16/05/05 16:33:26 INFO Client: Submitting application 151 to ResourceManager
16/05/05 16:33:26 INFO YarnClientImpl: Submitted application application_1461739406783_0151
16/05/05 16:33:26 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1461739406783_0151 and attemptId None
16/05/05 16:33:27 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:27 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1462458806216
final status: UNDEFINED
tracking URL: http://lnxbig05.cajarural.gcr:8088/proxy/application_1461739406783_0151/
user: bigotes
16/05/05 16:33:28 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:29 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:30 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:31 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:32 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:33 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:34 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:35 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:36 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:37 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:38 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:39 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:40 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED)
16/05/05 16:33:41 INFO Client: Application report for application_1461739406783_0151 (state: ACCEPTED) Is correct my code??? Thanks
... View more
Labels:
- Labels:
-
Apache Spark
05-04-2016
09:59 AM
Hi: finally i create a unique id like this: d = FOREACH c GENERATE
UniqueID() as id,
(chararray) group.$3 as canal,
(chararray) group.$0 as codtf,
(chararray) group.$2 as fechaoprcnf,
(int)COUNT (b) as frecuencia,
(chararray) group.$1 as codnrbeenf;
STORE d INTO 'hbase://canal_partitioned_v2' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage ('id,fijo:canal,fijo:codtf,fijo:fechaoprcnf,fijo:frecuencia,fijo:codnrbeenf');
and in the hbase look like this, i think i ROW -ID is not useful hbase(main):062:0> scan 'canal_partitioned_v2'
ROW COLUMN+CELL
0-0 column=fijo:canal, timestamp=1462355983610, value=BDPPM1KK
0-0 column=fijo:codtf, timestamp=1462355983610, value=2016-03-29
0-0 column=fijo:fechaoprcnf, timestamp=1462355983610, value=1
0-0 column=fijo:frecuencia, timestamp=1462355983610, value=3067
0-0 column=fijo:id, timestamp=1462355983610, value=03
0-1 column=fijo:canal, timestamp=1462355983615, value=BDPPM1KK
0-1 column=fijo:codtf, timestamp=1462355983615, value=2016-03-29
0-1 column=fijo:fechaoprcnf, timestamp=1462355983615, value=1
0-1 column=fijo:frecuencia, timestamp=1462355983615, value=3191
0-1 column=fijo:id, timestamp=1462355983615, value=03
0-2 column=fijo:canal, timestamp=1462355983615, value=BDPPM1RG
0-2 column=fijo:codtf, timestamp=1462355983615, value=2016-03-29
0-2 column=fijo:fechaoprcnf, timestamp=1462355983615, value=1
0-2 column=fijo:frecuencia, timestamp=1462355983615, value=3059
0-2 column=fijo:id, timestamp=1462355983615, value=03
0-3 column=fijo:canal, timestamp=1462355983616, value=DVI51OOU
0-3 column=fijo:codtf, timestamp=1462355983616, value=2016-03-29
0-3 column=fijo:fechaoprcnf, timestamp=1462355983616, value=2
0-3 column=fijo:frecuencia, timestamp=1462355983616, value=1554
my IMPORTANT question is, for agretations or frecuencias o wordcloud, i think is not goo to use HBASE, right???? thanks
... View more
05-04-2016
05:41 AM
Hi: Thanks for it, so, the row key need to be unique and not sequential... The problem is, I have a good row key, but is not unique... Please any default row key that incan apoly???? Thanks
... View more