Member since
11-01-2017
35
Posts
0
Kudos Received
0
Solutions
10-22-2018
12:53 PM
I am a newbie to Hadoop, I installed and configured a three-node hadoop cluster:
Master node : 8GB RAM Slave node1 : 8GB RAM Slave node2 : 4GB RAM My question is it okay if the slave2 has 4GB RAM? is the memory configuration right (for the master and slaves)?
... View more
Labels:
- Labels:
-
Apache Hadoop
10-05-2018
11:07 AM
I have developed a program code that executes an X task in sequential and in MAPREDUCE model. But I see that the running time taken by the mapreduce is bigger than the sequential one( It must be the opposite!!). I'm running mapreduce in a single node, so at least I should get the same running time?
... View more
Labels:
- Labels:
-
Apache Hadoop
10-01-2018
10:49 AM
I have executed a program in Hadoop MapReduce and I Wait to study the performance of my program with different input sizes, and I need the running time taken for each input? the results are in the attached file Thank youmapreduce.txt
... View more
Labels:
- Labels:
-
Apache Hadoop
12-15-2017
08:02 PM
I have this alert concerning the ambari metrics: Connection failed: [Errno 111] Connection refused to datanode1.xxxx:6188
... View more
12-15-2017
08:00 PM
I removed the ambari metrics using Ambari UI, and I add that service but I still get the same error !!
... View more
12-15-2017
10:02 AM
I'm using ubuntu 14, and the name of the package ambari-metrics-collector not known by ubuntu (Impossible to find ambari-metrics-collector package)??
... View more
12-14-2017
07:07 PM
What I have posted is the only message I got: resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf start' returned 255. ? ??? 14 18:19:50 WET 2017 Starting HBase.
starting master, logging to /var/log/ambari-metrics-collector/hbase-ams-master-datanode1.henceforth.out
Verifying ambari-metrics-collector process status...
ERROR: ambari-metrics-collector start failed. For more details, see /var/log/ambari-metrics-collector/ambari-metrics-collector.out:
====================
Erreur : impossible de trouver ou charger la classe principale org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
====================
Collector out at: /var/log/ambari-metrics-collector/ambari-metrics-collector.out
... View more
12-14-2017
01:12 PM
I got this error: ERROR: ambari-metrics-collector start failed. For more details, see /var/log/ambari-metrics-collector/ambari-metrics-collector.out:
====================
Erreur : impossible de trouver ou charger la classe principale org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
====================
... View more
Labels:
- Labels:
-
Apache Ambari
11-23-2017
02:21 PM
Output of ls -lrth /var/log/ | grep zookeeper is drwxr-xr-x 2 zookeeper hadoop 4.0K نوف 21 16:18 zookeeper Still nothing in /var/log/zookeeper
... View more
11-23-2017
02:07 PM
I dont what i have missed but the dir : /var/log/zookeeper and /usr/share/zookeeper a are empty!!
... View more
11-23-2017
02:01 PM
Output of "zookeeper-env.sh" : export JAVA_HOME=/usr/jdk64/jdk1.8.0_112 export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server export ZOO_LOG_DIR=/var/log/zookeeper export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid export SERVER_JVMFLAGS=-Xmx1024m export JAVA=$JAVA_HOME/bin/java export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
... View more
11-23-2017
01:21 PM
There is no log file created in ZooKeeper Log Dir :/var/log/zookeeper
... View more
11-23-2017
12:30 PM
I did what you said what the same problem zookeeper keeps to stops every 10 s. This the content of log4j.properties: # ZooKeeper Logging Configuration # DEFAULT: console appender only log4j.rootLogger=INFO, CONSOLE, ROLLINGFILE zookeeper.log.dir=. zookeeper.log.file=zookeeper.log log4j.rootLogger=${zookeeper.root.logger} # Example with rolling log file #log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE # Example with rolling log file and tracing #log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE # # Log INFO level and above messages to the console # log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender log4j.appender.CONSOLE.Threshold=INFO log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n # # Add ROLLINGFILE to rootLogger to get log file output # Log DEBUG level and above messages to a log file log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender log4j.appender.ROLLINGFILE.Threshold=DEBUG log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file} # Max log file size of 10MB log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_max_backup_size}}MB # uncomment the next line to limit number of backup files log4j.appender.ROLLINGFILE.MaxBackupIndex=10
... View more
11-23-2017
11:48 AM
I have the same problem; and I do not have the file: zookeeper.log!!
... View more
11-23-2017
11:05 AM
I get this error while trying to start zookeeper: /usr/hdfp/2.6.3-235/zookeeper/bin/zkServer.sh start /zkServer.sh: Permission denied
... View more
11-23-2017
10:08 AM
No the symbolic link is not broken zookeeper-server -> /usr/hdp/2.6.3.0-235/zookeeper
... View more
11-21-2017
02:23 PM
I get this error when I try to start zookeeper server on my 3 nodes resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdp/current/zookeeper-server/conf/zookeeper-env.sh ; env ZOOCFGDIR=/usr/hdp/current/zookeeper-server/conf ZOOCFG=zoo.cfg /usr/hdp/current/zookeeper-server/bin/zkServer.sh start' returned 127. env: /usr/hdp/current/zookeeper-server/bin/zkServer.sh: No such file or directory
... View more
- Tags:
- Hadoop Core
- Zookeeper
Labels:
11-21-2017
12:56 PM
My Internet connection uses a proxy server how can I use the above command with proxy address ?? this is what i got: HTTP/1.1 503 Service Unavailable Server: squid/3.3.8 Mime-Version: 1.0 Date: Tue, 21 Nov 2017 10:20:20 GMT Content-Type: text/html Content-Length: 3604 X-Squid-Error: ERR_CONNECT_FAIL 110 Vary: Accept-Language Content-Language: en X-Cache: MISS from cacheP1 X-Cache-Lookup: MISS from cacheP1:3128 X-Cache: MISS from cache X-Cache-Lookup: MISS from cache:3128 Via: 1.1 cacheP1 (squid/3.3.8), 1.1 cache (squid/3.5.20) Connection: keep-alive <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8"> <title>ERROR: The requested URL could not be retrieved</title> ....
... View more
11-21-2017
12:53 PM
this is what i tried: curl -k -u root:****** -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://10.23.**.**:8080/api/v1/clusters/ensias/hosts/masternode.ensias/host_components/slider
... View more
11-21-2017
11:11 AM
I'm about to install service in each node and I get this error while installing Slider Client in two nodes: resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/slider-client/lib/slider.tar.gz'] failed, parent directory /usr/hdp/current/slider-client/lib doesn't exit
... View more
- Tags:
- Hadoop Core
- slider
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
11-21-2017
09:48 AM
My problem is I already install a HDP cluster but i find some problem, then I tried to reinstall everything by removing the service files... Now I m enable to install it again, It always return the same problem on Zookeeper-server, and It should be a symlink but when I check /usr/hdp/current/zookeeper-server is a directory !!
... View more
11-21-2017
09:12 AM
I removed the directory "/usr/hdp/current/zookeeper-server, then I created the symlink as following: ln -s /usr/hdp/current/zookeeper-server ->/usr/hdp/2.6.3.0-235/zookeeper But when i try ls -l the symlink is not there?? and always return failed to install zookeeper server with the same error I mentioned before!!
... View more
11-20-2017
02:21 PM
I arrived to step 9 to set up a HDP cluster using Ambari. and the services failed to install for example zookeeper returns this error: resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/bin/hdp-select set zookeeper-server 2.6.3.0-235' returned 1. symlink target /usr/hdp/current/zookeeper-server for zookeeper already exists and it is not a symlink.
... View more
Labels:
11-09-2017
11:16 AM
I arrived to step9 to create a cluster and the chosen services are not enable to be installed in the host. I get this error: The 'hive-webhcat' component did not advertise a version. This may indicate a problem with the component packaging. However, the stack-select tool was able to report a single version installed (2.6.3.0-235). This is the version that will be reported.
Python script has been killed due to timeout after waiting 1800 secs
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
11-04-2017
01:14 PM
I arrived in the step where i have to download the Ambari Repository (I'm using Ubuntu as OS) https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambariinstallation/content/download_the_ambari_repo_ubuntu14.html and when i try this cmd: apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD I get this message: gpgkeys: key B9733A7A07513CAD can't be retrieved gpg: no valid OpendPGP data found?
... View more
Labels:
- Labels:
-
Apache Ambari
11-03-2017
05:26 PM
I tried what you said nothing I still get FAILED to Confirm hosts: this is the error i get: ==========================
Creating target directory...
==========================
Command start time 2017-11-03 20:03:09
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:11
==========================
Copying ambari sudo script...
==========================
Command start time 2017-11-03 20:03:11
scp /var/lib/ambari-server/ambari-sudo.sh
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:14
==========================
Copying common functions script...
==========================
Command start time 2017-11-03 20:03:14
scp /usr/lib/python2.6/site-packages/ambari_commons
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:16
==========================
Copying create-python-wrap script...
==========================
Command start time 2017-11-03 20:03:16
scp /var/lib/ambari-server/create-python-wrap.sh
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:18
==========================
Copying OS type check script...
==========================
Command start time 2017-11-03 20:03:18
scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:20
==========================
Running create-python-wrap script...
==========================
Command start time 2017-11-03 20:03:20
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:22
==========================
Running OS type check...
==========================
Command start time 2017-11-03 20:03:22
Cluster primary/cluster OS family is ubuntu14 and local/current OS family is ubuntu14
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:24
==========================
Checking 'sudo' package on remote host...
==========================
Command start time 2017-11-03 20:03:24
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:26
==========================
Copying repo file to 'tmp' folder...
==========================
Command start time 2017-11-03 20:03:26
scp /etc/apt/sources.list.d/ambari.list
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:28
==========================
Moving file to repo dir...
==========================
Command start time 2017-11-03 20:03:28
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:30
==========================
Changing permissions for ambari.repo...
==========================
Command start time 2017-11-03 20:03:30
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:32
==========================
Update apt cache of repository...
==========================
Command start time 2017-11-03 20:03:32
0% [En cours]
Atteint http://public-repo-1.hortonworks.com Ambari InRelease
0% [Attente des fichiers d'en-tête]
0% [InRelease gpgv 7 394 B] [Attente des fichiers d'en-tête] [Attente des fich
23% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP InRelease
23% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-tête
23% [InRelease gpgv 7 391 B] [Connexion à 10.23.201.11 (10.23.201.11)] [Attent
22% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP-UTILS InRelease
22% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-tête
22% [InRelease gpgv 7 439 B] [Connexion à 10.23.201.11 (10.23.201.11)] [Attent
22% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com Ambari/main amd64 Packages
22% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-tête
Atteint http://public-repo-1.hortonworks.com Ambari/main i386 Packages
22% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-tête
22% [Packages 4 187 B] [Attente des fichiers d'en-tête] [Attente des fichiers
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty InRelease
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
100% [InRelease gpgv 15,5 kB] [Attente des fichiers d'en-tête] [Attente des fi
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty InRelease
100% [Attente des fichiers d'en-tête]
100% [InRelease gpgv 15,5 kB] [Attente des fichiers d'en-tête]
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP/main amd64 Packages
100% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-têt
100% [Packages 160 kB] [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP/main i386 Packages
100% [Attente des fichiers d'en-tête]
100% [Packages 158 kB] [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty/main amd64 Packages
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
100% [Packages 16,0 kB] [Attente des fichiers d'en-tête] [Attente des fichiers
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP-UTILS/main amd64 Packages
100% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-têt
100% [Packages 33,6 kB] [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://public-repo-1.hortonworks.com HDP-UTILS/main i386 Packages
100% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-têt
100% [Packages 9 651 B] [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des
100% [Connexion à 10.23.201.11 (10.23.201.11)] [Attente des fichiers d'en-têt
Atteint http://ppa.launchpad.net trusty/main i386 Packages
100% [Attente des fichiers d'en-tête]
100% [Packages 16,0 kB] [Attente des fichiers d'en-tête] [Attente des fichiers
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty/main Translation-en
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
100% [Translation-en 8 386 B] [Attente des fichiers d'en-tête] [Attente des fi
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty/main amd64 Packages
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
100% [Packages 19,8 kB] [Attente des fichiers d'en-tête] [Attente des fichiers
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty/main i386 Packages
100% [Attente des fichiers d'en-tête]
100% [Packages 14,6 kB] [Attente des fichiers d'en-tête] [Attente des fichiers
100% [Attente des fichiers d'en-tête] [Attente des fichiers d'en-tête]
Atteint http://ppa.launchpad.net trusty/main Translation-en
100% [Attente des fichiers d'en-tête]
100% [Translation-en 6 192 B] [Attente des fichiers d'en-tête]
100% [Attente des fichiers d'en-tête]
100% [Attente des fichiers d'en-tête]
100% [Attente des fichiers d'en-tête]
Ign http://public-repo-1.hortonworks.com Ambari/main Translation-fr_FR
100% [En cours]
Ign http://public-repo-1.hortonworks.com Ambari/main Translation-fr
100% [En cours]
Ign http://public-repo-1.hortonworks.com Ambari/main Translation-en
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP/main Translation-fr_FR
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP/main Translation-fr
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP/main Translation-en
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP-UTILS/main Translation-fr_FR
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP-UTILS/main Translation-fr
100% [En cours]
Ign http://public-repo-1.hortonworks.com HDP-UTILS/main Translation-en
100% [En cours]
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 0%
Lecture des listes de paquets… 3%
Lecture des listes de paquets… 6%
Lecture des listes de paquets… 6%
Lecture des listes de paquets… 13%
Lecture des listes de paquets… 13%
Lecture des listes de paquets… 14%
Lecture des listes de paquets… 14%
Lecture des listes de paquets… 15%
Lecture des listes de paquets… 15%
Lecture des listes de paquets… 15%
Lecture des listes de paquets… 15%
Lecture des listes de paquets… 16%
Lecture des listes de paquets… 16%
Lecture des listes de paquets… 16%
Lecture des listes de paquets… 16%
Lecture des listes de paquets… 17%
Lecture des listes de paquets… 17%
Lecture des listes de paquets… 18%
Lecture des listes de paquets… 18%
Lecture des listes de paquets… 18%
Lecture des listes de paquets… 18%
Lecture des listes de paquets... Fait
W: Duplicate sources.list entry http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.2.0/ Ambari/main amd64 Packages (/var/lib/apt/lists/public-repo-1.hortonworks.com_ambari_ubuntu14_2.x_updates_2.5.2.0_dists_Ambari_main_binary-amd64_Packages)
W: Duplicate sources.list entry http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.2.0/ Ambari/main i386 Packages (/var/lib/apt/lists/public-repo-1.hortonworks.com_ambari_ubuntu14_2.x_updates_2.5.2.0_dists_Ambari_main_binary-i386_Packages)
W: Vous pouvez lancer « apt-get update » pour corriger ces problèmes.
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:37
==========================
Copying setup script file...
==========================
Command start time 2017-11-03 20:03:37
scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=datanode2.ensias, exitcode=0
Command end time 2017-11-03 20:03:39
==========================
Running setup agent script...
==========================
Command start time 2017-11-03 20:03:39
Host registration aborted. Ambari Agent host cannot reach Ambari Server 'masternode.ensias:8080'. Please check the network connectivity between the Ambari Agent host and the Ambari Server
Connection to datanode2.ensias closed.
SSH command execution finished
host=datanode2.ensias, exitcode=1
Command end time 2017-11-03 20:03:45
ERROR: Bootstrap of host datanode2.ensias fails because previous action finished with non-zero exit code (1)
ERROR MESSAGE: Connection to datanode2.ensias closed.
STDOUT: Host registration aborted. Ambari Agent host cannot reach Ambari Server 'masternode.ensias:8080'. Please check the network connectivity between the Ambari Agent host and the Ambari Server
... View more
11-03-2017
04:41 PM
get this error while confirming the hosts (cluster creation): Command start time 2017-11-03 19:18:20
chmod: cannot access ‘/var/lib/ambari-agent/data’: No such file or directory
... View more
Labels:
- Labels:
-
Apache Ambari