Member since
08-31-2017
30
Posts
0
Kudos Received
0
Solutions
12-23-2018
06:34 AM
I have a scenario where I need to read files from windows shared path using spark and scala. I tried with below but could not able to find the files:object ExternalFiles {
def main(args: Array[String]){
val conf = new SparkConf().setMaster("local").setAppName("External Files")
val sc = new SparkContext(conf)
val files = sc.textFile("\\\\sharedNetwork\\External Data\\testData.txt")
files.foreach(println)
}
} I tried using sc.textFile("file://sharedNetwork/External Data/testData.txt") but it shows below error in both cases: 18/12/23 11:57:57 WARN : Your hostname, name-21 resolves to a loopback/non-reachable address: 10.xx.xx.xxx, but we couldn't find any external IP address!
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://sharedNetwork/External Data/testData.txt
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) Can someone suggest me to resolve this solution to read files from shared drive using Spark and Scala code. Pls: suggest me on how to download files from NTFS windows shared path to linux machine through putty. Thanks, Chaitanya
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
01-06-2018
08:10 AM
below is my scenario: Object Sample {
def main(args:Array[String]){
val fileName = "SampleFile.txt"
val conf = new SparkConf().setMaster("local").setAppName("LoadToHivePart")
conf.set("spark.ui.port","4041")
val sc=new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val hc = new org.apache.spark.sql.hive.HiveContext(sc)
hc.setConf("hive.metastore.uris","thrift://127.0.0.1:9083")
test(hc,fileName)
sc.stop()
}
def test(hc:String, fileName: String){
//code.....
}
}
As per above code I am unable to pass hiveContext variable "hc" from main to another function. It is showing error. Kindly help me on the same. Thanks.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache Spark
01-05-2018
03:49 AM
My scenario is to check the fileName and then if the file contains specific word then I need to pick that file for processing. Eg: in mydirectory I have two filenames: file1: sample1.txt_processed file2: sample2.txt Now I need to check the file names with and without "_processed" keyword and pick only files without "_processed" text in file name. Can any one help me on this scenario.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
12-16-2017
05:22 PM
Hi Geoffrey Shelton Okot, These are the steps I followed while installing ambari server: 1. created one AWS EC2 instance and connected to that instance using .pem file and public DNS name from my local laptop running with ubuntu os. 2. The ambari server, mysql is running fine on EC2 instance. 3. Now I am trying to connect to ambari web from my local laptop ubuntu with ec2 publi domain name as shown below: http://ec2-54-218-134-113.us-west-2.compute.amazonaws.com:8080 but it is showing as "This site cant be reached. 4. I can able to ping to ec2 instance public ip: # ping 54-218-134-113 is working fine. but # ping ec2-54-218-134-113.us-west-2.compute.amazonaws.com is not working. so trying with http:54-218-134-113:8080 also but no use. also I have added below in my local laptop /etc/hosts file: 172.31.24.170 ec2-54-218-134-113.us-west-2.compute.amazonaws.com myambari also changed my local laptop hostnme to ec2-54-218-134-113.us-west-2.compute.amazonaws.com as shown below: chaithu@localhost:~$ hostname
ec2-54-218-134-113.us-west-2.compute.amazonaws.com chaithu@localhost:~$ hostname -f
ec2-54-218-134-113.us-west-2.compute.amazonaws.com Is any thing more I need to do that I missed. please suggest me onthe same. Thanks.
... View more
12-16-2017
01:58 PM
I tried using public DNS but the login page of ambari web is not accessable. I used as below: http://ec2-54-245-44-216.us-west-2.compute.amazonaws.com:8080/ also attaching the Inbound and Outbound rules defined in security groups. If there are any changes need to be done pls help me on the same.
... View more
12-16-2017
09:56 AM
@Geoffrey Shelton Okot Thanks for replying to me, Now the ambari-server is running fine as shown below: ubuntu@ip-172-31-24-170:~$ sudo ambari-server status
Using python /usr/bin/python
Ambari-server status
Ambari Server running
Found Ambari Server PID: 1389 at: /var/run/ambari-server/ambari-server.pid But I could not able to point web browser using any of the below: http://localhost:8080 http://ip-172-31-24-170.us-west-2.compute.internal:8080 http://ip-172-31-24-170:8080 how can I open ambari web please guide me onthe same. ubuntu@ip-172-31-24-170:~$ cat /etc/hostname
ip-172-31-24-170
ubuntu@ip-172-31-24-170:~$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu@ip-172-31-24-170:~$ hostname
ip-172-31-24-170.us-west-2.compute.internal
ubuntu@ip-172-31-24-170:~$ hostname -f
ip-172-31-24-170.us-west-2.compute.internal ubuntu@ip-172-31-24-170:~$ ifconfig
eth0 Link encap:Ethernet HWaddr 02:37:fe:68:1c:18
inet addr:172.31.24.170 Bcast:172.31.31.255 Mask:255.255.240.0
inet6 addr: fe80::37:feff:fe68:1c18/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:1156 errors:0 dropped:0 overruns:0 frame:0
TX packets:1036 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:100086 (100.0 KB) TX bytes:123808 (123.8 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:430 errors:0 dropped:0 overruns:0 frame:0
TX packets:430 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:73193 (73.1 KB) TX bytes:73193 (73.1 KB) Could you pls guide me to open ambari web, do I need to configure any thing here? Thanks in advance.
... View more
12-16-2017
06:56 AM
Trying to install and start ambari-server on EC2 ubuntu instance. Below is the error I am facing: ubuntu@ec2-34-209-250-154:~$ sudo ambari-server
start Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
Error in /var/log/ambari-server/ambari-server.log:
16 Dec 2017 05:53:27,745 INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.ViewInstanceEventCreator
16 Dec 2017 05:53:27,746 INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.ViewPrivilegeEventCreator
16 Dec 2017 05:53:29,631 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:981)at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:339)at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2254)at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2286)at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2085)at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) ubuntu@ec2-34-209-250-154:~$ hostname
ec2-34-209-250-154.us-west-2.compute.amazonaws.com
ubuntu@ec2-34-209-250-154:~$ hostname -f
ec2-34-209-250-154.us-west-2.compute.amazonaws.com For MYSQL port it is showing as below: ubuntu@ec2-34-209-250-154:~$ netstat -tnlpa | grep 3306
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN - kindly help me to start ambari-server.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
12-15-2017
02:27 PM
Can any one suggest me, is that possible to install HDP on AWS EC2 instances using public repository's. Is yes is there any doc which I can refer the same to install same. Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
12-04-2017
05:23 PM
Hi Aditya, Tried the same as you said. But when i restarted all services which impact changes it is taking more time(more than 1.5 hrs) but no task got completed/started. All showing in running state. Dont know what might be the issue(I did not found any issue log as well). Is there any other way to perform the same
... View more
12-02-2017
01:29 PM
After my ambari server installation and cluster setup. When I see the dashboard all the services are stopped. I tired Start All services from Actions dropdown. But it is failing at Name node start. Please find the attached screen shot. Kindly help me on the same to start all services and run ambari cluster. Error orrcured during NameNode Start: refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Name or service not known to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2017-12-02 18:14:28,474 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://localhost:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Name or service not known to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Temporary failure in name resolution to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2017-12-02 18:14:52,563 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://localhost:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Temporary failure in name resolution to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Name or service not known to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
2017-12-02 18:15:06,428 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://localhost:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From java.net.UnknownHostException: chaithu: chaithu: Name or service not known to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Command failed after 1 tries Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
12-01-2017
07:02 PM
Thankyou Aditya. You made my day 🙂 I made few changes and the cluster set up got completed. Thanks a lot Aditya. Really you are awesome.
... View more
12-01-2017
06:37 PM
I can login to ssh root@chaithu without password after performing the above said steps. So given Host: localhost and chaithu (also tried giving localhost alone) At ssh private key: choosen browse and given id_rsa at ssh user name: chaithu (also tried with root) ssh port number: 22 Is these details correct. As i am facing below failed error: ==========================
Creating target directory...
==========================
Command start time 2017-12-02 00:02:06
... View more
12-01-2017
06:03 PM
Thanks Aditya, I have chosen public repository with ubuntu 14. In Install options: given host as "localhost" and then added ssh private key (ssh/id_rsa file) and then ssh username: root, ssh port: 22. But the cluster setup fails with message: ==========================
Creating target directory...
==========================
Command start time 2017-12-01 23:32:41
Permission denied (publickey,password).
SSH command execution finished
host=localhost, exitcode=255
Command end time 2017-12-01 23:32:41
ERROR: Bootstrap of host localhost fails because previous action finished with non-zero exit code (255)
ERROR MESSAGE: Permission denied (publickey,password).
STDOUT:
Permission denied (publickey,password). any suggestions please.
... View more
12-01-2017
05:21 PM
I have installed Ambari-Server on ubuntu 14.04 and need to configute ambari cluster on my local machine need information on the below: 1. what is public/local repository. which i need to choose? 2. At Install Options: what i need to give in Target Hosts, and Host Registration information(SSH private key, SSH user account, ssh port number) 3. Is it posible to create multi node cluster with hadoop installation on a single machine if yes, how can i configure cluster for the same. Kindly help me on the same with steps. Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
12-01-2017
02:36 PM
Thanks Aditya... After changing to localhost as you suggested above it worked fine. Then I logged into ambari web cluster installation and trying to create user. Have few questions here: 1. At install Option as I am using my local laptop as a cluster can i give multiple hosts for namenode, snamenode, datanode1, datanode2 if yes hot do i need to give host names here? 2. In Host Registration Information: what SSH I need to give, how can i get this? 3. what i need to give near SSH user account and SSH port number? 4. After successful installation of hosts what will be the next steps i need to proceed? Thanks in Advance.
... View more
11-30-2017
02:01 PM
yes I changed /etc/mysql/my.cnf file as you said and as i could not able to use systemctl restart mysqld, I restarted mysql using "mysqld restart" and then performed below but error again 😞 root@chaithu:/etc/init.d# chmod 644 /usr/share/java/mysql-connector-java.jar
root@chaithu:/etc/init.d# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.
root@chaithu:/etc/init.d# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.
pidServer out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........
ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. I dont know why I am facing this error again and again. This is the error in /var/log/ambari-server/ambari-server.log: 29 Nov 2017 09:24:07,830 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor <br>com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure<br><br>The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.<br> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)<br> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)<br> at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)<br> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) If any one have complete end to end setup document, please share me the same so that i can follow the same.
... View more
11-29-2017
05:46 PM
yes after doing as Aditya said above again facing the same ambari-server issue: root@chaithu:~# ssh root@chaithuWelcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation: https://help.ubuntu.com/New release '16.04.3 LTS' available.Run 'do-release-upgrade' to upgrade to it.Last login: Wed Nov 29 22:43:44 2017 from localhost
root@chaithu:~# netstat -tnlpa | grep 3306tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1151/mysqld
root@chaithu:~# ambari-server status
Using python /usr/bin/pythonAmbari-server statusAmbari Server not running.
Stale PID File at: /var/run/ambari-server/ambari-server.pid Kindly help.
... View more
11-29-2017
04:39 PM
Thanks Jay Kumar, Followed the steps for ssh but facing below error: root@chaithu:/home/chaithu# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 36:65:71:ae:25:9c:88:e2:a1:0a:73:09:55:3b:fa:79 root@chaithu The key's randomart image is: +--[ RSA 2048]----+ | ... . . | | . . . o = | |. = . . * o | | . = + o + | |o = . S . | |.+ . . . . | |. o E | | . | | | +-----------------+ root@chaithu:/home/chaithu# .ssh/id_rsa.pub bash: .ssh/id_rsa.pub: Permission denied root@chaithu:/home/chaithu# sudo .ssh/id_rsa.pub sudo: .ssh/id_rsa.pub: command not found
Also tried with the command and able to see mysql:
root@chaithu:/home/chaithu# netstat -tnlpa | grep 3306
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1197/mysqld
and successfully able to list all tables using given commands. Help me to create ssh, which is showing above error.
... View more
11-29-2017
05:11 AM
I am trying to install ambari server on ubuntu 14.04 Below are few details for the same: root@chaithu:~# hostname
chaithu
root@chaithu:~# hostname -f
chaithu
root@chaithu:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 chaithu
# The following lines are desirable for IPv6 capable hosts::1 ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters
root@chaithu:~# ssh root@chaithu
root@chaithu's password:
(here it is asking for password, I have installed ssh also dont know why asking for password and what password i need to enter here)
Installed mysql and configured ambariuser successfully: mysql> CREATE USER 'ambariuser'@'%' IDENTIFIED BY 'ambaripassword'; mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambariuser'@'%'; mysql> CREATE USER 'ambariuser'@'localhost' IDENTIFIED BY 'ambaripassword'; mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambariuser'@'localhost'; mysql> CREATE USER 'ambariuser'@'chaithu' IDENTIFIED BY 'ambaripassword'; mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambariuser'@'chaithu'; mysql> FLUSH PRIVILEGES; $ mysql -u ambariuser -p mysql> create database ambaridatabase; mysql> use ambaridatabase; mysql> SOURCE Ambari-DDL-MySQL-CREATE.sql; Ambari Server setup as shown below: root@chaithu:/home/chaithu# ambari-server setup
Using python /usr/bin/pythonSetup ambari-serverChecking SELinux...WARNING: Could not run /usr/sbin/sestatus: OKCustomize user account for ambari-server daemon [y/n] (n)? n
Enter advanced database configuration [y/n] (n)? y
Enter choice (1): 3
Hostname (localhost): chaithu
Port (3306): 3306
Database name (ambari): ambaridatabase
Username (ambari): ambariuser
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? yExtracting system views......ambari-admin-2.6.0.0.267.jar........Adjusting ambari-server permissions and ownership...Ambari Server 'setup' completed successfully. Then I have started ambari- server but facing below error: root@chaithu:/home/chaithu# ambari-server start
Using python /usr/bin/pythonStarting
ambari-serverAmbari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
First Error in log: /var/log/ambari-server/ambari-server.log is as shown below:
29 Nov 2017 09:24:07,830 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
Kindly help me to configure the correct way to install ambari-server successfully. Thanks In Advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
11-27-2017
06:28 AM
@Jay Kumar SenSharma I have updated the same as per you suggestion and facing below error: root@node:/home/chaithu# ambari-server startUsing python /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
root@node:/home/chaithu# jps
2530 Jps root@node:/home/chaithu# hostname -f
node.chaithu.com root@node: more /var/log/ambari-server/ambari-server.log The first error shown is as give below: 27 Nov 2017 10:40:35,231 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:489) at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553) at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:254) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93)
... View more
11-27-2017
04:22 AM
Thanks for your reply. Yes I followed all the steps. Please have a look below: root@localhost:~# hostname -flocalhost
root@localhost:~# ssh root@localhost
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation: https://help.ubuntu.com/332 packages can be updated.231 updates are security updates.New release '16.04.3 LTS' available.Run 'do-release-upgrade' to upgrade to it.Last login: Mon Nov 27 09:21:46 2017 from localhost
root@localhost:~# apt-get install ambari-server
Reading package lists... DoneBuilding dependency tree Reading state information... Doneambari-server is already the newest version.0 upgraded, 0 newly installed, 0 to remove and 322 not upgraded.
root@localhost:~# ambari-server setup
Using python /usr/bin/pythonSetup ambari-serverChecking SELinux...WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...Do you want to change Oracle JDK [y/n] (n)? y[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7[3] Custom JDK==============================================================================Enter choice (1): 1JDK already exists, using /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gzInstalling JDK to /usr/jdk64/Successfully installed JDK to /usr/jdk64/JCE Policy archive already exists, using /var/lib/ambari-server/resources/jce_policy-8.zipInstalling JCE policy...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
Enter choice (2): 2
Hostname (localhost):
Port (1521):
Select Oracle identifier type:1 - Service Name2 - SID(1): Service Name (ambari): Username (ambari): Enter Database Password (bigdata): Configuring ambari database...
WARNING: Before starting Ambari Server, you must copy the Oracle JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.Press <enter> to continue.
ERROR: Before starting Ambari Server, you must copy the Oracle JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.
ERROR: Exiting with exit code -1.
REASON: Before starting Ambari Server, you must copy the Oracle JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.
root@localhost:~# ambari-server start
Using python /usr/bin/pythonStarting ambari-serverERROR: Exiting with exit code 1. REASON: Unable to detect a system user for Ambari Server.- If this is a new setup, then run the "ambari-server setup" command to create the user- If this is an upgrade of an existing setup, run the "ambari-server upgrade" command.Refer to the Ambari documentation for more information on setup and upgrade. I guess there is some error in database configuration step. Could you pls suggest me the same to resolve the issue.
... View more
11-26-2017
08:25 PM
I am installing Hortonworks ambari server on ubuntu 14.04 for the first using the setps given on hortoworks installation setps: Facing below error: root@localhost:~# ambari-server start
Using python /usr/bin/python
Starting ambari-server
ERROR: Exiting with exit code 1.
REASON: Unable to detect a system user for Ambari Server.
- If this is a new setup, then run the "ambari-server setup" command to create the user
- If this is an upgrade of an existing setup, run the "ambari-server upgrade" command.
Refer to the Ambari documentation for more information on setup and upgrade. root@localhost:~# ambari-server status
Using python /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid Kindly help me to install hortonworks with hadoop. I got stuck presently with hortonworks ambari server setup on ubuntu 14.04
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
11-01-2017
04:55 AM
Thankyou Geoffrey, Yes I have all the directories in hdfs with full rights and again tried with below sqoop command: sqoop import --bindir $SQOOP_HOME/lib --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234 --table customers -m 1 --class-name customers_part --target-dir /user/hduser/sqoop_import/customers/ and this time facing NullPointerException as shown below: 17/11/01 09:46:44 INFO mapred.LocalJobRunner: map task executor complete. 17/11/01 09:46:44 WARN mapred.LocalJobRunner: job_local270107642_0001 java.lang.Exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/hduser/sqoop_import/customers/_temporary/0/_temporary/attempt_local270107642_0001_m_000000_0/part-m-00000 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481) at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1337) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy11.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:440) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) at com.sun.proxy.$Proxy12.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1733) at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1536) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658) 17/11/01 09:46:45 INFO mapreduce.Job: Job job_local270107642_0001 running in uber mode : false 17/11/01 09:46:45 INFO mapreduce.Job: map 0% reduce 0% 17/11/01 09:46:45 INFO mapreduce.Job: Job job_local270107642_0001 failed with state FAILED due to: NA 17/11/01 09:46:45 INFO mapreduce.Job: Counters: 8 Map-Reduce Framework Map input records=2 Map output records=2 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=164 17/11/01 09:46:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/11/01 09:46:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 3.946 seconds (0 bytes/sec) 17/11/01 09:46:45 INFO mapreduce.ImportJobBase: Retrieved 2 records. 17/11/01 09:46:45 ERROR tool.ImportTool: Error during import: Import job failed! Kindly help me on the above issue.
... View more
10-31-2017
03:20 PM
I have a mysql table "customers" and tried importing data from mysql to hdfs location using sqoop import. Below are the versions I installed on my machine: sqoop version: 1.4.6 hive version: 2.3.0 Hadoop version: 2.8.1 Sqoop import Command: sqoop import --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234 --table customers -m 1 --target-dir /user/hduser/sqoop_import/customers1/ and showing below ClassNotFountException: Tue Oct 31 09:57:21 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.17/10/31 09:57:21 INFO db.DBInputFormat: Using read commited transaction isolation 17/10/31 09:57:21 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
17/10/31 09:57:21 INFO mapred.LocalJobRunner: map task executor complete.
17/10/31 09:57:22 WARN mapred.LocalJobRunner: job_local1437452057_0001java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403) at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:237)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:263) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:515)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class customers not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
... 12 more
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 running in uber mode : false 17/10/31 09:57:22 INFO mapreduce.Job: map 0% reduce 0%
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 failed with state FAILED due to: NA 17/10/31 09:57:22 INFO mapreduce.Job: Counters: 0
17/10/31 09:57:22 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 4.105 seconds (0 bytes/sec)
17/10/31 09:57:22 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Retrieved 0 records.
17/10/31 09:57:22 ERROR tool.ImportTool: Error during import: Import job failed! But when I tried listing tables using sqoop command it is working fine and showing customers table. Sqoop command: sqoop list-tables --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234; Output is displayed properly as shown: 17/10/31 10:07:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. Tue Oct 31 10:07:09 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. customers what might be the issue, why sqoop import from table is not recognizing the table from mysql. Kindly help me on the same. Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Sqoop
09-05-2017
01:37 PM
In short we can assume like a DataFrame is like a table which acts like a table in normal RDBMS. Here in spark we use Dataframe to write sql queries on data loaded/available in RDD's.
... View more
09-05-2017
03:32 AM
You can't load file into RC table directly. In order to load a file into RC Hive table first import the data from Teradata to temp table/Normal Hive table which stores as text and then insert into your actual RCTable using
INSERT OVERWRITE INTO RCTABLE select * from temp;
... View more
08-31-2017
05:40 PM
@Sami Ahmad : There is another way to perform the same if its ok to update records instead of only append then you can use below command: --table PARTON.TAB1 --split-by TAB1.TXN_ID --check-column txn_process_date --incremental lastmodified --last-value 0 --merge-key TXN_ID --hcatalog-database default --hcatalog-table PA_LANE_TXN_orc; If you don't want to update then you can write a --query to import only new data which will not update old records. Hope this helps.!
... View more
08-31-2017
02:21 PM
Sqoop import append works properly for normal hive table. But not sure how it works with Hive Partitioned table. Here I guess you are using same hive partitioned table so facing this issue.
... View more