Member since
08-31-2017
30
Posts
0
Kudos Received
0
Solutions
12-23-2018
06:34 AM
I have a scenario where I need to read files from windows shared path using spark and scala. I tried with below but could not able to find the files:object ExternalFiles {
def main(args: Array[String]){
val conf = new SparkConf().setMaster("local").setAppName("External Files")
val sc = new SparkContext(conf)
val files = sc.textFile("\\\\sharedNetwork\\External Data\\testData.txt")
files.foreach(println)
}
} I tried using sc.textFile("file://sharedNetwork/External Data/testData.txt") but it shows below error in both cases: 18/12/23 11:57:57 WARN : Your hostname, name-21 resolves to a loopback/non-reachable address: 10.xx.xx.xxx, but we couldn't find any external IP address!
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://sharedNetwork/External Data/testData.txt
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) Can someone suggest me to resolve this solution to read files from shared drive using Spark and Scala code. Pls: suggest me on how to download files from NTFS windows shared path to linux machine through putty. Thanks, Chaitanya
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
01-05-2018
03:49 AM
My scenario is to check the fileName and then if the file contains specific word then I need to pick that file for processing. Eg: in mydirectory I have two filenames: file1: sample1.txt_processed file2: sample2.txt Now I need to check the file names with and without "_processed" keyword and pick only files without "_processed" text in file name. Can any one help me on this scenario.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
12-01-2017
07:02 PM
Thankyou Aditya. You made my day 🙂 I made few changes and the cluster set up got completed. Thanks a lot Aditya. Really you are awesome.
... View more
12-01-2017
06:37 PM
I can login to ssh root@chaithu without password after performing the above said steps. So given Host: localhost and chaithu (also tried giving localhost alone) At ssh private key: choosen browse and given id_rsa at ssh user name: chaithu (also tried with root) ssh port number: 22 Is these details correct. As i am facing below failed error: ==========================
Creating target directory...
==========================
Command start time 2017-12-02 00:02:06
... View more
12-01-2017
06:03 PM
Thanks Aditya, I have chosen public repository with ubuntu 14. In Install options: given host as "localhost" and then added ssh private key (ssh/id_rsa file) and then ssh username: root, ssh port: 22. But the cluster setup fails with message: ==========================
Creating target directory...
==========================
Command start time 2017-12-01 23:32:41
Permission denied (publickey,password).
SSH command execution finished
host=localhost, exitcode=255
Command end time 2017-12-01 23:32:41
ERROR: Bootstrap of host localhost fails because previous action finished with non-zero exit code (255)
ERROR MESSAGE: Permission denied (publickey,password).
STDOUT:
Permission denied (publickey,password). any suggestions please.
... View more
12-01-2017
05:21 PM
I have installed Ambari-Server on ubuntu 14.04 and need to configute ambari cluster on my local machine need information on the below: 1. what is public/local repository. which i need to choose? 2. At Install Options: what i need to give in Target Hosts, and Host Registration information(SSH private key, SSH user account, ssh port number) 3. Is it posible to create multi node cluster with hadoop installation on a single machine if yes, how can i configure cluster for the same. Kindly help me on the same with steps. Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
12-01-2017
02:36 PM
Thanks Aditya... After changing to localhost as you suggested above it worked fine. Then I logged into ambari web cluster installation and trying to create user. Have few questions here: 1. At install Option as I am using my local laptop as a cluster can i give multiple hosts for namenode, snamenode, datanode1, datanode2 if yes hot do i need to give host names here? 2. In Host Registration Information: what SSH I need to give, how can i get this? 3. what i need to give near SSH user account and SSH port number? 4. After successful installation of hosts what will be the next steps i need to proceed? Thanks in Advance.
... View more
11-30-2017
02:01 PM
yes I changed /etc/mysql/my.cnf file as you said and as i could not able to use systemctl restart mysqld, I restarted mysql using "mysqld restart" and then performed below but error again 😞 root@chaithu:/etc/init.d# chmod 644 /usr/share/java/mysql-connector-java.jar
root@chaithu:/etc/init.d# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.
root@chaithu:/etc/init.d# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.
pidServer out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........
ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. I dont know why I am facing this error again and again. This is the error in /var/log/ambari-server/ambari-server.log: 29 Nov 2017 09:24:07,830 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor <br>com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure<br><br>The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.<br> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)<br> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)<br> at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)<br> at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) If any one have complete end to end setup document, please share me the same so that i can follow the same.
... View more
11-29-2017
05:46 PM
yes after doing as Aditya said above again facing the same ambari-server issue: root@chaithu:~# ssh root@chaithuWelcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation: https://help.ubuntu.com/New release '16.04.3 LTS' available.Run 'do-release-upgrade' to upgrade to it.Last login: Wed Nov 29 22:43:44 2017 from localhost
root@chaithu:~# netstat -tnlpa | grep 3306tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1151/mysqld
root@chaithu:~# ambari-server status
Using python /usr/bin/pythonAmbari-server statusAmbari Server not running.
Stale PID File at: /var/run/ambari-server/ambari-server.pid Kindly help.
... View more
11-29-2017
04:39 PM
Thanks Jay Kumar, Followed the steps for ssh but facing below error: root@chaithu:/home/chaithu# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 36:65:71:ae:25:9c:88:e2:a1:0a:73:09:55:3b:fa:79 root@chaithu The key's randomart image is: +--[ RSA 2048]----+ | ... . . | | . . . o = | |. = . . * o | | . = + o + | |o = . S . | |.+ . . . . | |. o E | | . | | | +-----------------+ root@chaithu:/home/chaithu# .ssh/id_rsa.pub bash: .ssh/id_rsa.pub: Permission denied root@chaithu:/home/chaithu# sudo .ssh/id_rsa.pub sudo: .ssh/id_rsa.pub: command not found
Also tried with the command and able to see mysql:
root@chaithu:/home/chaithu# netstat -tnlpa | grep 3306
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1197/mysqld
and successfully able to list all tables using given commands. Help me to create ssh, which is showing above error.
... View more