Member since 
    
	
		
		
		08-31-2017
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                30
            
            
                Posts
            
        
                0
            
            
                Kudos Received
            
        
                0
            
            
                Solutions
            
        
			
    
	
		
		
		12-23-2018
	
		
		06:34 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I have a scenario where I need to read files from windows shared path using spark and scala. I tried with below but could not able to find the files:object ExternalFiles {
  def main(args: Array[String]){
    val conf = new SparkConf().setMaster("local").setAppName("External Files")
    val sc = new SparkContext(conf)
    val files = sc.textFile("\\\\sharedNetwork\\External Data\\testData.txt")
    files.foreach(println)
  }
}  I tried using sc.textFile("file://sharedNetwork/External Data/testData.txt") but it shows below error in both cases:  18/12/23 11:57:57 WARN : Your hostname, name-21 resolves to a loopback/non-reachable address: 10.xx.xx.xxx, but we couldn't find any external IP address!
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://sharedNetwork/External Data/testData.txt
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251)  Can someone suggest me to resolve this solution to read files from shared drive using Spark and Scala code.  Pls: suggest me on how to download files from NTFS windows shared path to linux machine through putty.  Thanks,  Chaitanya 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Hadoop
- 
						
							
		
			Apache Spark
			
    
	
		
		
		01-05-2018
	
		
		03:49 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 My scenario is to check the fileName and then if the file contains specific word then I need to pick that file for processing.  Eg: in mydirectory I have two filenames:  file1: sample1.txt_processed  file2: sample2.txt  Now I need to check the file names with and without "_processed" keyword and pick only files without "_processed" text in file name.  Can any one help me on this scenario. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Hadoop
- 
						
							
		
			Apache Spark
			
    
	
		
		
		12-01-2017
	
		
		07:02 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thankyou Aditya. You made my day 🙂 I made few changes and the cluster set up got completed.  Thanks a lot Aditya. Really you are awesome. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-01-2017
	
		
		06:37 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I can login to ssh root@chaithu without password after performing the above said steps.  So given Host: localhost and chaithu (also tried giving localhost alone)  At ssh private key: choosen browse and given id_rsa  at ssh user name: chaithu (also tried with root)  ssh port number: 22  Is these details correct. As i am facing below  failed error:    ==========================
Creating target directory...
==========================
Command start time 2017-12-02 00:02:06   
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-01-2017
	
		
		06:03 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks Aditya, I have chosen public repository with ubuntu 14.  In Install options:  given host as "localhost" and then added ssh private key (ssh/id_rsa file) and then ssh username: root, ssh port: 22.  But the cluster setup fails with message:  ==========================
Creating target directory...
==========================
Command start time 2017-12-01 23:32:41
Permission denied (publickey,password).
SSH command execution finished
host=localhost, exitcode=255
Command end time 2017-12-01 23:32:41
ERROR: Bootstrap of host localhost fails because previous action finished with non-zero exit code (255)
ERROR MESSAGE: Permission denied (publickey,password).
STDOUT: 
Permission denied (publickey,password).  any suggestions please. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		12-01-2017
	
		
		05:21 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I have installed Ambari-Server on ubuntu 14.04 and need to configute ambari cluster on my local machine need information on the below:  1. what is public/local repository. which i need to choose?  2. At Install Options: what i need to give in Target Hosts, and Host Registration information(SSH private key, SSH user account, ssh port number)  3. Is it posible to create multi node cluster with hadoop installation on a single machine if yes, how can i configure cluster for the same.  Kindly help me on the same with steps. Thanks in Advance. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Ambari
- 
						
							
		
			Apache Hadoop
			
    
	
		
		
		12-01-2017
	
		
		02:36 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks Aditya... After changing to localhost as you suggested above it worked fine.  Then I logged into ambari web cluster installation and trying to create user. Have few questions here:  1. At install Option as I am using my local laptop as a cluster can i give multiple hosts for namenode, snamenode, datanode1, datanode2 if yes hot do i need to give host names here?  2. In Host Registration Information: what SSH I need to give, how can i get this?  3. what i need to give near SSH user account and SSH port number?  4. After successful installation of hosts what will be the next steps i need to proceed?  Thanks in Advance. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		11-30-2017
	
		
		02:01 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 yes I changed /etc/mysql/my.cnf file as you said and as i could not able to use systemctl restart mysqld, I restarted mysql using "mysqld restart"  and then performed below but error again 😞  root@chaithu:/etc/init.d# chmod 644 /usr/share/java/mysql-connector-java.jar
root@chaithu:/etc/init.d# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.
root@chaithu:/etc/init.d# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.
pidServer out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...........
ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.  I dont know why I am facing this error again and again.  This is the error in /var/log/ambari-server/ambari-server.log:  29 Nov 2017 09:24:07,830 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor <br>com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure<br><br>The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.<br>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br>   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br>   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)<br>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)<br>   at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)<br>   at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358)  If any one have complete end to end setup document, please share me the same so that i can follow the same. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		11-29-2017
	
		
		05:46 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 yes after doing as Aditya said above again facing the same ambari-server issue:  root@chaithu:~# ssh root@chaithuWelcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation:  https://help.ubuntu.com/New release '16.04.3 LTS' available.Run 'do-release-upgrade' to upgrade to it.Last login: Wed Nov 29 22:43:44 2017 from localhost 
 root@chaithu:~# netstat -tnlpa | grep 3306tcp  0  0 127.0.0.1:3306  0.0.0.0:*  LISTEN  1151/mysqld   
 root@chaithu:~# ambari-server status
Using python  /usr/bin/pythonAmbari-server statusAmbari Server not running. 
Stale PID File at: /var/run/ambari-server/ambari-server.pid  Kindly help. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		11-29-2017
	
		
		04:39 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks Jay Kumar, Followed the steps for ssh but facing below error:  root@chaithu:/home/chaithu# ssh-keygen  Generating public/private rsa key pair.  Enter file in which to save the key (/root/.ssh/id_rsa):   /root/.ssh/id_rsa already exists.  Overwrite (y/n)? y  Enter passphrase (empty for no passphrase):   Enter same passphrase again:   Your identification has been saved in /root/.ssh/id_rsa.  Your public key has been saved in /root/.ssh/id_rsa.pub.  The key fingerprint is:  36:65:71:ae:25:9c:88:e2:a1:0a:73:09:55:3b:fa:79 root@chaithu  The key's randomart image is:  +--[ RSA 2048]----+  |  ...  . .  |  | .  . . o =  |  |.  = . . * o  |  | . = +  o +  |  |o = .  S .  |  |.+ . . . .  |  |.  o E  |  |  .  |  |  |  +-----------------+  root@chaithu:/home/chaithu# .ssh/id_rsa.pub  bash: .ssh/id_rsa.pub: Permission denied  root@chaithu:/home/chaithu# sudo  .ssh/id_rsa.pub  sudo: .ssh/id_rsa.pub: command not found
Also tried with the command and able to see mysql:
root@chaithu:/home/chaithu# netstat -tnlpa | grep 3306
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1197/mysqld
  and successfully able to list all tables using given commands.  Help me to create ssh, which is showing above error. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
        









