About slachterman

slachterman · ‎06-28-2016

@Ravikumar Kumashi Make sure your VM is up and sshd is running and listening on port 2222: sudo netstat -anp | grep sshd Make sure no firewall rules are getting in the way. If confirmed, try using 127.0.0.1 instead of localhost and if that doesn't work try editing your hosts file so that sandbox.hortonworks.com resolves to 127.0.0.1 and then use the FQDN sandbox.hortonworks.com instead of localhost.

slachterman · ‎06-28-2016

@Ravikumar Kumashi in Cygwin, you can access the root of your C: drive by specifying the directory /cygdrive/c. So the path would be /cygdrive/c/Users/rnkumashi/Downloads/sample.txt This is one of the reasons I recommended pscp.

slachterman · ‎06-28-2016

@mayki wogno It's essentially non-HDFS data in dfs.datanode.data.dir. This could include log files, intermediate shuffle output from MapReduce jobs, local data files (if you put them on a data node), etc. You can use du or a similar tool to investigate further.

slachterman · ‎06-28-2016

@mayki wogno "Non DFS used" can be calculated by the following formula: Non DFS Used = Configured Capacity - Remaining Space - DFS Used Noting that Configured Capacity = Total Disk Space - Reserved Space Therefore, Non DFS Used = (Total Disk Space - Reserved Space) - Remaining Space - DFS Used Reserved Space is set by the property dfs.datanode.du.reserved

slachterman · ‎06-28-2016

I would recommend looking into pscp if you are on a Windows platform. If using Cygwin, you'll need to install scp, see http://stackoverflow.com/questions/18688502/how-do-i-download-scp-and-ssh-on-cygwin. scp is part of the openssh package as noted.

slachterman · ‎06-28-2016

@alain TSAFACK val VAL1 = "testcol" val df = HiveContext.sql(s"SELECT * FROM src WHERE col1 = $VAL1")

slachterman · ‎06-28-2016

@Ravikumar Kumashi yes, that is correct, you want to run the command from your local machine since that is where the file lives that you are scp'ing over to the sandbox. You can invoke via Cygwin or you can use pscp (from the makers of Putty) and run pscp from the Windows command line.

slachterman · ‎06-28-2016

@Simran Kaur an example of using Hive for data cleansing is in this article (see section 3.5 in particular). Regarding Spark, it is used widely for extract, transformation, and load logic and is usually well-suited for those kinds of use cases. Both MapReduce and Spark are very general computation paradigms. It would be helpful to know what data cleaning transformations you have in mind.

slachterman · ‎06-28-2016

@Ravikumar Kumashi the scp command is missing the port number (please notice that the "usage" text was returned by the command, which means the syntax was incorrect). Please try specifying 2222 after the -P switch.

slachterman · ‎06-27-2016

@Bharath Kumar K you may want to look into pscp if you want to run from the command line and resolve network drive mappings in the Windows fashion. I am not sure how you are running #1 (from Cygwin maybe?), but the syntax in the first example is essentially correct. #3 should work, what error are you receiving? One thing you might want to try is creating a hosts entry so that sandbox.hortonworks.com resolves to 127.0.0.1 and then using sandbox.hortonworks.com as your hostname/IP.

Online	Offline
Last Visited	‎05-03-2018 08:43 PM

Member Since	‎06-20-2016 02:58 PM
Last Visited	‎05-03-2018 08:43 PM
Posts	251
Kudos received	196

Cloudera Community

Re: PySpark and Python version (<3.6)?

Re: Ambari Server Start failure - Ranger Atlas Ta...

Re: Using underscore _ in a database name in HIVE

Re: Active directory as Directory Service and MIT ...

Re: 4 node cluster configuration

Re: Trying to copy file from my windows machine to...

Re: Trying to copy file from my windows machine to...

Re: [HDFS] - Non DFS used ?

Re: [HDFS] - Non DFS used ?

Re: Trying to copy file from my windows machine to...

Re: How to bind variable in Apache Spark SQL

Re: Trying to copy file from my windows machine to...

Re: Data cleaning before storing in HDFS

Re: Trying to copy file from my windows machine to...

Re: Copy file from my Windows to Sandbox