Member since
09-25-2016
11
Posts
0
Kudos Received
0
Solutions
07-10-2018
05:31 PM
@Vinit Pandey I suggest on your HDFS server you start a process (using shellscript for example) to execute kinit and after that, get these remote files using sftp or scp example # scp user@remoteserver:/remotepath/files localpath/ and # hdfs dfs -put localpath/files /hdfspath Note: To automate this process you can create a private/public ssh between these servers and create a crontab entry.
... View more
06-06-2018
01:34 PM
Vinit, Take a look at this example that I put up on github: https://github.com/bchagan/spark-sql-concepts/blob/master/src/main/java/com/hagan/brian/spark/NestedStructureProcessor.java
... View more
04-13-2018
11:03 PM
@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename. Please refer to the document for more details: "HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."
... View more
04-03-2019
05:16 AM
@Rahul Soni Hi, Currently i am using particular jar as follow : spark-shell --jars abc.jar, now i am trying build jar out my code, what is the way to add this jar (abc.jar)
... View more