@RAUI wholeTextFile() is not part of the HDFS API, I'm assuming you're using Spark, with which I'm not too familiar. I suggest you to post another question for this to HCC. ... View more

@RAUI No, it won't create it, the target directory must exist. However, if the target directory doesn't exist, it won't throw an exception, it will only indicate the error via the return value (as described in the documentation). So 1) you should create the target directory before you call rename() and 2) you should check the return value, like this: fs.mkdirs(new Path("/your/target/path")); boolean result = fs.rename( new Path("/your/source/path/your.file"), new Path("/your/target/path/your.file")); if (!result) { ... } ... View more

You can figure that out from the RM UI, or the RM/application logs. ... View more

@Himanshu Kukreja It's hard to figure out from your screenshot what kind of applications these are. I'd recommend you to dig deeper, find out as much about these as you can: more application info, application logs, etc. Then you should be able to stop them from spawning. ... View more

@Dinesh Chitlangia Unfortunately the native build on OS X is broken by HDFS-13403 at this moment on trunk. You have two options: If you don't need native build, you can build hadoop without the -Pnative option successfully. The build issue is fixed by HDFS-13534, but it's not merged yet (at the time of writing this answer). You can either wait until it gets merged, or apply it manually: wget https://issues.apache.org/jira/secure/attachment/12922534/HDFS-13534.001.patch git apply HDFS-13534.001.patch ... View more

@Jon Page Could you please provide more information about what kind of "job" are you trying to run through yarn? Are you using Spark? Custom native YARN App? Distributed Shell? Knit? Nevertheless, you could run a simple distributed shell app to see which python version YARN picks up: yarn jar path/to/hadoop-yarn-applications-distributedshell.jar -jar path/to/hadoop-yarn-applications-distributedshell.jar -shell_command python -shell_args -V Or you can check the same with the framework you are using. ... View more

@Anton P I'm glad it works. I'm not sure how exactly the "fair" ordering policy works inside one queue, but preemption is only for between queues. I assume, that it will try to give resources to the applications/users in the same queue equally, but once a container is running it will not preempt it. If you would like to achieve that, you should consider creating sub-queues. ... View more

@Jon Page You can't move just the usercache directory, but you can move its parent: set yarn.nodemanager.local-dirs to a different location. ... View more

@Nishant Verma In the first case your 'dummy' user can't authenticate to the Timeline Server to get the Delegation Token. In the second case you missed an important step: you need to kinit with your spark/hdfs user to get a TGT. You need to use a password or a keytab file to do so. ... View more

@Amithesh Merugu Try to use the IP address of the NameNode. And also add the port (default is 8020). ... View more

@Amithesh Merugu Use this method: copyFromLocalFile(Path src, Path dst). The first parameter is a path on your local disk (in your example /tmp/files) and the second is the HDFS path (hdfs://user/username). The documentation doesn't make it clear, but the source can be a dictionary and then the whole content is copied to the HDFS. FileSystem fs = FileSystem.get(hdfsUri, conf); fs.copyFromLocalFile(new Path("/tmp/files"), new Path("/user/username")); ... View more

@Gaurav Parmar Here is the documentation of the Cluster Applications API you are using. As you can see under the "Query Parameters Supported", to list jobs for a particular time frame you can use 4 parameters: startedTimeBegin startedTimeEnd finishedTimeBegin finishedTimeEnd All the parameters are specified in milliseconds since epoch, so you have to convert your time interval to Unix Timestamp. For example last week is 2017-12-04:00:00:01=1512345601000 - 2017-12-10:23:59:59=1512950399000. To list all the applications that were started and finished this week you can use http://hostname:8088/ws/v1/cluster/apps?startedTimeBegin=1512345601000&finishedTimeEnd=1512950399000&states=FINISHED ... View more

@Joe Karau What is the exact HDP version you are using? In 2.6 the -filters option should be available to exclude certain files. It is documented as "The path to a file containing a list of pattern strings, one string per line, such that paths matching the pattern will be excluded from the copy. Supports regular expressions specified by java.util.regex.Pattern." However, it's questionable whether the filtering happens before the exception. Can you give it a try? If it doesn't work, unfortunately I think the easiest way to fix this is to specify only the "correct" files to be copied. ... View more

@jpj To the best of my knowledge, MAPREDUCE-6304 is only part of HDP-2.6.2.0. Please check this document to upgrade. ... View more

.*(?<!History)$ ... View more

@swathi thukkaraju I'm not completely sure what you mean by 'incremental load format', but here are some hints: To read FTP server files you can simply use the builtin python module urllib, more specifically urlopen or urlretrieve To write to HDFS you can Use an external library, like HdfsCLI Use the HDFS shell and call it from python with subprocess Mount your HDFS with HDFS NFS Gateway and simply write with the normal write() method. Beware, that using this solution you won't be able to append! Here's an implementation for you using urlopen and HdfsCli. To try it first install HdfsCli with pip install hdfs. from urllib.request import urlopen from hdfs import InsecureClient # You can also use KerberosClient or custom client namenode_address = 'your namenode address' webhdfs_port = 'your webhdfs port' # default for Hadoop 2: 50070, Hadoop 3: 9870 user = 'your user name' client = InsecureClient('http://' + namenode_address + ':' + webhdfs_port, user=user) ftp_address = 'your ftp address' hdfs_path = 'where you want to write' with urlopen(ftp_address) as response: content = response.read() # You can also use append=True # Further reference: https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.Client.write with client.write(hdfs_path) as writer: writer.write(content ... View more

The default queue's AM limit is 6144 MB -> the default queue's capacity must be 7 GB (for 6 the limit would be 5 and for 8 it would be 7 with the maximum-am-resource-percent of 0.9). Since default.capacity = 60 the whole cluster's capacity equals to ~ 100 / 60 * 7, which could indicate 12 or 13 GBs in total, but the latter would be very unusual. Did you manage to overcome your issue with any of my suggestions? ... View more

@Dennis Hude Unfortunately I don't think there is, Hadoop is mainly designed for Linux clusters. But if you're interested, you can definitely try to write a ResourceCalculatorProcessTree implementation for Mac or just open a Jira ticket for it and see if someone else is interested. ... View more

@Karan Alang Based on this information I assume you have 12 GBs of memory and the minimum allocation is set to 1024 MB, the default queue has a configured capacity of 60%, 7 GBs. The AM limit is 6 GBs (7.2 * 0.9 rounded to GB), and it is full, probably three other AMs are running. Please correct me if I'm wrong! To get more memory, you might try these things: Add more memory to the cluster 😛 Increase the maximum-capacity of the default queue, so that it can use more resources when the LLAP queue doesn't use them Increase the maximum-am-resource-percent of the default queue to 1 Decrease the minimum-allocation-mb: this way the other AMs (and containers) might use less resources (e.g. if you need 1.2 GBs - just for the sake of the example - then with the default 1 GB minimum allocation you still need to get a 2 GB container) Kill other applications from the queue or wait until they finish ... View more

@Karan Alang Could you please share (at least the relevant part of) your Capacity Scheduler config and tell us how much memory your default queue in total should have? Based on your error your default queue's AM limit is in fact exceeded.. ... View more

@Paul Yang You can simply use the UI of the JobHistory server remotely, can't you? ... View more

@Dennis Hude What operating system are you using? CPU_MILLISECONDS, PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES are collected by ResourceCalculatorProcessTree which has two implementations: ProcfsBasedProcessTree (for Linux that uses /proc) and WindowsBasedProcessTree. You can check the syslogs from your application containers to see which one is used, grep for this message: "Using ResourceCalculatorProcessTree :". Not having an initiated ResourceCalculatorProcessTree instance explains your 0 values. If you have one, there might be a problem with that which requires further investigation (the container logs can help with that, too). ... View more

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_release-notes/content/upgrading_parent.html ... View more