@swathi thukkaraju You can use the dict.update() method: read the default config first to a dictionary, then read your config and call default.update(yours). But keep an eye on the fact that your two configs are not compatible: in your config "path_details" and "db_details" are the children of "parser_config", while in the default config they are on the same level. If we assume that the original config is correct, then you should update swathi_configure.json like this: [{ "parser_config": { "vista": { "Parser_info": { "module_name": "A", "class_name": "a", "standardformat1": { "transaction": "filename", "terminal": "filenme2", "session": "filename3" } } } }, "path_details": { "parent_path": "wasbs://XXXX@XXXXXstorage.blob.core.windows.net/tenantShortName/" }, "db_details": { "datawarehouse_url": "", "datawarehouse_username": "", "datawarehouse_password": "" } }] With these two files the following code does what you want: import json from pprint import pprint with open('config.json') as default_file, \ open('swathi_configure.json') as current_file: # [0]: take the first and only item from your list # If you have more items, use a for loop default = json.load(default_file)[0] current = json.load(current_file)[0] default.update(current) # default is your merged config now pprint(default) ... View more

@Triffids G The dfsadmin report is not relevant in this case, the "No space left on device" concerns the NameNode, not the DataNodes. Check "dfs.namenode.name.dir", I'm pretty sure that it points to a volume that is in fact full. Note that you can use comma separated paths, so I'd suggest to add a directory from the newly added partition too and restart the NameNode. ... View more

@btandel 1) Minimum user limit percentage, this is the definition from the documentation, I think this is as clear as it gets: "Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. If a third user submits an application, no single user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed. The default is 100. Value is specified as a integer." 2) Fair ordering policy. Check this documentation: Using Flexible Scheduling Policies The two both concerns a single queue's scheduling policy: minimum-user-limit-percentage defines how the queue's resources are distributed among users and the ordering policy defines in which order the submitted jobs will be executed. If the minimum user limit is 100%, it means that there are no actual limits in place, so the fair ordering policy will do its best to give all the jobs "fair" amount of resources. 3) "if i set the Minimum user limit to 50 % and, user1 job is utilizing 100 % of cluster resource, then user2 submit job who requires 20 % of cluster resource then will the resource get distributed as 80% and 20% or will it be 50% - 50%" It will be 80-20%, because user2 doesn't need any more resources. If they needed let's say 60%, or more, than the distribution would be 50-50%. ... View more

@Xiong Duan I'm afraid, as of now, there is no other way to remove dead/decommissioned datanodes from the WebUI (NameNode state) than restarting the NameNode. ... View more