You can put the s3 credentials in the s3 URI, or you can just pass the parameters on the command line, which is what I prefer, eg:   hadoop fs -Dfs.s3a.access.key="" -Dfs.s3a.secret.key="" -ls s3a://bucket-name/ Its also worth knowing that if you run the command like I have given above, it will override any other settings that are defined in the cluster config, such as core-site.xml etc. ... View more

Hi Leo,   Ok - its good that the Kerberos setup is all working, as that can be the hard part!   If you want to copy files such that you run the command on cluster1 then you will need to modify the cluster1 hdfs-site.xml to hold the information about the other cluster. To do this, you can list a new nameservice in the hdfs-site.xml on cluster1 that points at the Namenode hosts of the other cluster.   The nameservice on cluster1 is nameservice1. Lets create another nameservice called backupcluster. In the hdfs-site.xml on cluster1 add properties for the following:   dfs.ha.namenodes.backupcluster=nn1,nn2 dfs.namenode.rpc-address.backupcluster.nn1=cluster2_nn_host1:8020 dfs.namenode.rpc-address.backupcluster.nn2=cluster2_nn_host2:8020 dfs.client.failover.proxy.provider.backupcluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider   Assuming you are using CM, you can edit the /etc/hadoop/conf/hdfs-site.xml on your gateway node to test this out, but afterwards you should add the settings to the CM Cluster wide safety value for hdfs-site.xml.   If the above settings worked, from cluster1, you should be able to run: hadoop fs -ls hdfs://backupcluster/some/path   If that works, you can try distcp from cluster1:   hadoop distcp hdfs://nameservice1/path/on/cluster1 hdfs://backupcluster/target/path     ... View more

Assuming all the Kerberos trust relationships are setup correctly, it is possible to copy between two secured HA clusters with the same namespace. The following documentation talks about the Kerberos configuration required:   https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate.html#concept_hcs_srr_sr   The easiest way to do the copy, is to identify the active namenode of the target cluster, and run the command like:   hadoop distcp /path/to/source/file hdfs://namenode_host:8020/destination_path   Ie, intead of using the nameservice name, use the actual hostname of the active namenode. ... View more

You could try adding the following to the bottom of the yarn-site.xml:   <property> <description>Minimum allocation unit.</description> <name>yarn.scheduler.minimum-allocation-mb</name> <value>256</value> </property> Also, set the nodemanager memory a bit higher and the vcores a bit higher:   yarn.nodemanager.resource.memory-mb to 1536 yarn.nodemanager.resource.cpu-vcores to 4   And then add the following to the end of the mapred-site.xml and see if it gives better results:   <property> <name>mapreduce.map.memory.mb</name> <value>256</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>256</value> </property> <property> <description>Application master allocation</description> <name>yarn.app.mapreduce.am.resource.mb</name> <value>256</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx204m</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx204m</value> </property> <property> <description>Application Master JVM opts</description> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Xmx204m</value> </property> <property> <name>mapreduce.task.io.sort.mb</name> <value>50</value> </property> Note that it would be a good idea to make a copy of the original mapred-site and yarn-site before making these changes. After changing the settings reboot the Quickstart VM so the settings take effect. ... View more

It sounds like the jaas.conf is not being picked up. Could you try passing the full path to the jaas.conf to see if that works better, eg: -Djava.security.auth.login.config=/full/path/to/jaas.conf ... View more

Hi,   So the memory capacity of your single node is set to 1GB. When you run an oozie job, it always needs 2 containers - one for oozie and another to run the jobs, so it is likely you don't have enough memory allocated to have the job run. We will need to check some of your config settings.   In /etc/hadoop/conf/mapred-site.xml, what values are set for:   mapreduce.map.memory.mb mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb mapreduce.map.java.opts mapreduce.reduce.java.opts yarn.app.mapreduce.am.command-opts   What memory setting have you for the VM you are running? I am wondering if we can push up the yarn limits a little to let more containers run. ... View more

Sorry, I should have said, can you substitute your node_id for the one in my example, ie: yarn node -status taha99:8041 ... View more

Sounds like there is a chance there are not enough resources available in yarn to run the job. If you run the following command on the CDH host what is the output:   yarn node -list   If this returns a host, eg:   [vagrant@standalone ~]$ yarn node -list 16/11/29 12:50:32 INFO client.RMProxy: Connecting to ResourceManager at standalone/192.168.33.6:8032 Total Nodes:1 Node-Id Node-State Node-Http-Address Number-of-Running-Containers standalone:42578 RUNNING standalone:8042 Can you take the node-id and check what resources it has:   $ yarn node -status standalone:42578 ... View more

If you need to get your data from Oracle to HDFS in *near real time* then a Golden Gate solution is probably the best option. While you may need some additional redo logging for this to work, it probably results in the minimum overall impact on the database.   If you can tolerate some lag in the data appearing in HDFS, so you could setup some jobs to use Sqoop to pull data from Oracle at regular intervals. With the correct indexes and a way for Sqoop to identify new records this could be done quite efficiently.   Depending on how the applicaiton that writes the data to Oracle is architected, there could also be an option to change that applicaiton to write to Oracle and also write to Flume or Kafka, but that does require significant applicaiton changes which may not be feasible in your case. ... View more