The scenario I have is a HDFS cluster, and a separate Object Store. The Object store provides an HDFS Compatible File System jar that I can use from the native HDFS Cluster to read/write to the object store. However, referencing the Object store requires using a distinct URL i.e. For the HDFS Cluster I use hdfs dfs -ls hdfs://<namenode>:8020/... But for the Object store I have to use a custom url hdfs dfs -ls vipers://<namenode>:8020/... If I define a falcon mirroring job on the HDFS Cluster, but using paths on the HDFS Object store, then I get URL Exceptions thrown when the job is submitted. This is because the path URI is appended to the Cluster URI.... What I think is happening is as follows: hdfs://<namenode>:8020/ can only be used to address the HDFS Cluster Files and blocks. And, while I can make the cluster access a third party HDFS compatible Object store, there is no way to make a cluster name node 'proxy' for the Object store itself. Is that right?
... View more
