Hi, we are trying to use WebHDFS over Knox to access HDFS on our secured cluster from java. We are able to list files/folders there, but we are still struggling with the file creation. The problem is probably in Oracle's Java library, where the streaming does not seem to be supported when authentication is required:. In sun.net.www.protocol.http.HttpURLConnection.getInputStream0() there is something like if (j == 401) { /* 1635 */ if (streaming()) { /* 1636 */ disconnectInternal(); /* 1637 */thrownew HttpRetryException("cannot retry due to server authentication, in streaming mode", 401); /* */ } The streaming is not needed for some operation, like list/delete (and therefore it works), but it is required for file creation. Any suggestions how to handle this? Thanks a lot, Pavel ... View more

Hi, I am wondering if there is a general way how to determine what YARN applications were started by some application and vice-versa if some YARN application was started by some other. My use case is Oozie and sqoop where Oozie runs some launchers that in turn start some MR jobs to do the actual ingest. It is possible to browse through the logs to get ID of spawned application, but keep thinking that there should be some better way how to do it. This kind of relation must be stored somewhere, since when the Oozie workflow is killed all child processes are killed as well almost immediately. Thanks for any hints, Regards, Pavel ... View more

Hi, we are using org.apache.falcon.client.FalconClient API to update Falcon process from java: falconClient.update( EntityType.PROCESS.name(), <some-id>, <file-name>,true,doAs); where the local <file-name> is created like this: ... Marshaller marshaller = entityType.getMarshaller(); final File createTempFile = File.createTempFile(entityType.name().toLowerCase() + "_" + id, ".xml"); LOGGER.debug("Generated entity: {}", entity.toString()); marshaller.marshal(entity, createTempFile); return createTempFile.getPath(); and sometimes the update fails with this error: javax.xml.bind.UnmarshalException: [org.xml.sax.SAXParseException; Premature end of file.] at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:335) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:523) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:220) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:189) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:204) at org.apache.falcon.entity.parser.EntityParser.parse(EntityParser.java:94) ... 61 more Caused by: org.xml.sax.SAXParseException; Premature end of file. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:216) ... 65 more most of the updates pass, this error happens only sometimes, so I believe the file is created correctly on the client side and the error is caused possibly by some performance issue or race condition. Have you seen this behavior? Thanks, Pavel ... View more