@bgooley Does oozie email action needs a firewall to be opened between slave nodes to smtp or just the oozie server to smtp? ... View more

  Hi @csguna, CDH version is 5.13.2 ... View more

Thank you very much @Harsh J!   If I got it correctly these parameters   oozie.launcher.mapreduce.map.java.opts oozie.launcher.mapreduce.reduce.java.opts oozie.launcher.yarn.app.mapreduce.am.command-opts   control the maximum amount of memory allocated for the Oozie launcher. What are the equivalent parameters to control the memory allocated for the action instead (e.g. a Sqoop action), as shown in the image?           ... View more

Hi Ludof,   First of all, thanks a lot for the response, second my apologies that I could not respond you timely.   Actually I have very complex XSDs with >2000 elements in nested xsd complex types.  So, above solution would not work in my case. I can not create hive table manually with these number of elements and also Objects nested at 10th level   Sorry, I cannot share the code here but this is how I implemented the project.   Goal: Ingest XMLs data into HDFS and query using Hive/Impala   Solution: Convert XDS into Hive Avro table and keep pumping xml -> avro into hdfs.   I took all XSDs into XML Spy tool and generated sample xml I still had to fix some elements with default values in it because Spark was able to infer more correctly and intelligently. For example “0000” was being inferred to long which is correct as per the values but sine it is in double quotes I would expect it as String and this is how XML Spy generated the default values for alpha numeric fields. Now I have fully curated XML sample file Wrote a Spark-xml code Gave the sample xml as input and converted into Avro file. We know Avro file has schema in it. Took the Avro schema and created Hive table on top of it Finally wrote the Spark job It reads xml files from HDFS At time of reading I am asking Spark to infer xml schema as per my custom schema which I have gotten from sample xml Convert xml into Avro file Write Avro file to HDFS location. Query using Hive/Impala     Thanks, Jai ... View more

Check this https://issues.apache.org/jira/browse/SPARK-4105 ... View more

I am facing same problem, MySQL server is in remote location, they have created user, but I can not access with same user from hadoop cluster, how to solve this? ... View more

Thanks, I indeed end up using Maven and plugins.d folder on Flume. Forgot to update the topic, thank you guys for the help! ... View more

Do not specify driver in sqoop arguments. Using the –driver parameter will always force Sqoop to use the Generic JDBC Connector regardless of if a more specialized connector is available. For example, if the MySQL specialized connector would be used because the URL starts with jdbc:mysql://, specifying the –driver option will force Sqoop to use the generic connector instead. As a result, in most cases, you should not need to use the –driver option at all.     Thanks, Ankit Gaurav Sinha ... View more

Hello, We seem to have a similar problem with impala-shell also on CDH 5.13.2 (and now 5.13.3). We are using Active Directory KDCs with a one-way trust established between a writable Hadoop AD (KRB.DOMAIN.COM) and our main user AD (USERS.DOMAIN.COM).  In addition, our servers are deployed to a third domain (server.domain.com) which does not have an associated KDC. krb5.conf file:   [libdefaults] default_realm = KRB.DOMAIN.COM dns_lookup_kdc = true dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac permitted_enctypes = aes256-cts aes128-cts rc4-hmac udp_preference_limit = 1 kdc_timeout = 3000 rdns = false #default_ccache_name = KEYRING:persistent:%{uid} [realms] KRB.DOMAIN.COM = { kdc = krb.domain.com admin_server = krb.domain.com default_domain = krb.domain.com } USERS.DOMAIN.COM = { kdc = users.domain.com admin_server = users.domain.com } [domain_realm] krb.domain.com = KRB.DOMAIN.COM .krb.domain.com = KRB.DOMAIN.COM users.domain.com = USERS.DOMAIN.COM .users.domain.com = USERS.DOMAIN.COM   Below is how we invoke impala-shell and the tracing statements I'm seeing after manipulating the KRB5_TRACE env variable:   [user1@edgenode ~]$ \impala-shell -k --ssl -i daemonnode.server.domain.com:21000 Starting Impala Shell using Kerberos authentication Using service name 'impala' SSL is enabled. Impala server certificates will NOT be verified (set --ca_cert to change) [22712] 1524768162.661368: ccselect can't find appropriate cache for server principal impala/daemonnode.server.domain.com@ [22712] 1524768162.661450: Getting credentials user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ using ccache FILE:/tmp/krb5cc_738475 [22712] 1524768162.661513: Retrieving user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768162.661558: Retrying user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@USERS.DOMAIN.COM with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768162.661563: Server has referral realm; starting with impala/daemonnode.server.domain.com@USERS.DOMAIN.COM [22712] 1524768162.661627: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success [22712] 1524768162.661633: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768162.661640: Requesting tickets for impala/daemonnode.server.domain.com@USERS.DOMAIN.COM, referrals on [22712] 1524768162.661662: Generated subkey for TGS request: aes256-cts/56A9 [22712] 1524768162.661696: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac [22712] 1524768162.661790: Encoding request body and padata into FAST request [22712] 1524768162.661917: Sending request (9771 bytes) to USERS.DOMAIN.COM [22712] 1524768162.662009: Resolving hostname users.domain.com [22712] 1524768162.701885: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.739233: Sending TCP request to stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.836656: Received answer (351 bytes) from stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.836673: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88 [22712] 1524768164.121113: Response was not from master KDC [22712] 1524768164.121153: Decoding FAST response [22712] 1524768164.121209: TGS request result: -1765328377/Server not found in Kerberos database [22712] 1524768164.121230: Local realm referral failed; trying fallback realm SERVER.DOMAIN.COM [22712] 1524768164.121313: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768164.121366: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success [22712] 1524768164.121372: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768164.121417: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768164.121422: Requesting TGT krbtgt/SERVER.DOMAIN.COM@USERS.DOMAIN.COM using TGT krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768164.121434: Generated subkey for TGS request: aes256-cts/31FF [22712] 1524768164.121454: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac [22712] 1524768164.121530: Encoding request body and padata into FAST request [22712] 1524768164.121623: Sending request (9748 bytes) to USERS.DOMAIN.COM [22712] 1524768164.121649: Resolving hostname users.domain.com [22712] 1524768164.122515: Initiating TCP connection to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.126676: Sending TCP request to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.135666: Received answer (329 bytes) from stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.135673: Terminating TCP connection to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.137693: Response was not from master KDC [22712] 1524768164.137706: Decoding FAST response [22712] 1524768164.137731: TGS request result: -1765328377/Server not found in Kerberos database Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database) *********************************************************************************** Welcome to the Impala shell. (Impala Shell v2.10.0-cdh5.13.3 (15a453e) built on Sat Mar 17 03:48:31 PDT 2018) Want to know what version of Impala you're connected to? Run the VERSION command to find out! *********************************************************************************** [Not connected] > exit; Goodbye user1 I think the issue may be in what service principal impala-shell is requesting a ticket for.  It should be asking for impala/daemonnode.server.domain.com@KRB.DOMAIN.COM but I'm not sure if it ever does. When I obtain the keytab for that service principal and kinit against it, impala-shell connects. Additionally, I've been able to authenticate via Kerberos using Tableau (and the Cloudera ODBC driver), which grants me finer-grain control over what specific service principal needs to be authenticated. ... View more