@A C Just to understand, did you run the spark submit using yarn cluster as master/deploy mode? If so, let's try to check the job properties for the following parameter: ${resourceManager} Also, here it is another example regarding pyspark + oozie (using shell to submit spark). https://github.com/hgrif/oozie-pyspark-workflow Hope this helps ... View more

Quick question, does it work running outside of oozie? E.g. using directly the spark-submit. ... View more

Hello @n c! Would be like this: insert into testtbl select * from(select named_struct('tid','id21314','action','some action','createdts',current_timestamp()), '232402909', '123091203910') as dummy; Hope this helps. ... View more

Hello @Cody kamat! Could you run an explain in one of your queries and share with us the result? btw, a couple of questions related your queries: - Are you using TEZ? Or LLAP? - Are you using mapJoin? Vertex enable? - Is CBO enable as well? - Your tables has partitions/buckets? - Are you using any fileformat like (parquet, avro, orc)? Thanks ... View more

Hello @Kapil Kaushik! Did you check the path for the flow file of this specific node? If not, could you check with the following steps? #First identify where is the location of your nifi.properties being used by nifi process [root@node2 ~]# ps -ef | grep -i nifi.properties nifi 1319462 1319414 4 Sep07 ? 06:23:51 /usr/jdk64/jdk1.8.0_112/bin/java -classpath /usr/hdf/current/nifi/conf:/usr/hdf/current/nifi/lib/javax.servlet-api-3.1.0.jar:/usr/hdf/current/nifi/lib/jcl-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-nar-utils-1.5.0.3.1.2.0-7.jar:/usr/hdf/current/nifi/lib/jetty-schemas-3.1.jar:/usr/hdf/current/nifi/lib/jul-to-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/log4j-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-properties-1.5.0.3.1.2.0-7.jar:/usr/hdf/current/nifi/lib/logback-classic-1.2.3.jar:/usr/hdf/current/nifi/lib/logback-core-1.2.3.jar:/usr/hdf/current/nifi/lib/nifi-api-1.5.0.3.1.2.0-7.jar:/usr/hdf/current/nifi/lib/nifi-framework-api-1.5.0.3.1.2.0-7.jar:/usr/hdf/current/nifi/lib/slf4j-api-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-runtime-1.5.0.3.1.2.0-7.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dambari.application.id=nifi -Dambari.metrics.collector.url=http://node4:6188/ws/v1/timeline/metrics -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/usr/hdf/current/nifi/conf/nifi.properties -Dnifi.bootstrap.listen.port=45115 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi -K /usr/hdf/current/nifi/conf/sensitive.key #Then let's check the config which says where's the flow.xml.gz should be [root@node2 ~]# cat /usr/hdf/current/nifi/conf/nifi.properties | grep -i flow.xml nifi.flow.configuration.file=/var/lib/nifi/conf/flow.xml.gz After getting the path used for the flow.xml, try to replace it with your healthy flow.xml.gz. Hope this helps! ... View more

Hello @Teresa Tavernelli! Glad to know that you made it 🙂 So regarding the Mysql, you can give it a shot with the following parameters: hostname -> your-sandbox-hostname port -> 3306 user -> root ps: I didn't test it myself Also if it doesn't work, try with the hive configs for mysql (instead of using username as root, change it to hive) Hope this helps! ... View more

Hello @n c! Afaik you can use Avro with binary encoded since this binary encoded content has avro compatibility. To further details, take a look at this link: https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ You can give it a shot with this avro tools, to figure if your binary date coming from Kafka/Flume has avro schema embedded into the data. Lastly, take a look at this link, it says which encoding Avro accepts as data. https://avro.apache.org/docs/1.8.1/spec.html#Encodings Hope this helps ... View more

Hello @PPR Reddy. Did you try to escape by passing it's hexadecimal value? https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_large_objects I didn't give it a shot, but I'd say would be like this: sqoop import <YOUR ORACLE CONN STRING> --fields-terminated-by \0xD --the representation of ^M in hexdecimal Hope this helps! ... View more

Excellent explanation Matt! ... View more

Also, found this link: https://help.pentaho.com/Documentation/7.1/0H0/Set_Up_Pentaho_to_Connect_to_a_Hortonworks_Cluster Not sure if it's gonna help you 🙂 ... View more

Hi @Teresa Tavernelli! Could you try one thing? Test to change the localhost for the name of the sandbox machine (should be something like this: sandbox-hdp.hortonworks.com:10000). Since I'm not aware of Pentaho tools, do you've any config from Kettle where you can put the hive-site.xml (configs for Hive). Hope this helps! ... View more

Hello @Sai Krishna Makineni! I found a link with a similar error to yours https://community.hortonworks.com/content/supportkb/49525/receiving-error-javalangnullpointerexception-and-o.html Hope this helps ... View more

Hello @jessica Chen! Just to confirm, which language are you using in your OS? I'm asking this cause I've spot this link: https://community.hortonworks.com/questions/23409/there-is-a-problem-when-install-hdp-on-the-stepcon.html Hope this helps! ... View more

Hello @John King! Not sure if the issue persists, but I have a few questions related to your issue: - Are you running MiNifi as a service or agent (through bat) - If you're running through bat, are you running the cmd/PowerShell as administrator? Btw, could you enable the DEBUG mode for the TailFile? And share with us the logs? E.g. <logger name="org.apache.nifi.processors.standard.TailFile" level="DEBUG"/> Lastly, If you're able to make a quick test and install cygwin in the windows server, are you able to tail the file with the same user that's running minifi? Hope this helps ... View more

Hello @Thanuja Kularathna ! Did you try to add these properties in Custom HDFS-site? You can do it by going to: Ambari > HDFS > Config > Advanced > Custom hdfs-site > Go to the bottom of the Custom hdfs-site and Add Property And then save the new properties added and restart the service. Hope this helps! ... View more

Hi @A C. At first, glance, I can't see anything misconfig. Take a look at this article, to see if helps you on smtg: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.html Hope this helps ... View more

Hi @A C. You're right, this is the YARN WEB UI 🙂 Hm, so from what I can see, it looks like yarn didn't launch your spark application. Do you mind to share with us your oozie workflow xml? Thanks. ... View more

Hmm, @tauqeer khan good question. I've never tested it myself. But at first, glance, guess you can take a look at the following answer: https://community.hortonworks.com/questions/182344/how-to-copy-data-from-a-hive-table-recurrently-usi.html And try to build something similar 🙂 BTW, it's a good idea to test it, let us know if it works or if you face anything. Hope this helps! ... View more

Hello @A C. What do you see in the YARN UI? Is there any application_id running for your oozie workflow/Spark Job? Thanks. ... View more