About sunile_manjee

sunile_manjee · ‎08-07-2018

@Matt Clarke my properties file is owned by nifi user and the content is exactly as you posted but with my creds. However, on nifi it does not like the access key and secret key is empty and here is the properties in the processor: Again the creds are in the Credentials File so why it is asking me to input access key and secret key? those exists in credentials file.

sunile_manjee · ‎08-07-2018

I want to use the credential file in s3putfile so i don’t have to put access key and security key in the processor. However, when provide the path to my credentials file, NiFi complains the access key and security key are null. Isn’t that the entire point to have crendentials in a file instead of residing inside the processor. Feedback/help appreciated.

sunile_manjee · ‎08-07-2018

I am creating a flow in NiFi to migrated to minifi. It is a simple flow (only 2 processors), pulling from weather api and send to s3. When I migrate the flow to minifi, s3putobject credentials (access key and secret key) will be wiped out. How do I apply my access key and secret key in minifi?

sunile_manjee · ‎08-07-2018

Using cloudbreak to launch HDF (specifically NiFi), an error may prevent launching an instance. In the logs (/usr/hdf/current/nifi/logs/nifi-app.log) the following error is produced: pache.nifi.processors.hive.PutHiveStreaming could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) .... Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239) .... This is generally caused by by snappy library extracting to /tmp (generally defined in java.io.tmpdir). To fix this, do one of the following Grant nifi user access to /tmp Create a tmp directory where nifi user has access. For example, create tmp directory /home/tmp Then go into ambari and find parameter Template for bootstrap.conf Scroll past all the java.arg.* arguments Add the following java.arg.snappy=-Dorg.xerial.snappy.tempdir=/usr/hdf/current/nifi/tmp That's it!

sunile_manjee · ‎07-26-2018

After scanning the code here: https://github.com/hortonworks-spark/shc/tree/master/core/src/main/scala/org/apache/spark/sql/execution/datasources/hbase I don’t see any functionality to support HBase delete records. However, there is good example on how to delete HBase record using spark here: https://stackoverflow.com/questions/36183709/delete-hbase-cell-using-spark

sunile_manjee · ‎07-23-2018

I found the issue. Ambari alerts support JMX but only via HTTP. Having JMX alone does not allow for ambari alert to be created. Therefore, what I am trying to do is create ambari alert on kafka where jmx is available through jconsole connnection. However, those metrics are not expose via http and return json result. Basically, it can't be done unless JMX is exposed via HTTP

sunile_manjee · ‎07-19-2018

You are experiencing sparks lazy execution. When you execute your code, nothing in spark has been executed. You need to cache post execution. For example, a easy way to fix/test this is to run a something against your DF... (ie select * from df). store the results in a another DF and cache it thereafter.

sunile_manjee · ‎07-19-2018

@Ilya Li Your question has been answered here by @mpayne: https://community.hortonworks.com/questions/55454/can-nifi-promise-each-of-the-flowfiles-can-be-proc.html

sunile_manjee · ‎07-19-2018

Not according to that Jira. It is well documented.

sunile_manjee · ‎07-19-2018

@Prathamesh H Interesting question. This topic was proposed here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75977362 And default value seems to be implemented in Hive 3 https://issues.apache.org/jira/browse/HIVE-18726 Sunile - When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum.

Online	Offline
Last Visited	‎05-25-2022 10:07 AM

Member Since	‎05-30-2018 10:40 PM
Last Visited	‎05-25-2022 10:07 AM
Posts	1,322
Kudos received	713

Cloudera Community

Re: Iterate over ADLS files using spark?

Re: Install NiFi CA service post nifi cluster inst...

Re: Which storage format is optimum for training m...

Re: Ambari custom alert failing

Re: df.cache() is not working on jdbc table

Re: S3putobject credential file vs access key secr...

S3putobject credential file vs access key secret k...

minifi s3putobject credentials

NiFi org.xerial.snappy.SnappyError

Re: Spark HBase Connector (SHC) - How to delete ...

Re: Ambari custom alert failing

Re: df.cache() is not working on jdbc table

Re: "Exactly One" policy with PutSQL processor

Re: How to create a table with a one of the column...

Re: How to create a table with a one of the column...