Re: Transfer flow files using NiFi Rest API

stellium · ‎01-17-2019

Even, I cant find how one can get transactionId.. Any help greatly appreciated!

stellium · ‎10-24-2018

Ah! James Jones --- I have a Recommendation - Never believe recommendations from Ambari 😉

stellium · ‎07-24-2018

+1 I am looking for same exact thing! But strange, cannot find any! 😞

stellium · ‎06-01-2018

I had a similar problem with pyhive on my horton setup. The failure was always immediate and so it was not a timeout issue that some people on the net were pointing out. It turned out to be hive.server2.transport.mode. If this is set to binary, it works like a Charm. If it is 'http', PyHive does not work. Also found https://github.com/dropbox/PyHive/issues/69 which talks about this. HTH.

stellium · ‎05-04-2018

We had this issue... What I found from logs is that the livy client copies this zip from /tmp/sparkxxxxx to HDFS:/user/<username>/.sparkStaging/.... In our case, livy user impersonation was enabled. Hence <username> Livy wad able to create ".sparkStaging/<application-id>" directory... but that directory is owned by the <username> with 700 permission.... And then the zip file could not be copied to that directory by Livy. I found this by a writing a small shell script that continuously spins "ls -lR"ing the HDFS /user/<username>/.sparkStaging directory.. (while true:; do; hdfs dfs -ls -R /user/<username>/.sparkStaging; done;) --> Ran this as "livy" user.. So, we just gave blanket permissions in Ranger for "livy" to "/user" recursively. And that was it! We got it working!

stellium · ‎03-13-2018

Oh yeah! We too hit this problem on our Sandbox cluster running HDP 2.5.3 with Kerberos + AD integrated secure cluster. We can see that Ambari server is not sending the commands to Ambari agent (or) is hung on something with Ambari agent... Just the gear icon in the settings with 3 small icons never appear.... Man... Such a pain, this one is. Today we restarted the services and it took 2 hours for a 4-node cluster.... Hadoop means Elephant, right? Now, I believe it.... for a totally different reason though.... 🙂

stellium · ‎11-09-2017

Thanks for the info. Pretty useful.

stellium · ‎11-08-2017

@Jay Kumar SenSharma Also, we have not installed HDPSearch mpack (or) at least, we have not consciously done.. If it was installed without our notice, is there a way to check whethre HDPSearch is causing the issue?

stellium · ‎11-08-2017

@Jay Kumar SenSharma I need to upgrade to 2.5.1 from 2.4.2 I need 2.5.1 as it supports HDF 3.x Ambari 2.5.2 does not support HDF 3.x .... So, how do we resolve this paradox? Thanks for the answer much. Appreciate that.

stellium · ‎11-07-2017

@dmitro vasilenko, Were you able to get past the problem?

stellium · ‎09-21-2017

A learning I had was that when you export a table, use complete HDFS URI... In some cases, I found that it helped executing the command which otherwise was failed

stellium · ‎09-21-2017

Does this mean that if I upgrade from 2.5.3 to 2.6.2, I will lose my existing Solr install?

stellium · ‎09-21-2017

Is Solr removed from HDP install? I see the solr component listed for 2.5.3. So wondering why it has been removed in 2.6

stellium · ‎09-20-2017

We are Currently having HDP version 2.5.3. We have 2.6.0, 2.6.1 and 2.6.2 in front of us. Which is the best version to upgrade? The following points are important to us: 1. Configuration and Data should be unchanged to a large extent 2. New version must be less buggy. We had a tough time with Zeppelin on 2.5.3. Looking out for some experience from people who already done it.

stellium · ‎09-19-2017

We have a Spark SQL Thrift Server running with Kerberos enabled. While Kerberos helps in authenticate users, I am not sure how I could authorize who could use what tables. We use "Ranger" and it has "hive" policies. I think this is more for "HS2" access. Does this also protect access via Spark-SQL Thrift server?

stellium · ‎09-11-2017

@Rakesh Gupta Thank you very much. Smart debug sir. Saved me my day today.

stellium · ‎06-28-2017

What do you mean by "added host entries of the unsecured cluster in the secured cluster"?? Thats kind of weird. I thought we only add Hosts so that they can be a part of the current cluster.... If you did not add the unsecure host, what problem did you encounter?

stellium · ‎06-23-2017

Ah! Thanks for the input. I will definitely check with them.... Our IDs are numeric and so I see that it is covering some users and is advancing... I hope it will catch up. If not, I will ask the AD admin to check this out. Thanks again for sharing your experience.

stellium · ‎06-23-2017

We added a new user in our AD and Ranger is not syncing it at all. What could be the potential reason? I ambari-synced the user and the user is now reflected in Ambari users.... but the same user is not visible in Ranger for giving permissions. It has been more than 2 days and the sync has not happened. What should I do? How do I make it see?

stellium · ‎06-09-2017

@Ram Baskaran Thanks for coming back on this. HDP 2.5.3 is the version in the Hadoop Kerberized Cluster where Falcon server is running. HDP 2.4.2 is running in the Non-Kerberized Cluster So, When I add the cluster entity XML for the Non-Kerberized Cluster, here is the error message I receive. ERROR: Bad Request;default/Invalid storage server or port: hdfs://namenodehostOfNonKBCluster:8020, Cluster definition missing required namenode credential property: dfs.namenode.kerberos.principal CausedBy: Cluster definition missing required namenode credential property: dfs.namenode.kerberos.principal hdfs://namenodehostOfNonKBCluster:8020 is the verbatim copy of the "fs.defaultFS" property in the "Advanced core-site" config section in the Ambari server running on the Non-Kerberized cluster. Now, How do I tell Falcon not to look for the kerberos principal property? Also, I assume that the "locations" for staging, working etc.. for both the "K" and "NK" cluster are created in the cluster where Falcon server is running. In my setup, this is the "K" cluster. Kindly let me know on this. Thank you very much for your time.

stellium · ‎06-08-2017

We need to copy data from a non-kerberized (NK) cluster to a kerberized one (K). I am registering entities in "K" cluster. I was able to register the "K" cluster with the falcon server running on "K" cluster without any problems. While registering the "NK" cluster to the falcon server running on "K" , I run into multiple issues. Falcon seems to expect a "kerberos" principal by default. It is not clear if this property is for the "K" cluster or for the "NK" cluster..... Right now, I am using the namenode prinicipal in "K" cluster itself (not sure if this is right). And, while registering the NK entity, an error message says "Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections". ... So , I just added a property like this: " <property name="ipc.client.fallback-to-simple-auth-allowed" value="true" />". But this does not help either. Can some1 help how to do this with Falcon. Also, please kindly point me to some nice tutorials on Falcon. Thank you.

stellium · ‎05-26-2017

I found the answer to this myself. When you search for ID, it does not necessarily mean "Processor ID". In my case, it was the "Controller Service" ID which was present inside the Processor. Thats a good learning for the day. Thanks much.

stellium · ‎05-26-2017

Yo! Finally got it working. The "null" pointer exceptions in the log were from different Processors and not from what I was looking at.. I had a workaround from Horton support too. And, I implemented that, tested it and saw data streaming into Hive table. Thanks a lot! Your answer was helpful in getting my understanding right. Thanks a ton for that.

stellium · ‎05-26-2017

Thank you Sir. I tried that... and when I open the Processor that the search result refers to, it has a different ID... Why would it show me like that? Thanks again, for saving my time. Great feature.

stellium · ‎05-26-2017

Thanks..I am actually trying to debug a flow written by some1 else... So, I am learning stuff. I see that the "log" mentions the unique ID which I think corresponds to some processor... Now, my job is to find the processor that correspond to the IDs which are creating Null pointer exceptions..... I have been trying to click, click and click to find out... Is there any easier way out?

stellium · ‎05-26-2017

Ah... Thanks! Can you point me to where I can find these options? I will check NiFi manual anyway. Thanks a lot! Will post back if anything solved our problem

stellium · ‎05-25-2017

Hey, Need help on Kerberos integration... We have integrated NiFi with Kerberos by configuring it with necessary properties....and we are trying to use "Hive Streaming" processor and it is not working due to Kerberos issue. When I look at the nifi-app.log, I see the following error followed by Validator exception 2017-05-25 16:50:01,749 INFO [StandardProcessScheduler Thread-1] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=638acb69-015b-1000-0000-000037676c20] Hive Security Enabled, logging in as principal null with keytab null The Kerberos Principal and the Keytab files are both "null". Oops oops.... 1. nifi.kerberos.krb5.file is SET and is "accessible" for nifi account 2. nifi.kerberos.service.keytab.location is SET and the file is "accesible" for nifi account 3. nifi.kerberos.service.principal is SET (format is nifi/HOST@ClusterName) SPNEGO authentication is also set. But I guess that is for authenticating with NiFi I believe....(or Correct me here..) So, Any help appreciated. Thanks!

Online	Offline
Last Visited	‎01-17-2019 09:21 AM

Date Registered	‎05-25-2017 10:31 PM
Last Visited	‎01-17-2019 09:21 AM
Total Messages Posted	32

Re: Transfer flow files using NiFi Rest API

Re: How can I change the AMS hbase.rootdir in embe...

Re: How to list all domains, or entitytypes in the...

Re: pyhive connection error: thrift.transport.TTra...

Re: __spark_conf__.zip is not found

Re: Ambari is taking long time to restart slave se...

Re: Ambari-server upgrade from hdf-2.0 to hdf-3....

Re: Ambari-server upgrade from hdf-2.0 to hdf-3....

Re: Ambari-server upgrade from hdf-2.0 to hdf-3....

Re: Ambari-server upgrade from hdf-2.0 to hdf-3....

Re: EXPORT HIVE Database

Re: HDP 2.6 does not list Solr component.

HDP 2.6 does not list Solr component.

Best HDP version to upgrade

How do I authorize users for access of Spark SQL T...

Re: Can we not have HS2 and Spark Thrift Server (S...

Re: could not list the files in remote HDFS cluste...

Re: Ranger user-sync is not happening even after r...

Ranger user-sync is not happening even after resta...

Re: How to use Apache falcon to copy data from Non...

How to use Apache falcon to copy data from Non-Ker...

Re: NiFi Kerberos Integration

Re: NiFi Kerberos Integration

Re: NiFi Kerberos Integration

Re: NiFi Kerberos Integration

Re: NiFi Kerberos Integration

NiFi Kerberos Integration