Member since
05-25-2017
32
Posts
0
Kudos Received
0
Solutions
01-17-2019
07:33 AM
Even, I cant find how one can get transactionId.. Any help greatly appreciated!
... View more
10-24-2018
10:19 AM
Ah! James Jones --- I have a Recommendation - Never believe recommendations from Ambari 😉
... View more
07-24-2018
06:03 AM
+1 I am looking for same exact thing! But strange, cannot find any! 😞
... View more
06-01-2018
02:25 PM
I had a similar problem with pyhive on my horton setup. The failure was always immediate and so it was not a timeout issue that some people on the net were pointing out. It turned out to be hive.server2.transport.mode. If this is set to binary, it works like a Charm. If it is 'http', PyHive does not work. Also found https://github.com/dropbox/PyHive/issues/69 which talks about this. HTH.
... View more
05-04-2018
08:27 AM
We had this issue... What I found from logs is that the livy client copies this zip from /tmp/sparkxxxxx to HDFS:/user/<username>/.sparkStaging/.... In our case, livy user impersonation was enabled. Hence <username> Livy wad able to create ".sparkStaging/<application-id>" directory... but that directory is owned by the <username> with 700 permission.... And then the zip file could not be copied to that directory by Livy. I found this by a writing a small shell script that continuously spins "ls -lR"ing the HDFS /user/<username>/.sparkStaging directory.. (while true:; do; hdfs dfs -ls -R /user/<username>/.sparkStaging; done;) --> Ran this as "livy" user.. So, we just gave blanket permissions in Ranger for "livy" to "/user" recursively. And that was it! We got it working!
... View more
03-13-2018
09:34 AM
Oh yeah! We too hit this problem on our Sandbox cluster running HDP 2.5.3 with Kerberos + AD integrated secure cluster. We can see that Ambari server is not sending the commands to Ambari agent (or) is hung on something with Ambari agent... Just the gear icon in the settings with 3 small icons never appear.... Man... Such a pain, this one is. Today we restarted the services and it took 2 hours for a 4-node cluster.... Hadoop means Elephant, right? Now, I believe it.... for a totally different reason though.... 🙂
... View more
11-09-2017
01:08 AM
Thanks for the info. Pretty useful.
... View more
11-08-2017
11:16 AM
@Jay Kumar SenSharma
Also, we have not installed HDPSearch mpack (or) at least, we have not consciously done.. If it was installed without our notice, is there a way to check whethre HDPSearch is causing the issue?
... View more
11-08-2017
02:34 AM
@Jay Kumar SenSharma I need to upgrade to 2.5.1 from 2.4.2 I need 2.5.1 as it supports HDF 3.x Ambari 2.5.2 does not support HDF 3.x .... So, how do we resolve this paradox? Thanks for the answer much. Appreciate that.
... View more
11-07-2017
03:00 PM
@dmitro vasilenko, Were you able to get past the problem?
... View more
09-21-2017
11:27 AM
A learning I had was that when you export a table, use complete HDFS URI... In some cases, I found that it helped executing the command which otherwise was failed
... View more
09-21-2017
11:24 AM
Does this mean that if I upgrade from 2.5.3 to 2.6.2, I will lose my existing Solr install?
... View more
09-21-2017
10:52 AM
Is Solr removed from HDP install? I see the solr component listed for 2.5.3. So wondering why it has been removed in 2.6
... View more
Labels:
09-20-2017
02:33 PM
We are Currently having HDP version 2.5.3. We have 2.6.0, 2.6.1 and 2.6.2 in front of us. Which is the best version to upgrade? The following points are important to us: 1. Configuration and Data should be unchanged to a large extent 2. New version must be less buggy. We had a tough time with Zeppelin on 2.5.3. Looking out for some experience from people who already done it.
... View more
09-19-2017
11:16 AM
We have a Spark SQL Thrift Server running with Kerberos enabled. While Kerberos helps in authenticate users, I am not sure how I could authorize who could use what tables. We use "Ranger" and it has "hive" policies. I think this is more for "HS2" access. Does this also protect access via Spark-SQL Thrift server?
... View more
Labels:
- Labels:
-
Apache Ranger
-
Apache Spark
09-11-2017
08:54 AM
@Rakesh Gupta Thank you very much. Smart debug sir. Saved me my day today.
... View more
06-28-2017
07:31 AM
What do you mean by "added host entries of the unsecured cluster in the secured cluster"?? Thats kind of weird. I thought we only add Hosts so that they can be a part of the current cluster.... If you did not add the unsecure host, what problem did you encounter?
... View more
06-23-2017
07:58 AM
Ah! Thanks for the input. I will definitely check with them.... Our IDs are numeric and so I see that it is covering some users and is advancing... I hope it will catch up. If not, I will ask the AD admin to check this out. Thanks again for sharing your experience.
... View more
06-23-2017
07:14 AM
We added a new user in our AD and Ranger is not syncing it at all. What could be the potential reason? I ambari-synced the user and the user is now reflected in Ambari users.... but the same user is not visible in Ranger for giving permissions. It has been more than 2 days and the sync has not happened. What should I do? How do I make it see?
... View more
Labels:
- Labels:
-
Apache Ranger
06-09-2017
08:00 AM
@Ram Baskaran Thanks for coming back on this. HDP 2.5.3 is the version in the Hadoop Kerberized Cluster where Falcon server is running. HDP 2.4.2 is running in the Non-Kerberized Cluster So, When I add the cluster entity XML for the Non-Kerberized Cluster, here is the error message I receive. ERROR: Bad Request;default/Invalid storage server or port: hdfs://namenodehostOfNonKBCluster:8020, Cluster definition missing required namenode credential property: dfs.namenode.kerberos.principal
CausedBy: Cluster definition missing required namenode credential property: dfs.namenode.kerberos.principal hdfs://namenodehostOfNonKBCluster:8020 is the verbatim copy of the "fs.defaultFS" property in the "Advanced core-site" config section in the Ambari server running on the Non-Kerberized cluster. Now, How do I tell Falcon not to look for the kerberos principal property? Also, I assume that the "locations" for staging, working etc.. for both the "K" and "NK" cluster are created in the cluster where Falcon server is running. In my setup, this is the "K" cluster. Kindly let me know on this. Thank you very much for your time.
... View more
06-08-2017
05:43 PM
We need to copy data from a non-kerberized (NK) cluster to a kerberized one (K). I am registering entities in "K" cluster. I was able to register the "K" cluster with the falcon server running on "K" cluster without any problems. While registering the "NK" cluster to the falcon server running on "K" , I run into multiple issues. Falcon seems to expect a "kerberos" principal by default. It is not clear if this property is for the "K" cluster or for the "NK" cluster..... Right now, I am using the namenode prinicipal in "K" cluster itself (not sure if this is right). And, while registering the NK entity, an error message says "Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections". ... So , I just added a property like this: " <property name="ipc.client.fallback-to-simple-auth-allowed" value="true" />". But this does not help either. Can some1 help how to do this with Falcon. Also, please kindly point me to some nice tutorials on Falcon. Thank you.
... View more
Labels:
- Labels:
-
Apache Falcon
05-26-2017
08:08 PM
I found the answer to this myself. When you search for ID, it does not necessarily mean "Processor ID". In my case, it was the "Controller Service" ID which was present inside the Processor. Thats a good learning for the day. Thanks much.
... View more
05-26-2017
07:13 PM
Yo! Finally got it working. The "null" pointer exceptions in the log were from different Processors and not from what I was looking at.. I had a workaround from Horton support too. And, I implemented that, tested it and saw data streaming into Hive table. Thanks a lot! Your answer was helpful in getting my understanding right. Thanks a ton for that.
... View more
05-26-2017
07:12 PM
Thank you Sir. I tried that... and when I open the Processor that the search result refers to, it has a different ID... Why would it show me like that? Thanks again, for saving my time. Great feature.
... View more
05-26-2017
05:37 PM
Thanks..I am actually trying to debug a flow written by some1 else... So, I am learning stuff. I see that the "log" mentions the unique ID which I think corresponds to some processor... Now, my job is to find the processor that correspond to the IDs which are creating Null pointer exceptions..... I have been trying to click, click and click to find out... Is there any easier way out?
... View more
05-26-2017
05:04 PM
Ah... Thanks! Can you point me to where I can find these options? I will check NiFi manual anyway. Thanks a lot! Will post back if anything solved our problem
... View more
05-25-2017
10:44 PM
Hey, Need help on Kerberos integration... We have integrated NiFi with Kerberos by configuring it with necessary properties....and we are trying to use "Hive Streaming" processor and it is not working due to Kerberos issue. When I look at the nifi-app.log, I see the following error followed by Validator exception 2017-05-25 16:50:01,749 INFO [StandardProcessScheduler Thread-1] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=638acb69-015b-1000-0000-000037676c20] Hive Security Enabled, logging in as principal null with keytab null The Kerberos Principal and the Keytab files are both "null". Oops oops.... 1. nifi.kerberos.krb5.file is SET and is "accessible" for nifi account 2. nifi.kerberos.service.keytab.location is SET and the file is "accesible" for nifi account 3. nifi.kerberos.service.principal is SET (format is nifi/HOST@ClusterName) SPNEGO authentication is also set. But I guess that is for authenticating with NiFi I believe....(or Correct me here..) So, Any help appreciated. Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi