Re: Storm Run time issues

Srikaran · ‎01-11-2018

Agreed. Thanks for suggestion. For now it seems I have a work around by changing the run schedule from 0 seconds to 1 seconds and I dont see Lease holder exception. Even though there is a little latency in writing to HDFS unlike 0 seconds but error has gone. I will work on your suggestion for production. Thanks for help! Srikaran

Srikaran · ‎01-11-2018

@Bryan Bende I liked MergeContent option as you suggested. But Please clarify this. In production surveys will come real time as soon as customer write the survey we want to see in HDFS. So my use-case is during 24 hour period which is per day I want to see only 1 file in HDFS and as soon as Surveys were posted I should see that survey in HDFS. If I use Merge Content processor will that be still considered Real-Time? I am guessing it will wait until data reach certain threshold, upon which merge will happen and write to HDFS? During a day there will be times where no surveys at all or bunch of surveys coming at the same time or 1 survey per second. Thanks Srikaran.

Srikaran · ‎01-11-2018

@Bryan BendeHi Bryan. We are testing this in DEV and it has only 1 NIFI Node. However the puthdfs cluster has 4 datanodes. Prod we will have 2 nifi nodes and 5 datanodes. Thanks

Srikaran · ‎01-11-2018

HI We have a NIFI flow where we are sourcing the social media surveys from an API and writing them to HDFS via PutHDFS processor in with conflict resolution strategy as "append". This flow works if surveys are coming 1 by 1 with a second or 2 seconds delay. We want to test some 20000 surveys all coming at once and "PutHDFS" processor is failing for this scenario. Error is given below: WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.append: failed to create file XXXXXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXXX because current leaseholder is trying to recreate file. org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user@XXXXXXXXX (auth:KERBEROS) cause:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file XXXXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXX for client XXXXXXXX because current leaseholder is trying to recreate file. INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.append from XXXXXXXX Call#XXXXX Retry#0: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file XXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXX because current leaseholder is trying to recreate file. With these exception all the records are getting blocked in nifi queue to puthdfs and eventually they are not writing into HDFS. Is there a way to configure Nifi PutHDFS processor to accomodate this use-case? Rt now its configured under scheduling as "Timer Driven", Concurrent tasks as "1" and with run schedule as 0 seconds. Yield duration is 1 second. Please suggest. Thanks Srikaran

Srikaran · ‎01-04-2018

@Karl Fredrickson Hi Karl..Same issue after Stop and restart. I tried 1 hour and 4 hours for Kerberos relogin period as I am using the same relogin period for FetchHDFS/ListHDFS. This is happening only for "GetHDFS". I am assuming "GetHDFS" processor is trying to delete/move or write which might need some other permissions. The HDFS files are owned by hive:hive with 771 permissions. With the same 771 permissions and hive:hive fetchhdfs & listhdfs is working. Thanks

Srikaran · ‎01-04-2018

Hi I am using FetchHDFS nifi processor which is running fine to fetch the exact HDFS file. I want to get all HDFS files under a directory hence using GetHDFS by keeping the source file option as "True". But I am getting a Kerberos error saying "ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=XXXXXXXXXX] Error retrieving file hdfs://XXXXXXXXXXXXXXXXXXXX.0. from HDFS due to java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt): {} java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:311) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) I am wondering why Same Kerberos credentials are working for "FetchHDFS/ListHDFS" but not "GetHDFS". "GetHDFS" need additional setup? Please suggest. Thanks Srikaran

Srikaran · ‎12-06-2017

@Timothy Spann Thanks a lot. These are very helpful, Let me test the flow and will update accordingly. Thanks

Srikaran · ‎12-06-2017

@anarasimham Looks like GetHDFS will replace HDFS file. I am planning to use fetchHDFS and then invoke http processor. For now I am converting avro file to JSON on Hadoop end and fetching the json and posting it. I will directly test avro & other formats and will update. Thanks!

Srikaran · ‎12-04-2017

Hello. I have a HDFS file for which data needs to be posted to an outside URL (https), I have the user name and password for the URL; I can post a sample JSON via postman from my browser by using the user name and password. Now I have to use Ni-FI for this flow. Please let me know what are the exact nifi processors should I use to get the data from HDFS and post it into the URL via another ni-fi processor. Also kindly let me know what format the HDFS data should be in for these kind of use-cases. Thanks Srikaran

Srikaran · ‎09-14-2016

@Predrag Minovic Great options. It looks like from all the options above the 2nd ZK quorum should be installed manually outside Ambari and configure the Kafka accordingly? If that's the case when I do upgrades in future on this cluster I have to take care of 2nd manual ZK quorum upgrade as a separate effort rt? And I like the 2 clusters solution but what if some business logic on cluster 1 is dependent on kafka on cluster 2? In that case I guess "2 clusters " solution will not work rt? Please confirm! Thanks Sri.

Srikaran · ‎09-13-2016

Hi I am planning to build a HDP 2.4.2 Kerberized cluster via Ambari Blue-Prints and I am going to change the blue print to have 6 Zookeepers. The reason why I am having 6 zk's is I want to have two ZK quorums with 3 ZK's, 1 quorum I want to use for HDFS NN HA, Hbase and other services except for Kafka and for Kafka alone I want to have other ZK Quorum dedicated. I am assuming when I build the cluster with 6 ZK's initially I guess it will create only 1 ZK quorum with 6 ZK's in it. Can I change it to have to 2 ZK quorums after cluster installation from zkcli? or is there an option in Ambari blue print itself to create 2 ZK quorums with 3 ZK servers in each quorum? Please advice! Thanks

Srikaran · ‎06-28-2016

Alright @milind pandit will try it. Thanks!

Srikaran · ‎06-27-2016

@milind pandit I see topology.max.spout.pending

Srikaran · ‎06-27-2016

@milind pandit This is what I am giving for worker.childopts! Please correct if you see something weird.

Srikaran · ‎06-27-2016

Hi We are hitting the below Storm Run Time errors as below:

Srikaran · ‎06-07-2016

@Benjamin Leonhardi Thanks, This makes sense, so its always better to set the value to "True" rt?

Srikaran · ‎06-07-2016

Let me put the question this way. If I have hive.execution.engine=tez; why do I need the property hive.server2.tez.initialize.default.sessions to set it to "True"? Whats the use-case for this property? I ran multiple tests but my hive.execution.engine property drives how the query works and not this default sessions property.

Srikaran · ‎06-07-2016

@Ted Yu Thanks it worked.

Srikaran · ‎06-07-2016

Accepting this answer as well!

Srikaran · ‎06-07-2016

@Ted Yu Error message shows 60 seconds as default. Do we need to bump it to 90 seconds? thats what you mean? I dont see this property currently set so I guess its taking from default. Please confirm.

Srikaran · ‎06-07-2016

@Ted Yu Region servers are good.

Srikaran · ‎06-07-2016

Error: org.apache.hadoop.hbase.client.ScannerTimeoutException: 70218ms passed since the last invocation, timeout is currently set to 60000 at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:434) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364) at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:205) at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:147) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:216) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556) at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at

Srikaran · ‎06-07-2016

Sure. Thanks @Ted Yu

Srikaran · ‎06-07-2016

@Enis This worked. We just omitted TTL and created table. After that we did a Describe table and see TTL FOREVER. Thanks Enis

Srikaran · ‎06-06-2016

@Ted Yu Same error Ted. Please see below: ERROR: For input string: "MAX_VALUE"

Srikaran · ‎06-06-2016

@Ted Yu Please see above

Srikaran · ‎06-06-2016

We are trying to create the table in Hbase shell as below: Pasted the error. ERROR: For input string: "FOREVER" After this error we created table as below by removing TTL and adding hbase.store.delete.expired.storefile and it succeeded. CONFIGURATION => {'hbase.store.delete.expired.storefile' => 'false'}}

Srikaran · ‎06-06-2016

If both are not same how they are different? Can you please let me know?

Srikaran · ‎06-06-2016

Because customer is asking TTL => 'FOREVER' worked for them in HDP 2.2 and its not working in HDP 2.4 while creating the Hbase tables, So they have to give CONFIGURATION => {'hbase.store.delete.expired.storefile' => 'false'} instead of TTL.

Srikaran · ‎06-06-2016

Online	Offline
Last Visited	‎09-21-2016 04:39 PM

Date Registered	‎07-07-2016 01:28 PM
Last Visited	‎09-21-2016 04:39 PM
Total Messages Posted	53
Total Kudos Received	6

Re: Storm Run time issues

Re: putHDFS NIFI processor is getting failed with ...

Re: putHDFS NIFI processor is getting failed with ...

Re: putHDFS NIFI processor is getting failed with ...

putHDFS NIFI processor is getting failed with Leas...

Re: Kerberos issue for GetHDFS But Not FetchHDFS/L...

Kerberos issue for GetHDFS But Not FetchHDFS/ListH...

Re: NI-FI processors to get a file from HDFS and i...

Re: NI-FI processors to get a file from HDFS and i...

NI-FI processors to get a file from HDFS and invok...

Re: configuration of multiple zookeeper quorums on...

configuration of multiple zookeeper quorums on a s...

Re: Storm Run time issues

Re: Storm Run time issues

Re: Storm Run time issues

Storm Run time issues

Re: Hi..whats the difference between "Start Tez se...

Re: Hi..whats the difference between "Start Tez se...

Re: Hbase Copy table fails for 250 Million Records...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Re: Hbase Copy table fails for 250 Million Records...

Re: Hbase Copy table fails for 250 Million Records...

Hbase Copy table fails for 250 Million Records fro...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Hi..whats the difference between "Start Tez sessio...

Re: Hi..Is anything changed on TTL => 'FOREVER' on...

Hi..Is anything changed on TTL => 'FOREVER' on HDP...