About sindhu

sindhu · ‎01-17-2019

Outdated , Finally after doing these steps I get curl: (35) error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown Not working as of 2019 with HDP 2.6.3

sindhu · ‎12-31-2018

How can adjust nifi to change the username / permissions to 777 atleast . I think I have seen @Shu told few solution to tackle the problem However I cannot find this now .

sindhu · ‎12-29-2018

Hello everybody, Sorry for the basic question . We are processing the data and pushing the data to Hive using Puthive3streaming in Nifi . However it writing the data as nifi:hadoop as owner and group names respectively . Is there anyway I can change the owners to my userid:myorgID in Puthive3streaming in orc format. The problem here is we are unable to select the data in hive due to permission issue . select count(*) from tbl results in permission denied nifi:hadoop . How would you tackle the problem . Thanks ALL

sindhu · ‎12-22-2018

I have enabled the debug mode and Logs are clean

sindhu · ‎12-21-2018

Hello everybody, Table properties: -> Tried both Internal and External tables -> Stored as ORC -> 777 permission recursively for all the folders. -> executing the table as the same owner of the table -> transactional true(internal table only) ->orc compress zlib -> msck repair works fine . showing partition values and size of the folders are same as in prod I have moved the data from Server A to Server B and Server C to HDFS and built a table on top of hdfs data. All the three servers are in HDP 3.1 . Server A it is production server which is working fine from initial setup. moved the data to dev and test server respectively. Server B is test server which seems to be working without any change next day . Server C is Dev server is not at all working after three days . HS2 configs are compared . almost same for across the servers It is really strange to handle these type of scenario . Can someone help me in this

sindhu · ‎12-08-2018

Hello everybody, We are planning to add a new HDFS disk to existing cluster.How can we do that.

sindhu · ‎11-15-2018

Thanks @Matt Burgess . I would definitely have a look and get used to it . I'm thinking how do you identify the output of the processor . My flow which is working 2.x is listfile-fetchfile-customprocessor-partitionrecord-mergecontent-puthivestreaming. As you suggested I have added 1.7.1 Nar file . I apologies It HDF3.1 not HDF 3.2 . So basically you are asking me to use this path right -->listfile-fetchfile-customprocessor-partitionrecord-mergecontent-puthivestreaming(Avroreader,embdededschema)->puthive3streaming(Avroreader) . I believe I can use the the mergecontent->PHS alone instead of mergecontent->PH3S .. Getting o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutHive3Streaming[id=1666c555-0167-1000-0000-0000763cd449] due to uncaught Exception: java.lang.NullPointerException java.lang.NullPointerException: null and permission denied error . However I have given 777 permission . still getting permission denied error . HDF latest version seems to be a headache for the users.nifi.txt -rw-r-----1 nifi nifi 197MJul1213:41 nifi-hive3-nar-1.7.1.nar -rw-r-----1 nifi nifi 15KAug715:26 nifi-hive-services-api-nar-1.7.0.3.2.0.0-520.nar -rw-r-----1 nifi nifi 106MAug715:26 nifi-hive-nar-1.7.0.3.2.0.0-520.nar -rw-r-----1 nifi nifi 260MAug715:27 nifi-hive3-nar-1.7.0.3.2.0.0-520.nar

sindhu · ‎11-14-2018

Sure . Thanks for the Reply@Matt Burgess . As of now I'm using HDP Hive along with HDF Nifi . However I can see that Recordreader is included in the PutHive3Streaming . What would be option I need to use for PutHive3Streaming. This is seems little new to me. Do you have any sample document to configure PutHive3Streaming. Current version is HDF 3.2

sindhu · ‎11-14-2018

Hello Everybody , I am currently trying to test a flow using the PutHiveStreaming processor in NiFi-1.7.x . I'm getting the following error . However I'm able to execute the same flow in another/lower environment . It is not working in Prod environment and throws me this error in the Nifi-app.log kindly help me is that the problem with Latest HIveserver2 in HDP or is it nifi specific problem . Do I need to use PutHive3streaming instead of PutHivestreaming 2018-11-14 06:26:23,553 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e34c533f-04c8-37af-b523-02eee3925269] Error connecting to Hive endpoint: table performanc e_metrics at thrift://prodcluster:9083 2018-11-14 06:26:23,553 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e34c533f-04c8-37af-b523-02eee3925269] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://prodcluster:9083', database='renamed.db', table='renamed', partitionVals=[santaclara] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.a pache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://prodcluster:9083', database='renamed.db', table='renamed', partitionVals=[LLO1Y] } org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: F ailed connecting to EndPoint {metaStoreUri='thrift://prodcluster:9083', database='renamed.db', table='renamed, partitionVals=[santaclara] } at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:630) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:648) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:839) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2211) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2179) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:792) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:658) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:658) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://prodcluster:9083', database='renamed.db', table='renamed', parti tionVals=[] } at org.apache.nifi.util.hive.HiveWriter.newConnection(HiveWriter.java:249) at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:70) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46) at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1138) at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1049) at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113) at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:842) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) ... 18 common frames omitted

sindhu · ‎11-14-2018

Hello All , Thanks for helping us!! Our team planning to do fresh installation of the HDP 3.0.x in the machine . As of now we have HDP 2.6.x . We have several applications running inside the cluster and so many configurations has been modified over the period of time. Is there any way I can take the backup of the configurations from 2.6.x and push it to 3.x while during the install or after the Installation . Sorry for the noob question : Is it possible via Blueprints or how can we use BP as a best practise.

Online	Offline
Last Visited	‎03-12-2019 11:59 AM

Date Registered	‎11-14-2018 07:48 PM
Last Visited	‎03-12-2019 11:59 AM
Total Messages Posted	13
Total Kudos Received	2

Re: Rest call to ranger on wire encrypted cluster

Re: puthive3streaming permissions issue

puthive3streaming permissions issue

Re: Hive is unable to read the HDFS

Hive is unable to read the HDFS

Add a new HDFS disk to HDP 3.x

Re: Problem with PutHiveStreaming in HDP 3.x

Re: Problem with PutHiveStreaming in HDP 3.x

Problem with PutHiveStreaming in HDP 3.x

Fresh Installation of HDP 3.x