About sshringi003

sshringi003 · ‎08-05-2019

I want to merge source records from hive existing records based on some key and if records exist in hive table then only update the value of particular records. Example : i have a table in hive "Test" with records-1 (id : 1, name : abc, age : 20), records-2 (id : 2, name : xyz, age : 22) and i am getting the records from source (live sensor ). The same records i am getting from the sensor and doing some merge based on id, so if the records matched with existing records from hive table then only update new value else insert the key and value both. I want the way in java coding only, query i have for merging the records. merge into customer_name using ( select * from all_updates) sub on sub.id = customer_name.id when matched then update set name = sub.name, state = sub.state when not matched then insert values (sub.id, sub.name, sub.state); I did in java but not finding the way how to resolve this issue. Could anybody suggest me the way. Thanks.

sshringi003 · ‎01-04-2019

Thanks @Matt Clarke for quick repl. As per "nifi-bootstrap.log" i am getting ################## org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 21089 org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] org.apache.nifi.bootstrap.Command Apache NiFi is not running As per "nifi-app.log" i am getting ##################### ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection refused (Connection refused) java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100) at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83) at org.apache.nifi.NiFi.<init>(NiFi.java:102) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:296) 2019-01-03 10:14:26,252 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2019-01-03 10:14:26,253 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise). 2019-01-03 10:20:48,165 INFO [main] org.apache.nifi.NiFi Launching NiFi... 2019-01-03 10:20:48,509 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/root/nifi-1.8.0/./conf/nifi.properties' Now could you ple suggest where to and what to change in properties files. Thanks.

sshringi003 · ‎01-03-2019

Hi, I am running NiFi on centos-7, but its not running. I tried many times but the same issue came. I used command ./bin/nifi.sh restart and the output came "org.apache.nifi.bootstrap.Command Apache NiFi is not running" . If i am missing anything pls let me know. Java home: /usr/lib/jvm/java-1.8.0-openjdk NiFi home: /root/nifi-1.8.0 Thanks.

sshringi003 · ‎10-09-2018

Its working fine now . Thank you @Jay Kumar SenSharma

sshringi003 · ‎10-09-2018

@Jay Kumar SenSharma Thanks for your quick reply. I tried telnet $HS 10000 and nc -v $HS 10000, but nothing is coming. When i run the command "nc -v $HS Host 10000", i got only this Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connection refused. Then tried netstat -tnlpa | grep 10000 but nothing is coming. When i run i got below error ls -l /var/log/hive/hiveserver2.log Caused by: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=10096, maxMemory=7936. Could you pls let me know more about it.

sshringi003 · ‎10-09-2018

Hi, I installed HiveServer2 on ambari 2.6. Whlie accessing hive through terminal or webUI, its not working. The error is comming Error: Could not open client transport with JDBC Url: jdbc:hive2://ip:10000/transportMode=binary:Java.net.ConnectionException (Connection Refused). SET hive.exec.dynamic.partition = true; SET hive.exec.dynamic.partition.mode = nonstrict; SET hive.exec.max.dynamic.partitions = 10000; SET hive.exec.max.dynamic.partitions.pernode = 1000; these are the configuration which is available on config file. Could anybody help regarding this issue would be great. Thanks.

sshringi003 · ‎09-25-2018

@Geoffrey Shelton Okot it is also taking the same time. I run query select count(ayx) from test_table. It took 17 Sec. and while i run select count(1) from test_table, the time remains same. Could you please explore more about hive query fast processing. Thanks

sshringi003 · ‎09-08-2018

Hi thanks for quick reply @Felix Albani I tried but i didn't find out any way to connect. I am getting below error. Opened transport Fatal error: Uncaught exception 'Thrift\Exception\TApplicationException' with message 'Invalid method name: 'OpenSession'' in /var/www/html/hs2/hive/org/apache/hive/service/cli/thrift/TCLIService.php:89 Stack trace: #0 /var/www/html/hs2/hive/org/apache/hive/service/cli/thrift/TCLIService.php(56): org\apache\hive\service\cli\thrift\TCLIServiceClient->recv_OpenSession() #1 /var/www/html/hs2/HS2Client.php(40): org\apache\hive\service\cli\thrift\TCLIServiceClient->OpenSession(Object(org\apache\hive\service\cli\thrift\TOpenSessionReq)) #2 {main} thrown in /var/www/html/hs2/hive/org/apache/hive/service/cli/thrift/TCLIService.php on line 89 Could you pls let me know any solution.

sshringi003 · ‎09-07-2018

Hi, I have 4 node HDPcluster . In that second node I installed Hiveserver2 and apache lamp server. I want to connect HiveServer2 to PHP. Is there any way to connect. I tried but unable to resolve the issue. Could anybody suggest any idea or blog. Thanks.

sshringi003 · ‎09-06-2018

Hi @Vincent Jiang, How you solved this issue. Bcz i am facing same issue on Ambari server. could you post your config or solution here. Thanks.

sshringi003 · ‎09-05-2018

Thanks for such a nice community. I run bin/nifi.sh run and now i am able to run NiFi in background. Thanks @Matt Clarke

sshringi003 · ‎09-05-2018

sorry @Matt Clarke thanks for quick response. I was missed to put dot(.). I am using NiFi version 1.6.0. And my question is same which you posted.

sshringi003 · ‎09-05-2018

Hi, I installed NiFi 16.6 is my local system. I want, the NiFi should be working in background after shutdown the system and again starting the system we can directly open it through browser. No need to start again and again. So is there any way in NiFi so that i can make it 24*7 up-time. Help would be great appreciated. Thanks.

sshringi003 · ‎08-29-2018

Hi, I want to do below process in NiFi. Step 1) ConsumeMQTT using mosquitto Step 2) UpdateAttributes Step 3) EvaluateJsonPath Step 4) SplitJson - will produce the separate flowfile for each JSON array Step 5) PutHbaseJSON --> Store the Json Array to HBASE Hope you can now simulate the same or i'm having json array as {"weather": [ {"id":"1","latitude":"13.08","longitude":"80.27","temprature":"39","pressure":"1009mb","visibility":"7km","wind":"4mph","cloudcover":"50%","momentmagnitude":"9.1"}, {"id":"2","latitude":"14.08","longitude":"80.27","temprature":"39","pressure":"1009mb","visibility":"7km","wind":"4mph","cloudcover":"50%","momentmagnitude":"9.1"}, {"id":"3","latitude":"15.08","longitude":"80.27","temprature":"39","pressure":"1009mb","visibility":"7km","wind":"4mph","cloudcover":"50%","momentmagnitude":"9.1"}, {"id":"4","latitude":"16.08","longitude":"80.27","temprature":"39","pressure":"1009mb","visibility":"7km","wind":"4mph","cloudcover":"50%","momentmagnitude":"9.1"}, {"id":"5","latitude":"17.08","longitude":"80.27","temprature":"39","pressure":"1009mb","visibility":"7km","wind":"4mph","cloudcover":"50%","momentmagnitude":"9.1"} ] } i'm trying to insert the same to hbase, how to get the each array into hbase row. Any help would be appreciated, Thanks !!

sshringi003 · ‎08-29-2018

Hi @Patrick Maggiulli I tried the same flow, i am putting the data into HBase from HTTP. I have one CSV file that contains field (ID,Movie,Type), In GetFile processor i am taking this file and flow remains same as yours. In UpdateAttribute i am giving schema.name is "MoviesRecord". But getting error in ConvertRecord processor that ConvertRecord is failed to process StandardFlowFileRecord "will route to failure Field field_0 can not be null. " Any help that would be great. Thanks

sshringi003 · ‎08-29-2018

Thanks for your quick reply. I tried but unable to figure out the problem. The same error occurs while putting data into HBase.

sshringi003 · ‎08-25-2018

Should i copy all the xml's file from server nifi conf to local nifi conf server nifi/conf/.xml --> local nifi/conf/.xml or which files need to be copy, bcz i did not get actual answer. Pls ...

sshringi003 · ‎08-24-2018

Hi @Josh Elser I am putting my GPS device data using MQTT to kafka and from kafka to HBase. but i am getting stuck while putting data into HBase. Could you suggest any flow which can help me. MQTT --> Kafka --> HBase or HTTP --> Kafka --> HBase And how to define PutHBaseJSON processor property. I am running NiFi in localhost and i am putting data on cloud ambari HBase. any help it would be great. Thanks.

sshringi003 · ‎08-24-2018

Hi @Shu I am running NiFi in localhost. I want to store data on server instead of local. So how to define the server property in Hadoop Configuration Resources. Bcz it requires ip address and path of the core-site.xml. If you help it would be great. Thanks, Surendra

sshringi003 · ‎08-17-2018

Thanks for this great community. could you suggest any work flow to get data from GPS device to cassandra table with using Kafka.

sshringi003 · ‎08-17-2018

We are using unsecure Kafka connection and not using MiNiFi, instead of that we are using MQTT. this is our NiFi flow ..... MQTT --> PublishKafka --> ConsumeKafka --> PutCassandraQL. What you suggest to use NiFi flow for below process ... MQTT --> PublishKafka --> ConsumeKafka --> PutCassandraQL

sshringi003 · ‎08-17-2018

Hi @Matt Burgess thanks for your quick response. I tried as per your suggestion but the error remains same. Is there any template related to this.

sshringi003 · ‎08-17-2018

Hi @dbains, thanks for your quick response. I am using PublishKafka_0_10_1.0.0 processor. and i have given FQDN in kafka property but still getting an error. Do you have any demo template or any further help.

sshringi003 · ‎08-16-2018

Hi @dbains I am getting the same issue while pulling the data into PublishKafka procesor. My conf for PublishKafka is : Kafka Brokers: ip:6667 Security Protocol: PLAINTEXT Topic Name: test Delivery Guarantee: Best Effort Key Attribute Encoding: UTF-8 Encoded Max Request Size: 1 MB Acknowledgement Wait Time: 5 secs Max Metadata Wait Time: 5 secs It throws the error Failed to update metadata 5000 ms. Could you provide me any solution for this. Thanks you.

sshringi003 · ‎08-16-2018

Hi @Timothy Spann, I am doing the same but unable to figure out this problem. Could you share me demo flow for this project. Thank you.

sshringi003 · ‎08-13-2018

Hi @Matt Burgess thank you for your quick response. I tried as per your comment but problem still remains. Can you go through below attached image. I am not getting data in ConvertJSONToSQL processor. My main objective is to put data into cassandra table through NiFI flow.

sshringi003 · ‎08-10-2018

I am using ConsumerKafka processor to pull data from putkafka. Now i want to put same data into cassandra table. But i am getting errors. I am using PutCassandraQL processor to put data into cassandra table. Is there any way to put data directly from ConsumerKafka to PutCassandraQL. Response will be appreciated. Thanks.

sshringi003 · ‎02-02-2018

Thanks @kgautam. I will try the same and let you know.

sshringi003 · ‎01-30-2018

Hi, I want to use spark memory instead of hard disk. So is their any way to run spark job on memory only. I am getting space error in hard disk so want to run job on memory only not in hard disk.

sshringi003 · ‎01-13-2018

Thanks @Shu For such a nice help.

Online	Offline
Last Visited	‎08-06-2019 11:43 AM

Date Registered	‎12-26-2017 10:24 AM
Last Visited	‎08-06-2019 11:43 AM
Total Messages Posted	40
Total Kudos Received	9

How to merge source records with existing records ...

Re: Apache NiFi is not running on centos-7 ??

Apache NiFi is not running on centos-7 ??

Re: HiveServer2, automatically shutdown ?

Re: HiveServer2, automatically shutdown ?

HiveServer2, automatically shutdown ?

Re: How do you speed up count(*) on tables in Hive

Re: Hive_PHP | How to connect HiveServer2 with PHP...

Hive_PHP | How to connect HiveServer2 with PHP, us...

Re: Unable to start up a single NiFi node

Re: NiFi | Is there any way in NiFi local while sh...

Re: NiFi | Is there any way in NiFi local while sh...

NiFi | Is there any way in NiFi local while shutdo...

How to consume data from MQTT and put into HBase ?...

Re: Am I Configuring NiFi's AvroSchemaRegistry Cor...

Re: Standard way of giving Hadoop Configuration Re...

Re: Standard way of giving Hadoop Configuration Re...

Re: Can't Connect to HBase through NiFi HBaseClien...

Re: Standard way of giving Hadoop Configuration Re...

Re: Apache NiFi/ How to put data from kafka consum...

Re: TimeoutException while sending data from Nifi ...

Re: Apache NiFi/ How to put data from kafka consum...

Re: TimeoutException while sending data from Nifi ...

Re: TimeoutException while sending data from Nifi ...

Re: Part 2: IoT: Augmenting GPS Data with Weathe...

Re: Apache NiFi/ How to put data from kafka consum...

Apache NiFi/ How to put data from kafka consumer t...

Re: Is their any way of to use Spark memory instea...

Is their any way of to use Spark memory instead of...

Re: Problem : while parsing JSON formatted CSV fil...