quentin_toulou
Explorer

Re: ConvertJSONToSQL Processor INSERT Issue

by Explorer in Support Questions
‎08-25-2017 01:25 PM
‎08-25-2017 01:25 PM
Hi, If you really want use the ConvertJsonToSQL processor, you need to have a field in your json object with the SQL inner query. You can also use the EvaluateJSonPath -> replaceText -> putSQL flow to achieve that. ... View more

Re: Parse and modify data befor calling PutDatabas...

by Explorer in Support Questions
‎08-22-2017 04:46 PM
‎08-22-2017 04:46 PM
You can use the ReplaceText processor. Use ${attribute_name} to write an attribute in your text. ... View more

Re: Parse and modify data befor calling PutDatabas...

by Explorer in Support Questions
‎08-22-2017 01:24 PM
‎08-22-2017 01:24 PM
With the UpdateAttribute, you can add your own property. (With the '+' button in the 'Properties' tab) Add a property with the name : Sis_Grp.Dom and the value : ${ZComponent:substringAfter('"'):substringBefore('"')} (the argument for the substringBefore is double quote between simple quotes.) and a second with the name : Name and the value : ${ZComponent:substringAfter('Name="'):substringBefore('"')} You can find all the Nifi Expression language here ... View more

Re: Data replicated in postgreSQL while inject th...

by Explorer in Support Questions
‎08-21-2017 08:17 AM
‎08-21-2017 08:17 AM
Hi, Did you schedule your GetHTTP processor ? By default, it continually runs. So your data are continually fetched. You have to configure this property in the "Scheduling" tab. An easy way, in test : keep the "Timer driven" strategy and set the Run Schedule with a big interval. So you can start your flow and stop it before the second execution. ... View more

Re: How to Extract table values row by row, send t...

by Explorer in Support Questions
‎08-21-2017 07:20 AM
‎08-21-2017 07:20 AM
Do you have an error ? ... View more

Re: How to Extract table values row by row, send t...

by Explorer in Support Questions
‎08-18-2017 02:21 PM
‎08-18-2017 02:21 PM
1 - ExecuteSQL -> 2 - ConvertAvroToJSON -> 3 - splitJSon -> 4 - EvaluateJsonPath -> 5 - UpdateAttribute -> 6 ExecuteSQL -> 7 - replaceText -> 8- putSQL. 1 - 2 - 3 : query your table and split by row. You will have one flowfile by row. Use $ for the JsonPathExpression in the split processor. 4 - Retrieve the query_id and the query_execute in two attributes of each flowfile (values of your properties are $.query_id and $.query_execute) 5 - Create a new attribute query_startTime with the value ${now():format('HH:mm:ss') 6 - Execute the query ${query_execute} 7 - Write your insert query : INSERT INTO queryExecutionInfo VALUES (${query_id}, ${query_startTime},{${now():format('HH:mm:ss')); 8 - Execute your insert I think this flow works, maybe someone will have something efficient. ... View more

Re: Parse and modify data befor calling PutDatabas...

by Explorer in Support Questions
‎08-18-2017 09:10 AM
1 Kudo
‎08-18-2017 09:10 AM
1 Kudo
Hi A common flow to modify your data is : ConvertAvroToJson -> splitJson -> EvaluateJsonPath -> UpdateAttribute So you will have one flowfile by row and with the EvaluateJsonPath you convert your fields as attributes of this flowfile. After that you can use UpdateAttribute to modify an attribute or add new one. You can use Nifi Expression Language. ... View more

Re: How to aggregate csv or json data in Nifi?

by Explorer in Support Questions
‎08-16-2017 01:59 PM
1 Kudo
‎08-16-2017 01:59 PM
1 Kudo
Hi, I think you can do that with the QueryRecord processor. Add a new property, whatever the name and the content may be something like "select Subject, count(Name) as nb from flowfile group by Subject". Set correctly the schemas for the reader and the writer and you will find the result of the query as a new relationship of your processor. (A new relationship with the name of your property will appear) ... View more

Re: ConvertJSONToAvro not working despite successf...

by Explorer in Support Questions
‎08-11-2017 08:49 AM
‎08-11-2017 08:49 AM
Sorry, the splitJson processor is useless in your case. I assumed you have an array of several objects.. Please, remove the splitJson processor and share the full error you have. ... View more

Re: ConvertJSONToAvro not working despite successf...

by Explorer in Support Questions
‎08-10-2017 02:18 PM
‎08-10-2017 02:18 PM
If you want all the data and fields, it's : $ (corresponding to the root) ... View more

Re: ConvertJSONToAvro not working despite successf...

by Explorer in Support Questions
‎08-10-2017 01:00 PM
‎08-10-2017 01:00 PM
Hi, Try a SplitJson processor before the ConvertJsonToAvro one, it works like this for me ! ... View more

Re: hdf sandbox - schema registry

by Explorer in Support Questions
‎08-10-2017 08:57 AM
‎08-10-2017 08:57 AM
Hi, 1. You can find logs for Schema Registry under this path : /usr/hdf/current/registry/logs 2. Remove all the spaces you have in the name of your properties and it would be fine { "type" : "record", "namespace" : "poc", "name" : "Employee", "fields" : [ { "name" : "Name", "type" : "string"}, { "name" : "Age", "type" : "int"} ] } ... View more

Re: [HDF-3.0] Difference between Nifi and Stream b...

by Explorer in Support Questions
‎07-31-2017 08:00 AM
‎07-31-2017 08:00 AM
Thanks a lot for this detailed response ! @Greg Keys ... View more

[HDF-3.0] Difference between Nifi and Stream build...

by Explorer in Support Questions
‎07-28-2017 01:15 PM
1 Kudo
‎07-28-2017 01:15 PM
1 Kudo
With the new version of HDF (3.0), we find Nifi and the new application : Streaming analystics manager. What is the main difference between Nifi and the stream builder module of SAM ? How can/should we use these two applications together ? After a quick time to read the description of this module, it seems it's really close to Nifi. Thank you ! ... View more

Re: [NIFI] Can't connect to HiveMetastore with put...

by Explorer in Support Questions
‎06-22-2017 07:54 AM
‎06-22-2017 07:54 AM
Hi Matt, No i can't.. And when i try #ss -an | grep 9083 on the sandbox-host i have nothing ... View more

Re: [NIFI] Can't connect to HiveMetastore with put...

by Explorer in Support Questions
‎06-20-2017 02:02 PM
‎06-20-2017 02:02 PM
Yes i'm using the sandbox (with virtualbox) but i set up a network bridge so it seems there is no port forwading.. ... View more

Re: [NIFI] Can't connect to HiveMetastore with put...

by Explorer in Support Questions
‎06-20-2017 01:51 PM
‎06-20-2017 01:51 PM
Hi Pierre, it's done ! ... View more

[NIFI] Can't connect to HiveMetastore with putHive...

by Explorer in Support Questions
‎06-20-2017 01:51 PM
1 Kudo
‎06-20-2017 01:51 PM
1 Kudo
Hi, I run NiFi on my windows machine and i try to put some data in Hive (HDP 2.6). I can connect to hive with a putHiveQL processor (so the network is ok between the two machcines), so with a jdbc connection but when i use the putHiveStreaming processor it doesn't work. I can't connect to Hive Metastore. My configuration for the putHiveStreaming processor : (I added the link for my IP address in /etc/host) The error : 2017-06-20 10:46:09,900 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=c4a50533-015c-1000-0a44-9c01c3cd6345] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient: {} org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:535) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:542) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:674) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2125) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2095) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:628) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:552) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:552) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 16 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ... 26 common frames omitted Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:187) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at org.apache.thrift.transport.TSocket.open(TSocket.java:182) ... 35 more at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) ... 31 common frames omitted Any idea ? Thank you ! ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎08-25-2017 01:34 PM
Public Statistics
Date Registered ‎06-20-2017 09:16 AM
Last Visited ‎08-25-2017 01:34 PM
Total Messages Posted 19
Total Kudos Received 4