Re: Parse and modify data befor calling PutDatabas...

quentin_toulou · ‎08-25-2017

Hi, If you really want use the ConvertJsonToSQL processor, you need to have a field in your json object with the SQL inner query. You can also use the EvaluateJSonPath -> replaceText -> putSQL flow to achieve that.

quentin_toulou · ‎08-22-2017

You can use the ReplaceText processor. Use ${attribute_name} to write an attribute in your text.

quentin_toulou · ‎08-22-2017

With the UpdateAttribute, you can add your own property. (With the '+' button in the 'Properties' tab) Add a property with the name : Sis_Grp.Dom and the value : ${ZComponent:substringAfter('"'):substringBefore('"')} (the argument for the substringBefore is double quote between simple quotes.) and a second with the name : Name and the value : ${ZComponent:substringAfter('Name="'):substringBefore('"')} You can find all the Nifi Expression language here

quentin_toulou · ‎08-21-2017

Hi, Did you schedule your GetHTTP processor ? By default, it continually runs. So your data are continually fetched. You have to configure this property in the "Scheduling" tab. An easy way, in test : keep the "Timer driven" strategy and set the Run Schedule with a big interval. So you can start your flow and stop it before the second execution.

quentin_toulou · ‎08-21-2017

Do you have an error ?

quentin_toulou · ‎08-18-2017

1 - ExecuteSQL -> 2 - ConvertAvroToJSON -> 3 - splitJSon -> 4 - EvaluateJsonPath -> 5 - UpdateAttribute -> 6 ExecuteSQL -> 7 - replaceText -> 8- putSQL. 1 - 2 - 3 : query your table and split by row. You will have one flowfile by row. Use $ for the JsonPathExpression in the split processor. 4 - Retrieve the query_id and the query_execute in two attributes of each flowfile (values of your properties are $.query_id and $.query_execute) 5 - Create a new attribute query_startTime with the value ${now():format('HH:mm:ss') 6 - Execute the query ${query_execute} 7 - Write your insert query : INSERT INTO queryExecutionInfo VALUES (${query_id}, ${query_startTime},{${now():format('HH:mm:ss')); 8 - Execute your insert I think this flow works, maybe someone will have something efficient.

quentin_toulou · ‎08-18-2017

Hi A common flow to modify your data is : ConvertAvroToJson -> splitJson -> EvaluateJsonPath -> UpdateAttribute So you will have one flowfile by row and with the EvaluateJsonPath you convert your fields as attributes of this flowfile. After that you can use UpdateAttribute to modify an attribute or add new one. You can use Nifi Expression Language.

quentin_toulou · ‎08-16-2017

Hi, I think you can do that with the QueryRecord processor. Add a new property, whatever the name and the content may be something like "select Subject, count(Name) as nb from flowfile group by Subject". Set correctly the schemas for the reader and the writer and you will find the result of the query as a new relationship of your processor. (A new relationship with the name of your property will appear)

quentin_toulou · ‎08-11-2017

Sorry, the splitJson processor is useless in your case. I assumed you have an array of several objects.. Please, remove the splitJson processor and share the full error you have.

quentin_toulou · ‎08-10-2017

If you want all the data and fields, it's : $ (corresponding to the root)

quentin_toulou · ‎08-10-2017

Hi, Try a SplitJson processor before the ConvertJsonToAvro one, it works like this for me !

quentin_toulou · ‎08-10-2017

Hi, 1. You can find logs for Schema Registry under this path : /usr/hdf/current/registry/logs 2. Remove all the spaces you have in the name of your properties and it would be fine { "type" : "record", "namespace" : "poc", "name" : "Employee", "fields" : [ { "name" : "Name", "type" : "string"}, { "name" : "Age", "type" : "int"} ] }

quentin_toulou · ‎07-31-2017

Thanks a lot for this detailed response ! @Greg Keys

quentin_toulou · ‎07-28-2017

With the new version of HDF (3.0), we find Nifi and the new application : Streaming analystics manager. What is the main difference between Nifi and the stream builder module of SAM ? How can/should we use these two applications together ? After a quick time to read the description of this module, it seems it's really close to Nifi. Thank you !

quentin_toulou · ‎06-22-2017

Hi Matt, No i can't.. And when i try #ss -an | grep 9083 on the sandbox-host i have nothing

quentin_toulou · ‎06-20-2017

Yes i'm using the sandbox (with virtualbox) but i set up a network bridge so it seems there is no port forwading..

quentin_toulou · ‎06-20-2017

Hi Pierre, it's done !

quentin_toulou · ‎06-20-2017

Hi, I run NiFi on my windows machine and i try to put some data in Hive (HDP 2.6). I can connect to hive with a putHiveQL processor (so the network is ok between the two machcines), so with a jdbc connection but when i use the putHiveStreaming processor it doesn't work. I can't connect to Hive Metastore. My configuration for the putHiveStreaming processor : (I added the link for my IP address in /etc/host) The error : 2017-06-20 10:46:09,900 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=c4a50533-015c-1000-0a44-9c01c3cd6345] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient: {} org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:535) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:542) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:674) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2125) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2095) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:628) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:552) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:552) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 16 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ... 26 common frames omitted Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:187) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at org.apache.thrift.transport.TSocket.open(TSocket.java:182) ... 35 more at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) ... 31 common frames omitted Any idea ? Thank you !

Online	Offline
Last Visited	‎08-25-2017 01:34 PM

Date Registered	‎06-20-2017 09:16 AM
Last Visited	‎08-25-2017 01:34 PM
Total Messages Posted	19
Total Kudos Received	4

Re: Parse and modify data befor calling PutDatabas...

Re: How to aggregate csv or json data in Nifi?

Re: hdf sandbox - schema registry

Re: ConvertJSONToSQL Processor INSERT Issue

Re: Parse and modify data befor calling PutDatabas...

Re: Parse and modify data befor calling PutDatabas...

Re: Data replicated in postgreSQL while inject th...

Re: How to Extract table values row by row, send t...

Re: How to Extract table values row by row, send t...

Re: Parse and modify data befor calling PutDatabas...

Re: How to aggregate csv or json data in Nifi?

Re: ConvertJSONToAvro not working despite successf...

Re: ConvertJSONToAvro not working despite successf...

Re: ConvertJSONToAvro not working despite successf...

Re: hdf sandbox - schema registry

Re: [HDF-3.0] Difference between Nifi and Stream b...

[HDF-3.0] Difference between Nifi and Stream build...

Re: [NIFI] Can't connect to HiveMetastore with put...

Re: [NIFI] Can't connect to HiveMetastore with put...

Re: [NIFI] Can't connect to HiveMetastore with put...

[NIFI] Can't connect to HiveMetastore with putHive...