Member since
12-03-2017
156
Posts
26
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2070 | 11-03-2023 12:17 AM | |
| 4132 | 12-12-2022 09:16 PM | |
| 1559 | 07-14-2022 03:25 AM | |
| 2388 | 07-28-2021 04:42 AM | |
| 3310 | 06-23-2020 10:08 PM |
07-23-2018
09:19 AM
Hello, I have a json input for which I need to add one key-value("correlationId": "123456789") under 'tripSummaryMetaData' section dynamically. I tried with below spec and its working fine, but the issues is when 'tripSummaryMetaData' is not available the it is creating new 'tripSummaryMetaData' and adding my attribute. When 'tripSummaryMetaData' is not available the it should not add my new attribute, is that possible in JOLT? Thanks in advance. Input: {
"tripSummaryMetaData": {
"orgId": "demo",
"message_type_id": "MB-CD-MSG0001"
},
"eventSummary": {
"tripId": "2018072204cc7851d3e84f7ea8a1155c95f65b06",
"referenceData": "050bfa62-430a-45a5-b7a6-78d688d45167",
"gpsData": [{
"gpsTimeReceived": 1532319157338,
"gpsTime": "2018-07-22T21:12:38-07:00"
}]
}
} Output: {
"tripSummaryMetaData" : {
"orgId" : "demo",
"message_type_id" : "MB-CD-MSG0001",
"correlationId" : "123456789"
},
"eventSummary" : {
"tripId" : "2018072204cc7851d3e84f7ea8a1155c95f65b06",
"referenceData" : "050bfa62-430a-45a5-b7a6-78d688d45167",
"gpsData" : [ {
"gpsTimeReceived" : 1532319157338,
"gpsTime" : "2018-07-22T21:12:38-07:00"
} ]
}
} Spec : [{
"operation": "default",
"spec": {
"tripSummaryMetaData": {
"correlationId": "123456789"
}
}
}] Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
05-03-2018
09:33 AM
Yes, its possible. Alter table <table_name> set tblproperties ('transactional'='true');
... View more
05-03-2018
09:31 AM
Thanks for the nice explanation Ned. When I have created the flow, NiFi processor is able to store data to the table but its very slow in the cluster environment. I am able to push only 2-3 flowfile/records (few kb in size) per second. I tried with more thread counts as well, but no luck. Any suggestion will be much appreciated.
... View more
04-14-2018
06:00 PM
Thanks for replies, Issue was due to concurrent thread configuration. When we increase the max allowed thread count in controller settings, issue solved. We stopped all flows, increased the count, restarted the node and then flows.
... View more
04-14-2018
05:56 PM
hello, Facing an issue with 'InvokeHTTP' NiFi processor. Have configured below URL for GET operation - https://******/**/users/${userId} When I run the processor some time it will get proper response from url. But if I restart the processor, suddenly it will start telling 'URL not found' error and route the flow file to 'No Retry' relationship. If I try to restart the processor 2-3 times then once it will start responding correctly. Please suggest me to resolve the issue. screen shots of config - Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-29-2018
04:26 PM
Hi, I have a kafka consumer, that is consuming messages after that flowfiles are not moving at all. Suddenly this issue started. many messages are stuck at diff connection/queue in btw processors even though processor running it is not getting processing dataflow-struck.png Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-19-2018
11:09 AM
I am enriching my request payload with the payload attribute 'http.headers.orgId' and its working fine. SPEC: [{
"operation": "default", "spec": { "*": {
"orgId": "${http.headers.orgId}"
} } }] INPUT : {
"drivewiseDeviceData": {
"imei": "sampleimei",
"tripId": "3d9df81a-1828-40eb-940e-d2434eb5f50f",
"sumVer": null,
"payloadType": "Binary",
"packetInfo": "",
"payloadData": "g1bjSWX",
"recordinfo": null,
"sequenceNum": null,
"timeStamp": "2017-05-26T12:07:29.968Z" }
} OUTPUT: { "drivewiseDeviceData": { "imei": "sampleimei", "tripId": "3d9df81a-1828-40eb-940e-d2434eb5f50f", "sumVer": null, "payloadType": "Binary", "packetInfo": "", "payloadData": "g1bjSWX", "recordinfo": null, "sequenceNum": null, "timeStamp": "2017-05-26T12:07:29.968Z", "orgId": "Sample Org Name Here" } } ......................................................................................................................................... Now scenarios is I may get header attribute as 'http.headers.OrgID' or 'http.headers.orgId', So is there any way to handle both scenarios in my JOLT spec? If 'http.headers.orgId' value is null/empty/attribute-not-present then I should use 'http.headers.OrgID' for enrich.
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2018
10:47 AM
Thanks for the quick reply @Matt Clarke 🙂 One more query, I may have N number of files, so how can I find out the number of files in directory and ensure that all files reached Mergecontent processor before I proceed further to create trigger file and FTP ?
... View more
02-13-2018
10:39 AM
I have scenario where I need to transfer 10+ files from directory A to another server directory B using FTP at 10pm every day. After transferring all the files to directory B I have to create one trigger file in directory B to trigger the downstream process. How can I do this using NiFi ? Problem is creating trigger file in directory B at the end of all files transfer. please suggest. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
01-24-2018
12:11 PM
Hello All, Any idea what could be the reason to get this below error in NiFi app logs ? Also I am seeing same error on NiFi UI as well. ERROR : p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.Apple-tab-span {white-space:pre} 2018-01-24 07:07:10,663 WARN [StandardProcessScheduler Thread-2-SendThread(localhost:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2018-01-24 07:07:11,764 WARN [StandardProcessScheduler Thread-2-SendThread(localhost:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2018-01-24 07:07:11,865 WARN [StandardProcessScheduler Thread-2] o.a.h.h.zookeeper.RecoverableZooKeeper Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.h.zookeeper.RecoverableZooKeeper ZooKeeper exists failed after 4 attempts 2018-01-24 07:07:11,865 WARN [StandardProcessScheduler Thread-2] org.apache.hadoop.hbase.zookeeper.ZKUtil hconnection-0x6aa48200x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.hbase.zookeeper.ZooKeeperWatcher hconnection-0x6aa48200x0, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.h.h.c.ConnectionManager$HConnectionImplementation Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=5a504684-9c3a-10ae-ffff-ffffe9e6e06b] Failed to invoke @OnEnabled method due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase : {} org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:188) at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:408) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 19 common frames omitted Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:906) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:545) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1483) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 23 common frames omitted Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:895) ... 26 common frames omitted 2018-01-24 07:07:11,865 ERROR [StandardProcessScheduler Thread-2] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=5a504684-9c3a-10ae-ffff-ffffe9e6e06b] due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 24 07:07:11 EST 2018, RpcRetryingCaller{globalStartTime=1516795614116, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi
- « Previous
- Next »