Hi all, We have opened an ticket to hortonwork support but don't have any clues about this message, we have a lot of messages : 2018-04-12 14:18:10,277 WARN [Replicate Request Thread-42] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Response time from node1-nifi:9091 was slow for each of the last 3 requests made. To see more information about timing, enable DEBUG logging for org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator We have set some parameters : nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.cluster.node.protocol.max.threads=100 nifi.cluster.node.protocol.port=9088 nifi.cluster.node.protocol.threads=80 nifi.web.jetty.threads=600 After restart the nifi, all is ok for a few minutes before the message come back again. I don't use custom processors, and CPU use is around 5%, host has 32Go and Memory HEAD is set to minimum 8GB and max 16GB. NIFI 1.2 Any helps are welcome. ... View more

Hi all, I use ListSFTP+FETCHSFTP to get the files (the files could be one file or 3 files or 5 files by day) and put its with putHDFS. A file _SUCCESS needs to be create if all files have put on HDFS. The question how can I check if all files have been put on HDFS before create the empty _SUCCESS file ? regards ... View more

hi all, Somebody could explain what is mean the value "percentCompleted" when I make a POST /nifi-api/provenance ? Must I wait percentCompleted egal 100% before make a GET /nifi-api/provenance/${id} ? Here percentCompleted = 33% {"provenance":{"id":"a0e66f97-b318-1fa4-0000-0000241111aa","uri":"https://nifi001:9443/nifi-api/provenance/a0e66f97-b318-1fa4-0000-0000241111aa","submissionTime":"09/12/2017 11:29:16.606 CEST","expiration":"09/12/2017 11:59:16.606 CEST","percentCompleted":33,"finished":false,"request":{"searchTerms":{"Component ID":"7176ddeb-015e-1000-ffff-ffff8dee922b","Event Type":"DROP"},"startDate":"09/12/2017 10:00:00 CEST","endDate":"09/12/2017 11:59:59 CEST","maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"11:29:16 CEST","oldestEvent":"09/11/2017 12:00:06 CEST","bash-4.1# curl -v --compress -H Content-type:application/json --cacert nifi-cert.pem --cert ./nifi001.p12 -XGET https://nifi001:9443/nifi-api/provenance/a0e66f97-b318-1fa4-0000-0000241111aa thanks ... View more

Hi all, I'm trying use invokeHTTP to get a digest and token from an API, here my curl command : curl --silent -ik --url https://${OKAPI_HOST}/oauth/access_token -H "Authorization: Bearer $(base64 -w 0 < ${KEYTAB_FILE})" -d grant_type=client_credentials --data-urlencode scope=${SCOPE} how send parameter to invokeHTTP ? I'm thinking use updateAttribute to set scope, Authorization but it don't work. thanks ... View more

Hi all, I Have a cluster NIFI with 3 node, when I'm making GET dataprovenance with python script, only nifi-node003 sent me the result. So can't receive dataprovenance from nifi-node001 and nifi-node002 What's wrong ? thanks for helps "{\"provenance\":{\"request\":{\"maxResults\":1000,\"startDate\":\"08/23/2017 03:00:00 CEST\",\"endDate\":\"08/23/2017 03:59:59 CEST\",\"searchTerms\":{\"EventType\":\"SEND\",\"ProcessorID\":\"0dd63d71-8620-1ce1-be9d-587c5f6ec679\"}}}}" conn=httplib.HTTPSConnection(clusterNode, 9443, key_file=None, cert_file="/var/opt/hosting/nifi/conf/nifi-001.pem") headers = {"Content-Type": "application/json", "Accept": "application/json, text/javascript"} conn.request("POST", "/nifi-api/provenance", json.dumps(params), headers) response = conn.getresponse() result = json.loads(response.read()) conn.close() conn.request("GET", "/nifi-api/provenance/" + result['provenance']['id']) responseGet = conn.getresponse() resultGet = json.loads(responseGet.read()) conn.request("DELETE", "/nifi-api/provenance/" + resultGet['provenance']['id']) res = conn.getresponse() conn.close() ... View more

Hi all, Do you know why my node disconnected when I'm doing some POST/GET/DELETE with api ? params = {"provenance":{"request":{"maxResults":1000,"summarize":"true","incrementalResults":"false","startDate":"08/23/2017 00:00:00 CEST","endDate":"08/23/2017 23:59:59 CEST","clusterNodeId":"dae75344-81ca- 4251-81b3-3476e9750f3f","searchTerms":{"EventType":"SEND","ProcessorID":"0dd63d71-8620-1ce1-be9d-587c5f6ec679"}}}} #This curl to receice provenance ID curl -vv -H "Content-type: application/json" -XPOST 'https://nifi001:9443/nifi-api/provenance' --data-binary $params #this curl to get provenance curl -vv -H "Content-type: application/json" -XGET 'https://nifi001:9443/nifi-api/provenance/0dddacc8-015e-1000-0000-000025863d27' # this curl for delete curl -vv -H "Content-type: application/json" -XDELETE 'https://nifi001:9443/nifi-api/provenance/0dddacc8-015e-1000-0000-000025863d27' 08/23/2017 08:53:43 CEST: Node Status changed from CONNECTED to DISCONNECTED due to Failed to process request DELETE /nifi-api/provenance/0dddacc8-015e-1000-0000-000025863d27 thanks ... View more

Hi all, I'm try running a sqoop job and fall to killed job but i don't know why ? Here application log : 2017-08-21 14:21:15,179 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #4 tokens and #1 secret keys for NM use for launching container 2017-08-21 14:21:15,179 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 5 2017-08-21 14:21:15,179 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2017-08-21 14:21:15,563 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1501688338199_102303_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2017-08-21 14:21:15,567 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e71_1501688338199_102303_01_000002 taskAttempt attempt_1501688338199_102303_m_000000_0 2017-08-21 14:21:15,568 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1501688338199_102303_m_000000_0 2017-08-21 14:21:15,569 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host049:45454 2017-08-21 14:21:15,629 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1501688338199_102303_m_000000_0 : 13562 2017-08-21 14:21:15,630 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1501688338199_102303_m_000000_0] using containerId: [container_e71_1501688338199_102303_01_000002 on NM: [host049:45454] 2017-08-21 14:21:15,634 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1501688338199_102303_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2017-08-21 14:21:15,634 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1501688338199_102303_m_000000 Task Transitioned from SCHEDULED to RUNNING 2017-08-21 14:21:15,981 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1501688338199_102303: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:239616, vCores:77> knownNMs=57 2017-08-21 14:21:16,767 INFO [Socket Reader #1 for port 44108] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for user001 (auth:SIMPLE) 2017-08-21 14:21:16,774 INFO [Socket Reader #1 for port 44108] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for user001 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB 2017-08-21 14:21:16,826 INFO [IPC Server handler 0 on 44108] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Kill job job_1501688338199_102303 received from user001 (auth:TOKEN) at 10.99.224.106 2017-08-21 14:21:16,828 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1501688338199_102303Job Transitioned from RUNNING to KILL_WAIT 2017-08-21 14:21:16,842 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1501688338199_102303_m_000000 Task Transitioned from RUNNING to KILL_WAIT 2017-08-21 14:21:16,850 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1501688338199_102303_m_000000_0 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP 2017-08-21 14:21:16,850 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e71_1501688338199_102303_01_000002 taskAttempt attempt_1501688338199_102303_m_000000_0 2017-08-21 14:21:16,851 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1501688338199_102303_m_000000_0 2017-08-21 14:21:16,851 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : host049:45454 2017-08-21 14:21:16,866 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1501688338199_102303_m_000000_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 2017-08-21 14:21:16,867 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2017-08-21 14:21:16,873 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://clusterbigdata/user/user001/oozie-oozi/0005541-170818100252684-oozie-oozi-W/sqoop-d8b5--sqoop/output/_temporary/1/_temporary/attempt_1501688338199_102303_m_000000_0 2017-08-21 14:21:16,879 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1501688338199_102303_m_000000_0 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 2017-08-21 14:21:16,889 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1501688338199_102303_m_000000 Task Transitioned from KILL_WAIT to KILLED 2017-08-21 14:21:16,892 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2017-08-21 14:21:16,892 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1501688338199_102303Job Transitioned from KILL_WAIT to KILL_ABORT 2017-08-21 14:21:16,893 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 2017-08-21 14:21:16,904 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1501688338199_102303Job Transitioned from KILL_ABORT to KILLED 2017-08-21 14:21:16,904 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2017-08-21 14:21:16,905 INFO [Thread-67] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true Thanks for helps. ... View more

Hi all, Anybody already used google ControllerService and GCSBucket ? I don't know how use it to list bucket. Without NIFI, i've used gsutil and a configuration file how has these parameters gs_oauth2_refresh_token proxy proxy_port Any help welcome ... View more

Hi all, It seems that executeStreamCommand don't delete flowFile. The script bash worked but the queue will not empty after successful execution of script ... View more

Hi all, It seems that executeStreamCommand don't delete flowFile. The script bash worked but the queue will not empty after successful execution of script ... View more