Member since
β11-21-2017
16
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1442 | β12-17-2019 12:16 AM |
β12-17-2019
12:16 AM
well, This is not really a problem with NiFi per se: NiFi was installed with docker so I kept using the docker host as it was the correct machine. Embarassing.. π
... View more
β12-16-2019
08:42 AM
Disclaimer: crosspost from https://stackoverflow.com/questions/59359158/apache-nifi-1-10-how-to-execute-a-python-script-as-a-processor
In NiFi <= v1.9 I used the following NiFi setup:
`GetTwitter` generates a JSON for every tweet gathered and attached to it there used to be a `ExecuteStreamCommand` that has the following parameters:
This would allow me to get the flowfile as input, and I could analyse the content and do something (in this specific case I would put the content on Elasticsearch) - You could reach the same result if you do something like this `cat flowfile.json | python <scriptname>.py`
But now in >= v1.10 it seems I cannot do the same. it gives me some error about parameters (it seems it wants to use the flowfile as parameter on calling the command/script instead, as it used to be, pass the flowfile as input. It gives me the following error:
2019-12-16 14:28:44,953 ERROR [Timer-Driven Process Thread-2] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=ef1d15fb-016e-1000-3050-35fba1733f97] Transferring flow file StandardFlowFileRecord[uuid=c485d8aa-58e9-428c-84b1-b4acbd6275a7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576505352573-22, container=default, section=22], offset=663999, length=-1],offset=0,name=242802f9-5ffb-4a2c-90a5-27736459dfd0.json,size=0] to nonzero status. Executable command bash ended in an error: bash: line 1: $'{delete:status:id:1005156317567188992}\r': command not found
2019-12-16 14:28:44,962 ERROR [Timer-Driven Process Thread-2] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=ef1d15fb-016e-1000-3050-35fba1733f97] Transferring flow file StandardFlowFileRecord[uuid=5b617840-ba8e-47b3-9a27-06e8ec0f88bd,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576505352573-22, container=default, section=22], offset=663999, length=-1],offset=0,name=eb9e236a-1ccb-4243-aa1c-e5de43746f80.json,size=0] to nonzero status. Executable command bash ended in an error: bash: line 1: $'{delete:status:id:1139832600422277120}\r': command not found
It seems it takes the flowfile JSON as parameter instead of input . Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi
β11-21-2017
08:21 AM
1 Kudo
I didn't get why is this a problem for Cloudera. But excluding it, it actually worked!
... View more
β11-21-2017
08:13 AM
that works! and it is actually weird
... View more
β11-21-2017
07:33 AM
1 Kudo
I have cloudera Manager 5.12.1 and After having installed the CSD following the first steps of this guide: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html but after restarting the Cloudera Manager, even if it automatically added the remote parcel repository URL (effect from the installation of the CSD) . I cannot find the new parcel in order to download it. I digged into the logs and I found this warning 4:18:57.328 PM WARN ReadWriteDatabaseTaskCallable
Error while executing CmfEntityManager task
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
at com.google.common.collect.Collections2.filter(Collections2.java:92)
at com.cloudera.parcel.components.ParcelDownloaderImpl$RepositoryInfo.getParcelsWithValidNames(ParcelDownloaderImpl.java:673)
at com.cloudera.parcel.components.ParcelDownloaderImpl$RepositoryInfo.getSortedParcels(ParcelDownloaderImpl.java:691)
at com.cloudera.parcel.components.ParcelDownloaderImpl.syncRemoteRepos(ParcelDownloaderImpl.java:368)
at com.cloudera.parcel.components.ParcelDownloaderImpl$1.run(ParcelDownloaderImpl.java:438)
at com.cloudera.parcel.components.ParcelDownloaderImpl$1.run(ParcelDownloaderImpl.java:433)
at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) maybe it cannot download the parcels? why? I have no firewall or proxy activated. I have CENTOS7
... View more
Labels:
- Labels:
-
Apache Spark
-
Cloudera Manager
β04-30-2016
11:55 PM
1 Kudo
I tried to install Ambari into 16.04 Lubuntu. The ambari version was 2.2.1.1 While registering with the UI wizard, an Ambari agent⦠the following error occurred from: /var/log/ambari-agent/ambari-agent.log INFO 2016-05-01 01:01:03,673 NetUtil.py:60 - Connecting to https://salvob1:8440/connection_infoINFO 2016-05-01 01:01:04,165 security.py:99 - SSL Connect being called.. connecting to the serverINFO 2016-05-01 01:01:04,792 security.py:60 - SSL connection established. Two-way SSL authentication is turned off on the server.ERROR 2016-05-01 01:01:05,197 Controller.py:165 - Cannot register host with not supported os type, hostname=salvob2, serverOsType=ubuntu16, agentOsType=ubuntu16INFO 2016-05-01 01:01:05,199 Controller.py:394 - Registration response from salvob1 was FAILED I wonder if and when is gonna be available a fix for this
... View more
Labels:
- Labels:
-
Apache Ambari
β04-30-2016
11:48 PM
Thank you for the reply. I did some research and it is important to add the right repo.list This is a "better" guide . Plus, I was installing ambari in my Lubuntu 16.04 . I was sure it was fully compatible and almost the same with a 14.04 Ubuntu version. But I found out that it fails in registering a host with a "not-recognised os", here the stack error log from /var/log/ambari-agent/ambari-agent.log : INFO 2016-05-01 01:01:03,673 NetUtil.py:60 - Connecting to https://salvob1:8440/connection_info
INFO 2016-05-01 01:01:04,165 security.py:99 - SSL Connect being called.. connecting to the server
INFO 2016-05-01 01:01:04,792 security.py:60 - SSL connection established. Two-way SSL authentication is turned off on the server.
ERROR 2016-05-01 01:01:05,197 Controller.py:165 - Cannot register host with not supported os type, hostname=salvob2, serverOsType=ubuntu16, agentOsType=ubuntu16
INFO 2016-05-01 01:01:05,199 Controller.py:394 - Registration response from salvob1 was FAILED I hope Hortonworks add soon, even a small support to this new os
... View more
β04-30-2016
04:11 PM
I am trying to register a host with my Ambari server.
I am using ubuntu, and the version of the server is 2.2.1.0-161 but when trying to register the host, it fails with this error in bootstrap error: INFO:root:BootStrapping hosts ['salvob2'] using /usr/lib/python2.6/site-packages/ambari_server cluster primary OS: ubuntu16 with user 'root' sshKey File /var/run/ambari-server/bootstrap/1/sshKey password File null using tmp dir /var/run/ambari-server/bootstrap/1 ambari: salvob1; server_port: 8080; ambari version: 2.2.1.0; user_run_as: root
INFO:root:Executing parallel bootstrap
ERROR:root:ERROR: Bootstrap of host salvob2 fails because previous action finished with non-zero exit code (1)
ERROR MESSAGE: Connection to salvob2 closed.
STDOUT: Desired version (2.2.1.0) of ambari-agent package is not available. Repository has following versions of ambari-agent:2.2.1.0,2.2.1.0,
Connection to salvob2 closed.
INFO:root:Finished parallel bootstrap
... View more
Labels:
- Labels:
-
Apache Ambari