Member since
11-26-2016
9
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2451 | 11-26-2016 09:34 PM |
02-01-2021
07:16 AM
Thanks for you answer Dennis. As I need the value of the attribute I don't understand how anymatchingattribute could help What I need is to find an attribute based on it's name (doing a case insensitive search). So casting to lower case all attributes, then searching the one I need and then getting the value is indeed what i was looking for, but without scripting, using el only Wkr Peter
... View more
01-27-2021
02:36 AM
Hello, I have a HandleHttpRequest processor configured. I can access the headers as documented but unfortunately the flow file properties (http.headers.XXX) are case sensitive, so if I want to use one in some later processor using EL, I need to know the exact case used (e.g. http.headers.x-My-Header). As it's not possible to control the end that sends the http request I am stuck (unless I add some script processing to extract what I need). Also http spec wise, the header names are case insensitive. So, question - is there a 'standard EL way' to get hold of a header value that came with a HandleHttpRequest processor regardless of the case of the header name? Many thanks! Peter
... View more
Labels:
- Labels:
-
Apache NiFi
02-21-2020
11:23 AM
I ended up building from source based on 1.11.1 tag and then it works fine
... View more
02-14-2020
04:40 AM
Hello
I used tls toolkit to generate keys and certificates
tls-toolkit.sh standalone -n localhost -C 'CN=admin,OU=NIFI'
Started the nifi through the 'latest' docker image
docker run --name securednifi \
-v /home/peter/playground/nifirenderdemo/certs:/opt/certs \
-p 8443:8443 \
-e AUTH=tls \
-e KEYSTORE_PATH=/opt/certs/localhost/keystore.jks \
-e KEYSTORE_TYPE=JKS \
-e KEYSTORE_PASSWORD="xyaCZWLb8GHM20/dghr6nL0p2Z8r6sK5Q81cRZAA/A0" \
-e TRUSTSTORE_PATH=/opt/certs/localhost/truststore.jks \
-e TRUSTSTORE_PASSWORD="4fmyZFm1FevdAmeYEsCHxjzSB/S8sAb+/9w7VQZ0Lms" \
-e TRUSTSTORE_TYPE=JKS \
-e INITIAL_ADMIN_IDENTITY='CN=admin,OU=NIFI' \
-d \
apache/nifi:latest
Passwords were copied from the locahost/nifi.properties file generated by tls-toolkit Certificates was imported in browser Trying to open nifi gives
Insufficient Permissions
Unknown user with identity 'CN=admin, OU=NIFI'. Contact the system administrator.
Checked - nifi.security properties in nifi.properties - users.xml - identity - authorizations.xml - identifier of policy elements matches users.xml
Noticed - The host in nifi.properties configured for nifi.remote.input.host refers to the docker host name (ca....) - In nifi-users.log there is
2020-02-14 10:56:53,454 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: CN=admin,OU=NIFI
...
2020-02-14 10:57:43,063 WARN [NiFi Web Server-75] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.
java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.
at org.apache.nifi.web.api.AccessResource.createAccessTokenFromTicket(AccessResource.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
2020-02-14 10:57:43,362 WARN [NiFi Web Server-75] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response.
java.lang.IllegalStateException: OpenId Connect is not configured.
at org.apache.nifi.web.api.AccessResource.oidcExchange(AccessResource.java:298)
...
2020-02-14 10:57:43,472 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=admin, OU=NIFI) GET https://localhost:8443/nifi-api/flow/current-user (source ip: 172.17.0.1)
2020-02-14 10:57:43,481 INFO [NiFi Web Server-21] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=admin, OU=NIFI
2020-02-14 10:57:43,817 INFO [NiFi Web Server-21] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=admin, OU=NIFI], groups[] does not have permission to access the requested resource. Unknown user with identity 'CN=admin, OU=NIFI'. Returning Forbidden response.
Not sure any of this is a real problem. I also do not immediately find information on the above in the context of two way ssl without ldap or registry
At this point I am out of inspiration tbh So, all tips really appreciated! Tx
Peter
... View more
Labels:
- Labels:
-
Apache NiFi
12-24-2019
03:43 AM
1 Kudo
Hello, Tx for your reaction Dennis. I was able to continue. The reason I was having issues seemed to be related to the fact that the Groovy script was a standard class with a static main method. It makes sense that Nifi does not accept that I guess, but the process of diagnosing such issue might be made easier imo. It was a trial and error task, and as you suggested, by starting from an example. Anyway, I am good now 🙂 Regards Peter
... View more
12-19-2019
10:17 PM
Hello
I am trying to add an ExecuteGroovyScript processor. A simple example test works fine, but when I try to add my script it does not initiate. Exception (on hoover over) is really brief: InstantiationException thrown by groovy invokehelper. Script uses some external jars. As far as I can see I added those to the classpath.
I am probably doing something stupid but I was wondering how to debug this issue? The log files do not seem to contain anything wrt this error which makes troubleshooting very difficult
All tips warmly welcomed!
Tx
Peter
... View more
Labels:
- Labels:
-
Apache NiFi
03-10-2017
04:55 PM
Hello Having the same issue 😞 I find in the hive meta store log 2017-03-10 16:50:52,164 INFO [main]: zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=hive No idea where this is coming from though All tips appreciated!
Peter
... View more
11-26-2016
09:34 PM
It seems that by removing the call to master('yarn') when building the SparkSession, the issue is gone.
... View more
11-26-2016
04:17 PM
1 Kudo
Hello
I am trying to port a spark application from hdp2.3 to hdp2.5 and switch to spark2.
I always seem to run into an issue where the worker(s) cannot find pyspark Traceback (most recent call last):
File "t.py", line 14, in <module>
print (imsi_stayingtime.collect())
File "/usr/hdp/current/spark2-client/python/pyspark/rdd.py", line 776, in collect
port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", line 933, in __call__
File "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py", line 312, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 22, ip-10-0-0-61.eu-west-1.compute.internal): org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named pyspark
PYTHONPATH was:
/hadoop/yarn/local/filecache/13/spark2-hdp-yarn-archive.tar.gz/spark-core_2.11-2.0.0.2.5.0.0-1245.jar
I can easily reproduce it with a very simple hive/spark test app e.g. import pyspark
from pyspark.sql import SparkSession
from operator import add
spark = SparkSession \
.builder \
.master('yarn') \
.appName("Ttt...111") \
.enableHiveSupport() \
.getOrCreate()
report = spark.sql("select imsi,tacs,sum(estimated_staying_time) as total_group_stayingtime from ... where ... group by tacs,imsi")
imsi_stayingtime = report.select('imsi','total_group_stayingtime').rdd.reduceByKey(add)
print (imsi_stayingtime.collect())
I tried to add the zip files (addPyFile), change the enviroment shell file and change spark.yarn.dist.files but nothing seems to help All tips are extremely welcome indeed! Tx Peter
... View more
Labels: