Re: Read Avro record for the QueryRecord processor

markus_wilhelm · ‎11-02-2018

Hey there, I'm currently trying to get Spark dynamic allocation to work, but sadly it doesn't work. I configured following properties like documented in the documentation: spark.dynamicAllocation.enabled true spark.dynamicAllocation.initialExecutors 3 spark.dynamicAllocation.minExecutors 3 spark.dynamicAllocation.maxExecutors 30 yarn.nodemanager.aux-services.spark2_shuffle.classpath {{stack_root}}/${hdp.version}/spark2/aux/* yarn.nodemanager.aux-services mapreduce_shuffle,spark2_shuffle,{{timeline_collector}} yarn.nodemanager.aux-services.spark2_shuffle.class org.apache.spark.network.yarn.YarnShuffleService It doesn't matter if I try to use Spark in a Zeppelin notebook, with the spark-shell or spark-submit. The job stays at 3 executors and doesn't increase executors, even if the job is taking extremely long. Is there a way I can test if dynamic allocation is activated? As of now it seems for me like it is not. Best regards, Markus

markus_wilhelm · ‎09-17-2018

Hey there, I'm trying to install a new Hadoop cluster with Ambari 2.7. During the master selection of the services I get following error in the web interface: 0 status code received on POST method for API: /api/v1/stacks/HDP/versions/3.0/recommendations After that the next button grays out and I can't continue. I tried looking in the ambari-server log, but there gets nothing logged during this error. I would appreciate any help. Best regards, Markus

markus_wilhelm · ‎08-22-2018

Hey there, I'm trying to use user impersonation for the Spark interpreter in yarn-cluster mode to reduce the load of the Zeppelin host. The HDP cluster is kerberized and Zeppelin is in version 0.8.0. After trying to execute some spark code I get the following error: java.lang.RuntimeException: Error: Only one of --proxy-user or --principal can be provided. A short Google search shows that you can't use spark-submit with both a principal and proxy user, what Zeppelin is trying to do I guess? Does anyone know how I can fix this and use the Spark interpreter in yarn-cluster mode with user impersonation on a kerberized cluster? Any help would be appreciated. Best regards, Markus

markus_wilhelm · ‎07-26-2018

Seems like a error in the SQL statement threw the NullPointerException. Not really the place I would have looked with this kind of error.

markus_wilhelm · ‎07-24-2018

Hey there, I'm trying to query an Avro record with the QueryRecord processor. The Avro gets created with a ExecuteSQL processor. The output of the ExecuteSQL seems fine and I can convert it in e.g. JSON without any problems. But when I try to query it I get a NullPointerException. Anyone able to help me with the problem? QueryRecord queryrecord.png AvroReader avroreader.png AvroRecordSetWriter avrorecordsetwriter.png Exception in the QueryRecord processor 2018-07-24 14:27:37,277 ERROR [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=ac799474-0164-1000-ffff-ffffc83bf6e1] Unable to query StandardFlowFileRecord[uuid=78d37332-b122-4433-b16d-099757d8d839,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532435241579-3, container=default, section=3], offset=0, length=108732158],offset=0,name=6117269883610207,size=108732158] due to java.lang.NullPointerException: {} java.lang.NullPointerException: null at Baz$5$1.current(Unknown Source) at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.next(Linq4j.java:672) at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46) at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:227) at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:51) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:313) at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:306) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

markus_wilhelm · ‎07-12-2018

Thanks for your answer. Your mention of using quotes in the ticket gave me the idea of stating the table name as "time" in the processor. With that it seems to work now. I don't know if this leads to other problems though. Edit: It seems like the where clause doesn't get insert in the statements. I don't know if it needs that though? If I try other kind of columns like numbers it puts a WHERE <column> <= <max_value> in the generated statement.

markus_wilhelm · ‎07-12-2018

Hey there, I want to configure a data flow where I load new data as they get saved in a postgres database. For that I want to use the timestamp column in the GenerateTableFetch processor as maximum value column. Sadly the processor is not able to generate the right sql statements. Unable to execute SQL select query SELECT COUNT(*), MAX(time) time FROM log_table due to ERROR: syntax error at or near "time" It seems to have a problem with the alias, as would need the 'as' statement in postgres, but it seems to use no alias statement at all and fails because of this. Is there a way to fix that? Any help would be appreciated. Best regards, Markus

markus_wilhelm · ‎07-06-2018

NiFi needs to be configured for HTTPS access for the SSO redirecting to work.

markus_wilhelm · ‎07-05-2018

Hey there, I'm currently trying to read files from the HDFS in the HDP cluster where NiFi is running (with HDF management pack). Is there a way that HDFS processors automatically get the configuration for core-site, hdfs-site and kerberos, without having to configure them every time I'm using the HDFS processors? Best regards, Markus

markus_wilhelm · ‎06-26-2018

Hey there, currently trying to configure KnoxSSO authentication for NiFi. Sadly NiFi doesn't redirect to the KnoxSSO login site. The SSO part should work, because I'm already using it for the Ambari login. # Apache Knox SSO Properties # nifi.security.user.knox.url=https://knoxgateway.local:8443/gateway/knoxsso/api/v1/websso nifi.security.user.knox.publicKey=/certs/nifi/knox.pem nifi.security.user.knox.cookieName=hadoop-jwt nifi.security.user.knox. Above is the relevant properties I configured. I didn't find anything else to do in the installation guide. Sadly there aren't any errors, so I really don't know where to go from here. Is there any configuration I may missed which is required for the redirect? Best regards, Markus

markus_wilhelm · ‎05-16-2018

Hey there, I'm trying to kerberize my cluster. I installed Kerberos 5 server following this guide. Everything there seemed to work. After that I tried to kerberize my cluster following the official documentation, but it fails after testing the Kerberos clients with the following error: 500 status code received on POST method for API: /api/v1/clusters/BIKW/requests Error message: An internal system exception occurred: Failed to execute the command: Broken pipe Sadly I couldn't find another post where someone had a similar problem. I attached the log with the complete error stack. ambari-server.zip I would appreciate any help. Best regards, Markus

markus_wilhelm · ‎03-15-2018

Yes this option is configured true.

markus_wilhelm · ‎03-14-2018

@Vitalii Vitrenko Hey there, I'm having the same problem. Did you perhaps find a solution to this problem?

markus_wilhelm · ‎01-26-2018

livy-stderr.txt Hey there, I'm trying to use Spark2 with Livy in Apache Zeppelin. But sadly I can get it to work, because the Livy session always shuts down with the error that it can't initialize the SparkContext. I can't seem to find the problem. The YARN resource manager doesn't even gets a request to start an application (as far as I can see in the logs), so I don't think the problem lies there. Also all other applications work just fine with YARN, including Spark submit and shell requests. Here is the livy.conf: livy.environment production livy.impersonation.enabled true livy.repl.enableHiveContext true livy.rsc.rpc.server.address 192.168.1.22 livy.server.csrf_protection.enabled true livy.server.port 8999 livy.server.recovery.mode recovery livy.server.recovery.state-store filesystem livy.server.recovery.state-store.url /livy2-recovery livy.server.session.timeout 3600000 livy.spark.master yarn livy.superusers zeppelin, livy I included the stderr log for the failed attempt. I would appreciate any help with this problem. Best regards, Markus

markus_wilhelm · ‎12-13-2017

Somehow I can't answer your last post. I managed to add the NiFi service to the Ambari web view, but how do I configure it to actually use it with the web view? Currently I can't start / stop the service or configure it, probably because Ambari doesn't know on which machine in the cluster it runs?

markus_wilhelm · ‎12-12-2017

I checked again and it seems NiFi is installed and works. This still leaves the problem that it doesn't show up in Ambari though. Any suggestions what I could do to add it in Ambari?

markus_wilhelm · ‎12-11-2017

Really can't say. It was a new installation which used the local repositories with the version specified above. I double checked again to make sure.

markus_wilhelm · ‎12-11-2017

HDF 3.0.2.0-76 Ambari 2.6.0.0-267

markus_wilhelm · ‎12-11-2017

Using the public repositories is sadly no possibility in my environment. But I also used the local ones on my HDP cluster without any problems, so I really don't think that the problem lies there.

markus_wilhelm · ‎12-11-2017

No it is a fresh cluster. I tried installing all of HDF3 and NiFi only, the same problem for both of them.

markus_wilhelm · ‎12-08-2017

Hey, I'm trying to install HDF3 on a newly installed CentOS machine. After following the documentation and installing Ambari server it fails with the following error instantly during the service installation: stderr: /var/lib/ambari-agent/data/errors-25.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module> BeforeAnyHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook import params File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module> hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True) TypeError: get_hadoop_dir() got an unexpected keyword argument 'force_latest_on_upgrade' Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-25.txt 2017-12-08 15:01:42,833 - Cannot find stack features for the stack named HDF Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] 2017-12-08 15:01:42,842 - Cannot find stack features for the stack named HDF Command failed after 1 tries I have installed the newest stable version of Ambari and HDF. The repositories are local ones, which are functioning. I already have an HDP Cluster which worked without any problems. Any help would be appreciated.

markus_wilhelm · ‎12-07-2017

Hey there, I'm trying to install Apache NiFi on a HDP 2.6 cluster with Ambari. For that I followed the documentation at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_installing-hdf-on-hdp/bk_installing-hdf-on-hdp.pdf. Now that I want to instal NiFi I get the error that "Skipping stack-select on NIFI because it does not exist in the stack-select package structure" Here are the log files for the installation: output-1056.txt errors-1056.txt I really don't know what to do right now, so I would appreciate any help. Best regards, Markus

Online	Offline
Last Visited	‎01-29-2019 01:47 PM

Date Registered	‎12-07-2017 03:50 PM
Last Visited	‎01-29-2019 01:47 PM
Total Messages Posted	23
Total Kudos Received	2

Re: Read Avro record for the QueryRecord processor

Re: NiFi not redirecting to KnoxSSO

Spark dynamic allocation not working

cluster installation - can't proceed during master...

User impersonation for Spark yarn-cluster mode not...

Re: Read Avro record for the QueryRecord processor

Read Avro record for the QueryRecord processor

Re: Can't use GenerateTableFetch with maximum valu...

Can't use GenerateTableFetch with maximum value co...

Re: NiFi not redirecting to KnoxSSO

Using HDFS processors without additional configura...

NiFi not redirecting to KnoxSSO

Can't kerberize cluster - broken pipe error

Re: Ambari Hive view does not reuse YARN(TEZ) appl...

Re: Ambari Hive view does not reuse YARN(TEZ) appl...

Apache Zeppelin Livy can't initialize SparkContext...

Re: Can't install Apache NiFi on a HDP cluster wit...

Re: Can't install Apache NiFi on a HDP cluster wit...

Re: Can't install HDF3

Re: Can't install HDF3

Re: Can't install Apache NiFi on a HDP cluster wit...

Re: Can't install HDF3

Can't install HDF3

Can't install Apache NiFi on a HDP cluster with Am...