markus_wilhelm
Explorer
My Accepted Solutions
Show All

Spark dynamic allocation not working

by Explorer in Support Questions
‎11-02-2018 07:24 AM
‎11-02-2018 07:24 AM
Hey there, I'm currently trying to get Spark dynamic allocation to work, but sadly it doesn't work. I configured following properties like documented in the documentation: spark.dynamicAllocation.enabled true spark.dynamicAllocation.initialExecutors 3 spark.dynamicAllocation.minExecutors 3 spark.dynamicAllocation.maxExecutors 30 yarn.nodemanager.aux-services.spark2_shuffle.classpath {{stack_root}}/${hdp.version}/spark2/aux/* yarn.nodemanager.aux-services mapreduce_shuffle,spark2_shuffle,{{timeline_collector}} yarn.nodemanager.aux-services.spark2_shuffle.class org.apache.spark.network.yarn.YarnShuffleService It doesn't matter if I try to use Spark in a Zeppelin notebook, with the spark-shell or spark-submit. The job stays at 3 executors and doesn't increase executors, even if the job is taking extremely long. Is there a way I can test if dynamic allocation is activated? As of now it seems for me like it is not. Best regards, Markus ... View more
Labels:

cluster installation - can't proceed during master...

by Explorer in Support Questions
‎09-17-2018 02:49 PM
‎09-17-2018 02:49 PM
Hey there, I'm trying to install a new Hadoop cluster with Ambari 2.7. During the master selection of the services I get following error in the web interface: 0 status code received on POST method for API: /api/v1/stacks/HDP/versions/3.0/recommendations After that the next button grays out and I can't continue. I tried looking in the ambari-server log, but there gets nothing logged during this error. I would appreciate any help. Best regards, Markus ... View more

User impersonation for Spark yarn-cluster mode not...

by Explorer in Support Questions
‎08-22-2018 11:42 AM
‎08-22-2018 11:42 AM
Hey there, I'm trying to use user impersonation for the Spark interpreter in yarn-cluster mode to reduce the load of the Zeppelin host. The HDP cluster is kerberized and Zeppelin is in version 0.8.0. After trying to execute some spark code I get the following error: java.lang.RuntimeException: Error: Only one of --proxy-user or --principal can be provided. A short Google search shows that you can't use spark-submit with both a principal and proxy user, what Zeppelin is trying to do I guess? Does anyone know how I can fix this and use the Spark interpreter in yarn-cluster mode with user impersonation on a kerberized cluster? Any help would be appreciated. Best regards, Markus ... View more

Scroll past end of Zeppelin notebook

by Explorer in Support Questions
‎08-16-2018 08:06 AM
‎08-16-2018 08:06 AM
Hey there, just a short question: Is it possible to configure that you can scroll past the ending of a notebook? Right now it is pretty bad because if you work on a notebook you are always at the bottom of the screen. I'm used to having code in the mid of the screen and not in the last inches of the screen. Any help would be appreciated. Best regards, Markus ... View more
Labels:

Re: Read Avro record for the QueryRecord processor

by Explorer in Support Questions
‎07-26-2018 07:34 AM
‎07-26-2018 07:34 AM
Seems like a error in the SQL statement threw the NullPointerException. Not really the place I would have looked with this kind of error. ... View more

Read Avro record for the QueryRecord processor

by Explorer in Support Questions
‎07-24-2018 12:46 PM
‎07-24-2018 12:46 PM
Hey there, I'm trying to query an Avro record with the QueryRecord processor. The Avro gets created with a ExecuteSQL processor. The output of the ExecuteSQL seems fine and I can convert it in e.g. JSON without any problems. But when I try to query it I get a NullPointerException. Anyone able to help me with the problem? QueryRecord queryrecord.png AvroReader avroreader.png AvroRecordSetWriter avrorecordsetwriter.png Exception in the QueryRecord processor 2018-07-24 14:27:37,277 ERROR [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=ac799474-0164-1000-ffff-ffffc83bf6e1] Unable to query StandardFlowFileRecord[uuid=78d37332-b122-4433-b16d-099757d8d839,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532435241579-3, container=default, section=3], offset=0, length=108732158],offset=0,name=6117269883610207,size=108732158] due to java.lang.NullPointerException: {} java.lang.NullPointerException: null at Baz$5$1.current(Unknown Source) at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.next(Linq4j.java:672) at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46) at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:227) at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:51) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:313) at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:306) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more
Labels:

Re: Can't use GenerateTableFetch with maximum valu...

by Explorer in Support Questions
‎07-12-2018 02:41 PM
‎07-12-2018 02:41 PM
Thanks for your answer. Your mention of using quotes in the ticket gave me the idea of stating the table name as "time" in the processor. With that it seems to work now. I don't know if this leads to other problems though. Edit: It seems like the where clause doesn't get insert in the statements. I don't know if it needs that though? If I try other kind of columns like numbers it puts a WHERE <column> <= <max_value> in the generated statement. ... View more

Can't use GenerateTableFetch with maximum value co...

by Explorer in Support Questions
‎07-12-2018 11:08 AM
‎07-12-2018 11:08 AM
Hey there, I want to configure a data flow where I load new data as they get saved in a postgres database. For that I want to use the timestamp column in the GenerateTableFetch processor as maximum value column. Sadly the processor is not able to generate the right sql statements. Unable to execute SQL select query SELECT COUNT(*), MAX(time) time FROM log_table due to ERROR: syntax error at or near "time" It seems to have a problem with the alias, as would need the 'as' statement in postgres, but it seems to use no alias statement at all and fails because of this. Is there a way to fix that? Any help would be appreciated. Best regards, Markus ... View more
Labels:

Re: NiFi not redirecting to KnoxSSO

by Explorer in Support Questions
‎07-06-2018 08:26 AM
‎07-06-2018 08:26 AM
NiFi needs to be configured for HTTPS access for the SSO redirecting to work. ... View more

Using HDFS processors without additional configura...

by Explorer in Support Questions
‎07-05-2018 12:06 PM
‎07-05-2018 12:06 PM
Hey there, I'm currently trying to read files from the HDFS in the HDP cluster where NiFi is running (with HDF management pack). Is there a way that HDFS processors automatically get the configuration for core-site, hdfs-site and kerberos, without having to configure them every time I'm using the HDFS processors? Best regards, Markus ... View more

NiFi not redirecting to KnoxSSO

by Explorer in Support Questions
‎06-26-2018 06:11 AM
‎06-26-2018 06:11 AM
Hey there, currently trying to configure KnoxSSO authentication for NiFi. Sadly NiFi doesn't redirect to the KnoxSSO login site. The SSO part should work, because I'm already using it for the Ambari login. # Apache Knox SSO Properties # nifi.security.user.knox.url=https://knoxgateway.local:8443/gateway/knoxsso/api/v1/websso nifi.security.user.knox.publicKey=/certs/nifi/knox.pem nifi.security.user.knox.cookieName=hadoop-jwt nifi.security.user.knox. Above is the relevant properties I configured. I didn't find anything else to do in the installation guide. Sadly there aren't any errors, so I really don't know where to go from here. Is there any configuration I may missed which is required for the redirect? Best regards, Markus ... View more

Can't kerberize cluster - broken pipe error

by Explorer in Support Questions
‎05-16-2018 06:15 AM
‎05-16-2018 06:15 AM
Hey there, I'm trying to kerberize my cluster. I installed Kerberos 5 server following this guide. Everything there seemed to work. After that I tried to kerberize my cluster following the official documentation, but it fails after testing the Kerberos clients with the following error: 500 status code received on POST method for API: /api/v1/clusters/BIKW/requests Error message: An internal system exception occurred: Failed to execute the command: Broken pipe Sadly I couldn't find another post where someone had a similar problem. I attached the log with the complete error stack. ambari-server.zip I would appreciate any help. Best regards, Markus ... View more

Re: Ambari Hive view does not reuse YARN(TEZ) appl...

by Explorer in Support Questions
‎03-15-2018 10:10 AM
‎03-15-2018 10:10 AM
Yes this option is configured true. ... View more

Re: Ambari Hive view does not reuse YARN(TEZ) appl...

by Explorer in Support Questions
‎03-14-2018 06:59 AM
‎03-14-2018 06:59 AM
@Vitalii Vitrenko Hey there, I'm having the same problem. Did you perhaps find a solution to this problem? ... View more

Apache Zeppelin Livy can't initialize SparkContext...

by Explorer in Support Questions
‎01-26-2018 09:06 AM
‎01-26-2018 09:06 AM
livy-stderr.txt Hey there, I'm trying to use Spark2 with Livy in Apache Zeppelin. But sadly I can get it to work, because the Livy session always shuts down with the error that it can't initialize the SparkContext. I can't seem to find the problem. The YARN resource manager doesn't even gets a request to start an application (as far as I can see in the logs), so I don't think the problem lies there. Also all other applications work just fine with YARN, including Spark submit and shell requests. Here is the livy.conf: livy.environment production livy.impersonation.enabled true livy.repl.enableHiveContext true livy.rsc.rpc.server.address 192.168.1.22 livy.server.csrf_protection.enabled true livy.server.port 8999 livy.server.recovery.mode recovery livy.server.recovery.state-store filesystem livy.server.recovery.state-store.url /livy2-recovery livy.server.session.timeout 3600000 livy.spark.master yarn livy.superusers zeppelin, livy I included the stderr log for the failed attempt. I would appreciate any help with this problem. Best regards, Markus ... View more

Re: Can't install Apache NiFi on a HDP cluster wit...

by Explorer in Support Questions
‎12-13-2017 08:07 AM
‎12-13-2017 08:07 AM
Somehow I can't answer your last post. I managed to add the NiFi service to the Ambari web view, but how do I configure it to actually use it with the web view? Currently I can't start / stop the service or configure it, probably because Ambari doesn't know on which machine in the cluster it runs? ... View more

Re: Can't install Apache NiFi on a HDP cluster wit...

by Explorer in Support Questions
‎12-12-2017 08:27 AM
‎12-12-2017 08:27 AM
I checked again and it seems NiFi is installed and works. This still leaves the problem that it doesn't show up in Ambari though. Any suggestions what I could do to add it in Ambari? ... View more

Re: Can't install HDF3

by Explorer in Support Questions
‎12-11-2017 09:55 AM
‎12-11-2017 09:55 AM
Really can't say. It was a new installation which used the local repositories with the version specified above. I double checked again to make sure. ... View more

Re: Can't install HDF3

by Explorer in Support Questions
‎12-11-2017 08:20 AM
‎12-11-2017 08:20 AM
HDF 3.0.2.0-76 Ambari 2.6.0.0-267 ... View more

Re: Can't install Apache NiFi on a HDP cluster wit...

by Explorer in Support Questions
‎12-11-2017 07:20 AM
‎12-11-2017 07:20 AM
Using the public repositories is sadly no possibility in my environment. But I also used the local ones on my HDP cluster without any problems, so I really don't think that the problem lies there. ... View more

Re: Can't install HDF3

by Explorer in Support Questions
‎12-11-2017 07:18 AM
‎12-11-2017 07:18 AM
No it is a fresh cluster. I tried installing all of HDF3 and NiFi only, the same problem for both of them. ... View more

Can't install HDF3

by Explorer in Support Questions
‎12-08-2017 02:14 PM
1 Kudo
‎12-08-2017 02:14 PM
1 Kudo
Hey, I'm trying to install HDF3 on a newly installed CentOS machine. After following the documentation and installing Ambari server it fails with the following error instantly during the service installation: stderr: /var/lib/ambari-agent/data/errors-25.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module> BeforeAnyHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook import params File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module> hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True) TypeError: get_hadoop_dir() got an unexpected keyword argument 'force_latest_on_upgrade' Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-25.txt 2017-12-08 15:01:42,833 - Cannot find stack features for the stack named HDF Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] 2017-12-08 15:01:42,842 - Cannot find stack features for the stack named HDF Command failed after 1 tries I have installed the newest stable version of Ambari and HDF. The repositories are local ones, which are functioning. I already have an HDP Cluster which worked without any problems. Any help would be appreciated. ... View more

Can't install Apache NiFi on a HDP cluster with Am...

by Explorer in Support Questions
‎12-07-2017 03:58 PM
‎12-07-2017 03:58 PM
Hey there, I'm trying to install Apache NiFi on a HDP 2.6 cluster with Ambari. For that I followed the documentation at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_installing-hdf-on-hdp/bk_installing-hdf-on-hdp.pdf. Now that I want to instal NiFi I get the error that "Skipping stack-select on NIFI because it does not exist in the stack-select package structure" Here are the log files for the installation: output-1056.txt errors-1056.txt I really don't know what to do right now, so I would appreciate any help. Best regards, Markus ... View more
Contact Me
Online
Offline
Last Visited
‎01-29-2019 01:47 PM
Public Statistics
Date Registered ‎12-07-2017 03:50 PM
Last Visited ‎01-29-2019 01:47 PM
Total Messages Posted 23
Total Kudos Received 1