Member since
12-07-2017
23
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1002 | 07-26-2018 07:34 AM | |
666 | 07-06-2018 08:26 AM |
11-02-2018
07:24 AM
Hey there, I'm currently trying to get Spark dynamic allocation to work, but sadly it doesn't work. I configured following properties like documented in the documentation: spark.dynamicAllocation.enabled true spark.dynamicAllocation.initialExecutors 3 spark.dynamicAllocation.minExecutors 3 spark.dynamicAllocation.maxExecutors 30 yarn.nodemanager.aux-services.spark2_shuffle.classpath {{stack_root}}/${hdp.version}/spark2/aux/* yarn.nodemanager.aux-services mapreduce_shuffle,spark2_shuffle,{{timeline_collector}} yarn.nodemanager.aux-services.spark2_shuffle.class org.apache.spark.network.yarn.YarnShuffleService It doesn't matter if I try to use Spark in a Zeppelin notebook, with the spark-shell or spark-submit. The job stays at 3 executors and doesn't increase executors, even if the job is taking extremely long. Is there a way I can test if dynamic allocation is activated? As of now it seems for me like it is not. Best regards, Markus
... View more
Labels:
- Labels:
-
Apache Spark
09-17-2018
02:49 PM
Hey there, I'm trying to install a new Hadoop cluster with Ambari 2.7. During the master selection of the services I get following error in the web interface: 0 status code received on POST method for API: /api/v1/stacks/HDP/versions/3.0/recommendations After that the next button grays out and I can't continue. I tried looking in the ambari-server log, but there gets nothing logged during this error. I would appreciate any help. Best regards, Markus
... View more
Labels:
08-22-2018
11:42 AM
1 Kudo
Hey there, I'm trying to use user impersonation for the Spark interpreter in yarn-cluster mode to reduce the load of the Zeppelin host. The HDP cluster is kerberized and Zeppelin is in version 0.8.0. After trying to execute some spark code I get the following error: java.lang.RuntimeException: Error: Only one of --proxy-user or --principal can be provided.
A short Google search shows that you can't use spark-submit with both a principal and proxy user, what Zeppelin is trying to do I guess? Does anyone know how I can fix this and use the Spark interpreter in yarn-cluster mode with user impersonation on a kerberized cluster? Any help would be appreciated. Best regards, Markus
... View more
Labels:
07-26-2018
07:34 AM
Seems like a error in the SQL statement threw the NullPointerException. Not really the place I would have looked with this kind of error.
... View more
07-24-2018
12:46 PM
Hey there, I'm trying to query an Avro record with the QueryRecord processor. The Avro gets created with a ExecuteSQL processor. The output of the ExecuteSQL seems fine and I can convert it in e.g. JSON without any problems. But when I try to query it I get a NullPointerException. Anyone able to help me with the problem? QueryRecord queryrecord.png AvroReader avroreader.png AvroRecordSetWriter avrorecordsetwriter.png Exception in the QueryRecord processor 2018-07-24 14:27:37,277 ERROR [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=ac799474-0164-1000-ffff-ffffc83bf6e1] Unable to query StandardFlowFileRecord[uuid=78d37332-b122-4433-b16d-099757d8d839,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1532435241579-3, container=default, section=3], offset=0, length=108732158],offset=0,name=6117269883610207,size=108732158] due to java.lang.NullPointerException: {}
java.lang.NullPointerException: null
at Baz$5$1.current(Unknown Source)
at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.next(Linq4j.java:672)
at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:227)
at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:51)
at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:313)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571)
at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:306)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache NiFi
07-12-2018
02:41 PM
Thanks for your answer. Your mention of using quotes in the ticket gave me the idea of stating the table name as "time" in the processor. With that it seems to work now. I don't know if this leads to other problems though. Edit: It seems like the where clause doesn't get insert in the statements. I don't know if it needs that though? If I try other kind of columns like numbers it puts a WHERE <column> <= <max_value> in the generated statement.
... View more
07-12-2018
11:08 AM
Hey there, I want to configure a data flow where I load new data as they get saved in a postgres database. For that I want to use the timestamp column in the GenerateTableFetch processor as maximum value column. Sadly the processor is not able to generate the right sql statements. Unable to execute SQL select query SELECT COUNT(*), MAX(time) time FROM log_table due to ERROR: syntax error at or near "time" It seems to have a problem with the alias, as would need the 'as' statement in postgres, but it seems to use no alias statement at all and fails because of this. Is there a way to fix that? Any help would be appreciated. Best regards, Markus
... View more
Labels:
- Labels:
-
Apache NiFi
07-06-2018
08:26 AM
NiFi needs to be configured for HTTPS access for the SSO redirecting to work.
... View more
07-05-2018
12:06 PM
Hey there, I'm currently trying to read files from the HDFS in the HDP cluster where NiFi is running (with HDF management pack). Is there a way that HDFS processors automatically get the configuration for core-site, hdfs-site and kerberos, without having to configure them every time I'm using the HDFS processors? Best regards, Markus
... View more
06-26-2018
06:11 AM
Hey there, currently trying to configure KnoxSSO authentication for NiFi. Sadly NiFi doesn't redirect to the KnoxSSO login site. The SSO part should work, because I'm already using it for the Ambari login. # Apache Knox SSO Properties #
nifi.security.user.knox.url=https://knoxgateway.local:8443/gateway/knoxsso/api/v1/websso
nifi.security.user.knox.publicKey=/certs/nifi/knox.pem
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.knox.
Above is the relevant properties I configured. I didn't find anything else to do in the installation guide. Sadly there aren't any errors, so I really don't know where to go from here. Is there any configuration I may missed which is required for the redirect? Best regards, Markus
... View more
Labels:
05-16-2018
06:15 AM
Hey there, I'm trying to kerberize my cluster. I installed Kerberos 5 server following this guide. Everything there seemed to work. After that I tried to kerberize my cluster following the official documentation, but it fails after testing the Kerberos clients with the following error: 500 status code received on POST method for API: /api/v1/clusters/BIKW/requests
Error message: An internal system exception occurred: Failed to execute the command: Broken pipe
Sadly I couldn't find another post where someone had a similar problem. I attached the log with the complete error stack. ambari-server.zip I would appreciate any help. Best regards, Markus
... View more
Labels:
03-15-2018
10:10 AM
Yes this option is configured true.
... View more
03-14-2018
06:59 AM
@Vitalii Vitrenko Hey there, I'm having the same problem. Did you perhaps find a solution to this problem?
... View more
01-26-2018
09:06 AM
livy-stderr.txt Hey there, I'm trying to use Spark2 with Livy in Apache Zeppelin. But sadly I can get it to work, because the Livy session always shuts down with the error that it can't initialize the SparkContext. I can't seem to find the problem. The YARN resource manager doesn't even gets a request to start an application (as far as I can see in the logs), so I don't think the problem lies there. Also all other applications work just fine with YARN, including Spark submit and shell requests. Here is the livy.conf: livy.environment production
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.rsc.rpc.server.address 192.168.1.22
livy.server.csrf_protection.enabled true
livy.server.port 8999
livy.server.recovery.mode recovery
livy.server.recovery.state-store filesystem
livy.server.recovery.state-store.url /livy2-recovery
livy.server.session.timeout 3600000
livy.spark.master yarn
livy.superusers zeppelin, livy I included the stderr log for the failed attempt. I would appreciate any help with this problem. Best regards, Markus
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
12-13-2017
08:07 AM
Somehow I can't answer your last post. I managed to add the NiFi service to the Ambari web view, but how do I configure it to actually use it with the web view? Currently I can't start / stop the service or configure it, probably because Ambari doesn't know on which machine in the cluster it runs?
... View more
12-12-2017
08:27 AM
I checked again and it seems NiFi is installed and works. This still leaves the problem that it doesn't show up in Ambari though. Any suggestions what I could do to add it in Ambari?
... View more
12-11-2017
09:55 AM
Really can't say. It was a new installation which used the local repositories with the version specified above. I double checked again to make sure.
... View more
12-11-2017
07:20 AM
Using the public repositories is sadly no possibility in my environment. But I also used the local ones on my HDP cluster without any problems, so I really don't think that the problem lies there.
... View more
12-11-2017
07:18 AM
No it is a fresh cluster. I tried installing all of HDF3 and NiFi only, the same problem for both of them.
... View more
12-08-2017
02:14 PM
1 Kudo
Hey, I'm trying to install HDF3 on a newly installed CentOS machine. After following the documentation and installing Ambari server it fails with the following error instantly during the service installation: stderr: /var/lib/ambari-agent/data/errors-25.txt Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 35, in <module>
BeforeAnyHook().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py", line 26, in hook
import params
File "/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/params.py", line 101, in <module>
hadoop_home = stack_select.get_hadoop_dir("home", force_latest_on_upgrade=True)
TypeError: get_hadoop_dir() got an unexpected keyword argument 'force_latest_on_upgrade'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', ''] stdout: /var/lib/ambari-agent/data/output-25.txt 2017-12-08 15:01:42,833 - Cannot find stack features for the stack named HDF
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-25.json', '/var/lib/ambari-agent/cache/stacks/HDF/2.0/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-25.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
2017-12-08 15:01:42,842 - Cannot find stack features for the stack named HDF
Command failed after 1 tries I have installed the newest stable version of Ambari and HDF. The repositories are local ones, which are functioning. I already have an HDP Cluster which worked without any problems. Any help would be appreciated.
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)
12-07-2017
03:58 PM
Hey there, I'm trying to install Apache NiFi on a HDP 2.6 cluster with Ambari. For that I followed the documentation at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_installing-hdf-on-hdp/bk_installing-hdf-on-hdp.pdf. Now that I want to instal NiFi I get the error that "Skipping stack-select on NIFI because it does not exist in the stack-select package structure" Here are the log files for the installation: output-1056.txt errors-1056.txt I really don't know what to do right now, so I would appreciate any help. Best regards, Markus
... View more
Labels: