Member since
06-09-2016
529
Posts
129
Kudos Received
104
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1373 | 09-11-2019 10:19 AM | |
8329 | 11-26-2018 07:04 PM | |
1940 | 11-14-2018 12:10 PM | |
4047 | 11-14-2018 12:09 PM | |
2646 | 11-12-2018 01:19 PM |
08-13-2018
06:48 PM
@Harun
Zengin please let me know if the above has helped you? Thanks.
... View more
07-25-2018
02:10 PM
Thanks a lot @Felix Albani...you solved my issue.
... View more
07-20-2018
03:03 PM
Thanks for sharing the detailed information and will try to update the spark-env to use default mode as yarn.
... View more
07-19-2018
04:03 PM
@David Pocivalnik yes, that is correct! Please if you are satifsfied with the answer remember to login and mark as accepted.
... View more
05-21-2019
01:46 PM
I have NIFI API Link https://nifi.apache.org/docs/nifi-docs/rest-api/index.html But dont understand how to use it? How do you know when to stop NIFI Job.. i mean how do you know all processors have processed the data? Could you please share the documentation if any about it.
... View more
07-09-2018
01:57 PM
I'm not sure if it matters; but it seems there is a 1.5 hour difference between the 2 ps -ef -s ; it might be possible that somehow one of the hs2 instances have not shut down cleanly earlier; I would recommend to try shutting both hs2 down; and checking that they are both stopped by hand ; check that the configs are updated - and start both of them...
... View more
07-04-2018
08:08 PM
1 Kudo
@Zeev Grinberg Here is the link for latest odbc driver for mac: https://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/hive-odbc/2.1.12.1017/OSX/hive-odbc-native.dmg You can find all drivers under HDP Add-Ons in the following URL: https://hortonworks.com/downloads/ HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.
... View more
02-23-2019
06:00 PM
for example, in one of my DAG, all that those task do is Sort WithinPartition (so no shuffle) still it spills data on disk because partition size is huge and spark resort to ExternalMergeSort. As a result, I have a high Shuffle Spill (memor) and also some Shuffle Spill(Disk). There is no shuffle here. but on the other hand you can argue that Sorting process moves data in order to sort so it's kind of internal shuffle 🙂
... View more
07-03-2018
11:39 AM
@Vladislav Shcherbakov I think using InvokeScriptedProcessor may be a good option on this case. Please take a look here: http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html Also there is a similar article that you could review here: https://community.hortonworks.com/articles/193822/parsing-web-pages-for-images-with-apache-nifi.html HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.
... View more
06-20-2018
03:56 PM
2 Kudos
This article with cover step by step how to configure HDP Search Solr with Ranger Plugin Step 1 Download and install the 2.2.9+ mpack - FYI: Previous mpack versions don't support integration of HDP Search Solr with Ranger. Mpack 2.2.9 includes configurable section for the solr-security on ambari which allows to add the authorization information. wget 'http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service-mpack-2.2.9.tar.gz'; -O /tmp/solr-service-mpack-2.2.9.tar.gz
ambari-server install-mpack --mpack=/tmp/solr-service-mpack-2.2.9.tar.gz Step 2 On HDP solr host run yum install ranger-solr-plugin.noarch
cd /usr/hdp/2.6.2.0-205/ranger-solr-plugin Edit install.properties and make sure the following settings at least are properly configured: POLICY_MGR_URL=http://<ranger-host>:6080
SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar Edit solr-plugin-install.properties and set correct value for install dir: COMPONENT_INSTALL_DIR_NAME=/opt/lucidworks-hdpsearch/solr/server Next source the environment and enable the plugin: source /etc/hadoop/hadoop-env.sh
./enable-solr-plugin.sh Step 3 Update security znode with ranger authorization class kinit -kt solr.service.keytab solr/<host>@REALM.COM
/opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -z '<zookeeper>:2181' -cmd put /solr/security.json '{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"},"authorization":{"class": "org.apache.ranger.authorization.solr.authorizer.RangerSolrAuthorizer"}}' Also on Ambari->Solr-Config->Advance solr-security set: {
"authentication":{"class": "org.apache.solr.security.KerberosPlugin"},
"authorization":{"class": "org.apache.ranger.authorization.solr.authorizer.RangerSolrAuthorizer"}
} Save and restart, on the operation start output you should see - call['/opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper1>:2181,<zookeeper2>:2181,<zookeeper3>:2181 -cmd get /solr/security.json'] {'timeout': 60, 'env': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112'}}
- call returned (0, '{\"authentication\":{\"class\": \"org.apache.solr.security.KerberosPlugin\"},\"authorization\":{\"class\": \"org.apache.ranger.authorization.solr.authorizer.RangerSolrAuthorizer\"}}')
- Solr Security Json was found, it will not be overridden Step 4 Fix clustername for solr plugin cd /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/WEB-INF/classes/ Edit ranger-solr-audit.xml and add following property <property>
<name>ranger.plugin.solr.ambari.cluster.name</name>
<value>YOUR_CLUSTER_NAME</value>
</property> Restart solr for changes to reflect. Step 5 Open Ranger Admin UI and edit solr repository add New Configurations tag.download.auth.users = solr
policy.download.auth.users = solr
ambari.service.check.user = ambari-qa
... View more
Labels: