About gmartin

gmartin · ‎09-25-2017

@rmr1989 In-place migrations from RHL6 to RHL7 are challenging. Would advise to upgrade HDP 2.4 to 2.6.2 as is, and use swing kit to install RHL7 on new nodes, which you add to the cluster. You then migrate the services off the RHL6 nodes onto the new RHL7 asap. It is advised to run with a mixed cluster (RHL6 and RHL7) for as short a time as possible. As usual, read the docs, and test as much as possible.

gmartin · ‎09-19-2017

@ketan kunde Some comments: - HDCloud is an AWS specific Provisioning capability. It is just a layer built upon Cloudbreak, which is open-source. - SmartSense is more of a service. Cluster logs, metrics, configs, volumetrics, etc. are captured in a bundle and analyzed to produce a set of recommendations/reports. It is more of a process or a service, than an opensource project. - HCP, aka CyberSecurity, is based on Apache Metron. There are other Apache Projects in HCP - Ni-Fi, Storm, Kafka, Ambari, Ranger (i.e. HDP and HDF).

gmartin · ‎08-01-2017

@Zubair Jaleel There are many Kappa and other case studies presented at the DataWorks Summit (e.g. Ford, Yahoo, etc.). Videos and Slides are available for most sessions: https://dataworkssummit.com/san-jose-2017/agenda/

gmartin · ‎08-01-2017

@Smart Data Ranger can be used to sync users with LDAP/AD. Credentials are stored in LDAP/AD, and Ranger configured to access. Knox is used as a proxy, but more for REST API service calls, and some UIs. It is not meant to proxy high volume traffic like Kafka messages.

gmartin · ‎08-01-2017

@Smart Data Atlas is more Governance related, security to a less extent. You secure Kafka via Kerberos for authentication, and Ranger for authorization: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/index.html#bk_security

gmartin · ‎07-28-2017

@Gagan SinghThe ports are different to the HDP Sandbox. Ambari is on port 9080, please see the tutorial below:https://hortonworks.com/tutorial/real-time-event-processing-in-nifi-sam-schema-registry-and-superset/

gmartin · ‎07-28-2017

@Kiran Kumar Should all be answered in the below: https://hortonworks.com/agreements/support-services-policy/

gmartin · ‎07-25-2017

One comment is RHL 7.3 is not supported with Ambari 2.2.2/HDP2.4.3

gmartin · ‎07-25-2017

@somesh sriramoju In the list of connectors, there is a drop down option for HDP: https://datascience.ibm.com/docs/content/local/createconnectionslocal.html?linkInPage=true Regards,

gmartin · ‎07-25-2017

@Irfan syafii - Take a look at the Ambari Metrics to understand some of the core cluster metrics. Current versions of HDP have Grafana Dashboards to give more detail. - There a number of benchmark options: https://gist.github.com/ace-subido/0a9b219b2348921f6a87 https://github.com/hortonworks/hive-testbench TESTDFSIO being a good option to get an understanding of read/write performance (lots of articles on that). The second link relates to Hive and TPC type benchmarks. - For more advanced users, there is also SmartSense with HDP, which can make recommendations on performance issues. Regards,

gmartin · ‎07-18-2017

@Mihai Lucaciu The IDs and password is provided below - admin/hadoophadoop https://hortonworks.com/tutorial/real-time-event-processing-in-nifi-sam-schema-registry-and-superset/#step7

gmartin · ‎07-18-2017

@Maurice Hickey With HDP 3.x this will be much easier, but AFAIK this isn't as elegant prior to that. Keep in mind that there is one RDBMS maintaining the Hive Metadata - and therefore you can have only one Database/Table entry. In theory, you could have 2 x HS2/Metastores (QA/DEV) pointing to a different Metastore RDBMS each - but you lose the ability to query between the two (as DEV and QA are separate) unless you federate at a level above the cluster. A common approach would be to split at the database level, so you can keep the same table names. You need to change for PROD, so would advise changing DB name is part of the promotion process to PROD anyway, so should be done from DEV to QA).

gmartin · ‎06-21-2017

@Roberto Sancho This error can occur if you do not have adequate permissions on the file. I would suggest you run the 'fsck' command to determine if there are any corrupt blocks on HDFS. Is it only certain queries that fail, or all queries? Would also locate the under-lying files on HDFS, and check permissions, etc.

gmartin · ‎06-21-2017

@priyal patel Have you considered using Ni-Fi instead. Lots of articles on Ni-Fi capturing API data, including from FB: https://community.hortonworks.com/questions/65310/apache-nifi-facebook-integration.html

gmartin · ‎06-20-2017

@Saby SS We always advise the Knox is not designed to be a proxy/sentry for bulk data xfer. It is fine as an ODBC interface, passing summary data back to the client, but not designed for large volumes. For large volumes, you might be better served with an edge node, or jump box inside the secure zone and close to the cluster.

gmartin · ‎06-19-2017

@Farrukh Munir The HA Guide covers this, and yes Ambari can be used to install, configure, and manage the services. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/index.html#bk_hadoop-high-availability

gmartin · ‎06-15-2017

@Qinglin Xia Checking you created the Tag Service in Ranger. You need to define the Tag Service to enable tags flowing from Atlas to Ranger. And always worth checking all the components are online - Ranger Tag-sync, Kafka, etc.

gmartin · ‎06-15-2017

@ahmed bejaoui I would recommend: https://community.hortonworks.com/questions/103514/nifi-and-storm-withour-kafka.html and also to look at the Partner Demo Kit: https://community.hortonworks.com/articles/75341/partner-demo-kit-1.html which has a packaged demo of the IoT Trucking App - which includes Ni-Fi, Kafka, and Storm. All ready to go in a few easy steps.

gmartin · ‎06-11-2017

@Hemil Shah 1. Correct, each DC would have its own Zookeeper ensemble. The offsets are stored here. 2. Intra-Data Centre data redundancy can be provided by Kafka Replication. Cross-Data Centre requires MirrorMaker. For reference below, Mirror Maker best Practices. https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html

gmartin · ‎06-04-2017

@nbalaji-elangovan If you perform a search on tensorflow in the HCC Search, there are a number of excellent articles, such as: https://community.hortonworks.com/articles/54954/setting-up-gpu-enabled-tensorflow-to-work-with-zep.html plus articles on DZone: https://dzone.com/articles/tensorflow-on-the-edge-part-2-of-5

gmartin · ‎06-01-2017

@Vishal Gupta Would always advise to review the documentation on the Ambari and HDP upgrade. Generally, no, you don't need to re-gen all of the keytabs or SSL in the cluster as part of an upgrade - though Ambari will generate keytabs as required post-upgrade. https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-upgrade/content/upgrading_HDP_post_upgrade_tasks_ranger_kerberos.html There are certain cases which need some attention (e.g. Kafka at 2.2, Ranger HA, etc.). https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/index.html You can raise a ticket with Hortonworks Support prior to the upgrade, as they can inform you on any known issues (if you have a subscription). Also, would recommend to move to 2.5.5 if moving to 2.5. And testing is imperative. Would advise to test all of the components under representative use cases.

gmartin · ‎05-31-2017

@Sharon Kirkham Nothing obvious come to mind....so - Re-cycle of the Cloud Controller - Kill your browser, and re-try. - Same issue for different Images (e.g. the Data Science image)? - Same issue for different regions (or can you only provision on one region)? - Reducing the size of the cluster?

gmartin · ‎05-31-2017

@Sharon Kirkham Exact same config, and it works fine. The ssh key you are using is working ok, you can ssh onto an AWS host with it?

gmartin · ‎05-31-2017

@Sharon Kirkham Looking at another json (same config, worked 2 days ago), all is the same, except for spot price (maybe check if it works by removing), and the ClusterAndAmbariPassword is empty (though maybe you have manually removed before publishing?). To confirm, you have access/authority to provision resources on AWS also?

gmartin · ‎05-30-2017

@katelyn thomas If you use the 'asparagus' chart below (even if you are not using HDP), you can select the combination of components that have been tested with each other. Not just Storm, ZK, but also Kafka, HDFS, etc., depending on what your source/targets are: https://hortonworks.com/products/data-center/hdp/ The Hortonworks Sandbox contains all of the HDP components installed, configured, and ready to use. The Partner Demo kit also has a Storm use case pre-built: https://community.hortonworks.com/articles/75341/partner-demo-kit-1.html

gmartin · ‎05-30-2017

@Sanaz Janbakhsh Assigning the Solr standalone (you can still install via Ambari Management pack) to the slaves, at least separates the Solr-Cloud Infra, from the Solr standalone. The other consideration is what storage are you using to store the indexes on Solr standalone. There are a number of threads where this is discussed on HCC. In summary storing the indexes on HDFS makes it easier to manage. If performance and response time it important, than you should review storing the indexes on fast disk, like SSD.

gmartin · ‎05-29-2017

@Saurabh FYI https://community.hortonworks.com/questions/82135/how-to-limit-access-to-zeppelin-webui-based-for-sp.html

gmartin · ‎05-29-2017

@Karl Can you check: - you are able to run a sparkR shell, e.g. /usr/hdp/current/spark2-client/bin/sparkR - in the /var/log/spark2, /var/log/spark, /var/log/zeppelin directories, are there any error messages. - all the related services and components are running (Spark, Spark2, Zeppelin, HDFS, etc.) - what R packages are installed?

gmartin · ‎05-28-2017

@Allen Niu For reference, see this thread with a number of comments around this topic: https://community.hortonworks.com/questions/91789/traditional-etl-vs-open-source.html#answer-91800

gmartin · ‎05-25-2017

@Christophe Vico I recommend you download the Sandbox: https://hortonworks.com/products/sandbox/ From Zeppelin, and in the one notebook, you can run different versions of Spark (1.6.3 or 2.1) as per your choice of interpreter: %spark.spark or %spark2.spark You can review the settings used in the Interpreter screen. Regards,

Online	Offline
Last Visited	‎08-15-2019 06:35 AM

Date Registered	‎07-07-2016 10:30 AM
Last Visited	‎08-15-2019 06:35 AM
Total Messages Posted	79
Total Kudos Received	17

Re: Kappa Architecture Using HDP/HDF

Re: Atlas : how to secure Kafka ?

Re: HDP Versions end of support roadmap

Re: difference between nifi, storm and kafka and t...

Re: Does Kerberos needs to be redone after a Hadoo...

Re: HDP & Linux upgrade

Re: Which all products of Horton Works are 100 % o...

Re: Kappa Architecture Using HDP/HDF

Re: Atlas : how to secure Kafka ?

Re: Atlas : how to secure Kafka ?

Re: Unable to access web client 127.0.0.1:14200 an...

Re: HDP Versions end of support roadmap

Re: Hosts are not added to the Ambari with HA serv...

Re: how do i configure IBM DSX Local in existing H...

Re: I am a new user in hadoop system, i want to be...

Re: Incorrect HDF Sandbox user/passwords for Super...

Re: Mutliple Hive Metastores.

Re: hdfs block corrupt

Re: which flume source is used to transfer data fr...

Re: KNOX Hive performance issue

Re: Deploy HA Multinode Cluster using Ambari

Re: Atlas-Ranger Tag based policy not working

Re: difference between nifi, storm and kafka and t...

Re: Does Kafka Mirroring needs different zookeeper...

Re: Looking for steps to install and run tensorflo...

Re: Does Kerberos needs to be redone after a Hadoo...

Re: AWS HDC create cluster button grayed out

Re: AWS HDC create cluster button grayed out

Re: AWS HDC create cluster button grayed out

Re: What is Apache Storm Compatibility?

Re: Assign Solr to the Master and Slaves nodes

Re: Can we give access to users for only specific ...

Re: Getting SparkR to work on Zeppelin (HDP2.6 San...

Re: Data Transformation: Tools (Nifi, Talend) or ...

Re: HDP2.6 / Zeppelin0.7 / How to have both Spark ...