Member since
09-25-2015
72
Posts
61
Kudos Received
20
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2198 | 03-10-2017 01:57 PM | |
788 | 12-14-2016 01:22 PM | |
748 | 12-12-2016 10:54 AM | |
1758 | 11-07-2016 04:24 PM | |
387 | 09-23-2016 12:32 PM |
03-13-2017
08:24 PM
1 Kudo
Hi @SC SC, DS2v2 is too small for data nodes and not properly sized for benchmarking. Please see https://community.hortonworks.com/articles/22376/recommendations-for-microsoft-azure-hdp-deployment-1.html for Azure sizing recommendations and HDP. /Best regards, Mats
... View more
03-10-2017
02:26 PM
2 Kudos
Hi @Greg Lawrance, The only bridge AFAIK is MITI Meta Bridge which supports both PDI and Atlas in Beta. There are a couple of projects on Github which will give an idea on how you can do it (REST API, Java). https://github.com/shivajid/atlas https://github.com/yhemanth/atlas-demo https://github.com/vakshorton/NifiAtlasBridge /Best regards, Mats
... View more
03-10-2017
01:57 PM
1 Kudo
Hi @rahul gulati, Atlas GUI works for Chrome and Firefox. I'm using HDP 2.5.3 and the latest Chrome version without any problems. hdfs_path is an Atlas metadata type but unfortunately it's no HDFS hook available in 2.5 which automatically populates the hdfs_path so you probably do not have any hdfs entity data in Atlas. HDP 2.6 will include some functionality for HDFS entity creation. You can always build your own hook - a good doc to understand the Atlas model is http://atlas.incubator.apache.org/0.7.1-incubating/AtlasTechnicalUserGuide.pdf. /Best regards, Mats
... View more
03-10-2017
01:07 PM
2 Kudos
HI @Mehul Shah, You probably missing Java JCE support or UNlimited Strength Chryptographic plocies (USC) or have some mismatch with your keys for OpenSSL. Please visit http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.2/bk_dataflow-ambari-installation/content/distribute_and_install_the_jce.html (This is for HDF 2.1 installed with Ambari but same doc is available for other install options). There are also some info around PKCS12/OpenSSL in https://issues.apache.org/jira/browse/NIFI-3062. /Best regards, Mats
... View more
12-27-2016
01:52 PM
2 Kudos
Hi @vamsi valiveti, Oozie is a scheduler and Flume is not working on a schedule basis instead Flume is treating the data when it receives it. So you use teh Flume configuration to tell for example that each time there is a file in a certain directory Flume will put it in hdfs (if you use the spooldir source) and so on. /Best regards, Mats
... View more
12-14-2016
01:22 PM
1 Kudo
Hi @Roger Young, Haven't played with Minifi but with Nifi in general source origin is a Flow File attribute which you can find under the Attributes tab when viewing a Flow File under the data provenance menu. /Best regards, Mats
... View more
12-14-2016
12:37 PM
1 Kudo
Hi @Arsalan Siddiqi, I recommend you to use Ambari as it will give you access to all configuration parameters and also to data files in HDFS by using the Ambari File View. Please be aware that if you are using Ambari, any changes to config needs to be done through the Ambari GUI or API and editing the config files directly will not work (they are just a copy of the Ambari config). /Best regards, Mats
... View more
12-12-2016
03:50 PM
1 Kudo
Hi @anand maurya, Please have a look at the Hortonworks-Securosis White Paper regarding Security in Hadoop. The paper can be found at: http://hortonworks.com/info/securing-hadoop/ /Best regards, Mats
... View more
12-12-2016
11:17 AM
Hi @subash sharma, Make sure your Java environment is configured properly - for example JAVA_HOME. Please have a look at https://community.hortonworks.com/questions/39839/how-to-import-metadata-from-hive-into-atlas-and-th.html /Best regards, Mats
... View more
12-12-2016
10:54 AM
1 Kudo
Hi @Avijeet Dash, The Solr index requires persistent storage as well. There are several options to read Hbase from Hive and Solr from Hive and they all include storage handlers and SerDes such as https://github.com/lucidworks/hive-solr and https://github.com/chimpler/hive-solr. Also for Hive/Hbase integration there is https://cwiki.apache.org/confluence/display/Hive/StorageHandlers Hope this helps. /Best regards, Mats
... View more
11-07-2016
04:24 PM
2 Kudos
Hi @Edgar Daeds, Hive Metsatore is designed for Innodb and do not support MyISAM or ndbcluster as database type in MySQL. You can still replicate your Metastore by using MySQL replication of the binlogs.
... View more
11-07-2016
04:01 PM
Hi @vshukla, How do I enable impersonation for %sh? I can't find it in the Zeppelin GUI or in the docs? /Best regards, Mats
... View more
11-04-2016
01:05 PM
Hi, Docs says: "Shell interpreter uses Apache Commons Exec to execute external processes. In Zeppelin notebook, you can use %sh in the beginning of a paragraph to invoke system shell and run commands. Note : Currently each command runs as the user Zeppelin server is running as." Is there any way we can execute the shell interpreter as the current Zeppelin user? Like introducing a special shell for Apache Commons Exec, passing a $USER parameter etc? /Best regards, Mats
... View more
Labels:
- Labels:
-
Apache Zeppelin
09-23-2016
12:32 PM
1 Kudo
Hi @Shishir Jaiswal, Assuming HDP components on the Edge nodes such as clients and Knox then mixed versions are not supported. Knox for example only supports specific versions of an Hadoop service (see section 2.8.1.2 in http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/bk_security-20160829.pdf). Mixed OS versions do most times not affect things as long as it's fullfil all requirements around libraries, JVM versions etc. /Best regards, Mats
... View more
09-23-2016
12:06 PM
1 Kudo
Hi @Mohana Murali Gurunathan, Unfortunately not, Ranger have currently plugins and support for HDFS, Hive, HBase, Kafka, Knox, YARN, Storm and Atlas. For mor einfo please visit: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/overview_ranger_ambari_install.html /Best regards, Mats
... View more
09-23-2016
08:51 AM
Hi @rama, You can delete the old container logs. If yarn.nodemanager.log-dir is full no new containers will start on that node. See also yarn.nodemanager.disk-health-checker.min-healthy-disks yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb /Best regards, Mats
... View more
09-22-2016
02:54 PM
1 Kudo
Hi @sivasaravanakumar k, The rate of replication work is throttled by HDFS to not interfere with cluster traffic when failures happen during regular cluster load. Some properties controlling this are dfs.namenode.replication.work.multiplier.per.iteration , dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit . The foremost controls the rate of work to be scheduled to a DN at every heartbeat that occurs, and the other two further limit the maximum parallel threaded network transfers done by a DataNode at a time. Some description of this is available at https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml /Best regards, Mats
... View more
09-22-2016
02:39 PM
1 Kudo
Hi @Mahesh Mallikarjunappa, The Flume agent is typically listen to a logfile etc until it's aborted by an operator so that is where the"long-lived process" comes in. For more info on Flume please visit - https://flume.apache.org/FlumeUserGuide.html /Best regards, Mats
... View more
09-19-2016
09:32 AM
1 Kudo
Hi @R c, The exam is handled by PSI Services / Innovative Exams so you need to contact them. You can contact them at examsupport@examslocal.com, or call +1-888-504-9178, +1-312-612-1049. /Best regards, Mats
... View more
09-16-2016
02:57 PM
2 Kudos
HI @Ron Buckley, Driver above are the ones which Hortonworks distribute and support. As AIX not is binary compatible with Linux any of the above drivers will not work with AIX. If you can get the source code you can compile it for AIX using the AIX Linux affinity libraries which according to IBM makes AIX Linux source code compatible. I also found Download ODBC connectors which is commercial AIX Hive ODBC driver from Progress Data Direct. /Best regards, Mats
... View more
09-16-2016
10:39 AM
Hi @Jitendra, Please have a look at http://hadoop.apache.org/versioning.html - It cover Ambari as well. /Best regards, Mats
... View more
09-16-2016
10:31 AM
3 Kudos
Hi @Sunil Mukati, Yes it works with the "Generic" Database Type but it do not accept Netezza specific syntax. You also need to setup a DBCPConnectionPool and JDBC connection for your Netezza box. /Best regards, Mats
... View more
09-14-2016
11:35 AM
Sorry, too quick on the trigger. Normally SPARK_HOME=/usr/hdp/current/spark-client In my (old) sandbox the start_master.sh script is located at /usr/hdp/2.4.0.0-169/spark/sbin/start-master.sh.
... View more
06-14-2016
09:37 AM
Hi, Not sure what you mean with Spark on R? If that is RHadoop, MS R Server or DistributedR those are not really comparable just like that with SparkR, ie no drop-in replacement, as the functionality differs. So before doing a performance comparison you have to look into which functionality you would like to compare. /Best regards, Mats
... View more
06-14-2016
08:16 AM
Sounds ok as long as you continue to use /usr/hdp as the directory path for the new drive and shut down the cluster/node while doing the change.
... View more
06-14-2016
08:11 AM
1 Kudo
Hi @Uday Vakalapudi, HDP do not require any special sizing for /dev as long as we follow the OS recommendations. The "baseline" we present includes general Linux requirements for system partitions. So in this case I recommend to follow the requirements for /dev issued by the operating system vendor.
... View more
05-10-2016
02:26 PM
@Predrag Minovic - Is it possible to remove the auditlog in hdfs as well without affecting lineage etc?
... View more
05-09-2016
08:05 AM
3 Kudos
Is there any best practices on how to handle the ranger_audit.xa_access_audit table over time when it's growing big. In particular can we truncate/delete rows without affecting lineage?
... View more
Labels:
- Labels:
-
Apache Ranger
04-25-2016
06:49 PM
Looks like hadoop-yarn-common-x.x.x.jar is missing.
... View more