Member since
03-29-2016
46
Posts
25
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
219 | 06-12-2017 09:28 PM | |
532 | 03-30-2017 12:06 PM | |
294 | 03-28-2017 04:12 PM | |
1190 | 12-16-2016 06:34 PM |
06-16-2017
11:35 AM
There is a Sqoop atlas hook if that is the way you are pulling the data from the MySql/Oracle, that will track the lineage. You also have a full set of API's into Atlas (here is a swagger page that will give perspective - http://atlas.incubator.apache.org/api/v2/ui/index.html ), also there are third party Gov tooling that integrates, growing every quarter, and with the latest extended partnership with IBM we will see more around Atlas in the near future - little info at here - http://www.kmworld.com/Articles/News/News/IBM-and-Hortonworks-Expand-Partnership-118808.aspx
... View more
06-14-2017
11:49 AM
2 Kudos
This diagram may help also, and the most recent announcement with deeper IBM partnership will also help excel this space, much more to come.
... View more
06-14-2017
11:38 AM
1 Kudo
Hi Saba, this hive architecture doc from apache gives a good breakdown, if this does not answer your question, please follow up and I can dive a little deeper. http://atlas.incubator.apache.org/Bridge-Hive.html
... View more
06-13-2017
12:11 PM
1 Kudo
Hi Simon, you can try some of the import control options, you can find them here - https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html - i.e - The --null-string and --null-non-string arguments are optional.\ If not specified, then the string "null" will be used.
... View more
06-12-2017
09:28 PM
2 Kudos
Not yet, but it coming. Latest release just announced can now handle HDF and HDP in same ambari instance. Here are the jira's to give you perspective, the multi cluster ambari depends on the multi-everything-architecture - https://issues.apache.org/jira/browse/AMBARI-14714 ambari jira - https://issues.apache.org/jira/browse/AMBARI-20466
... View more
06-12-2017
08:30 PM
Hi, you take a look at the Tez view in ambari and also smartsense activity analysis - https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.4.0/bk_installation/content/activity_analysis.html
... View more
03-30-2017
10:02 PM
Hi Laura, HDP2.6 will be coming out in April, maybe in the next few weeks. I am currently using it internally and both sets of API's seem to be working fine. Nice area to see the new API's if you have not seen it already is here http://atlas.apache.org/api/v2/index.html and you can try it out with swagger. The rest of the details around the release will be coming out very soon, more tag based policy support, performance, +. Better wait on more.
... View more
03-30-2017
12:06 PM
Hi Padmanabhan, you can pull all the client configs using Ambari from the action button download all client configs and put them where you need them. In my 2.4 instance it is in /usr/hdp/2.4.0.0-169/etc/sqoop/conf.dist/sqoop-site.xml
... View more
03-30-2017
10:51 AM
Hi Padmanabhan, this link will give you some insight, you can use shell script/local os variables to do this - https://community.hortonworks.com/questions/64361/how-to-implement-and-deploying-our-own-sqoop-frame.html
... View more
03-30-2017
10:41 AM
Hi Saravanan, yes you can run Pig in local mode without hadoop, it will use the local file system and a single VM.
... View more
03-29-2017
04:34 PM
1 Kudo
Hi maybe this link may help under Use the Parallel Features
(pig.exec.reducers.bytes.per.reducer, pig.exec.reducers.max)
and the PARALLEL key word. https://pig.apache.org/docs/r0.7.0/cookbook.html#Use+the+PARALLEL+Clause Running it under Tez, article may
help also. https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html
... View more
03-29-2017
01:31 PM
Hi Sasitharan, this is a little dated, but may be worth a try - https://hortonworks.com/hadoop-tutorial/connecting-sap-products-to-sandbox/#hana link from within tuttorial - https://2xbbhjxc6wk3v21p62t8n4d4-wpengine.netdna-ssl.com/wp-content/uploads/2013/09/Demo-Tutorial-Leveraging_SAP_HANA__HDP_Jan_2014.pdf
... View more
03-29-2017
12:35 PM
Hi Houssem, maybe these links to the spark on hbase connector may help if that is what you meant by your question. https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/ example - https://hortonworks.com/hadoop-tutorial/spark-hbase-dataframe-based-hbase-connector/
... View more
03-29-2017
12:07 PM
Hi Yair, this does not answer your question, but may help to understand where HDP 3.0 is heading this year with these blog posts - https://hortonworks.com/blog/data-lake-3-0-part-2-multi-colored-yarn/ the 4th blog has links to the other three here - https://hortonworks.com/blog/data-lake-3-0-part-4-cutting-storage-overhead-in-half-with-hdfs-erasure-coding/
... View more
03-28-2017
09:06 PM
This little dated articles may help https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ and https://hortonworks.com/blog/elephants-can-remember-mapreduce-job-history-in-hdp-2-0/ . Also take a look at the timeline server doc. May also be worth taking a look at Tez timeline server if you are using Tez for your jobs. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_yarn_resource_mgt/content/ch_timeline_server.html https://tez.apache.org/tez-ui.html
... View more
03-28-2017
05:14 PM
yah. same for me, sorry about that, I should have checked content
... View more
03-28-2017
04:12 PM
2 Kudos
Hive LLAP is GA with HDP 2.6 which will be out in a few weeks. Has performed very well for me on my testing using the TPC-DS.
... View more
03-28-2017
03:57 PM
Hi, for me it does not work from my chrome browser but does from my safari browser, if that helps, so the file is there. I will dig a little why later and let you know.
... View more
03-28-2017
03:43 PM
Have you looked at the visualization capabilities with zeppelin, may be an option. Can also incorporate some D3.js or additional process with graphx, etc. into zeppelin if needed. Just food for thought.
... View more
01-22-2017
04:17 PM
2 Kudos
These HCC links may help if you have not seen them https://community.hortonworks.com/questions/77703/is-there-an-existing-feature-to-stop-user-from-kil.html And this thread has some info that may help through the links https://community.hortonworks.com/questions/1705/storm-multi-tenancy-best-practices.html
... View more
12-20-2016
02:11 PM
Good question, I have only used Sqoop with the mapper aspect, Sqoop 1 is client based tool so eval tool is more than likely just using jdbc from client jvm. There are other ways to import data and apply structure, if that is what you are looking for. The Hive view can give you some ideas around quickly importing data and applying structure, if you are thinking the self service type aspects. The Tez engine with LLAP have really increased the access times around query.
... View more
12-19-2016
08:12 PM
Curious why the mapper aspect is a concern for you. Sqoop will build mappers for you and run them in parallel with you deciding the number you want (default 4, but no reason why you could not use 1). It will evaluate how best to split the jobs. It will also save off the mapper code in case you want to use it or tweak it for future work.
... View more
12-19-2016
07:01 PM
If you are using the sandbox, or you can check your Hive config Hive_Database value, you may be using MySQL for that, if that helps
... View more
12-16-2016
06:34 PM
These links from previous questions may help https://community.hortonworks.com/questions/58916/utf-8-hive.html https://community.hortonworks.com/articles/58548/processing-files-in-hive-using-native-non-utf8-cha.html
... View more
12-16-2016
02:03 PM
1 Kudo
Can ambari Zeppelin View propagate the Ambari authentication into Zeppelin client view, or is a separate authentication within the iframe required? I will give this a try myself, just curious of insights
... View more
Labels:
12-16-2016
12:04 PM
Quick question, curious of perspective, does it make sense to use falcon snapshot support to just manage the snapshots for a single cluster and not necessarily the DR replication aspects.
... View more
09-29-2016
07:11 PM
1 Kudo
Nice step by step, will save a bunch of time for people
... View more
09-26-2016
07:40 PM
Hi Artem, trying the new virtual box VM, one I had was same build, oh well, different issue, with zookeeper/hbase it looks like, does not start, with no stderr per se, log showing below. Few threads on this, let you know if I figure it out, input welcome 2016-09-26 19:06:47,804 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/172.17.0.2:61181. Will not attempt to authenticate using SASL (unknown error) 2016-09-26 19:06:47,804 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
... View more
09-26-2016
11:21 AM
1 Kudo
Hi Vasilis, thanks for getting topic going. An article that would help for sure, and maybe I can help when I get back, is showing the taxonomy aspects and adding terms to assets. The earlier versions of tech preview had samples already running.
... View more
09-20-2016
11:19 PM
Ambari Metrics Server not coming up with HDP 2.5 sandbox on my Virtual Box VM, anybody having a similar problem? OSError: [Errno 2] No such file or directory: '/var/lib/ambari-metrics-collector/hbase-tmp/Jetty_0_0_0_0_61310_master____36m3iw/.active' - log below. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 148, in <module>
AmsCollector().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 46, in start
self.configure(env, action = 'start') # for security
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 41, in configure
hbase('master', action)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py", line 105, in hbase
recursive_ownership = True,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 199, in action_create
recursion_follow_links=self.resource.recursion_follow_links, safemode_folders=self.resource.safemode_folders)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 73, in _ensure_metadata
sudo.chown_recursive(path, _user_entity, _group_entity, recursion_follow_links)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 53, in chown_recursive os.lchown(os.path.join(root, name), uid, gid)
... View more
Labels: