About andrew_sears

andrew_sears · ‎10-01-2016

Hi there, You can get the jar from here. http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-examples/ cheers, Andrew

andrew_sears · ‎08-11-2016

CoarseGrainedExecutorBackend should be in spark-assembly. Might be relevant to you... https://issues.apache.org/jira/browse/OOZIE-2482 https://community.hortonworks.com/articles/49479/how-to-use-oozie-shell-action-to-run-a-spark-job-i.html https://developer.ibm.com/hadoop/2015/11/05/run-spark-job-yarn-oozie/ Try setting SPARK_HOME variable in hadoop-env.sh cheers, Andrew

andrew_sears · ‎07-21-2016

You can see examples of the various alert definitions by getting them using API. The JSON returned can be used as a template to create your own definition. https://docs.daplab.ch/ambari_cheat_sheet/ eg. curl -v -X GET -u ${ambari_credentials} -H 'X-Requested-By:ambari' https://admin.daplab.ch/api/v1/clusters/DAPLAB02/alert_definitions?AlertDefinition/service_name=HIVE The 5 alert types are here. https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Users_Guide/content/_alert_definitions_and_instances.html

andrew_sears · ‎05-13-2016

If you are looking for a way to monitor the job and determine which nodes it ran on, how many executors, etc, you can see this in the Spark Web UI located at <sparkhost>:4040 http://spark.apache.org/docs/latest/monitoring.html http://stackoverflow.com/questions/35059608/pyspark-on-cluster-make-sure-all-nodes-are-used cheers, Andrew

andrew_sears · ‎04-22-2016

FROM salary employees will syntax above alias the salary table as employees. I guess your query needs to join on employees table with left or inner join? Also you will probably not want to overwrite the table you are selecting from? INSERT OVERWRITE TABLE employees SELECT employees.<all columns but salary_date>, salary.salary_date FROM salary inner join employees on salary.employee_number = employee.employee_number;

andrew_sears · ‎04-22-2016

You can use lateral view & explode or inline keyword to get at the data in the struct column. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView https://www.qubole.com/resources/cheatsheet/hive-function-cheat-sheet/ cheers, Andrew

andrew_sears · ‎03-24-2016

Further details on this. Configuring the Spark History Server to Use HDFS https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/config-shs-hdfs.html

andrew_sears · ‎03-16-2016

Does Atlas 0.5 support ALTER statements in Hive Bridge or is it reflected in a later version? These statements don't seem to be reflected in schema for table. ALTER TABLE test SET TBLPROPERTIES ("comment"="test comment"); ALTER TABLE test CHANGE a a string comment 'test column comment'; ALTER TABLE test add COLUMNS (b string comment 'test column comment 2'); In addition, transient_lastDdlTime is set to Jan 17, 1970. This statement shows updated metadata. CREATE TABLE test2 (b string comment 'test column comment 2'); thanks, Andrew

andrew_sears · ‎03-15-2016

Updating link above. https://github.com/apache/incubator-atlas/blob/master/repository/src/main/scala/org/apache/atlas/query/QueryParser.scala

andrew_sears · ‎03-06-2016

Explicit denial is mentioned in next version 0.6 of Ranger. https://cwiki.apache.org/confluence/display/RANGER/Deny-conditions+and+excludes+in+Ranger+policies

Online	Offline
Last Visited	‎02-16-2017 02:23 AM

Member Since	‎12-15-2015 06:28 PM
Last Visited	‎02-16-2017 02:23 AM
Posts	16
Kudos received	9

Cloudera Community

Re: Spark application fails on slaves when launchi...

Re: hadoop-examples-1.1.0-SNAPSHOT.jar - MISSING?

Re: Spark application fails on slaves when launchi...

Re: How to create and register custom ambari alert...

Re: distributed processing operation of dataframe ...

Re: Overwriting a column in HIVE

Re: Creating a HIVE table with org.apache.hcatalog...

Re: Yarn timeline server periodically fails

Atlas 0.5 and Alter Scripts

Re: Where can we find the SQL syntax reference for...

Re: protecting database - Best practice