About georg_kf_heiler

georg_kf_heiler · ‎10-05-2018

How can I manage multiple versions of a JAR using Oozie share-libs? Think about the following scenario. My company is currently stuck on using HDP x which supports spark in version y. How can I configure Oozie share-libs and oozie-spark-action to execute against a custom version of spark (custom build, or i.e. latest 2.4 community snapshot) ?

georg_kf_heiler · ‎05-04-2018

Further digging around in the hive source code I have found: https://github.com/apache/hive/commit/8ce0118ffe517f0c622571778251cbd9f760c4f5#diff-a0e344e574e0fe542ad8a98e64c967cf in particular https://github.com/apache/hive/blob/1eea5a80ded2df33d57b2296b3bed98cb18383fd/ql/src/test/queries/clientpositive/reloadJar.q leads me to believe that hfs should be supported. --! qt:dataset:src dfs -mkdir ${system:test.tmp.dir}/aux; dfs -cp ${system:hive.root}/data/files/identity_udf.jar ${system:test.tmp.dir}/aux/udfexample.jar; SET hive.reloadable.aux.jars.path=${system:test.tmp.dir}/aux; RELOAD; CREATE TEMPORARY FUNCTION example_iden AS 'IdentityStringUDF'; EXPLAIN SELECT example_iden(key) FROM src LIMIT 1; SELECT example_iden(key) FROM src LIMIT 1; DROP TEMPORARY FUNCTION example_iden; dfs -rm -r ${system:test.tmp.dir}/aux; EDIT It appears, that CREATE TEMPORARY FUNCTION example_iden AS 'IdentityStringUDF'; throws a Warning of : WARN. . permanent functions created without USIJNG clause will not be replicated so I assume the USING /path/to/jar.jar is mandatory for permanent UDFs even when reloadable flag is set.

georg_kf_heiler · ‎05-03-2018

Hi,I want to update Hive UDFs without requiring a restart of hive. According to: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_mc_hive_udf.html#concept_zb2_rxr_lw setting hive.reloadable.aux.jars.path is required. I have set it to /user/hive/libs/udf (which resides on HDFS). However following their documentation I see: file:///usr/lib/hive/lib/foo.jar which is confusing me. Does this property only work for files residing on the local file system? Do I understand correctly. that I should execute beelines reload manually? Also in case this property works for HDFS does it automatically pick up the classes in the jar (load them) and no longer requires to specify the CREATE FUNCTION foo AS 'my/path/to/jar-1.jar'? Desired behaviour: 1. copy jar to HDFS /user/hive/lib/udf/foo-1.jar 2. add function to hive: DROP FUNCTION IF EXISTS foo; CREATE FUNCTION foo AS 'my.class.path.in.jar.FooUDF' USING JAR '/user/hive/lib/udf/foo-1.jar'; 3. add a new jar to HDFS /user/hive/lib/udf/foo-2.jar 4. update function in hive: DROP FUNCTION IF EXISTS foo; CREATE FUNCTION foo AS 'my.class.path.in.jar.FooUDF' USING JAR '/user/hive/lib/udf/foo-2.jar'; This currently does not work and requires a restart of hive. It results in round robin seeing the updated UDF (or still the old one). How can I get hive to not require a restart when updating UDF? Also I do not want to put the UDF locally into a directory. It should reside on HDFS. Best, Georg

georg_kf_heiler · ‎04-16-2018

confirmed that set to false, also Maximum Total Concurrent Queries > 0. The problem seems to have resolved itself after restarting a couple of times.

georg_kf_heiler · ‎04-12-2018

When executing the following query on LLAP: WITH first as (SELECT 1 as a, 1 as b) SELECT * FROM first as f JOIN first as f2 ON f.a=f2.a Hive fails with: IllegalArgumentException: max parallelism must be positive, currently is 0

georg_kf_heiler · ‎02-19-2018

I face problems with a hive udf and java dependency hell: 2018-02-19 10:11:49,328 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster java.lang.VerifyError: Bad return type Exception Details: Location: org/apache/hadoop/hdfs/DFSClient.getQuotaUsage(Ljava/lang/String;)Lorg/apache/hadoop/fs/QuotaUsage; @94: areturn Reason: Type 'org/apache/hadoop/fs/ContentSummary' (current frame, stack[0]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage' (from method signature) Obviously, somehow dependencies are clashing with wrong versions. When looking at the jar via jar tf myjar.jar | grep hdfs no contents are returned. The same for fd This seems strange to me as the exception clearly states that these should be involved in the problem. The error occurs only on Tez, not on regular hive.

georg_kf_heiler · ‎07-05-2017

I want to use sparks JDBC connection to write a data frame to oracle. My data frame has a string column which is very long. At least longer than the default 255 characters the spark will allocate when creating the schema. How can I still write to the oracle table? Does it work when I manually create the schema first with CLOB datatypes? If yes, how can I get spark to only `TRUNCATE` instead of overwrite the table?

georg_kf_heiler · ‎06-28-2017

When following along with https://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-dataset and trying to register my own kryo encoder for a data frame I face an issue of Schema for type com.esri.core.geometry.Envelopeis not supported There is a function which will parse a String (WKT) to an geometry object like: def mapWKTToEnvelope(wkt: String): Envelope = { val envBound = new Envelope() val spatialReference = SpatialReference.create(4326) // Parse the WKT String into a Geometry Object val ogcObj = OGCGeometry.fromText(wkt) ogcObj.setSpatialReference(spatialReference) ogcObj.getEsriGeometry.queryEnvelope(envBound) envBound } This is applied to an UDF like: implicit val envelopeEncoder: Encoder[Envelope] = Encoders.kryo[Envelope] val ST_Envelope = udf((wkt: String) => mapWKTToEnvelope(wkt)) However, the UDF will compile but throw a runtime error of: [error] Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type com.esri.core.geometry.Envelope is not supported [error] at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:733) [error] at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:671) [error] at org.apache.spark.sql.functions$.udf(functions.scala:3076) Whereas val first = df[(String, String)].first val envBound = new Envelope() val ogcObj = OGCGeometry.fromText(first._1) ogcObj.setSpatialReference(spatialReference) ogcObj.getEsriGeometry.queryEnvelope(envBound) spark.createDataset(Seq((envBound)))(envelopeEncoder) Works just fine: root |-- value: binary (nullable = true) +--------------------+ | value| +--------------------+ |[01 00 63 6F 6D 2...| +-------------------- How can I get it to work in the UDF as well?

georg_kf_heiler · ‎04-24-2017

Thanks a lot for the great tutorial. How could this be extended to not only listen to a web socket, but rather periodically send control commands like: https://blockchain.info/api/api_websocket for example `{"op":"unconfirmed_sub"}`?

georg_kf_heiler · ‎04-07-2017

I see some hdp will only ship default spark? I mean it is just a compile switch for spark. Having that would be nice. Can I just replace the hdp binaries?

georg_kf_heiler · ‎04-07-2017

Well, at least spark has the option to include the machine learning options. I am just interested if hdp is deploying these optimizations or if manual work is required.

georg_kf_heiler · ‎04-06-2017

Hi, is there any possibility in the newly released HDP 2.6 to install NIFI in cluster mode via ambari?

georg_kf_heiler · ‎04-04-2017

There is a similar question: https://community.hortonworks.com/questions/52359/netlib-java-and-anaconda-for-spark-ml.html without netlike lgpl answered.

georg_kf_heiler · ‎04-04-2017

Hi, I wonder if the spark version provided by HDP will actually provide http://spark.apache.org/docs/latest/ml-guide.html mentioned optimizations compiled in: MLlib uses the linear algebra package Breeze, which depends on netlib-java for optimised numerical processing. If native libraries1 are not available at runtime, you will see a warning message and a pure JVM implementation will be used instead.

georg_kf_heiler · ‎03-27-2017

How can I use NIFI to select some (ranked) list of records from a database? I want to use every k-th record (or a better ranking in the future) to select some records which should be processed by NIFI as a request to a web service. The Webservice will provide some information (enrich) each record. NIFI should remember which ones were already sent to the rate limited Webservice. But, as this web service will provide more up to date information for each record every 2 months I would like NIFI to automatically refresh the information of each record after initially fetching some values. How could something like this be achieved in NIFI, and if not possible out of the box what operators would come close? I.e. maybe just take every k-th record from a database until all rows were read, then start again but with a different k (i.e. after 2 months)

georg_kf_heiler · ‎03-27-2017

How can I batch read data from a source (lets say CSV or JDBC) into NIFI to query a webservice? I want to read only every k-th row (lets say 10th row) and read 50.000 records into a single batch. However, I must keep a state, i.e. know which records have already been processed by NIFI.

georg_kf_heiler · ‎02-20-2017

Where is the environment variable $ACCUMULO_HOME defined for HDP 2.5.3? In the config I can only find references to this variable for further directories but not the actual definition. In /usr/hdp/current/ There no longer is a single accumulo folder, rather I see several folders (accumulo-master, accumulo-gc, accumulo-tracer, ...)

georg_kf_heiler · ‎02-16-2017

Great. Do you know when 2.5.4 should be released?

georg_kf_heiler · ‎02-15-2017

Ambari will support the installation of spark 2.0.0 as a technical preview. This version of spark contains a lot of bugs which are fixed in 2.0.2 or 2.1.0 how can I install any of these versions via ambari?

georg_kf_heiler · ‎02-14-2017

Is it possible to export a blueprint of the current cluster configuration?

georg_kf_heiler · ‎02-14-2017

Initially, I deployed a blueprint to ambari.Having used the nice ambari UI to create some configuration changes I would like to know how to export the current cluster configuration as a blueprint. If this is not possible, how can I access the "default" configuration to know the config values which need to be passed in a blueprint.

georg_kf_heiler · ‎02-13-2017

I used``` "policymgr_external_url" : "https://{% if isSingleNode %} {{ groups[cluster_name+'_mn01'][0] }} {% else %} {{ groups[cluster_name+'_mn03'][0] }} {% endif %}:6182", {% else %} "policymgr_external_url" : "http://{% if isSingleNode %} {{ groups[cluster_name+'_mn01'][0] }} {% else %} {{ groups[cluster_name+'_mn03'][0] }}{% endif %}:6080", {% endif %} ``` to create the JSON. It is adopted from https://github.com/bushnoh/ansible-hadoop-asap/blob/master/blueprints/bare_cluster.bp.j2 When removing the spaces inn all places ineed, finally ambari successfully installs ranger.

georg_kf_heiler · ‎02-13-2017

When installing Ranger via a blueprint I get the following exception: 2017-02-13 18:22:10,754 [E] create_dbversion_catalog.sql file import failed! 2017-02-13 18:22:40,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mn01.vagrant :3306/ranger -u 'ranger' -p '********' -noheader -trim -c \; -query "show tables like 'x_db_version_h';" SQLException : SQL state: 3D000 java.sql.SQLException: No database selected ErrorCode: 1046 SQLException : SQL state: 3D000 java.sql.SQLException: No database selected ErrorCode: 1046 2017-02-13 18:22:41,149 [I] Table x_db_version_h does not exist in database ranger 2017-02-13 18:22:41,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mn01.vagrant :3306/ranger -u 'ranger' -p '********' -noheader -trim -c \; -input /usr/hdp/current/ranger-admin/db/mysql/create_dbversion_catalog.sql Error executing: create table if not exists x_db_version_h ( id bigint not null auto_increment primary key, version varchar(64) not null, inst_at timestamp not null default current_timestamp, inst_by varchar(256) not null, updated_at timestamp null default null, updated_by varchar(256) not null, active ENUM('Y', 'N') default 'Y' ) ; java.sql.SQLException: No database selected SQLException : SQL state: 3D000 java.sql.SQLException: No database selected ErrorCode: 1046 2017-02-13 18:22:41,556 [E] create_dbversion_catalog.sql file import failed! The blueprint's json can be found here: https://gist.github.com/geoHeil/bbe4eb9cef4f4e6c2feca743f2b19bc8 The complete input json is found here https://gist.github.com/geoHeil/b54181d35c0d4549c0da25465cc93e29 and the full output txt https://gist.github.com/geoHeil/6b3d08d748e03703b35ddce424b108c1

georg_kf_heiler · ‎02-08-2017

@Ashnee Sharma is there any documentation how to perform this task?

georg_kf_heiler · ‎02-08-2017

Can I use ambari to install the very latest and for hdp still unsupported versions e.g. Of spark or accumulo? I don't mean the technical previews here but rather the current version of these services.

georg_kf_heiler · ‎02-02-2017

So what would sync sort offer over plain spark sql then?

georg_kf_heiler · ‎02-02-2017

I recently read this news announcement http://hortonworks.com/partner/syncsort/ and am wondering why sync sort is proposed instead of nifi.

georg_kf_heiler · ‎10-22-2016

Hi, is there a component in HDP similar to http://predictionio.incubator.apache.org? If not, what approach would you suggest? I would like to build a predictive API based on spark and gradient boosting and deploy it as an API.

Online	Offline
Last Visited	‎10-05-2018 05:26 AM

Date Registered	‎10-22-2016 09:04 AM
Last Visited	‎10-05-2018 05:26 AM
Total Messages Posted	28
Total Kudos Received	5

manage multiple versions of a jar i.e. spark via O...

Re: updating hive UDF without restarting the clust...

updating hive UDF without restarting the cluster

Re: Hive LLAP parallelism configuration

Hive LLAP parallelism configuration

hive UDF java dependency hell

spark jdbc oracle long string fields and truncate

spark custom kryo encoder not providing schema for...

Re: NiFi WebSocket support

Re: Does HDP spark include netLib optimizations?

Re: Does HDP spark include netLib optimizations?

HDF NIFI on existing HDP in cluster mode on edge n...

Re: Does HDP spark include netLib optimizations?

Does HDP spark include netLib optimizations?

NFI JDBC batches remember state

Nifi batch read every k-th row form a source

HDP 2.5 $ACCUMULO_HOME

Re: Spark 2 Technical preview with patches

Spark 2 Technical preview with patches

Re: Automate HDP installation using Ambari Bluepri...

ambari export blueprint

Re: Install Ranger via blueprint

Install Ranger via blueprint

Re: Install unsupported latest versions

Install unsupported latest versions

Re: Why use syncsort instead of nify?

Why use syncsort instead of nify?

model as predictive API