About TimothySpann

TimothySpann · ‎07-08-2016

hive> add jar urldetector-1.0-jar-with-dependencies.jar; add jar urldetector-1.0-jar-with-dependencies.jar Added [urldetector-1.0-jar-with-dependencies.jar] to class path Added resources: [urldetector-1.0-jar-with-dependencies.jar] hive> CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector'; CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector' OK Time taken: 0.032 seconds hive> select urldetector(remote_host), remote_host from AccessLogs limit 100; select urldetector(remote_host), remote_host from AccessLogs limit 100 OK _c0 remote_host Sent: 68.180.229.154 Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.Text com.dataflowdeveloper.detection.URLDetector.evaluate(org.apache.hadoop.io.Text) throws java.io.IOException on object com.dataflowdeveloper.detection.URLDetector@5dbd1a3e of class com.dataflowdeveloper.detection.URLDetector with arguments {68.180.229.154:org.apache.hadoop.io.Text} of size 1 Time taken: 0.147 seconds This is when trying to load a Hive UDF.

TimothySpann · ‎07-08-2016

https://nifi.apache.org/docs.html see this article https://community.hortonworks.com/articles/28341/converting-csv-to-avro-with-apache-nifi.html

TimothySpann · ‎07-08-2016

have you looked at alluxio as a virtual layer over hdfs and s3

TimothySpann · ‎07-08-2016

See this post: https://community.hortonworks.com/questions/21011/how-i-extract-attribute-from-json-file-using-nifi.html And some more information on creating directories: https://kisstechdocs.wordpress.com/2015/01/18/simple-tasks-in-nifi-file-objects-by-date/ Look at the split route example here: https://community.hortonworks.com/articles/961/a-collection-of-nifi-examples.html Also useful for routing and scripting based on values: https://blogs.apache.org/nifi/entry/indexing_tweets_with_nifi_and

TimothySpann · ‎07-08-2016

I don't know why it was picked but Apache Storm is very production ready, mature and very well instrumented for metrics, debugging and running real production code. It is also real streaming not micro-batching. Hortonworks also has committers on that project. Spark and Flink are very new and maturing.

TimothySpann · ‎07-08-2016

It is easier to run Squirrel from your PC or Mac and point it at the Sandbox Phoenix server (make sure ports are open). If you installed the Linux version on the sandbox, you will need to be in the VM graphical UI to see Squirrel Sql run as it is a GUI program and not a Web UI. Can you tell me how you ran it? I downloaded it via wget. Then [root@sandbox demo]# java -jar squirrel-sql-3.7.1-standard.jar Welcome to the installation of SQuirreL SQL Client 3.7.1! - Gerd Wagner <gerdwagner@users.sourceforge.net> - Rob Manning <manningr@users.sourceforge.net> - Colin Bell <colbell@users.sourceforge.net> The homepage is at: http://www.squirrelsql.org press 1 to continue, 2 to quit, 3 to redisplay ... (pick a lot of options) [ Unpacking finished ] Install was successful application installed on /opt/demo/squirrelsql [ Console installation done ] After that you run with /opt/demo/squirrelsql/ ./squirrel-sql.sh Unfortunately if you do not have a graphical UI in your Sandbox (HDP 2.5 is just command-line and web UI) you will not be able to run Squirrel SQL from there. Install it on your Host machine and run from there. The VirtualBox image does not have a graphical UI (no XWindows by default).

TimothySpann · ‎07-08-2016

The brokers were up, I shutdown everything and restarted the box and everything was okay. This is for the sandbox.

TimothySpann · ‎07-07-2016

Adding HDF (with Apache NiFi) to your HDP 2.5 Sandbox is very quick, painless and easy. Get the most recent Hortonworks DataFlow (download😞 wget http://d3d0kdwqv675cq.cloudfront.net/HDF/centos6/1.x/updates/1.2.0.1/HDF-1.2.0.1-1.tar.gz tar -xvf HDF-1.2.0.1-1.tar.gz cd HDF-1.2.0.1-1/nifi/ Then change the port used by NiFi in the conf/nifi.properties file to: nifi.web.http.port=8090 Install NiFi as a Linux Service bin/nifi.sh install sudo service nifi start NiFi home: /opt/HDF-1.2.0.1-1/nifi Bootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf 2016-07-04 02:18:00,005 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-07-04 02:18:00,006 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /opt/HDF-1.2.0.1-1/nifi You can check the status of single NiFi server via status command: [root@sandbox nifi]# sudo service nifi status nifi.sh: JAVA_HOME not set; results may vary Java home: NiFi home: /opt/HDF-1.2.0.1-1/nifi Bootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf 2016-07-04 02:18:42,527 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 43184, PID=4391 Make sure you add port 8090 to the sandbox networking. You are now ready to go. Now start flowing.

TimothySpann · ‎07-07-2016

[root@sandbox kafka]# ls -lt /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ total 100 -rw-r--r-- 1 root root 1216 2016-06-25 16:17 stack_advisor.pyc drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SQOOP drwxr-xr-x 2 root root 4096 2016-06-23 23:25 STORM drwxr-xr-x 2 root root 4096 2016-06-23 23:25 TEZ drwxr-xr-x 3 root root 4096 2016-06-23 23:25 YARN drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ZOOKEEPER drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER_KMS drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SLIDER drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SPARK drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KNOX drwxr-xr-x 2 root root 4096 2016-06-23 23:25 MAHOUT drwxr-xr-x 2 root root 4096 2016-06-23 23:25 OOZIE drwxr-xr-x 2 root root 4096 2016-06-23 23:25 PIG drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FALCON drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FLUME drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HBASE drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HDFS drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HIVE drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KAFKA drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KERBEROS drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ACCUMULO drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ATLAS -rwxr-xr-x 1 root root 1190 2016-06-23 17:47 stack_advisor.pyo -rwxr-xr-x 1 root root 850 2016-06-23 17:41 stack_advisor.py [root@sandbox kafka]# ls -lt /var/lib/ambari-server/resources/stacks/HDP/2.5/services total 300 -rw-r--r-- 1 root root 51905 2016-06-26 21:45 stack_advisor.pyc -rwxr-xr-x 1 root root 95429 2016-06-26 21:45 stack_advisor.py drwxr-xr-x 7 root root 4096 2016-06-24 00:35 NIFI drwxr-xr-x 3 root root 4096 2016-06-23 23:25 YARN drwxr-xr-x 3 root root 4096 2016-06-23 23:25 ZEPPELIN drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ZOOKEEPER drwxr-xr-x 3 root root 4096 2016-06-23 23:25 STORM drwxr-xr-x 3 root root 4096 2016-06-23 23:25 TEZ drwxr-xr-x 3 root root 4096 2016-06-23 23:25 SPARK drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SPARK2 drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SQOOP drwxr-xr-x 3 root root 4096 2016-06-23 23:25 RANGER drwxr-xr-x 4 root root 4096 2016-06-23 23:25 RANGER_KMS drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SLIDER drwxr-xr-x 2 root root 4096 2016-06-23 23:25 OOZIE drwxr-xr-x 2 root root 4096 2016-06-23 23:25 PIG drwxr-xr-x 3 root root 4096 2016-06-23 23:25 KNOX drwxr-xr-x 2 root root 4096 2016-06-23 23:25 MAHOUT drwxr-xr-x 5 root root 4096 2016-06-23 23:25 HIVE drwxr-xr-x 3 root root 4096 2016-06-23 23:25 KAFKA drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KERBEROS drwxr-xr-x 3 root root 4096 2016-06-23 23:25 HBASE drwxr-xr-x 3 root root 4096 2016-06-23 23:25 HDFS drwxr-xr-x 3 root root 4096 2016-06-23 23:25 FALCON drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FLUME drwxr-xr-x 3 root root 4096 2016-06-23 23:25 ATLAS drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ACCUMULO -rwxr-xr-x 1 root root 51651 2016-06-23 17:47 stack_advisor.pyo [root@sandbox kafka] It won't install and states install failed. It leaves things in a bad state and changes a bunch of YARN and HDFS configs.

TimothySpann · ‎07-07-2016

GetKafka[id=1f42d69d-b02b-4555-85d7-77f8163e4002] GetKafka[id=1f42d69d-b02b-4555-85d7-77f8163e4002] failed to process session due to java.lang.IllegalStateException: java.util.concurrent.ExecutionException: kafka.common.ConsumerRebalanceFailedException: f31464b5-467a-4b0b-b227-7c81bda87835_sandbox.hortonworks.com-1467599409455-2e5b48f4 can't rebalance after 4 retries: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: kafka.common.ConsumerRebalanceFailedException: f31464b5-467a-4b0b-b227-7c81bda87835_sandbox.hortonworks.com-1467599409455-2e5b48f4 can't rebalance after 4 retries Any idea? Is there a way around this? Can I force rebalance? It's a new Kafka instance, nothing to save

Online	Offline
Last Visited	‎05-20-2024 05:42 PM

Member Since	‎01-07-2019 11:58 AM
Last Visited	‎05-20-2024 05:42 PM
Posts	1,973
Kudos received	1122

Cloudera Community

Re: Has anyone tried NiFi consuming (JMSConsume) f...

Re: NiFi Crash after runing chain of lookups

Re: Recommend approach for listening to RSS Feed i...

Re: NiFi ListenFTP Processor Default Data Port

Re: Nifi: Kafka Producer with Avro format in both ...

Exception when trying to run Hive UDF

Re: Nifi - AVRO to CSV or Json to CSV,NIFI - conve...

Re: HDFS Heterogeneous Storage - Using AWS S3 as s...

Re: IS THERE A PROCESSOR IN NIFI THAT CAN CREATE M...

Re: Why does HDF come with Storm and not Spark?

Re: Nothing happens when I run squirrel-sql.sh fro...

Re: Sending a Message to Kafka Error

Quickly Adding HDF to HDP 2.5 Sandbox

Re: Adding NiFi Server to HDP 2.5 Sandbox

Sending a Message to Kafka Error