Member since
10-01-2015
3933
Posts
1150
Kudos Received
374
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3362 | 05-03-2017 05:13 PM | |
2792 | 05-02-2017 08:38 AM | |
3070 | 05-02-2017 08:13 AM | |
3002 | 04-10-2017 10:51 PM | |
1512 | 03-28-2017 02:27 AM |
02-23-2017
04:59 PM
@Aruna Sameera can you paste a sample txt file you are using as well as your Hive DDL statement.
... View more
02-23-2017
04:54 PM
1 Kudo
@Gunjan Dhawas based by the wiki, it will take 8 bytes. INT/INTEGER (4-byte signed integer, from -2,147,483,648 to 2,147,483,647)
BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-IntegralTypes(TINYINT,SMALLINT,INT/INTEGER,BIGINT) and https://cwiki.apache.org/confluence/display/Hive/Tutorial#Tutorial-TypeSystem
... View more
02-23-2017
04:46 PM
@Dmitro Vasilenko I'm going to attempt to answer this as there are far better experts here. You can improve performance in a few ways, you can parallelize by running PutHiveStreaming processor on a couple of nodes or tweak some of the parameters in the processor. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hive.PutHiveStreaming/ Probably one property I'd tweak first is Transactions per batch. See if yours is set too low?
... View more
02-23-2017
04:12 PM
1 Kudo
@manpreet kaur this is a nice little riddle of a task. First of all, I had to figure out why I wasn't able to parse the string using Java as it was less obvious in Pig. import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.logging.Level;
import java.util.logging.Logger;
/**
*
* @author aervits
*/
public class NewClass1 {
public static void main(String [] args) {
String formatStr = "MM/dd/yyyy HH:mm:ss";
SimpleDateFormat format = new SimpleDateFormat(formatStr);
String dateStr = "01/05/2017 15:29:19.4980";
try {
Date formatted = format.parse(dateStr);
System.out.println("Date: " + formatted);
String formatStr1 = "yyyy-MM-dd HH:mm:ss";
format = new SimpleDateFormat(formatStr1);
String format2 = format.format(formatted);
System.out.println("Date: " + format2);
} catch (ParseException ex) {
Logger.getLogger(NewClass.class.getName()).log(Level.SEVERE, ex.getMessage());
}
}
}
Date: Thu Jan 05 15:29:19 EST 2017
Date: 2017-01-05 15:29:19 So I had to make two hoops to get it to the format you expect. Then I applied the same logic in Pig to get to this a = load 'date' using PigStorage('.') as (dateStr:chararray);
b = foreach a generate ToDate(dateStr, 'MM/dd/yyyy HH:mm:ss') as (dateobj:DateTime);
c = foreach b generate ToString(dateobj, 'yyyy-MM-dd HH:mm:ss') as (dateStr2:chararray);
You can also rewrite the code in two lines a = load 'date' using PigStorage('.') as (dateStr:chararray);
b = foreach a generate ToString(ToDate(dateStr, 'MM/dd/yyyy HH:mm:ss'), 'yyyy-MM-dd HH:mm:ss') as (dateStr2:chararray); (2017-01-05 15:29:19) The problem is in your string having millisecond precision, notice my load statement, I truncated it by using PigStorage('.'). Then I take your string and create a DateTime object from it with a SimpleDateFormat that understands your string, then I'm converting the DateTime object back to String with format I expect. I didn't spend a lot of time on this to figure out how to parse your String in one shot via Java and hence via Pig but it gets you at least where you want to be.
... View more
02-23-2017
02:29 PM
@Anders Boje yes that is a Workflow Manager view but an old version which is not mean to be production ready. Workflow Manager View AKA Oozie View will be available in Ambari 2.5. Ambari 2.5 is not released yet, it will be released in a month or two. As you noted, there are bugs and we are still working through fixing some issues so it is stable. My tutorials do cover a lot of WFM but the ideas can be applied to working with Oozie XML. You should just wait for Ambari 2.5 release if this is more what you want to do.
... View more
02-23-2017
01:59 PM
@Anders Boje when you say you're using Oozie view from 2.5, are you referring to Workflow Manager View in Ambari 2.4.2? http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-views/content/ch_workflow_designer_view.html If so, I am using the same tool only from Ambari 2.5 (unreleased version). This is certainly not Hue. You just have to wait for it to be released soon as you pointed out we are still working out through bugs and usability issues.
... View more
02-23-2017
12:51 PM
1 Kudo
along with Ambari infra, you need Kafka and HBase to be started.
... View more
02-23-2017
12:44 PM
Can you run the command below, looks like some ports associated with namenode are in use. Check the rest of the ports, can't find more recent doc but most ports still apply https://ambari.apache.org/1.2.3/installing-hadoop-using-ambari/content/reference_chap2_1.html sudo netstat -tulnp | grep 8020
... View more
02-23-2017
12:31 AM
you can create an email alias with many accounts associated to it.
... View more
02-22-2017
09:36 PM
@Vladislav Falfushinsky about to finish writing a new article on WFM https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.html check back in 30min.
... View more