Member since
05-24-2017
9
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1047 | 09-20-2017 12:22 PM |
09-20-2017
02:44 PM
Hi, here is a sample of my files in hdfs. thank´s. flumedata1505877214592tmp.zip
... View more
09-20-2017
02:14 PM
My problem is in the construction of the table in HIVE to receive the data from the twitter file, someone seeing the header that I put can help me in building the script that receives the data?
I'm still walking in that part and having that difficulty. /usr/root/flume/test File: FlumeData.1505877214590 the header of my file in hdfs is as follows: Header: ------------------------------------------------------------------------------- Objavro.schemaä
{"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}
... View more
09-20-2017
01:47 PM
Hi, When I remove one of the time_zone columns, I get the error below, which is the same as the previous script, how can I handle this, and this data comes straight from twitter: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.serde2.SerDeException: org.codehaus.jackson.JsonParseException: Unexpected character ('O' (code 79)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: java.io.ByteArrayInputStream@6f5728d5; line: 1, column: 2]
... View more
09-20-2017
12:55 PM
Hello everyone, I know that there are already several topics here about my doubt, the point is that, none resolved. I have the following situation. I was able to get twitter data via flume. They are already on HDFS. But I can not turn them into Hive tables. the path of my files is in: /usr/root/flume/test
File: FlumeData.1505877214590 the header of my file in hdfs is as follows: Header: ------------------------------------------------------------------------------- {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]} ------------------------------------------------------------------------------- I've tried the script below, or other variants of it and nothing. ------------------------------------------------------------------------------- CREATE EXTERNAL TABLE tweets (
id BIGINT,
created_at STRING,
source STRING,
favorited BOOLEAN,
retweet_count INT,
retweeted_status STRUCT<
text:STRING,
userr:STRUCT<screen_name:STRING,name:STRING>>,
entities STRUCT<
urls:ARRAY<STRUCT<expanded_url:STRING>>,
user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>,
hashtags:ARRAY<STRUCT<text:STRING>>>,
text STRING,
userr STRUCT<
screen_name:STRING,
name:STRING,
friends_count:INT,
followers_count:INT,
statuses_count:INT,
verified:BOOLEAN,
utc_offset:INT,
time_zone:STRING>,
in_reply_to_screen_name STRING
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION '/user/root/flume/test'; ERROR: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.serde2.SerDeException: org.codehaus.jackson.JsonParseException: Unexpected character ('O' (code 79)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@1b1f2ed4; line: 1, column: 2] -------------------------------------------------------------------------------
or for the other script ------------------------------------------------------------------------------- CREATE EXTERNAL TABLE tweets1 (
id BIGINT,
created_at STRING,
source STRING,
favorited BOOLEAN,
retweet_count INT,
retweeted_status STRUCT<
text:STRING,
userr:STRUCT<screen_name:STRING,name:STRING>>,
entities STRUCT<
urls:ARRAY<STRUCT<expanded_url:STRING>>,
user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>,
hashtags:ARRAY<STRUCT<text:STRING>>>,
text STRING,
userr STRUCT<
time_zone: STRING,
screen_name:STRING,
name:STRING,
friends_count:INT,
followers_count:INT,
statuses_count:INT,
verified:BOOLEAN,
utc_offset:INT,
time_zone:STRING>,
in_reply_to_screen_name STRING
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION '/user/root/flume/test'; ERROR: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.IllegalArgumentException: Field named time_zone already exists ------------------------------------------------------------------------------- Can someone help me with this?
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache Hive
09-20-2017
12:25 PM
thanks for the tip, helped me to get the answer.
... View more
09-20-2017
12:22 PM
I have decided as follows: I downloaded the postgresql-42.1.1.jar file, then put it inside the "/ usr / share / java /" folder. For security, I also unpacked it inside the JAVA_HOME folder, "= / usr / java / jdk1.8.0_131", the decompression is done with the command "jar -xfv postgresql-42.1.1.jar". I hope to help someone else who goes through it.
... View more
09-20-2017
12:14 PM
Hi Geoffrey, The answer is good, solved my problem.
... View more
08-11-2017
11:35 AM
Hi all, When I start the HDP virtual machine, this message is displayed in my browser and I can not access the ambari dashboard to test the applications, how do I solve this? There is no button to accept the terms, can you help me? (I did not put the whole message, it's too big.) ORACLE DATABASE 10g EXPRESS EDITION LICENSE AGREEMENT To use this license, you must agree to all of the following terms (by either clicking the accept button or installing and using the program): ELIGIBILITY EXPORT RESTRICTIONS I am not a citizen, national or resident of, and am not under the control of, the government of: Cuba, Iran, Sudan, Iraq, Libya, North Korea, Syria, nor any other country to which the United States has prohibited export. I will not download or otherwise export or re-export the Programs, directly or indirectly, neither to the above mentioned countries nor to citizens, nationals or residents of those countries. ...
... View more
Labels:
05-24-2017
01:05 PM
Hi guys,
I'm trying to install ambari, but I'm having problems with the postgresql driver. I am doing an installation without internet, so I downloaded the packages needed for this task. Here's the mistake I'm having, have anyone been through this? ]# whereis java java: /usr/bin/java /usr/share/man/man1/java.1 ]# rpm -Uhv jpackage-utils-5.0.0-7.2.src.rpm warning: jpackage-utils-5.0.0-7.2.src.rpm: Header V3 DSA/SHA1 Signature, key ID abde7ce2: NOKEY
Updating / installing...
1:jpackage-utils-0:5.0.0-7.2 ################################# [100%] ]# rpm -Uhv postgresql-jdbc-9.2.1002-5.el7.noarch.rpm warning: postgresql-jdbc-9.2.1002-5.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
error: Failed dependencies:
jpackage-utils is needed by postgresql-jdbc-9.2.1002-5.el7.noarch ]# Write failed: Connection reset by peer
... View more
Labels:
- Labels:
-
Apache Ambari