Member since
03-09-2018
12
Posts
1
Kudos Received
0
Solutions
02-17-2019
06:11 PM
In fact the problem is related to the java Orc api when parallelism is activated (multi-thread) I use Flink and when I set a parallelism > 1 on the Sink that generates Orc files, I face to this issue: data are unreadable. I've seen some tickets about this issue like this one: https://jira.apache.org/jira/browse/ORC-361 At the moment I use a parallelism of 1 but I have to fix this issue in order to scale my ingest pipeline. All help is welcome. Thx
... View more
02-11-2019
02:34 PM
Hello, I face to one error when I try to read my Orc files from Hive (external table) or Pig or with hive --orcfiledump .. These files are generated with Flink using the Orc Java API with Vectorize column. If I create these files locally (/tmp/...), push them to hdfs, then I can read the content of these files from Pig or with the use of External table in Hive. If I change the path and use hdfs directly, then I face to this error : Failure while running task:java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54)
at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:302)
at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$TimestampTreeReader.next(TreeReaderFactory.java:1105)
at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2079)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1082)
at org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat$OrcRecordReader.nextKeyValue(OrcNewInputFormat.java:108) And the same if I get these files locally.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Pig
03-21-2018
11:06 PM
One HCC thread related to this kind of error: https://community.hortonworks.com/questions/84101/failed-to-start-hive-server2-illegal-character-in.html
... View more
03-21-2018
11:02 PM
I pursue in the resolution of my problem and my understanding.. When I activate HDFS Audit in Ranger Configs and restart a Namenode to apply config, I obtain these errors in the log file '
/var/lib/ambari-agent/data/output-XX.txt':
call returned (3, '') 2018-03-21
23:44:37,745 - Will retry 7 time(s), caught exception: (u"Execution of
'curl --location-trusted -k --negotiate -u : -b
/var/lib/ambari-agent/tmp/cookies/7768 3cd0-2605-4b42-a750-509ecc713ba9
-c
/var/lib/ambari-agent/tmp/cookies/77683cd0-2605-4b42-a750-509ecc713ba9
-w '%{http_code}' '{{policymgr_mgr_url}}/login.jsp' --connect -timeout
10 --max-time 12 -o /dev/null 1>/tmp/tmp21P7Ww 2>/tmp/tmpfrrHEy'
returned 3. curl: (3) [globbing] nested braces not supported at pos
2\n",). Sleeping for 8 sec(s)
... View more
03-21-2018
04:19 PM
Hello, I'm still trying to deploy a "kerberized" cluster HDP with Ambari blueprint. With the Ranger service but with all plugins desactivated, it's ok. A soon as I try to activate plugins, the cluster never starts (Namenodes are still stopped) : "ranger-hdfs-plugin-enabled" : "Yes"... A JVM related to the Namenode is started but that's all. This is the command to format Namenode: org.apache.hadoop.hdfs.server.namenode.NameNode -format -nonInteractive Then, there is no log file related to the namenode in /var/log/hadoop/hdfs/ If someone has some advices or an example of blueprint that works with Kerberos and Ranger (+ plugins) I will be eternally grateful 🙂
... View more
03-20-2018
02:30 PM
I think the problem is related to missing steps during post install but I'm not sure because the error is not really obvious. Example for Hdfs: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/hdfs_plugin_kerberos.html Questions about that : - If these steps are required, does I have to create this 'rangerlookup' user manually and its keytab (cluster is kerberized) ? - Is it blueprint compliant ?
... View more
03-12-2018
12:24 AM
Despite the copy of these ranger-* files before application of Kerberos, I still obtain the same error.
... View more
03-09-2018
11:25 PM
Thanks for your reply It seems ok. I spawn a new cluster from scratch and give a feedback.
... View more
03-09-2018
10:11 PM
Hello, I face to a problem related to Ranger during application of Kerberos with Ambari onto my HDP cluster (2.6.4) FATAL conf.Configuration (Configuration.java:loadResource(2712)) -
error parsing conf file:/.../hadoop/2.6.4.0-91/0/xasecure-audit.xml java.io.FileNotFoundException: /.../hadoop/2.6.4.0-91/0/xasecure-audit.xml (No such file or directory) I've found a temporary work-around by desactivate audit in Ranger configuration. How can I resolve this problem in order to reactivate audit. Thanks in advance, Regards, David
... View more
Labels: