Is it possible to treat empty strings (or string with only white spaces) as NULL values in ORC table? I will give the example. I import data from MS SQL to Hive using Sqoop. When I use TEXTFILE everything works good. All empty strings are treated as NULL if I use paremeter: 'serialization.null.format'='' But this parameter does not work with ORC tables. Is there any equivalent of this parameter in ORC format? ... View more

Is there any way to use list or regex in Hive table parameter 'serialization.null.format'? I know that I can use single character as null by using this command: tblproperties('serialization.null.format'='null') But what if I want Hive to treat two characters as null? For example: tblproperties('serialization.null.format'=['null', '\\N']); OR tblproperties('serialization.null.format'=' *'); Does anyone know if it is possible? ... View more

Hi. I have a question regarding to Hadoop architecture. I have 10 node cluster and I want to create some kind of sandboxes inside the cluster. What do I mean by sandbox? Separate space from the whole cluster resources when business users could create some temporary databases, files and run jobs. A simple solution would be creating a technical user for every sandbox, but I don't want to do it that way. Business users have their own accounts and I want them to run jobs using those accounts. Maybe the picture will say more: As you can see, the problem is that some jobs should be run by users in main cluster space and some of jobs should be run in specific sandbox. The question is - how I can achieve this? Does anyone have the idea? ... View more

Hi. I have a problem with exporting Hive table to Oracle database. I wanna encrypt and hide password using jceks. I read great article about using jceks while importing data using Sqoop: Storing Protected Passwords in Sqoop It works great when I import data from Oracle to Hive. But the problem is that when I try to export data from Hive to Oracle I get an error: Unable to process alias My Sqoop command which I try to run: sqoop export \ -Dhadoop.security.credential.provider.path=jceks://hdfs/user/hdfs/pass-enc.jceks \ --connect jdbc:oracle:thin:@1.1.1.1:2222:SID \ --table hive_temp_table_orc \ --username orc_user \ --password-alias oracle.password \ --hcatalog-database default \ --hcatalog-table hive_temp_table \ --hive-partition-key col1 \ --hive-partition-value 2011-01-01 My question is - is that possible to use jceks and --password-alias parameter with Sqoop export command? Or is it an option only when I importing data? ... View more

I have a problem with PXF and external tables in HAWQ. I have seven data nodes in cluster (and seven HAWQ segments) and I installed PXF on each of them. It looks like that: Node1 - NameNode, HiveMetastore, HAWQ master, Hive Client, HCat Client, PXF Node2 - SNameNode, DataNode, HAWQ Segment, PXF Node3-7 - DataNode, HAWQ Segment, PXF Node8-9 - HiveClient, HCat Clients I created table based on "Test PXF" on this site. But I have a problem with accesing data. When I try to run a simple query: SELECT * FROM pxf_hdfs_textsimple; I get error: ERROR: remote component error (0): (libchurl.c:897) That's all. In HAWQ Master Node Log I see this: 2018-01-04 18:43:06.902998 CET,"hawq","postgres",p19781,th-319940160,"[local]",,2018-01-04 18:43:06 CET,16768,con26,,seg-10000,,,x16768,sx1,"LOG","00000","no master mirroring standby configuration found",,,,,,,0,,"cdblink.c",159, 2018-01-04 18:43:10.759145 CET,"hawq","poligon",p19820,th-319940160,"[local]",,2018-01-04 18:43:10 CET,0,con27,,seg-10000,,,,,"LOG","00000","getLocalTmpDirFromMasterConfig session_id:27 tmpdir:/tmp",,,,,,,0,,"postinit.c",470, 2018-01-04 18:43:20.600046 CET,"hawq","poligon",p19820,th-319940160,"[local]",,2018-01-04 18:43:10 CET,16773,con27,cmd7,seg-10000,,,x16773,sx1,"ERROR","XX000","remote component error (0): (libchurl.c:897)",,,,,,"select * from pxf_hdfs_textsimple ;",0,,"libchurl.c",897,"Stack trace: 1 0x8c165e postgres errstart (??:?) 2 0x8c34fb postgres elog_finish (??:?) 3 0x5124d6 postgres check_response_code (??:?) 4 0x512686 postgres churl_read_check_connectivity (??:?) 5 0x517b22 postgres <symbol not found> (pxfutils.c:?) 6 0x517d66 postgres call_rest (??:?) 7 0x5168c0 postgres <symbol not found> (pxfmasterapi.c:?) 8 0x516f97 postgres get_data_fragment_list (??:?) 9 0x512ff5 postgres map_hddata_2gp_segments (??:?) 10 0x73f8a2 postgres <symbol not found> (createplan.c:?) 11 0x73fdb5 postgres <symbol not found> (createplan.c:?) 12 0x741dec postgres create_plan (??:?) 13 0x74d1a6 postgres <symbol not found> (planner.c:?) 14 0x74eb3c postgres subquery_planner (??:?) 15 0x74f177 postgres <symbol not found> (planner.c:?) 16 0x74f72e postgres planner (??:?) 17 0x7e496a postgres pg_plan_queries (??:?) 18 0x7e4e05 postgres <symbol not found> (postgres.c:?) 19 0x7e6560 postgres PostgresMain (??:?) 20 0x799860 postgres <symbol not found> (postmaster.c:?) 21 0x79c5e9 postgres PostmasterMain (??:?) 22 0x4a2dff postgres main (??:?) 23 0x7fdfe9e2ab35 libc.so.6 __libc_start_main (??:0) 24 0x4a2e7c postgres <symbol not found> (??:?) " When I try access data in Hive I get error too. Query: select count(*) from hcatalog.default.aaa; Error: ERROR: remote component error (500) from '127.0.0.1:51200': type Exception report message java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException description The server encountered an internal error that prevented it from fulfilling this request. exception javax.servlet.ServletException: java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException (libchurl.c:897) LINE 1: select count(*) from hcatalog.default.aaa; Does anyone know what I am doing wrong? What can cause a problem with accesing data from HDFS, Hive by PXF? ... View more

Hi. Configuration management tools are really popular these days. I wanted to deploy one of them, for example Ansible, on my cluster. Is there a way to deploy Ansible (or other CM tool) to existing HDP cluster? I found a couple of articles about Ansible and HDP, but all of them show how to deploy HDP cluster using Ansible. But what about existing cluster? ... View more

Hi. I want to change ambari-agent user to non-root. Documentation says that I should manually install ambari-agent. What if I installed it using Ambari (it is not manually install)? Do I have to remove ambari-agent and install it manually then to configure it for non-root user? ... View more

I installed HDP using Ambari as root user. Due to security I want to change it. As I read, there is no problem with running Ambari Agent as non-root user (How to Configure an Ambari Agent for Non-Root). But what about Ambari Server? During the Ambari Server setup process, when prompted to Customize user account for ambari-server daemon? , I chose n . Is there any way to change user for Ambari Server? Or do I have to setup Ambari Server one more time? ... View more

Hi. I've got a problem with YARN and Capacity Scheduler. I created two queues: 1. default - 60% 2. q_apr_general - 40% There is one Spark Streaming job in queue 'q_apr_general'. Processing time for every single batch is ~2-6 seconds. In the default queue I started Zeppelin with preconfigured resources. I added to zeppelin-env.sh one line: export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.4.2.0-258 -Dspark.executor.instances=75 -Dspark.executor.cores=6 -Dspark.executor.memory=13G The problem is that when I execute Spark SQL query in Zeppelin, processing time is ~20-30 seconds. It is weird for me, because Zeppelin process and Spark streaming are in different queues. Spark streaming process should not depend on Zeppelin process in another queue. Does anyone know what is the reason of my problem? ... View more

Hi. I have three Spark streaming applications. One of them saves data to Hive table (parquet format). The other two read data from that table and cache it every hour at the same moment. Both of them have the same code to read data. Writing to table and reading from it is never done at the same time. After couple of hours one of them seems to read only part of data. You can see below how it looks in Storage tab. First application reads all data. The second one omnited one partition. Do you know what is the reason of this issue? UPDATE Read: Two reading applications have the same part of code, which cache table. sqlContext.clearCache() df = sqlContext.sql('select timestamp, col2 from table where timestamp > time') df.cache() This code is executed every hour. Write: df.registerTempTable('temp') sqlContext.sql("INSERT INTO TABLE table SELECT * FROM temp") Code above is also executed every hour. I checked and I am sure that these codes finish properly. Every hour when I insert new data into table Hive creates new partition for it, so every partition have data for only one hour. When I read table I want data from last hour (last added partition). The problem is that Spark streaming seems to not update number of partition in table. If we have desired data in one partition there's a chance Spark will not read it. Maybe I should do something with the way I insert data into Hive? Spark version: 1.6 ... View more