Hi all! Also, have this error. ... View more

Hi! this in express upgrade with "Skip all Service Check failures" Documentation: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/ambari-managed-hdf-upgrade/content/upgrade-hdf.html ... View more

Hi! Thanks you , I check and take an ansver . ... View more

Hi! Have same problem, but Ranger not installed [root@serv12 ~]# grep -i Ranger /var/lib/ambari-agent/data/output-869.txt 2018-10-03 17:49:28,216 - Ranger admin not installed ... View more

Thanks you! check logs ... View more

Hi! Try One of this changes: Zeppelin NIFI ExecuteSparkInteractive preds = spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’) preds = spark.sql("select * from andbox.CHURN_PRP_D_PREDS_S") dir = ‘ /user/CHURN_PRP_D’ dir = "hdfs:/user/CHURN_PRP_D" date_df = spark.sql("select to_date('{}') as score_date".format(date_calc)) ttt = spark.createDataFrame([(date_calc,)], ["t"]) date_df = ttt.select(to_date(ttt.t)) sqlContext.registerDataFrameAsTable(train, "train") train.registerTempTable("train") ... View more

Thanks you!!! In my case need route “CSV” content, example: Id date_col name 1|2018:06:2108:40:00|Ukraine 2|2018:06:2108:15:00|USA If date_col less then 25 minutes of now(), route to rule1 If date_col great equivalent 25 minutes of now(), route to rule2 now() = 2018:06:21 08:50:00 Result 2018:06:2108:40:00 to rule1 <<<<<<<<< 2018:06:21 08:15:00 to rule2 <<<<<<<<< ... View more

Hi! one more question, how control count "livy-session-"" ? ... View more

Problem was in "sqlContext.registerDataFrameAsTable" What investigate similar (HTTP) errors ? ... View more

Do you have example run Zeppelin REST API ? ... View more

Thank you! But what to do with processor , for example FetchFTP , after import change (version control) from DEV to PROD property: Password "No value set" Can add pasword to variables (on processor group), but this is not secured. ... View more

Hi! finded this tables! Thank you! ... View more

Thanks you ! But I do not see this tables in postgres database: postgres=# \connect ambari You are now connected to database "ambari" as user "postgres". ambari=# \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+------------------------ ambari | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres + | | | | | postgres=CTc/postgres + | | | | | ambari=CTc/postgres postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | ranger | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres + | | | | | postgres=CTc/postgres + | | | | | rangerdba=CTc/postgres template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (5 rows) ambari=# \dt[S+] List of relations Schema | Name | Type | Owner | Size | Description ------------+-------------------------+-------+----------+------------+------------- pg_catalog | pg_aggregate | table | postgres | 40 kB | pg_catalog | pg_am | table | postgres | 40 kB | pg_catalog | pg_amop | table | postgres | 64 kB | pg_catalog | pg_amproc | table | postgres | 56 kB | pg_catalog | pg_attrdef | table | postgres | 64 kB | pg_catalog | pg_attribute | table | postgres | 784 kB | pg_catalog | pg_auth_members | table | postgres | 0 bytes | pg_catalog | pg_authid | table | postgres | 40 kB | pg_catalog | pg_cast | table | postgres | 48 kB | pg_catalog | pg_class | table | postgres | 192 kB | pg_catalog | pg_collation | table | postgres | 256 kB | pg_catalog | pg_constraint | table | postgres | 120 kB | pg_catalog | pg_conversion | table | postgres | 56 kB | pg_catalog | pg_database | table | postgres | 8192 bytes | pg_catalog | pg_db_role_setting | table | postgres | 16 kB | pg_catalog | pg_default_acl | table | postgres | 0 bytes | pg_catalog | pg_depend | table | postgres | 560 kB | pg_catalog | pg_description | table | postgres | 280 kB | pg_catalog | pg_enum | table | postgres | 0 bytes | pg_catalog | pg_extension | table | postgres | 40 kB | pg_catalog | pg_foreign_data_wrapper | table | postgres | 0 bytes | pg_catalog | pg_foreign_server | table | postgres | 0 bytes | pg_catalog | pg_foreign_table | table | postgres | 0 bytes | pg_catalog | pg_index | table | postgres | 104 kB | pg_catalog | pg_inherits | table | postgres | 0 bytes | pg_catalog | pg_language | table | postgres | 40 kB | pg_catalog | pg_largeobject | table | postgres | 0 bytes | pg_catalog | pg_largeobject_metadata | table | postgres | 0 bytes | pg_catalog | pg_namespace | table | postgres | 40 kB | pg_catalog | pg_opclass | table | postgres | 48 kB | pg_catalog | pg_operator | table | postgres | 144 kB | pg_catalog | pg_opfamily | table | postgres | 48 kB | pg_catalog | pg_pltemplate | table | postgres | 40 kB | pg_catalog | pg_proc | table | postgres | 536 kB | pg_catalog | pg_range | table | postgres | 40 kB | pg_catalog | pg_rewrite | table | postgres | 496 kB | pg_catalog | pg_seclabel | table | postgres | 8192 bytes | pg_catalog | pg_shdepend | table | postgres | 48 kB | pg_catalog | pg_shdescription | table | postgres | 48 kB | pg_catalog | pg_shseclabel | table | postgres | 0 bytes | pg_catalog | pg_statistic | table | postgres | 872 kB | pg_catalog | pg_tablespace | table | postgres | 40 kB | pg_catalog | pg_trigger | table | postgres | 160 kB | pg_catalog | pg_ts_config | table | postgres | 40 kB | pg_catalog | pg_ts_config_map | table | postgres | 48 kB | pg_catalog | pg_ts_dict | table | postgres | 40 kB | pg_catalog | pg_ts_parser | table | postgres | 40 kB | pg_catalog | pg_ts_template | table | postgres | 40 kB | pg_catalog | pg_type | table | postgres | 160 kB | pg_catalog | pg_user_mapping | table | postgres | 0 bytes | (50 rows) ambari=# ... View more

Hi All! One more question, how add user mapping on “sub-queue” ? ... View more

Hi! Thank you, it is works. One more question, how add user mapping on “sub-queue” ? : Example like this not working : yarn.scheduler.capacity.queue-mappings=u:airflow:aggregate.airflow yarn.scheduler.capacity.maximum-am-resource-percent=0.6 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=10 yarn.scheduler.capacity.root.default.maximum-capacity=40 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=0.7 yarn.scheduler.capacity.root.queues=aggregate,default,llap,load,reports yarn.scheduler.capacity.root.aggregate.queues=airflow <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< yarn.scheduler.capacity.queue-mappings=u:airflow:aggregate.airflow <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<, yarn.scheduler.capacity.queue-mappings-override.enable=true yarn.scheduler.capacity.root.aggregate.acl_submit_applications=* yarn.scheduler.capacity.root.aggregate.capacity=35 yarn.scheduler.capacity.root.aggregate.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.aggregate.maximum-applications=20000 yarn.scheduler.capacity.root.aggregate.maximum-capacity=60 yarn.scheduler.capacity.root.aggregate.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.aggregate.state=RUNNING yarn.scheduler.capacity.root.aggregate.user-limit-factor=0.7 yarn.scheduler.capacity.root.aggregate.airflow.acl_submit_applications=* yarn.scheduler.capacity.root.aggregate.airflow.capacity=100 yarn.scheduler.capacity.root.aggregate.airflow.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.aggregate.airflow.maximum-applications=20000 yarn.scheduler.capacity.root.aggregate.airflow.maximum-capacity=100 yarn.scheduler.capacity.root.aggregate.airflow.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.aggregate.airflow.state=RUNNING yarn.scheduler.capacity.root.aggregate.airflow.user-limit-factor=0.5 yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.default.maximum-applications=20000 yarn.scheduler.capacity.root.default.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.default.priority=0 yarn.scheduler.capacity.root.llap.acl_administer_queue=* yarn.scheduler.capacity.root.llap.acl_submit_applications=* yarn.scheduler.capacity.root.llap.capacity=0 yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.llap.maximum-capacity=0 yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.llap.ordering-policy=fifo yarn.scheduler.capacity.root.llap.priority=0 yarn.scheduler.capacity.root.llap.state=STOPPED yarn.scheduler.capacity.root.llap.user-limit-factor=1 yarn.scheduler.capacity.root.load.acl_submit_applications=* yarn.scheduler.capacity.root.load.capacity=45 yarn.scheduler.capacity.root.load.maximum-am-resource-percent=1 yarn.scheduler.capacity.root.load.maximum-applications=20000 yarn.scheduler.capacity.root.load.maximum-capacity=80 yarn.scheduler.capacity.root.load.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.load.ordering-policy=fair yarn.scheduler.capacity.root.load.ordering-policy.fair.enable-size-based-weight=false yarn.scheduler.capacity.root.load.priority=1 yarn.scheduler.capacity.root.load.state=RUNNING yarn.scheduler.capacity.root.load.user-limit-factor=0.7 yarn.scheduler.capacity.root.ordering-policy=priority-utilization yarn.scheduler.capacity.root.priority=0 yarn.scheduler.capacity.root.reports.acl_submit_applications=* yarn.scheduler.capacity.root.reports.capacity=10 yarn.scheduler.capacity.root.reports.maximum-am-resource-percent=0.4 yarn.scheduler.capacity.root.reports.maximum-capacity=10 yarn.scheduler.capacity.root.reports.minimum-user-limit-percent=5 yarn.scheduler.capacity.root.reports.priority=0 yarn.scheduler.capacity.root.reports.state=RUNNING yarn.scheduler.capacity.root.reports.user-limit-factor=0.5 <br> ... View more

In log see next errors : [yarn@ks-dmp03 ~]$ grep -i ERROR /tmp/application_1518080001967_51536.log | grep -v HADOOP_ROOT_LOGGER | grep -v Dtez.root.logger | awk '{print $1 $3 $4 $5 $6 $7 $8 $9 $10 $11 $12 $13 $14 $15 $16}' | sort -u 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000034_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000035_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000036_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000037_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000038_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000039_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000040_0accordingtoscheduler.Taskreported ****************************** 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000107_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000108_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000109_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000110_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000111_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000112_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000114_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000115_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000116_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000117_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000118_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000119_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000120_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000121_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000122_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TaskSchedulerEventHandlerThread]|rm.TaskSchedulerEventHandler|:Nocontainerallocatedtotask:attempt_1518080001967_51536_3_02_000124_0accordingtoscheduler.Taskreported 2018-03-07[ERROR][TezChild]|tez.ReduceRecordProcessor|:Hiterrorwhileclosingoperators-failingtree 2018-03-07[ERROR][TezChild]|tez.TezProcessor|:java.lang.InterruptedException [yarn@ks-dmp03 ~]$ ... View more

Vertex dependency in root stage Map 1 <- Map 3 (BROADCAST_EDGE) Reducer 2 <- Map 1 (SIMPLE_EDGE) Stage-3 Stats-Aggr Operator Stage-0 Move Operator partition:{"hday":"2018-03-06","src":"OTHER_DETAIL"} table:{"name:":"agg_dpi.agg_dpi_traffic_daily_l","input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde"} Stage-2 Dependency Collection{} Stage-1 Reducer 2 File Output Operator [FS_15] compressed:true Statistics:Num rows: 419376708 Data size: 435732400043 Basic stats: COMPLETE Column stats: NONE table:{"name:":"agg_dpi.agg_dpi_traffic_daily_l","input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde"} Select Operator [SEL_13] outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16","_col17","_col18"] Statistics:Num rows: 419376708 Data size: 435732400043 Basic stats: COMPLETE Column stats: NONE Group By Operator [GBY_12] | aggregations:["sum(VALUE._col0)","sum(VALUE._col1)","sum(VALUE._col2)","sum(VALUE._col3)","sum(VALUE._col4)","sum(VALUE._col5)","sum(VALUE._col6)","sum(VALUE._col7)","count(DISTINCT KEY._col5:0._col0)","count(DISTINCT KEY._col5:1._col0)","count(DISTINCT KEY._col5:2._col0)","count(DISTINCT KEY._col5:3._col0)"] | keys:KEY._col0 (type: varchar(16)), KEY._col1 (type: decimal(10,0)), KEY._col2 (type: decimal(10,0)), KEY._col3 (type: decimal(1,0)), KEY._col4 (type: string) | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16"] | Statistics:Num rows: 419376708 Data size: 435732400043 Basic stats: COMPLETE Column stats: NONE |<-Map 1 [SIMPLE_EDGE] Reduce Output Operator [RS_11] key expressions:_col0 (type: varchar(16)), _col1 (type: decimal(10,0)), _col2 (type: decimal(10,0)), _col3 (type: decimal(1,0)), _col4 (type: string), _col5 (type: decimal(20,0)), _col6 (type: decimal(20,0)), _col7 (type: decimal(20,0)), _col8 (type: decimal(20,0)) Map-reduce partition columns:_col0 (type: varchar(16)), _col1 (type: decimal(10,0)), _col2 (type: decimal(10,0)), _col3 (type: decimal(1,0)), _col4 (type: string) sort order:+++++++++ Statistics:Num rows: 838753416 Data size: 871464800087 Basic stats: COMPLETE Column stats: NONE value expressions:_col9 (type: decimal(31,0)), _col10 (type: decimal(31,0)), _col11 (type: decimal(31,0)), _col12 (type: decimal(31,0)), _col13 (type: decimal(21,0)), _col14 (type: decimal(21,0)), _col15 (type: decimal(21,0)), _col16 (type: decimal(21,0)) Group By Operator [GBY_10] aggregations:["sum(CASE WHEN ((_col5 = 2)) THEN ((_col6 + _col7)) ELSE (0) END)","sum(CASE WHEN ((_col5 = 1)) THEN ((_col6 + _col7)) ELSE (0) END)","sum(CASE WHEN ((_col5 = 0)) THEN ((_col6 + _col7)) ELSE (0) END)","sum((_col6 + _col7))","sum(CASE WHEN ((_col5 = 2)) THEN ((_col8 - _col9)) ELSE (0) END)","sum(CASE WHEN ((_col5 = 1)) THEN ((_col8 - _col9)) ELSE (0) END)","sum(CASE WHEN ((_col5 = 0)) THEN ((_col8 - _col9)) ELSE (0) END)","sum((_col8 - _col9))","count(DISTINCT CASE WHEN ((_col5 = 2)) THEN (_col10) END)","count(DISTINCT CASE WHEN ((_col5 = 1)) THEN (_col10) END)","count(DISTINCT CASE WHEN ((_col5 = 0)) THEN (_col10) END)","count(DISTINCT _col10)"] keys:_col0 (type: varchar(16)), _col1 (type: decimal(10,0)), _col2 (type: decimal(10,0)), _col4 (type: decimal(1,0)), _col3 (type: string), CASE WHEN ((_col5 = 2)) THEN (_col10) END (type: decimal(20,0)), CASE WHEN ((_col5 = 1)) THEN (_col10) END (type: decimal(20,0)), CASE WHEN ((_col5 = 0)) THEN (_col10) END (type: decimal(20,0)), _col10 (type: decimal(20,0)) outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16","_col17","_col18","_col19","_col20"] Statistics:Num rows: 838753416 Data size: 871464800087 Basic stats: COMPLETE Column stats: NONE Select Operator [SEL_8] outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10"] Statistics:Num rows: 838753416 Data size: 871464800087 Basic stats: COMPLETE Column stats: NONE Map Join Operator [MAPJOIN_19] | condition map:[{"":"Left Outer Join0 to 1"}] | HybridGraceHashJoin:true | keys:{"Map 3":"_col1 (type: string)","Map 1":"_col10 (type: string)"} | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col11"] | Statistics:Num rows: 838753416 Data size: 871464800087 Basic stats: COMPLETE Column stats: NONE |<-Map 3 [BROADCAST_EDGE] vectorized | Reduce Output Operator [RS_22] | key expressions:_col1 (type: string) | Map-reduce partition columns:_col1 (type: string) | sort order:+ | Statistics:Num rows: 65114 Data size: 3581293 Basic stats: COMPLETE Column stats: NONE | value expressions:_col0 (type: varchar(40)) | Select Operator [OP_21] | outputColumnNames:["_col0","_col1"] | Statistics:Num rows: 65114 Data size: 3581293 Basic stats: COMPLETE Column stats: NONE | TableScan [TS_3] |alias:dim_dpi_ip_service |Statistics:Num rows: 65114 Data size: 3581293 Basic stats: COMPLETE Column stats: NONE |<-Select Operator [SEL_2] outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10"] Statistics:Num rows: 762503089 Data size: 792240710181 Basic stats: COMPLETE Column stats: NONE TableScan [TS_0] ACID table:true alias:dpi_other_detail_ufdr Statistics:Num rows: 762503089 Data size: 792240710181 Basic stats: COMPLETE Column stats: NONE Time taken: 3.479 seconds, Fetched: 70 row(s) hive> ... View more

[yarn@ks-dmp03 ~]$ hive log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAp Logging initialized using configuration in file:/etc/hive/2.6.3.0-235/0/hive-log hive> explain WITH DPI AS (SELECT SERVICE_NAME, trim(IP_ADDRESS) AS IP_ADDR FROM OTHER_SOURCES.DIM_DPI_IP_SERVICE), DSD AS (SELECT MSISDN, NVL(PROT_TYPE,0) AS PROTOCOL_ID, NVL(PROT_CATEGORY,0) AS PROTOCOL_CATEGORY_ID, TETHERING_FLAG, RAT, NETWORK_UL_TRAFFIC, NETWORK_DL_TRAFFIC, END_TIME_N, BEGIN_TIME_N, SID, trim(SERVER_IP) AS SERV_IP FROM OTHER_SOURCES.DPI_OTHER_DETAIL_UFDR WHERE HDAY="2018-03-06"), JN AS (SELECT DSD.MSISDN, DSD.PROTOCOL_ID, DSD.PROTOCOL_CATEGORY_ID, NVL(DPI.SERVICE_NAME,"Other") AS CUSTOMIZED_CATEGORY, DSD.TETHERING_FLAG, DSD.RAT, DSD.NETWORK_UL_TRAFFIC, DSD.NETWORK_DL_TRAFFIC, DSD.END_TIME_N, DSD.BEGIN_TIME_N, DSD.SID FROM DSD LEFT JOIN DPI ON DSD.SERV_IP = DPI.IP_ADDR) INSERT INTO AGG_DPI.AGG_DPI_TRAFFIC_DAILY_l PARTITION(HDAY="2018-03-06",SRC="OTHER_DETAIL") SELECT JN.MSISDN, JN.PROTOCOL_ID, JN.PROTOCOL_CATEGORY_ID, JN.CUSTOMIZED_CATEGORY, JN.TETHERING_FLAG, ROUND(SUM(CASE WHEN JN.RAT = 2 THEN JN.NETWORK_UL_TRAFFIC+JN.NETWORK_DL_TRAFFIC ELSE 0 END)/ 1024, 2) AS DATA_VOL_2G_AMT, ROUND(SUM(CASE WHEN JN.RAT = 1 THEN JN.NETWORK_UL_TRAFFIC+JN.NETWORK_DL_TRAFFIC ELSE 0 END)/ 1024, 2) AS DATA_VOL_3G_AMT, ROUND(SUM(CASE WHEN JN.RAT = 0 THEN JN.NETWORK_UL_TRAFFIC+JN.NETWORK_DL_TRAFFIC ELSE 0 END)/ 1024, 2) AS DATA_VOL_0_AMT, ROUND(SUM(JN.NETWORK_UL_TRAFFIC+JN.NETWORK_DL_TRAFFIC)/ 1024, 2) DATA_VOL_TOTAL_AMT, SUM(CASE WHEN JN.RAT = 2 THEN END_TIME_N - BEGIN_TIME_N ELSE 0 END) DATA_SESSIONS_DUR_2G, SUM(CASE WHEN JN.RAT = 1 THEN END_TIME_N - BEGIN_TIME_N ELSE 0 END) DATA_SESSIONS_DUR_3G, SUM(CASE WHEN JN.RAT = 0 THEN END_TIME_N - BEGIN_TIME_N ELSE 0 END) DATA_SESSIONS_DUR_0, SUM(END_TIME_N - BEGIN_TIME_N) DATA_SESSIONS_DUR, COUNT(DISTINCT CASE WHEN JN.RAT = 2 THEN JN.SID END) AS DATA_SESSIONS_2G_CNT, COUNT(DISTINCT CASE WHEN JN.RAT = 1 THEN JN.SID END) AS DATA_SESSIONS_3G_CNT, COUNT(DISTINCT CASE WHEN JN.RAT = 0 THEN JN.SID END) AS DATA_SESSIONS_0_CNT, COUNT(DISTINCT JN.SID) AS DATA_SESSIONS_CNT, 1 AS LOAD_ID, CURRENT_TIMESTAMP AS INS_DT FROM JN GROUP BY JN.MSISDN, JN.PROTOCOL_ID, JN.PROTOCOL_CATEGORY_ID, JN.TETHERING_FLAG, JN.CUSTOMIZED_CATEGORY; OK Plan not optimized by CBO due to missing statistics. Please check log for more details. ... View more

Application log very big: [yarn@serv03 ~]$ ls -hl /tmp/application_1518080001967_51536.log -rw-r--r-- 1 yarn hadoop 89M Mar 7 18:10 /tmp/application_1518080001967_51536.log ... View more