Member since
05-04-2017
8
Posts
0
Kudos Received
0
Solutions
02-07-2018
06:48 AM
Launching 2 jobs in parallel allows me to finish the etl. 3 jobs in parallel creates a deadlock. There must be something wrong with my scheduler config
... View more
02-07-2018
06:43 AM
I'm having the same problem, I killed some jobs, but tez UI saus some of them are still running; no App ID or DAG ID. Query was launched from Hive HDP 2.6.0.3-8 YARN 2.7.3 Tez 0.7.0
... View more
11-08-2017
07:14 AM
Hello,
I'm designing ETL for my company.
The source database is Oracle. We have HDP 2.6.
To make things simple, I have 100s of tables to extract data from. I wrote a parameterised sqoop workflow that I call 100s times - each time with different table name as parameter. The calls are from a fork in another oozie workflow.
The problem I'm having is that all the resources of the cluster is being taken up by the oozie launcher and oozie action is not started as there are no resources left. Currently I'm using the default queue for everything.
So the question is; Is my design/understanding wrong? Any help would be appreciated. <fork name="startAllExtracts">
<path start="sqoop_action_list"/>
<path start="sqoop_card"/>
...
</fork>
<join name="endAllExtracts" to="end"/>
<action name="sqoop_action_list">
<sub-workflow>
<app-path>${pathWF}/pwc_05_sqoop</app-path>
<propagate-configuration/>
<configuration>
<property>
<name>table_name</name>
<value>action_list</value>
</property>
</configuration>
</sub-workflow>
<ok to="endAllExtracts"/><error to="fail"/>
</action>
<action name="sqoop_card">
<sub-workflow>
<app-path>${pathWF}/pwc_05_sqoop</app-path>
<propagate-configuration/>
<configuration>
<property>
<name>table_name</name>
<value>card</value>
</property>
</configuration>
</sub-workflow>
<ok to="endAllExtracts"/><error to="fail"/>
</action>
...
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache YARN
06-16-2017
12:13 PM
Thanks @mqureshi, however it gives me the same thing; beeline -u jdbc:hive2:// --hivevar rdate=112211 -e "select 9${hivevar:rdate}9"
Connecting to jdbc:hive2://
Connected to: Apache Hive (version 1.2.1000.2.5.0.0-1245)
Driver: Hive JDBC (version 1.2.1000.2.5.0.0-1245)
Transaction isolation: TRANSACTION_REPEATABLE_READ
OK
+------+--+
| _c0 |
+------+--+
| 99 |
+------+--+
1 row selected (4.268 seconds)
Beeline version 1.2.1000.2.5.0.0-1245 by Apache Hive
Closing: 0: jdbc:hive2:// I'm on HDP 2.5 btw. Test server, 1 node.
... View more
06-16-2017
12:13 PM
Made it to work. Had to use the full database path, i.e. Doesnt work: -u jdbc:hive2:// Works: -u jdbc:hive2://localhost:10000
... View more
06-16-2017
12:13 PM
Managed to get hive to substitute the variable, but not beeline; hive --hivevar rdate=112211 -e 'select 9${hivevar:rdate}9'
Logging initialized using configuration in file:/etc/hive/2.5.0.0-1245/0/hive-log4j.properties
OK
91122119
Time taken: 3.76 seconds, Fetched: 1 row(s)
... View more
06-16-2017
12:13 PM
beeline -u jdbc:hive2:// --hivevar rdate=112211 -e "set rdate" Returns rdate is underfined, but hive --hivevar rdate=112211 -e "set rdate"
returns: Logging initialized using configuration in file:/etc/hive/2.5.0.0-1245/0/hive-log4j.properties
rdate=112211 However, using the variable in hive still does not evaluate: hive --hivevar rdate=112211 -e "select 9${hivevar:rdate}9"
Logging initialized using configuration in file:/etc/hive/2.5.0.0-1245/0/hive-log4j.properties
OK
99
Time taken: 3.178 seconds, Fetched: 1 row(s)
... View more
06-16-2017
12:13 PM
Hello, I've been struggling with parameters in command line. Parameters work in interactive shell but I cannot figure out what is wrong. This is the code I'm running. Output should be 9911221199, but only 99 is displayed. [hive@bitest ~]$ beeline -u jdbc:hive2:// --hivevar rdate=112211 -e "select 9${hivevar:rdate}9"
Connecting to jdbc:hive2://
Connected to: Apache Hive (version 1.2.1000.2.5.0.0-1245)
Driver: Hive JDBC (version 1.2.1000.2.5.0.0-1245)
Transaction isolation: TRANSACTION_REPEATABLE_READ
OK
+------+--+
| _c0 |
+------+--+
| 99 |
+------+--+
1 row selected (3.818 seconds)
Beeline version 1.2.1000.2.5.0.0-1245 by Apache Hive
Closing: 0: jdbc:hive2://
... View more
Labels:
- Labels:
-
Apache Hive