Member since
07-31-2013
98
Posts
54
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2424 | 11-23-2016 07:37 AM | |
2108 | 05-18-2015 02:04 PM | |
4456 | 05-13-2015 07:33 AM | |
3158 | 05-12-2015 05:36 AM | |
3654 | 04-06-2015 06:05 AM |
10-28-2020
04:48 AM
Hi What i have seen is tht the share option only gives you read or read+modify permisson. There is nothing as such execute? If i give read+modify other users will be ale to run the oozie workflow. I have seen it does not happen. As the permission on the underlying hdfs folder for the workflow is only for my user and it does not get modified. drwxrwx--- - kuaksha hue 0 2020-10-28 10:42 /user/hue/oozie/workspaces/hue-oozie-1520605312.96 Please elaborate and help. Regards Akshay
... View more
12-04-2019
04:38 AM
thanks. its worked nicely
... View more
05-10-2019
12:44 PM
That particular package only comes from the MySQL Community repository. Not from the OS repos or from the SCL repos.
... View more
04-12-2019
06:14 AM
this workflow is working? Just workflow and it will send to email?
... View more
12-10-2018
02:07 AM
Hi. How can you find your workflow from list of all workflows? They are nameless...
... View more
03-22-2018
05:59 AM
Looks like you already have another thread opened: http://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Hive-Safety-Valve-configuration-is-not-applied-HiveConf-of-name/td-p/64037 Will follow up there.
... View more
01-16-2018
06:20 AM
Hi @cconner I've connected to the Hue database on Mysql. I see all the the tables prefixed with oozie_ however I do not see any meaninful data in these tables. Can you explain where in this DB schema the workflow definitions are stored? Thanks Shak
... View more
09-12-2017
12:09 PM
Partitioner is not invoked when used in oozie mapreduce action (Creating workflow using HUE). But works as expected when running using hadoop jar commad in CLI, I have implemented secondary sort in mapreduce and trying to execute it using Oozie (From Hue). Though I have set the partitioner class in the properties, the partitioner is not being executed. So, I'm not getting output as expected. The same code runs fine when run using hadoop command. And here is my workflow.xml <workflow-app name="MyTriplets" xmlns="uri:oozie:workflow:0.5">
<start to="mapreduce-598d"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="mapreduce-598d">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.output.dir</name>
<value>/test_1109_3</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/</value>
</property>
<property>
<name>mapred.input.format.class</name>
<value>org.apache.hadoop.hive.ql.io.RCFileInputFormat</value>
</property>
<property>
<name>mapred.mapper.class</name>
<value>PonRankMapper</value>
</property>
<property>
<name>mapred.reducer.class</name>
<value>PonRankReducer</value>
</property>
<property>
<name>mapred.output.value.comparator.class</name>
<value>PonRankGroupingComparator</value>
</property>
<property>
<name>mapred.mapoutput.key.class</name>
<value>PonRankPair</value>
</property>
<property>
<name>mapred.mapoutput.value.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.reduce.output.key.class</name>
<value>org.apache.hadoop.io.NullWritable</value>
</property>
<property>
<name>mapred.reduce.output.value.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
</property>
<property>
<name>mapred.partitioner.class</name>
<value>PonRankPartitioner</value>
</property>
<property>
<name>mapred.mapper.new-api</name>
<value>False</value>
</property>
</configuration>
</map-reduce>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/> When running using hadoop jar command, I set the partitioner class using JobConf.setPartitionerClass API. Not sure why my partitioner is not executed when running using Oozie. Inspite of adding <property>
<name>mapred.partitioner.class</name>
<value>PonRankPartitioner</value>
</property>
... View more
11-23-2016
07:37 AM
Unfortunately there is no way to provide super user access to an entire group today. It must be done one user at a time. There is a feature request to add this in a future release. You could do it programatically to make it a little easier, see the section "How to make a certain user a Hue admin" http://gethue.com/password-management-in-hue/ You could create a list of users and iterate through them in hue shell. Make sure to set HUE_CONF_DIR=/var/cloudera-scm-agent/process/id-hue-HUE_SERVER where id is most recent. ON CDH 5.5 and above you also have to set: export HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 export HUE_DATABASE_PASSWORD=huedatabasepassword Hope this helps.
... View more
12-03-2015
09:22 AM
Hue sends queries to HiveServer2, so you would want to implement it in the CM configuration for HS2, which is this safety valve in the Hive service: HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml
... View more
08-06-2015
10:46 AM
First, try to index only couple of records. If that doesn't work then recheck the data values and data types used. If it works then index the data in small batches ... I faced a similar issue if more than 200 records are indexed.
... View more
06-19-2015
04:46 PM
If you use CM you don't need to use the Hue safety valve. Make sure the Trift Server is added and started Make sure in the Hue configuration that the HBase service and the HBase thrift service are checked Restart Hue and that's it! Romain
... View more
05-18-2015
02:24 PM
I decided to do a quick check on the attempt processing one of oozie's hive actions: org.apache.hadoop.mapred.YarnChild 10.0.0.9 33024 attempt_1431965085423_0015_m_000000_0 2 ...and collected a quick stack trace which shows the hive client does indeed run inproc with the JVM running the attempt: "main" prio=10 tid=0x00007f4dbc027800 nid=0xac2c in Object.wait() [0x00007f4dc5c2b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000fe43fa48> (a org.apache.hadoop.ipc.Client$Call) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.ipc.Client.call(Client.java:1454) - locked <0x00000000fe43fa48> (a org.apache.hadoop.ipc.Client$Call) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy32.getJobReport(Unknown Source) at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:323) - locked <0x00000000ff3bafd0> (a org.apache.hadoop.mapred.ClientServiceDelegate) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:422) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:603) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:611) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:636) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:288) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:345) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:443) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:459) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Thanks
... View more
05-12-2015
05:36 AM
You can do the following: 1. Spawn the Hue shell: export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk '{print $9}'`"
cd /opt/cloudera/parcels/CDH/lib/hue (or /usr/lib/hue if using packages)
./build/env/bin/hue shell 2. Paste the following python in the shell replacing <username> with the user you want to become superuser: from django.contrib.auth.models import User
a = User.objects.get(username='<username>')
a.is_staff = True
a.is_superuser = True
a.save()
... View more
04-06-2015
10:28 AM
I ended up going with option 2 which solved the immediate problem. Thanks.
... View more
03-07-2015
06:11 PM
Great! And for information, look at the Hue configuration section and look at Sentry, you should just need to check it and it will configure Hue automatically (no need to use any safety valve or put the Sentry service on the same host as Hue) Romain
... View more
07-03-2014
12:16 AM
1 Kudo
Hi, to read data in avro format from Hive you have to use an Avro SerDe. Maybe a good starting point will be http://www.michael-noll.com/blog/2013/07/04/using-avro-in-mapreduce-jobs-with-hadoop-pig-hive/ But this is not related to this topic since the solr sink will put data into Solr. I'd suggest to use just a HDFS sink to put your data on HDFS and create an (external or not) Hive table afterwards. You do not need Solr and/or Morphlines for this. best, Gerd
... View more
06-10-2014
09:22 AM
1 Kudo
Hey In Step 4 are we editing the file /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/etc/oozie/conf.dist/oozie-site.xml manually? We are using CM. I see changes in CM but not in the oozie-site.xml or on Oozie admin UI . Getting error [JA020: Could not load credentials of type [hbase] with name [hbase_credentials]]; perhaps it was not defined in oozie-site.xml?] Thanks
... View more
04-04-2014
01:34 PM
That's right, you'll have to put the following in your Hue Service->Configuration->Hue Server->Advanced-> Hue Server Configuration Safety Valve for hue_safety_valve_server.ini: [hadoop] [[mapred_clusters]] [[[default]]] jobtracker_host=cdh45-2.qa.test.com thrift_port=9290 jobtracker_port=8021 submit_to=true hadoop_mapred_home={{HADOOP_MR1_HOME}} hadoop_bin={{HADOOP_BIN}} hadoop_conf_dir={{HADOOP_CONF_DIR}} security_enabled=true logical_name=logicaljt [[[jtha]]] jobtracker_host=cdh45-1.qa.test.com thrift_port=9290 jobtracker_port=8021 submit_to=true hadoop_mapred_home={{HADOOP_MR1_HOME}} hadoop_bin={{HADOOP_BIN}} hadoop_conf_dir={{HADOOP_CONF_DIR}} security_enabled=true logical_name=logicaljt Leaving off "secuirty_enabled=true" if you're not using kerberos.
... View more
04-01-2014
02:59 AM
Hi just to follow up on this, I have now solved the problem. There were two things that I needed to do: 1. In addition to adding oozie.libpath to my job.properties, I also needed to include oozie.use.system.libpath=true 2. Before I was using the following line to add files to the DistributedCache: FileStatus[] status = fs.listStatus("/application/lib");
if (status != null) {
for (int i = 0; i < status.length; ++i) {
if (!status[i].isDir()) {
DistributedCache.addFileToClassPath(status[i].getPath(), job.getConfiguration(), fs);
}
}
} This appeared to be causing a classpath issue because it was adding hdfs://hostname before the hdfs path. Now I am using the following to remove that and only add the absolute hdfs path: FileStatus[] status = fs.listStatus("/application/lib");
if (status != null) {
for (int i = 0; i < status.length; ++i) {
if (!status[i].isDir()) { Path distCachePath = new Path(status[i].getPath().toUri().getPath());
DistributedCache.addFileToClassPath(distCachePath, job.getConfiguration(), fs);
}
}
} Thankyou to those that replied to my original query for pointing me in the right direction. Andrew
... View more
03-04-2014
05:24 AM
Hi Chris, thanks, restarting the MGMT services solved the test issues. bye...Gerd...
... View more
01-13-2014
01:25 PM
Ok great. I restarted all MapReduce and Hbase daemons in addition to the HDFS daemons and it is working properly now. Thank you! Previously I had only restarted HDFS.
... View more
10-28-2013
10:44 AM
No, I stoped all jobes before changing configuration in MapReduce service, and restarted all cluster. I also checked folder mapred/local/taskTracker/hdfs/jobcache and i am sure it was empty. Thank you for the link, but I found nothing about jobcache folder. Also, after the job failes or completed job folder are deleted from jobcache folder whith all attemp_xxxx folders.
... View more
10-04-2013
08:14 AM
Hi Chris, I tried your suggestions... and various alternatives: [zk: localhost:2181(CONNECTED) 8] ls /
[configs, solr,n1:2181, zookeeper, solr.xml, solr,n1.hdc:2181, solr]
[zk: localhost:2181(CONNECTED) 9] ls /solr
[clusterstate.json, aliases.json, solr.xml, live_nodes, overseer, collections, overseer_elect]
[zk: localhost:2181(CONNECTED) 10] ls /solr/collections
[]
[zk: localhost:2181(CONNECTED) 11] ls /solr,n1:2181
[solr]
[zk: localhost:2181(CONNECTED) 12] ls /solr,n1:2181/solr
[configs]
[zk: localhost:2181(CONNECTED) 13] ls /solr,n1:2181/solr/configs
[collection4]
[zk: localhost:2181(CONNECTED) 14] ls /solr,n1.hdc:2181/solr
[configs]
[zk: localhost:2181(CONNECTED) 15] ls /solr,n1.hdc:2181/solr/configs
[collection2, collection1]
[zk: localhost:2181(CONNECTED) 16] Btw, my nodes are all in the domain hdc.
... View more
09-30-2013
11:53 PM
Thanks Chris.. I understood the process when i spent more time on it. I have tried the same and it did work.. Probably i might have asked question little too early.. 🙂 Thanks for your resonse...
... View more