Member since
06-10-2016
18
Posts
2
Kudos Received
0
Solutions
06-03-2019
03:07 PM
Hello, where I can find updated information upon how to install a small cluster in Azure? Preferably a video. I understand that using cloudbreak looks like the most promising way. After I have my cluster, is it possible to store the parameters and be able to launch the cluster on Azure at will, without having to spend a lot of time configuring the same things? Or even better do you know if there is a way to launch already preconfigured clusters from a blueprint? Say for example that I have a script which automatically provides 5 or 6 VMs on Azure and then launce HDP automatically? Thank you
... View more
Labels:
11-16-2017
03:21 PM
use <env-var>HADOOP_USER_NAME=${wf:user()}</env-var> in the workflow.xml I found the solution here: https://community.hortonworks.com/content/kbentry/26593/permission-denied-useryarn-accesswrite-oozie-shell.html
... View more
11-15-2017
09:16 PM
hello, I use an oozie shell action to create a file aaaknowledge.txt in /tmp and then copy it to HDFS as per script.sh: echo "`date -u`" >> /tmp/aaaknowledge.txt hadoop fs -copyFromLocal /tmp/aaaknowledge.txt hdfs://sandbox.hortonworks.com:8020/user/maria_dev/oozie/log However the created directory has yarn as owner, thus the user cannot access it. How can I setup user rights for oozie actions?
... View more
08-11-2017
09:04 PM
1 Kudo
Hello, I want a file in HDFS created/updated after a successful completion of an oozie job. I thought of having the last node being a shell action. Are there other alternatives? It is running a script like: echo "Data process ok `date -u`" >> hdfs://sandbox.hortonworks.com:8020/user/maria_dev/output.oozie This is always killed.. Then, I thought that perhaps oozie cannot create a file in HDFS, so I tried instead: echo "Data process ok `date -u`" >> /home/maria_dev/output.oozie which again is always killed... Interestingly enough, if I use a different directory it works: echo "Data process ok `date -u`" >> /tmp/output.oozie So, I guess that is a problem with oozie permissions ??? I use the sandbox, HDP 2.5 my workflow.xml, looks like: <?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell-action">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>myscript.sh</exec>
<file>${nameNode}/user/maria_dev/oozie/myscript.sh#myscript.sh</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>
... View more
Labels:
07-24-2017
07:14 PM
thanks guys, in my case, it is 192.168.163.134 port 2222 It works, but after I had to download and install again the sandbox
... View more
07-20-2017
04:22 PM
Hello, as a matter of fact, it seems that I cannot use the command line for any Hadoop related commands... I have an HDP 2.5 sandbox and I cannot login as hdfs user. I connect through ssh at 192.168.163.134 (as per my VM display), port 22 when I type: su hdfs I get : su: user hdfs does not exist I tried also port 2222 , but I get access denied when I try to login as root/hadoop I also tried to use the built-in web client (http://192.168.163.134:4200) but I receive : 192.168.163.134 refused to connect. any ideas?
... View more
06-09-2017
10:16 AM
Hello Is there an archive from where I can download a HDP 2.5 sandbox? currently in https://hortonworks.com/downloads/#sandbox I can find only HDP 2.6 thank you
... View more
Labels:
06-08-2017
03:48 PM
thank you Mark
... View more
06-08-2017
03:37 PM
Hello, I currently use spark submit to execute an SBT created jar. How can I automate the process in order to let it start periodically
(i.e. every night or every few hours?) thank you
... View more
Labels:
06-08-2017
03:35 PM
Hello, I currently use spark submit to execute an SBT created jar. How can I automate the process in order to let it start periodically
(i.e. every night or every few hours?) Thank you
... View more
Labels:
07-27-2016
03:28 PM
Thanx Simon, do you know perhaps of a tool that can enable me to read the ORC files in windows environment (I am not supposed to have access to Linux terminal)? and even hopefully automate the data (I have hundrends of ORC files)?
... View more
07-27-2016
03:15 PM
thanx I get : waiting for AM container to be allocated, launched and register with RM
... View more
07-27-2016
03:03 PM
they are actually very small test files 10-30 rows and 2-10 columns. I can save them in HDFS and then create a table in hive and load the data from HDFS, but in this case it works but my string data are inside " " (as in the original csv) ... and thus there is a need to clean them afterwards. In contrast, when I use Ambari to upload a table, the wizard removes them automatically ...
... View more
07-27-2016
12:01 PM
thanx Mukesh, is there a way to see the log from ambari? or a web browser? I do not have credentials for a linux terminal...
... View more
07-27-2016
10:48 AM
Hello community, I try to upload csv files to Hive using the ambari upload functionality and although the file preview is fine and the upload process starts nice, and the first 3 steps are completed with success, it freezes at upload error: waiting for insertion of rows from temporary table to actual table... I left it overnight and nothing happened... The final status remains undefined (from servername:8088/cluster) Interesting I do not get any errors, warnings from Hive... In Hive I get only the new table but without data. Sometimes I can also see the temp table with data.
... View more
Labels:
07-27-2016
10:26 AM
1 Kudo
Hello community, How can I upload ORC files to Hive? I was given an ORC file to import into hive. The upload table functionality in Ambari, which I always used, supports only csv, json and xml. I thought of saving the file in HDFS in the respective database directory and then create table in hive and load the data. Unfortunately, I cannot open the the file (I have only notepad) to see it's schema. Thank you.
... View more
Labels:
06-10-2016
11:25 AM
Hello @Ryan Cicak I cannot find the calculator either... Also the self paced course from the university has a broken link: "Hortonworks provides some basic help in an
online Hadoop Cluster Configuration Guide available at http://hortonworks.com/cluster-sizing-guide/ " 404 error
... View more