Member since
01-11-2016
355
Posts
230
Kudos Received
74
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7869 | 06-19-2018 08:52 AM | |
2972 | 06-13-2018 07:54 AM | |
3411 | 06-02-2018 06:27 PM | |
3606 | 05-01-2018 12:28 PM | |
5095 | 04-24-2018 11:38 AM |
04-25-2016
11:43 PM
1 Kudo
Hi @Revathy Mourouguessane, You can use IsEmpty to check if A1 is empty or not. Try something like this grouped = COGROUP ..... ;
filtered = FILTER grouped BY not IsEmpty($2);
DUMP filtered;
Here's an example that shows how this work for something similar: cat > owners.csv
adam,cat
adam,dog
alex,fish
david,horse
alice,cat
steve,dog
cat > pets.csv
nemo,fish
fido,dog
rex,dog
paws,cat
wiskers,cat
owners = LOAD 'owners.csv' USING PigStorage(',') AS (owner:chararray,animal:chararray);
pets = LOAD 'pets.csv' USING PigStorage(',') AS (name:chararray,animal:chararray);
grouped = COGROUP owners BY animal, pets by animal;
filtered = FILTER grouped BY not IsEmpty($2);
DUMP grouped;
(cat,{(alice,cat),(adam,cat)},{(wiskers,cat),(paws,cat)})
(dog,{(steve,dog),(adam,dog)},{(rex,dog),(fido,dog)})
(horse,{(david,horse)},{})
(fish,{(alex,fish)},{(nemo,fish)})
DUMP filtered;
(cat,{(alice,cat),(adam,cat)},{(wiskers,cat),(paws,cat)})
(dog,{(steve,dog),(adam,dog)},{(rex,dog),(fido,dog)})
(fish,{(alex,fish)},{(nemo,fish)})
... View more
04-22-2016
05:05 PM
Hi @AKILA VEL, Please check this tutorial on how you can do a wordcount with Spark on HDP 2.3: http://fr.hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ Section 1 shows how to upgrade Spark to 1.6 version. You can ignore it and go directly to section 2. I hope this will help you.
... View more
04-21-2016
12:38 PM
Can you delete this question please since it's a duplicate. Thanks
... View more
04-21-2016
12:36 PM
Hi @Klaus Lucas, The VM has Ambari installed and configured so you should get Ambari UI at port 8080. Can you check your VM settings (port redirection, network, etc) and see if you can get access to Ambari ?
... View more
03-29-2016
07:47 PM
4 Kudos
Hi @Vadim, OpenCV is famous for image processing in general. They have several tools for image and face recognition. Here is an example of how to do face recognition with OpenCV: tutorial. In terms of integration with Hadoop, there's a framework called HIPI developed by University of Virginia for leveraging HDFS and MapReduce for large scale image processing. This framework supports OpenCV too. Finally, for image processing in motion, you can use HDF with an OpenCV processor like the one published here
... View more
03-16-2016
05:12 PM
Hi @Lubin Lemarchand Try to change the parameter through Ambari. Go to HDFS -> Config and search for dfs.permissions.superusergroup Ambari stores the configuration in a database which is the truth of configuration. If you directly modify configuration files that are managed by Ambari, it will update the file and delete your modification at service restart. See this link doc
... View more
03-06-2016
10:21 PM
5 Kudos
@Abha R Panchal What user are you currently logged in as ? the user dev_maria doesn't have admin access so you will not have the add service button. To add services, you have to log in with admin. The admin user has been deactivated in HDP 2.4 sandbox. To activate it use the following command: ambari-admin-password-reset
... View more
03-05-2016
03:26 PM
2 Kudos
@Kyle Prins The sandbox gives you an easy way to have a working Hadoop installation in a VM. If you need a multi nodes cluster my advice is to install an HDP cluster by yourself. This way, you will understand what have been installed and how it was configured. Use Ambari for the installation, it's straightforward and quick : http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_Installing_HDP_AMB/content/index.html If you want to have all nodes as VMs in your local machine, you can use Vagrant too. Look at these links to have an idea on how to do it http://uprush.github.io/hdp/2014/12/29/hdp-cluster-on-your-laptop/ and https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide
... View more
03-05-2016
03:00 PM
@vinay kumar Maybe you have problem in disk partitioning. Can you check how much space you have allocated for partitions used by HDP? Here's a link for partitioning recommendations http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_cluster-planning-guide/content/ch_partitioning_chapter.html
... View more
03-04-2016
04:25 PM
3 Kudos
Hi @Prakash Punj
You can use NiFi to supervise a directory and ingest each new file to HDFS (GetFile and PutHDFS processors). https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetFile/index.html You can do Spark in a browser with Zeppelin. You can have it in Ambari with the Zeppelin view. Some tutorials here http://hortonworks.com/hadoop/zeppelin/#tutorials To avoid a SPOF you need HDFS HA. Federation is having multiple NNs for managing very big clusters and reducing the stress on a single NN. In Ambari you can have admin users and simple users. Simple users have less power in Ambari.
... View more
- « Previous
- Next »