Member since
02-08-2017
20
Posts
1
Kudos Received
0
Solutions
03-29-2017
10:58 AM
Hi Everyone, I am planning for HCA - HORTONWORKS CERTIFIED ASSOCIATE Certification in somtime. I have been referring to the exam objective as mentioned below - examobjectives-hcassociate.pdf I need some tips to make more solid foundation on my preparation for this exam. Considering there is no sample test or question available , I am struggling to understand the depth , I need to go through for each topic. Considering the exam broad defines itself into below categorties as mentioned in examobjectives-hcassociate.pdf 1. Data Access 2. Data Management 3. Data Governance and Workflow 4. Operations 5. Security My query is whether we need to just focus on objective component - What each component does and How it does ? (first two sections) or whether we need to see and understand all the tutorials , blogs and go in depth. Considering it being very vast and covering many Ecosystem component , understanding this will be really helpful. Thanks, Ankur
... View more
- Tags:
- hadoop
- Hadoop Core
Labels:
- Labels:
-
Apache Hadoop
03-08-2017
11:40 AM
Hi Fernando, I am sorry if I am asking basic question here. How to know the container Id where our content is present. Moreover I need to copy stuff on Nifi , how to know path with in container. (/file/path/within/container). Many thanks, Ankur
... View more
03-06-2017
08:03 PM
@Fernando Lopez Bello - Can you please advise how you pulled out what needed from inside the docker machine, that will be really helpful. Thanks, Ankur
... View more
03-06-2017
03:18 PM
@Fernando Lopez Bello - Please advise whether you are able to solve this issue? I am facing the exact same issue and don't know , how to resolve it. Thanks, Ankur
... View more
03-06-2017
02:44 PM
Hi @glupu, I got an error mentioned as below. [root@sandbox ~]# sudo docker start sandbox Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:89: jailing process inside rootfs caused \\\\\\\"can't create pivot_root dir , error mkdir /var/lib/docker/overlay/f02fb928677e5786db91f4f0e8d4d39ec6c267c63a67e2b0d30f0b5129f37505/merged/.pivot_root864353686: read-only file system\\\\\\\"\\\"\"\n"
Error: failed to start containers: sandbox [root@sandbox ~]# Please advise. Thanks, Ankur
... View more
03-06-2017
02:22 PM
Hi @glupu, Thanks for the reply again [root@sandbox ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES I believe that sandbox docker container is not up and running. Please advise how to restart this? I checked # ip a sh , it is still pointing to old ip - 192.168.216.130 ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:4f:f9:b5 brd ff:ff:ff:ff:ff:ff inet 192.168.216.130/24 brd 192.168.216.255 scope global dynamic ens32 valid_lft 1622sec preferred_lft 1622sec inet6 fe80::20c:29ff:fe4f:f9b5/64 scope link valid_lft forever preferred_lft forever Thanks, Ankur
... View more
03-06-2017
01:56 PM
Hi @glupu, Many thanks for the reply and I did as you have advised. I found output of [root@sandbox network-scripts]# ip a sh ens interface - ens32 You were right that I have a mismatch between the config files and the interface name. I did all the steps mentioned and created ifcfg-ens32 [root@sandbox network-scripts]# cat ifcfg-ens32 Generated by dracut initrd NAME="ens32" DEVICE="ens32" ONBOOT=yes NETBOOT=yes IPV6INIT=yes BOOTPROTO=dhcp TYPE=Ethernet
Now I am able to restart by executing the command - systemctl restart network [root@sandbox ~]# systemctl status network.service ● network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network) Active: active (exited) since Mon 2017-03-06 13:43:42 UTC; 37s ago Docs: man:systemd-sysv-generator(8) Process: 2978 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS) Process: 3220 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS) Mar 06 13:43:41 localhost.localdomain systemd[1]: Starting LSB: Bring up/down networking... Mar 06 13:43:42 localhost.localdomain network[3220]: Bringing up loopback interface: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 06 13:43:42 localhost.localdomain network[3220]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 06 13:43:42 localhost.localdomain network[3220]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 06 13:43:42 localhost.localdomain network[3220]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 06 13:43:42 localhost.localdomain network[3220]: [ OK ] Mar 06 13:43:42 sandbox.hortonworks.com network[3220]: Bringing up interface ens32: Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2) Mar 06 13:43:42 sandbox.hortonworks.com network[3220]: [ OK ] Mar 06 13:43:42 sandbox.hortonworks.com systemd[1]: Started LSB: Bring up/down networking.
After that I restarted the VM machine , hoping it will fix my issues of logging into sandbox. [root@sandbox network-scripts]# ssh root@sandbox.hortonworks.com -p 2222 ssh: connect to host sandbox.hortonworks.com port 2222: Connection refused Please provide me pointers of connection refused error if possible.
... View more
03-06-2017
08:55 AM
@Jay SenSharma Many thanks for the reply. I did check and the file etc/sysconfig/network-scripts/ifcfg-lo exist on the sandbox. Snapshot attached for the reference. snapshot-of-ifcfg-lo.gif [root@sandbox network-scripts]# service network restart Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.
[FAILED] [root@sandbox etc]# ssh root@sandbox.hortonworks.com -p 2222 ssh: connect to host sandbox.hortonworks.com port 2222: Connection refused Thanks, Ankur
... View more
03-05-2017
09:59 PM
Hi Friends, I need help here. I have been practicing the tutorials from 2-3 months through HDP2.5 VMware version. Today , my laptop has got hung and the VM machine has been stopped abruptly. After that I am not able to login to sandbox. Error Snapshot [root@sandbox ~]# ssh root@sandbox.hortonworks.com -p 2222
ssh: connect to host sandbox.hortonworks.com port 2222: Connection refused error-snapshot-1.gif ifconfig.gif [root@sandbox ~]# systemctl status network.service ● network.service - LSB: Bring up/down networking Loaded: loaded (/etc/rc.d/init.d/network) Active: failed (Result: exit-code) since Sun 2017-03-05 21:36:01 UTC; 3min 34s ago Docs: man:systemd-sysv-generator(8)
Process: 592 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE) Mar 05 21:36:01 sandbox.hortonworks.com network[592]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 05 21:36:01 sandbox.hortonworks.com network[592]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 05 21:36:01 sandbox.hortonworks.com network[592]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Mar 05 21:36:01 sandbox.hortonworks.com network[592]: [ OK ] Mar 05 21:36:01 sandbox.hortonworks.com network[592]: Bringing up interface ens33: Error: Connection activation failed: No suitable device found for this connection. Mar 05 21:36:01 sandbox.hortonworks.com network[592]: [FAILED] Mar 05 21:36:01 sandbox.hortonworks.com systemd[1]: network.service: control process exited, code=exited status=1 Mar 05 21:36:02 sandbox.hortonworks.com systemd[1]: Failed to start LSB: Bring up/down networking. Mar 05 21:36:02 sandbox.hortonworks.com systemd[1]: Unit network.service entered failed state. Mar 05 21:36:02 sandbox.hortonworks.com systemd[1]: network.service failed. I am attaching the error snapshot and also ifconfig for your reference. Please help!! Thanks, Ankur
... View more
02-26-2017
05:22 PM
1 Kudo
Hi Friends, Need help on TUTORIAL 3: REAL TIME DATA INGESTION IN HBASE AND HIVE USING STORM I ran the attached logs for deploying a new Storm Topology for Truck Events. logs.txt. I have noticed some of the errors - log4j:ERROR A "org.apache.log4j.xml.DOMConfigurator" object is not assignable to a "org.apache.log4j.spi.Configurator" variable. log4j:ERROR The class "org.apache.log4j.spi.Configurator" was loaded by log4j:ERROR [org.apache.atlas.plugin.classloader.AtlasPluginClassLoader@2715644a] whereas object of type
log4j:ERROR "org.apache.log4j.xml.DOMConfigurator" was loaded by [sun.misc.Launcher$AppClassLoader@5e2de80c].
log4j:ERROR Could not instantiate configurator [org.apache.log4j.xml.DOMConfigurator]. But when I checked in the Storm View located in the Ambari User Views list , I found new Topology truck-event-processor. After that I ran the NiFi DataFlow to generate events which publishes the data to Kafka queue. Attached Nifi Flow.gif for reference. nififlow.gif Return to the Storm View and click on truck-event-processor topology in the list of topologies to drill into it. I am not seeing any progress on Sprouts and Bolts. Attaching Storm.GIF for reference.storm.gif Moreover on the topology , I am seeing symbols of sprouts being used completely for entire flow with no bolt symbol. Kindly advise if there is any mistake I am doing. I am new to Storm , so any advise / pointer will definately help.
... View more
Labels:
- Labels:
-
Apache Storm
02-13-2017
08:39 AM
Hi @mqureshi, I should have been specific before. My requirement is that the big xml files are coming in real time and it is needed to be ingested through Nifi to covert into avro format. I had attached some of the xmls for your reference. Kindly have a look at those and advise. I have been reading and found --> 1. TransformXML processor - convert xml to json format easily , but it requires us to know XSLT format. Kindly advise. Thanks, Ankur
... View more
02-12-2017
05:52 PM
Hi Friends, Needed a help here. I was able to covert one of xml file lying in the zip - trafficLocs_data_for_simulator.zip to avro schema by able to define its structure in EvaluteXPath (image attached for reference). Many thanks to @mqureshi for his help in solving my last question. Now I want to understand , how we handle bigger xmls , do we need to define its structure in EvaluateXPath completely or is there is a simpler way to handle this? How we handle conversion of these big xml's into avro which really exist in the real life. Please advise. Attached some xmls for your reference. Many Thanks, Ankur evaluate-xpath.png xmls.zip
... View more
02-12-2017
09:25 AM
Hi @mqureshi , Many thanks brother. I am pleased to say it is working now. You have really been helpful. Now , I want to solve some bigger questions , the xml file lying in the zip - trafficLocs_data_for_simulator.zip was a small xml. Hence we are able to define its structure in EvaluteXPath , suppose we are having bigger xmlxmls.zip How we process these big xml's which really exist in the real life. Please advise. Attached some xmls for your reference. Many Thanks, Ankur
... View more
02-11-2017
11:52 PM
Hi @mqureshi, Many thanks for your reply. I had actually tried this option of using infer avro schema before but did not work. I wanted to check whether you have used XML file as a source or CSV file. My requirement is to convert the XML file into Avro format. If you see tutorials on Nifi , I had picked up one xml file lying in the zip - trafficLocs_data_for_simulator.zip. I am attaching some snapshots also warning message which is coming. Please have a look. 1-entire-flow.gif2-getfile.gif4-evaluatexpath.gif7-inferavroschema.gifwarning-message.png Thanks, Ankur
... View more
02-11-2017
05:47 PM
Hi @mqureshi, Many thanks for the reply. I have removed space but it is still showing the same issue Failed to Parse Record Schema. The Record schema is mentioned below - {
"type" : "record",
"name" : "first",
"namespace" : "xml",
"fields" : [{"name" : "Direction_of_Travel", "type" : "string" },
{ "name" : "Latitude", "type" : "float" },
{ "name" : "Longitude", "type" : "float" },
{ "name" : "Vehicle_ID", "type" : "integer" },
{ "name" : "Vehicle_Speed","type" : "integer" }]
} Please advise. Thanks, Ankur
... View more
02-11-2017
01:18 PM
Hi Friends, Needed a help here. I have picked up one of the xmls lying in the trafficLocs_data_for_simulator.zip. My objective is to convert XML to Avro format. The tutorials has shown us how to convert XML to JSON format by using AttributesToJSON processor. I am facing issue while configuring ConvertJSONToAvro processor. I have used the Record Schema as mentioned below - {
"type": "record",
"name" : "first",
"namespace" : "xml",
"fields" : [{"name" : "Direction_of_Travel", "type" : "string" },
{ "name" : "Latitude", "type" : "float" },
{ "name" : "Longitude", "type" : "float" },
{ "name" : "Vehicle_ID", "type" : "integer" },
{ "name" : "Vehicle_Speed"," type" : "integer" }]
}
I am getting red alert symbol mentioning it as Failed to Parse Record Schema. Any pointers will help. Thanks, Ankur K
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)