Member since
05-07-2018
331
Posts
45
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6597 | 09-12-2018 10:09 PM | |
2605 | 09-10-2018 02:07 PM | |
8893 | 09-08-2018 05:47 AM | |
2884 | 09-08-2018 12:05 AM | |
3881 | 08-15-2018 10:44 PM |
09-25-2018
02:33 AM
Hey @Vinicius Higa Murakami .. I now get it . The issue was with the minifi 0.3 tool kit . When i tried to create the config.yml file from minifi tool kit version 0.3 , i kept on getting this issue. As soon as I switched to the latest version for tool kit , this problem got fixed and the port id was correct. Thanks for the help. I have accepted your answer as well
... View more
09-17-2018
05:46 AM
@A C Just to understand, did you run the spark submit using yarn cluster as master/deploy mode? If so, let's try to check the job properties for the following parameter: ${resourceManager} Also, here it is another example regarding pyspark + oozie (using shell to submit spark). https://github.com/hgrif/oozie-pyspark-workflow Hope this helps
... View more
09-13-2018
10:16 PM
Hello @Teresa Tavernelli! Glad to know that you made it 🙂 So regarding the Mysql, you can give it a shot with the following parameters: hostname -> your-sandbox-hostname
port -> 3306
user -> root ps: I didn't test it myself Also if it doesn't work, try with the hive configs for mysql (instead of using username as root, change it to hive) Hope this helps!
... View more
08-29-2018
06:01 PM
2 Kudos
How to use make a simple flow using SNMPSET/SNMPGET Append A - Troubleshooting common mistakes Pre-requisites: - Nifi Cluster Installed (I'm using HDF 3.1.2.0 hence with Nifi 1.5) - Centos7 With your NIFI cluster installed, you're up to initiate this step-by-step. First of all, we'll need to install a snmp server to retrieve/set values of/to MIB's. In my case I'm using net-snmp tool, and it comes with some MIB samples to play. Here we're going to use the SNMPv2-MIB.txt under the /usr/share/snmp/mibs path 1)Install net-snmp yum install -y net-snmp net-snmp-utils net-snmp-libs 2)Give full access of read-write to anyone at public community (DON'T DO IT IN PROD, please) echo > /etc/snmp/snmpd.conf
printf "agentAddress udp:161\nrwcommunity public\nrwuser public\nrocommunity public default system\nrouser public" >> /etc/snmp/snmpd.conf 3)Start the SNMP SERVER DAEMON - port 161 service snmpd start 4)Test a simple snmpwalk to see if those changes at step 2 are working snmpwalk -v2c -mALL -c public localhost system 5)Test a simple snmpget snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 6)Setting a simple value to overwrite the value above, and check again to see if the value has been replaced with the new value snmpset -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 = "Vinicius Higa Murakami"
snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 7)Login as nifi user and take the same steps as above (just to ensure that the nifi user has access to read/write) su - nifi
snmpset -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 = "Nifi it's here"
snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 8)Now we're ready to use nifi and draw the simple flow, using SNMP processors. Go to nifi UI and draw the following components: GenerateFlowFile Property SetSNMP Property LogAttribute Property GetSNMP Property LogAttribute Property 9)Run the nifi flow (snmpset and snmpget) and check if your value is showing up in the nifi-app.log Append A To troubleshoot the snmp, you can enable the DEBUG log by adding the following line to the /etc/sysconfig/snmpd OPTIONS="-A -p /var/run/snmpd -a -LF 7 /var/log/snmpd.log" And then make some snmpget (step 7) to check if it's logging the connections into the /var/log/snmpd.log. Should appear 2 lines: Connection from UDP: [127.0.0.1]:40769->[127.0.0.1]:161
Received SNMP packet(s) from UDP: [127.0.0.1]:40769->[127.0.0.1]:161 If you're having issues with the SNMPSET like the below. Check if your snmpset it's able to reach the snmp server and send snmp packets, to do this, you can use tail -f /var/log/snmpd.log to monitor the connections and start the flow to watch the behaviour. Other attention point is to check if your nifi has permissions to set/get values from MIB's (make sure you did the step 7). And lastly is to check if your snmp$oid is valid. And here's the template used: template-kb-snmp.xml
... View more
Labels:
08-29-2018
11:56 AM
Thanks a lot! @Vinicius Higa Murakami, I have one small question, how I can connect to livy2 server? I need to know some parameters to connect(look at the screenshot), but I don't know where I could find them. I'm work at SandBox 2.6.5(last version) Please help me, if u can. Thanks!
... View more
08-22-2018
08:35 AM
Hi @Vinicius Higa Murakami, That worked. After installing the hive interact module i was able to create the external table. Thank you so much! Sorry for the late response but i went on vacation 😛
... View more
08-21-2018
10:08 AM
Update: It is found that .hiverc was used in hive user for HIVE CLI so the difference was found. hive.exec.scratchdir=/user/hive/scratch hive.exec.stagingdir=/user/hive/staging The issue is hdfs /user/hive directory is encrypted with Ranger and hdfs /tmp/hive directory is non-encrypted and can read/write by all user in hadoop group. hive-site.xml <property> <name>hive.security.authorization.sqlstd.confwhitelist.append</name> <value>hive\.exec\.scratchdir|hive\.exec\.stagingdir</value> <description>append conf property in white list followed by pipeline</description> </property> Restart the metastore and hiveserver. I tested with beeline with session level change . The execution is fast like HIVE CLI . hive.exec.scratchdir=/user/hive/scratch hive.exec.stagingdir=/user/hive/staging I tested with HIVE CLI with session level change. The execution is slow with MAP reduce Job for moving data. hive.exec.scratchdir=/tmp/hive/scratch hive.exec.stagingdir=/tmp/hive/staging So the root cause is data are encrypted in /user/hive and not encrypted in /tmp/hive. Solution is to make ssession level change to use same encryption zone. So below INFO log will be printed if the encryption zones are different. metadata.Hive: Copying source hdfs://edhcluster/tmp/hive/staging_hive_2018-08-07_16- 29-12_750_8973639287951385407-1/-ext-10000/000001_0 to hdfs://edhcluster/user/hive/warehouse/temp_tro/000001_0 because HDFS encryption zones are different. Thanks, Manjil
... View more
07-29-2018
03:15 AM
Good to know @Mohammad Shamim! Please if the issue is solved, I'd kindly ask you to accept the answer. This will help the other HCC users to find the best answer faster and will encourage the other users to keep doing a good job as well 🙂
... View more
07-05-2018
05:41 AM
Hi @heta desai! Yes you can 🙂 Here's a link with more details: http://druid.io/docs/latest/ingestion/data-formats.html Hope this helps!
... View more
06-27-2018
03:56 AM
@Raj ji You can use ExecuteProcess (doesn't allow any incoming connections) (or) ExecuteStreamCommand processors to trigger the shell script. ExecuteProcess configs: As your executable script is on Machine 4 and NiFi installed on Machine1 so create a shell script on Machine 1 which ssh into Machine 4 and trigger your Python Script. Refer to this and this links describes how to use username/password while doing ssh to remote machine. As you are going to store the logs into a file, so you can use Tail file processor to tail the log file and check is there any ERROR/WARN, by using RouteText Processor then trigger mail. (or) Fetch the application id (or) application name of the process and then use yarn rest api to get the status of the job Please refer to how to monitor yarn applications using NiFi and Starting Spark jobs directly via YARN REST API and this link describes yarn rest api capabilities.
... View more