Cloudera Community
saikrishna_tara
user rank
Super Collaborator

Hive RegEx finding second pattern

by Super Collaborator in Support Questions
‎04-12-2017 09:13 PM
‎04-12-2017 09:13 PM
Hi , I have line entries in my HDFS file as these , i created an external table on this file. Mar 12 04:03:01 172.16.3.1 %ASA-6-106100: access-list FW-DATA permitted tcp FW-DATA/172.16.1.4(59289) -> OUTSIDE/52.87.195.145(22) hit-cnt 1 first hit [0xe37d63e0, 0x0] i am trying break this into different columns using the HiveQL below. select regexp_extract(log_entry, '[A-Z][a-z]+\s\d+\s\d+:\d+:\d+', 0) , regexp_extract(log_entry, '\d+\.\d+\.\d\.\d', 0) , regexp_extract(log_entry, '%ASA-6-106100', 0) , regexp_extract(log_entry, '\w+-\w+\s+\w+-\w+', 0) , regexp_extract(log_entry, 'permitted|denied', 0) , regexp_extract(log_entry, 'tcp|udp', 0) , regexp_extract(log_entry, '[A-Z]+-[A-Z]+', 0) , regexp_extract(log_entry, '\d+\.\d+\.\d\.\d', 0) from temp_firewall_log values for _c1 and _c7 are same , showing the first pattern it found. but how can i show second pattern in _c7.? and getting output like below.. ... View more
Labels:

Re: Help with Hive Regex extract.

by Super Collaborator in Support Questions
‎04-06-2017 09:54 PM
‎04-06-2017 09:54 PM
I think I found the answer..looks like we need to use double slashes in hive.. this is working when I replaced log_entry with the text from above.. select regexp_extract(log_entry, '[A-Z][a-z]+\\s\\d+\\s\\d+:\\d+:\\d+', 0) , regexp_extract(log_entry, '\\d+\\.\\d+\\.\\d\\.\\d', 0) , regexp_extract(log_entry, '%ASA-6-106100', 0) , regexp_extract(log_entry, '\\w+-\\w+\\s+\\w+-\\w+', 0) ... View more

Re: Help with Hive Regex extract.

by Super Collaborator in Support Questions
‎04-06-2017 08:53 PM
‎04-06-2017 08:53 PM
Hi @Bala Vignesh N V when I use the following SQL its not returning me anything..i am not getting any error. select regexp_extract('Mar 12 04:03:01 172.16.3.1 %ASA-6-106100: access-list FW-DATA permitted tcp FW-DATA/172.16.1.4(59289) -> OUTSIDE/52.87.195.145(22) hit-cnt 1 first hit', '([A-Z][a-z]+ \d+\s\d+:\d+:\d+)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+(\%ASA-6-106100):\s+(\w+-\w+\s+\w+-\w+)\s+(\w+)\s+(\w+)\s+(\w+-\w+)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+->\s+(\w+-?\w+?)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+(\w+-\w+\s+\d+\s+\w+\s+\w+)',0) How do I get my input string broken in to multiple columns based on the regex.? ... View more

Help with Hive Regex extract.

by Super Collaborator in Support Questions
‎04-06-2017 04:38 PM
‎04-06-2017 04:38 PM
Hi, i have a firewall log with entries like this.. Mar 12 04:03:01 172.16.3.1 %ASA-6-106100 access-list FW-DATA permitted tcp FW-DATA 172.16.1.4 59289 OUTSIDE 52.87.195.145 22 hit-cnt 1 first hit i created an external table in hive for this log file and i am trying to use HIVE SQL and regexp_extract to extract column out of this lines by using this regular expression. ([A-Z][a-z]+ \d+\s\d+:\d+:\d+)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+(\%ASA-6-106100):\s+(\w+-\w+\s+\w+-\w+)\s+(\w+)\s+(\w+)\s+(\w+-\w+)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+->\s+(\w+-?\w+?)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+(\w+-\w+\s+\d+\s+\w+\s+\w+) when i tested this at different sites (like https://regex101.com/) ..it is working fine..breaking this line into many different groups shown below. but when i try to use that in HIVE its not working..any help is appriciated. Mar 12 04:03:01 172.16.3.1 %ASA-6-106100 access-list FW-DATA permitted tcp FW-DATA 172.16.1.4 59289 OUTSIDE 52.87.195.145 22 hit-cnt 1 first hit ... View more
Labels:

Help with Hive Regex extract.

by Super Collaborator in Support Questions
‎04-06-2017 04:30 PM
‎04-06-2017 04:30 PM
Hi, i have a firewall log with entries like this.. Mar 12 04:03:01 172.16.3.1 %ASA-6-106100 access-list FW-DATA permitted tcp FW-DATA 172.16.1.4 59289 OUTSIDE 52.87.195.145 22 hit-cnt 1 first hit i created an external table in hive for this log file and i am trying to use HIVE SQL and regexp_extract to extract column out of this lines by using this regular expression. ([A-Z][a-z]+ \d+\s\d+:\d+:\d+)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+(\%ASA-6-106100):\s+(\w+-\w+\s+\w+-\w+)\s+(\w+)\s+(\w+)\s+(\w+-\w+)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+->\s+(\w+-?\w+?)\/?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\((\d+)\)\s+(\w+-\w+\s+\d+\s+\w+\s+\w+) when i tested this at different sites (like https://regex101.com/) ..it is working fine..breaking this line into many different groups shown below. but when i try to use that in HIVE its not working..any help is appriciated. Mar 12 04:03:01 172.16.3.1 %ASA-6-106100 access-list FW-DATA permitted tcp FW-DATA 172.16.1.4 59289 OUTSIDE 52.87.195.145 22 hit-cnt 1 first hit ... View more
Labels:

Re: GetHDFS Missing block exception

by Super Collaborator in Support Questions
‎04-04-2017 03:41 PM
‎04-04-2017 03:41 PM
Looks like only the Name nodes are opened for connectivity to\from HDF server , the data nodes are not . We are trying to fix and will test after that. ... View more

Re: GetHDFS Missing block exception

by Super Collaborator in Support Questions
‎04-04-2017 02:30 PM
‎04-04-2017 02:30 PM
@Bryan Bende I didn't try from command line , but it is accessible from Hue. will try from command-line. ... View more

GetHDFS Missing block exception

by Super Collaborator in Support Questions
‎04-03-2017 06:59 PM
‎04-03-2017 06:59 PM
Hi, I am trying to connect to our HADOOP cluster from a new HDF Server outside of our HADOOP cluster.i am trying to do some simple tests before i move all my flows onto the server from old server. I am experiencing issues when i try to access file using getHDFS..i copied the config files onto HDF server. both servers are using same kerberos KDC , so i am using same keytabs. here is the error message from applog. i was told all the ports for HDFS,HIVE etc are open for communication between the 2 servers.do i need to change anything in the config files.? 2017-04-03 13:14:36,648 ERROR [Timer-Driven Process Thread-4] o.apache.nifi.processors.hadoop.GetHDFS org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.hadoop.hdfs.client.HdfsDataInputStream@389fa2c0 for StandardFlowFileRecord[uuid=ee9aa74d-21eb-45cd-b6d5-6440c1a95093,claim=,offset=0,name=7412824162032320,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-306710789-172.16.3.5-1445707884245:blk_1075144155_1404007 file=/user/putarapasa/OCA_Nestac_XRef_Old.xlsx at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2690) ~[na:na] at org.apache.nifi.processors.hadoop.GetHDFS.processBatchOfFiles(GetHDFS.java:369) [nifi-hdfs-processors-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.processors.hadoop.GetHDFS.onTrigger(GetHDFS.java:315) [nifi-hdfs-processors-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-306710789-172.16.3.5-1445707884245:blk_1075144155_1404007 file=/user/putarapasa/OCA_Nestac_XRef_Old.xlsx at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2683) ~[na:na] ... 14 common frames omitted Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-306710789-172.16.3.5-1445707884245:blk_1075144155_1404007 file=/user/putarapasa/OCA_Nestac_XRef_Old.xlsx at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:984) ~[hadoop-hdfs-2.7.3.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642) ~[hadoop-hdfs-2.7.3.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882) ~[hadoop-hdfs-2.7.3.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934) ~[hadoop-hdfs-2.7.3.jar:na] at java.io.DataInputStream.read(DataInputStream.java:100) ~[na:1.8.0_112] at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) ~[nifi-utils-1.1.0.2.1.0.0-165.jar:1.1.0.2.1.0.0-165] at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:700) ~[na:na] at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2680) ~[na:na] ... View more
Labels:

Re: NiFi jetty server

by Super Collaborator in Support Questions
‎03-29-2017 02:29 PM
‎03-29-2017 02:29 PM
Hi @Matt Burgess i know nifi uses port 8080..i was wondering if Jetty it self runs on a different port. thank you. ... View more

NiFi jetty server

by Super Collaborator in Support Questions
‎03-29-2017 02:00 PM
‎03-29-2017 02:00 PM
Hi, I know NiFi uses jetty server , do we know on which port Jetty run on.? and also can we change it to use Apache or IIS.? Regards, Sai ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎10-26-2018 04:07 PM
Community Statistics
Member Since ‎07-08-2016 07:25 PM
Last Visited ‎10-26-2018 04:07 PM
Posts 260
Kudos received 44