Member since
05-24-2016
45
Posts
4
Kudos Received
0
Solutions
02-11-2019
12:21 PM
Hi, I want to disable services of a process group with cli.sh. But the command doesn't change the state of the services. All the 54 processors of the processGroup are stopped. Services stay at "ENABLED" state. Has anyone tried this command with success ? Commands line in attachement file Thanks for your feedback Thierry
... View more
Labels:
- Labels:
-
Apache NiFi
06-07-2018
08:07 AM
Hi @Vinicius Higa Murakami Thanks Vinicius. You're right. I didn't see the "ARMING" records with the "select...limit 2;" because the's a lot of "client" records and few "ARMING" records. But now, how exclude other records and have only "ARMING" rows in the final table and no row "NULL" ?
... View more
06-06-2018
10:26 AM
Hi, I would like to create an external table with an Hive regex expression by selecting lines containing "ARMING" (in uppercase). The records HDFS look like this 2018-06-06T11:28:54+02:00 sazqye named[980]: ARMING trigger on (action:LOG) (T6-Recursive-attacks recursive-time: 1283) 2018-06-06T11:20:27+02:00 sazqyd named[92960]: client (1.debian) ... My request : CREATE EXTERNAL TABLE my_arming_table (
dc_syslog_date STRING,
dc_syslog_hostname STRING,
dc_syslog_process STRING,
dc_logtype STRING, dc_message STRING)
PARTITIONED BY (yearbrut INT, monthbrut INT, daybrut INT) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES(
'input.regex'='^(\\S+)\\s(\\S+)\\s(\\S+)\\s(ARMING)\\s(.*)')
STORED AS TEXTFILE; The result is KO : > select * from my_arming_table limit 2 ; OK
NULL NULL NULL NULL NULL 0 0 0 NULL NULL NULL NULL NULL 0 0 0 And if I try this request (with client in lowercase) CREATE EXTERNAL TABLE my_client_table (
dc_syslog_date STRING,
dc_syslog_hostname STRING,
dc_syslog_process STRING,
dc_logtype STRING,
dc_message STRING)
PARTITIONED BY (yearbrut INT, monthbrut INT, daybrut INT) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES(
'input.regex'='^(\\S+)\\s(\\S+)\\s(\\S+)\\s(client)\\s(.*)')
STORED AS TEXTFILE; The result is OK > select * from my_client_table limit 2 ;
OK
2018-06-06T11:12:55+02:00 sazqyd named[92960]: client (swza6z) 0 0 0 2018-06-06T11:13:10+02:00 sazqyd named[92960]: client (osce01) 0 0 0 Does anybody knows why it doesn't work with uppercase in the regex expression ? Thanks
... View more
Labels:
- Labels:
-
Apache Hive
10-25-2017
01:36 PM
@Aditya Sirna, Of course... I'm going to try this. Thanks
... View more
10-25-2017
01:04 PM
Thanks but it doesn't work for the same reason. When you "mv /mydirectory /targetdirectory" the result is always /targetdirectory/mydirectory.
... View more
10-25-2017
12:22 PM
Thanks Not possible because the result is /targetdirectory/mydirectory and I expect all the files moved in path /targetdirectory/*
... View more
10-25-2017
12:07 PM
Hello, I've got 30 thousand of files to move to another hdfs directory. Do you know a better way than "hdfs dfs -mv /mydirectory/* /targetdirectory" to go faster ? Average size of a file : 10 Kb. And I can't merge the files in a bigger one before. Thanks for your feedback
... View more
Labels:
- Labels:
-
Apache Hadoop
10-23-2017
11:35 AM
1 Kudo
Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.
for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done
... View more
10-23-2017
06:40 AM
Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.
... View more
10-19-2017
04:14 PM
Hello, I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*" For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command. hdfs@host:~>
hadoop dfs -ls /mydirectory Found 1942
items -rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1 -rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2 -rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3 -rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4 -rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5 … Thanks for your feedbacks
... View more
Labels:
- Labels:
-
Apache Hadoop
06-14-2017
02:35 PM
@Matt Clarke The second suggestion works as well. I kepp the third one for a next usage. Thanks for all Matt TV.
... View more
06-14-2017
12:59 PM
@Matt Clarke Thanks for these suggestions. I'm going to try number 2. And could you give me an example of properties for the number 3 and detectduplicate processor ? Thanks, TV
... View more
06-14-2017
12:12 PM
1 Kudo
Hi everybody, I use Nifi 1.0.0 on AIX server. My ListFile processor gives the same file in two different dataflows. It schedules every 15 seconds. The file O27853044.1135 begins to fill at 11:35 and ends at 11:45. Is it normal that the processor creates a dataflow at 11:42 ? How avoid ListFile processor to create a dataflow before the end of file's update ? Thanks for you help
... View more
Labels:
- Labels:
-
Apache NiFi
05-23-2017
06:23 AM
@Wynner Hi, Memory settings already to 8Go for both. This morning I tried to set Number min and max of Entries and it works !!! Thanks for your help Wynner. Regards Parameters set (so the merge create about 1000 files) Minimum Number of Entries
1
Maximum Number of Entries
2500
Minimum Group Size
0 B
Maximum Group Size
No value set
Max Bin Age
5 min
Maximum number of Bins
100
Delimiter Strategy
Text
... View more
05-22-2017
01:14 PM
@Wynner Thanks for your feddback, Here are the properties (I've also tried with 200MB before but it doesn't work). Merge
Strategy Bin-Packing
Algorithm Merge
Format Binary
Concatenation Attribute
Strategy Keep Only Common
Attributes Correlation
Attribute Name No value set Minimum
Number of Entries 1 Maximum
Number of Entries No value set Minimum
Group Size 20 MB Maximum
Group Size 20 MB Max Bin Age
5 min Maximum
number of Bins 100 Delimiter
Strategy Text Header No value set Footer No
value set Demarcator Compression
Level 1 Keep Path false After an hour Nifi fails with a outOfmemory 2017-05-22 12:35:44,770 WARN [NiFi Web Server-22-acceptor-0@2c439296-ServerConnector@ccf1486{HTTP/1.1,[http/1.1]}{0.0.0.0:28080}] o.eclipse.jetty.server.AbstractConnector
java.lang.OutOfMemoryError: Java heap space
2017-05-22 12:35:44,784 WARN [NiFi Web Server-21] org.eclipse.jetty.servlet.ServletHandler Error for /nifi-api/flow/controller/bulletins
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder.toString(StringBuilder.java:407) ~[na:1.8.0_66]
at java.net.Inet4Address.numericToTextFormat(Inet4Address.java:373) ~[na:1.8.0_66]
at java.net.Inet4Address.getHostAddress(Inet4Address.java:328) ~[na:1.8.0_66]
at org.eclipse.jetty.server.Request.getRemoteAddr(Request.java:1193) ~[na:na]
at javax.servlet.ServletRequestWrapper.getRemoteAddr(ServletRequestWrapper.java:275) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:62) ~[classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) ~[na:na]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) ~[spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE]
... View more
05-22-2017
10:04 AM
Exactlty The queue after GetMongo has 2,500,000,evt (1,3GB). The following processor (MergeContent) can not empty this queue. And I don't understand why ?
... View more
05-19-2017
02:53 PM
I've set it to 100 or 1000 or 2000. After about 10 minutes, the processor reads with success all the collection but in one shot and whatever the value of the propoerty. Is it normal ?
... View more
05-19-2017
09:17 AM
Hi, I try to get 2,500,000 records from a MongoDB collection with 1. these parameters in bootstrap.conf # JVM memory settings java.arg.2=-Xms6144m java.arg.3=-Xmx6144m 2.the following properties processor SSL Context Service
No value set
Client Auth NONE Query
No value set Projection
No value set Sort
No value set Limit No value set Batch Size
No value set and with 2 concurrent tasks scheduling I've tried wtih 4Go, then 6Go for the memory settings. Nifi failed with OutofMemory errors With 4Go - Starting GetMongo processor at 09:05 and Error at 09:24 2017-05-19 09:24:40,150 ERROR [Timer-Driven
Process Thread-4] o.a.nifi.processors.mongodb.GetMongo
GetMongo[id=4ee5171c-1006-115b-5dc0-6ef54c1e9a73]
GetMongo[id=4ee5171c-1006-115b-5dc0-6ef54c1e9a73] failed to process due to
java.lang.OutOfMemoryError: Java heap space; rolling back session:
java.lang.OutOfMemoryError: Java heap space 2017-05-19 09:24:40,167 ERROR [Timer-Driven
Process Thread-4] o.a.nifi.processors.mongodb.GetMongo java.lang.OutOfMemoryError : Java heap space With 6Go - Starting GetMongo processor at 09:42 and Error at 10:28 2017-05-19
10:28:50,336 ERROR [NiFi logging handler] org.apache.nifi.StdErr 2017-05-19
10:28:50,337 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception:
java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"pool-2-thread-1" Do you have any suggestions to help me ? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-23-2016
09:24 AM
Hi @Pierre Villard Thanks for your answer.
... View more
06-23-2016
09:17 AM
Hi @jwit The idea is to match on multiple files. Instead of create several processors with one file to tail for each one, I'd like to create only one processor with a "pattern" to tail which adresses several files. Obviously, we except for each flow ingested the right properties "path, filename ... " of the original file tailed in this flow. So the properties values may change with each flow. Let me know if it's clear to you. Thanks
... View more
06-22-2016
08:49 AM
Hi, I've got more than one hundred files to tail and I would like to use patterns in the File to Tail property. For example : /donapp/*/p00/wi*/log/*.?.*log.2016-06-22 Do you think it's possible to improve this processor ? Or is there another way to do it ? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-16-2016
09:34 AM
Hi We'd like to get/ingest WMI windows events ... with NIFI. Is it possible ? Has anyone tried to do it ? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-06-2016
01:08 PM
Hi @Pierre Villard Which value in your test for property "Initial Start Position" ? Is the Nifi release important ? We use "0.5.1". Thanks
... View more
06-03-2016
01:03 PM
@Pierre Villard Results are the same even with the new pattern "wls.log.*" and also with all the value of "initial start position". I think there's a problem. All I want is to receive (tail) the new records that are written in the wls.log file. And also when the log file rotates and keeps the same name "wls.log". This case seems very simple yet. But I can't achieve it with Nifi. So I think there's a little bug. Don't you think the same ? Please let me know Thanks
... View more
06-03-2016
12:35 PM
No it's not what I did. I'm going to test again with Rolling Filename Pattern : wls.log.*.
... View more
06-03-2016
09:08 AM
Hi @Pierre Villard and @Bryan Bende I also tested with rolling file pattern property = wls.log and initial start = beggining of file Test1 the same result Test2 the same resut Test3 .... the same result than without file pattern property (see my previous comment to Bryan) Nifi Flow Data Provenance shows : 06/03/2016 08:44:24.759 CEST RECEIVE 389968cd-a1b6-497d-9e0a-c0e93ffe90f6 4 bytes . But I want to complete my comment. The 4 bytes (of the 32) "789" are the differnece beetween the size of wls.log file (before its init with "cp empty.txt wls.log") minus the size of the new record. So if the size before init of wls.log is 10 MB (for example), the next new records won't be ingested as the new records size will not reach 10 MB. . I think there's a bug... So do you think it's possible to contact Hortonworks support to improve Nifi product in this case ? . Thanks for your feedback.
... View more
06-03-2016
07:05 AM
Hi @Bryan Bende So with the second echo "record0123456789" >> ... (I've made a mistake in my original question, it's an echo instead of a cat command)... the results are Test1 the same Test2 the same Test3 .... not the same result cp empty.txt wls.log -rw-r--r-- 1 itve6530 userdsi 0 Jun 3 08:44 wls.log echo "record33333333333xxxx0123456789" >> wls.log Nifi Flow Data Provenance shows : 06/03/2016 08:44:24.759
CEST RECEIVE 389968cd-a1b6-497d-9e0a-c0e93ffe90f6 4 bytes And the 4 bytes are "789" the last of the echo command which contains 32 bytes ... funny, isn't it ? So I wrote another different record echo "record33333333333xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0123456789" >> wls.log Nifi Flow Data Provenance shows :06/03/2016 08:45:32.218
CEST RECEIVE fb6d2013-03ab-4576-989e-02e2bd9981f4 64 bytes with all the 64 bytes of the echo command... in progress ! So I wrote another different record echo "record44" >> wls.log OK in Nifi Flow Data Provenance : 06/03/2016 08:57:58.926 CEST RECEIVE e253f961-3a8a-43a2-97b1-25d54526a894 9 bytes echo "record55" >> wls.log KO in Nifi Flow Data Provenance (same size than previous record) echo "record5566" >> wls.log OK in Nifi Flow Data Provenance (differnent siez). Now it shows the last and the one before the last. 06/03/2016 08:58:28.013 CEST RECEIVE 3c646f8c-84c8-4353-af66-e7e06fc76560 11 bytes 06/03/2016
08:58:10.944 CEST RECEIVE e158b0ce-a856-44f9-b7cf-4769ce8b0f69 9 bytes Very interesting ...
... View more
06-03-2016
05:57 AM
Hi @Pierre Villard I've tried this in a similar situation (see the question posted Nifi : How avoid ingesting an old rolling file in TailFileProcessor ?) And I've got a problem with the odl rolled file that is always ingested when a new record is written into the new file. But there was a little difference. So I'll test in this situation and give you my feedback.
... View more
06-03-2016
05:53 AM
Hi @Bryan Bende I' gonna test this and give you a feedback. Please could you also have a look to Pierre's answer ?
... View more