Member since
10-22-2017
7
Posts
1
Kudos Received
0
Solutions
05-29-2018
02:12 PM
1 Kudo
Hi, I am running Nifi from CentOS , which I configured to run my Cloudera. To start Nifi, I change the Java version, exporting manually and start Nifi. I open the browser and could not load the Nifi GUI( http://localhost:8080/nifi/). I start tail -f for the log "tail -f /home/cloudera/raja/nifi-1.5.0/logs/nifi-app.log" . After a long time , it redirects to http://localhost:8080/nifi/. So if I have to transfer Kafka topics from CentOS Cloudera to Nifi, HDP, it may be misleading for me as the latency is so high. Please suggest. ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2018-05-29 06:57:50,071 INFO [Timer-Driven Process Thread-1] o.a.kafka.common.utils.AppInfoParser Kafka version : 0.10.2.1
2018-05-29 06:57:50,071 INFO [Timer-Driven Process Thread-1] o.a.kafka.common.utils.AppInfoParser Kafka commitId : e89bffd6b2eff799
2018-05-29 06:58:01,912 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d306d27{/nifi-api,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-api-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.5.0.war}
2018-05-29 06:58:04,866 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1800ms
2018-05-29 06:58:04,881 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
2018-05-29 06:58:05,001 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5317a7ef{/nifi-content-viewer,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-content-viewer-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.5.0.war}
2018-05-29 06:58:05,039 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@53c39950{/nifi-docs,null,AVAILABLE}
2018-05-29 06:58:05,200 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=87ms
2018-05-29 06:58:05,331 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath
2018-05-29 06:58:05,364 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@3d056418{/nifi-docs,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-docs-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.5.0.war}
2018-05-29 06:58:05,547 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=103ms
2018-05-29 06:58:05,623 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath
2018-05-29 06:58:05,650 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@35ab0bf3{/,file:///home/cloudera/raja/nifi-1.5.0/work/jetty/nifi-web-error-1.5.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.5.0.war}
2018-05-29 06:58:05,795 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@239ce32{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-05-29 06:58:05,795 INFO [main] org.eclipse.jetty.server.Server Started @467767ms
2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer http://192.168.142.134:8080/nifi
2018-05-29 06:58:05,832 INFO [main] org.apache.nifi.web.server.JettyServer http://127.0.0.1:8080/nifi
2018-05-29 06:58:05,855 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2018-05-29 06:58:05,855 INFO [main] org.apache.nifi.NiFi Controller initialization took 116886957819 nanoseconds (116 seconds).
2018-05-29 06:58:36,124 WARN [NiFi Web Server-31] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi]
2018-05-29 06:58:36,126 WARN [NiFi Web Server-31] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi]
2018-05-29 06:58:36,300 WARN [NiFi Web Server-28] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico]
2018-05-29 06:58:36,301 WARN [NiFi Web Server-28] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico]
2018-05-29 06:58:36,308 WARN [NiFi Web Server-30] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [192.168.142.134:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico]
2018-05-29 06:58:36,310 WARN [NiFi Web Server-30] o.a.nifi.web.server.HostHeaderHandler Request host header [192.168.142.134:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico]
2018-05-29 06:58:52,020 WARN [NiFi Web Server-25] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi]
2018-05-29 06:58:52,023 WARN [NiFi Web Server-25] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi]
2018-05-29 06:58:52,153 WARN [NiFi Web Server-31] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico]
2018-05-29 06:58:52,153 WARN [NiFi Web Server-31] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico]
2018-05-29 06:58:52,179 WARN [NiFi Web Server-26] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/favicon.ico]
2018-05-29 06:58:52,180 WARN [NiFi Web Server-26] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico]
2018-05-29 06:58:58,281 WARN [NiFi Web Server-30] o.a.n.w.s.HostHeaderSanitizationCustomizer Request host header [127.0.0.1:8080] different from web hostname [(:8080)]. Overriding to [:8080/nifi]
2018-05-29 06:58:58,281 WARN [NiFi Web Server-30] o.a.nifi.web.server.HostHeaderHandler Request host header [127.0.0.1:8080] different from web hostname [localhost(:8080)]. Overriding to [localhost:8080/nifi]
2018-05-29 06:59:43,195 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@17d45cfb checkpointed with 7 Records and 0 Swap Files in 41 milliseconds (Stop-the-world time = 7 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID 20
... View more
Labels:
- Labels:
-
Apache NiFi
03-12-2018
05:30 AM
Its working ever since then. It's just the values in configurations of Kafka, setting. I remember , few years back , when I consumed from a contunuous file system as input and streamed to hdfs sink, the size configiguration has to be proper, else it will give some strange output.
... View more
02-14-2018
01:34 AM
My zookeeper, Kafka runs fine and tested both consumer and producer and data flows thru but when I configured in Flume as below, it does not sink to hdfs. Can anyone see where I miss the configurations? agent1.sources = kafka-source agent1.channels = memory-channel agent1.sinks = hdfs-sink agent1.sources.kafka-source.bootstrap.servers = localhost:9092 agent1.sources.kafka-source.batchSize = 1 agent1.sources.kafka-source.type = org.apache.flume.source.kafka.KafkaSource agent1.sources.kafka-source.zookeeperConnect = localhost:2181 agent1.sources.kafka-source.topic = Namaskar-vanakam-Hello agent1.sources.kafka-source.groupId = flume agent1.sources.kafka-source.channels = memory-channel agent1.sources.kafka-source.interceptors = i1 agent1.sources.kafka-source.interceptors.i1.type = timestamp agent1.sources.kafka-source.kafka.consumer.timeout.ms = 100 agent1.channels.memory-channel.type = memory agent1.channels.memory-channel.capacity = 10 agent1.channels.memory-channel.transactionCapacity = 10 agent1.sinks.hdfs-sink.type = hdfs agent1.sinks.hdfs-sink.hdfs.path = hdfs://quickstart.cloudera:8020/tmp/kafka/ agent1.sinks.hdfs-sink.hdfs.rollInterval = 0 agent1.sinks.hdfs-sink.hdfs.rollSize = 0 agent1.sinks.hdfs-sink.hdfs.rollCount = 1 agent1.sinks.hdfs-sink.hdfs.fileType = DataStream agent1.sinks.hdfs-sink.channel = memory-channel agent1.sinks.hdfs-sink.writeFormat = Text agent1.sinks.agent-sink.hdfs.idletimeout = 0 agent1.sinks.agent-sink.hdfs.batchSize = 1
... View more
02-14-2018
01:30 AM
I miss the configuration in Kafka streaming to hdfs via Flume.Any pointer? My zookeeper, Kafka runs fine and tested both consumer and producer and data flows thru but when I configured in Flume as below, it does not sink to hdfs. Can anyone see where I miss the configurations? agent1.sources = kafka-source agent1.channels = memory-channel agent1.sinks = hdfs-sink agent1.sources.kafka-source.bootstrap.servers = localhost:9092 agent1.sources.kafka-source.batchSize = 1 agent1.sources.kafka-source.type = org.apache.flume.source.kafka.KafkaSource agent1.sources.kafka-source.zookeeperConnect = localhost:2181 agent1.sources.kafka-source.topic = Namaskar-vanakam-Hello agent1.sources.kafka-source.groupId = flume agent1.sources.kafka-source.channels = memory-channel agent1.sources.kafka-source.interceptors = i1 agent1.sources.kafka-source.interceptors.i1.type = timestamp agent1.sources.kafka-source.kafka.consumer.timeout.ms = 100 agent1.channels.memory-channel.type = memory agent1.channels.memory-channel.capacity = 10 agent1.channels.memory-channel.transactionCapacity = 10 agent1.sinks.hdfs-sink.type = hdfs agent1.sinks.hdfs-sink.hdfs.path = hdfs://quickstart.cloudera:8020/tmp/kafka/ agent1.sinks.hdfs-sink.hdfs.rollInterval = 0 agent1.sinks.hdfs-sink.hdfs.rollSize = 0 agent1.sinks.hdfs-sink.hdfs.rollCount = 1 agent1.sinks.hdfs-sink.hdfs.fileType = DataStream agent1.sinks.hdfs-sink.channel = memory-channel agent1.sinks.hdfs-sink.writeFormat = Text agent1.sinks.agent-sink.hdfs.idletimeout = 0 agent1.sinks.agent-sink.hdfs.batchSize = 1
... View more
10-22-2017
09:28 AM
If I have 50 users. 10 users are updating and inserting the same base table simultaneusly and 40 users are just querying simultaneously, will there be any locking? How is the concurrency working? I do not want to turn on the transactional or ACID features. Please let me know. Thanks, Raja
... View more