Member since
10-20-2016
28
Posts
9
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1257 | 07-13-2017 12:47 PM | |
1693 | 06-30-2017 01:37 PM | |
1941 | 06-30-2017 05:18 AM | |
677 | 06-29-2017 03:15 PM | |
1471 | 06-23-2017 01:51 PM |
02-20-2018
11:50 AM
1 Kudo
@Ravikiran Dasari Have you tried looking at NiFi and its capabilities. NiFi provides a lot of processors which can help you automate your tasks and to create a flow for performing those tasks. You can create a flow to pickup data from a source and dump it in a different location. You can check the following example written by one of our NiFi experts @Matt Clarke on - How you can use NiFi to pull data from an FTP server. How-to: Retrieve files from a SFTP server using NiFi (GetSFTP vs. ListSFTP) Along with the processors mentioned in the article above, you can use PutHDFS processors explained in below docs to dump the data in HDFS. PutHDFS - NiFi docs Hope this helps
... View more
07-13-2017
12:47 PM
@Mehedi Hasan SASL_SSL is not available in the older kafka consumer api. The new kafka consumer api integration is in Flume v.1.7.0 as noted here: https://issues.apache.org/jira/browse/FLUME-2821. What is the version of Flume you are using?
... View more
07-06-2017
09:26 AM
@siva karna It appears that your processor is configured to run only on 'primary node'. Please check the scheduling tab in the processor configuration, and if it is configured to run only on 'Primary Node' configure it to run on 'All nodes'. Check screenshot.
... View more
07-04-2017
07:12 PM
@Simran Kaur I believe currently oozie does not support the functionality of tagging jobs with prioirities Check the following links for more details Oozie Workflow Functional Spec OOZIE-2892
... View more
07-04-2017
06:28 AM
@Simran Kaur Oozie does not handle workflow jobs priority. As soon as a workflow job is ready to do a transition, Oozie will trigger the transition. Workflow transitions and action triggering are assumed to be fast and lightweight operations.
In the capacity scheduler you could set a high priority queue providing the maximum resources of the cluster with extension( max. Capacity) to 100%. And while running the oozie workflow, you can provide the queue name in Job.properties and workflow.xml so that the job is always submitted to a high priority queue. In Job.Properties File queueName=<queue-name> Use the queueName in workflow.xml <configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
... View more
07-04-2017
05:16 AM
@Adda FuentesAwesome. Good to hear. You can mark your answer as "Accepted" so that if someone faces this issue in future, the can try and debug around the same lines.
... View more
07-03-2017
05:14 AM
@Adda Fuentes What is the kafka version on the other end? Please add a screenshot of your configuration of PutKafka, Jaas.conf file and the entry you have added in bootstrap.conf
... View more
06-30-2017
03:09 PM
@Bharadwaj Bhimavarapu There are no attachments to the question. Can you please check again? Do you see any error message on the processor or in the bulletin board, or in nifi-app.log?
... View more
06-30-2017
03:00 PM
2 Kudos
PROBLEM DESCRIPTION: Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully.
The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari. # Flume agent config
agent1.sinks = HdfsSink1
agent1.channels = channel1
agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannel
agent1.channels.channel1.brokerList=node11.openstacklocal:6667
agent1.channels.channel1.kafka.topic=test
agent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181
agent1.channels.channel1.capacity=10000
agent1.channels.channel1.transactionCapacity=1000
agent1.channels.channel1.parseAsFlumeEvent=false
agent1.channels.channel1.kafka.consumer.group.id=test.hdfs-c
agent1.sinks.HdfsSink1.channel=channel1
agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000
agent1.sinks.HdfsSink1.hdfs.batchSize=1000
agent1.sinks.HdfsSink1.hdfs.callTimeout=10000
agent1.sinks.HdfsSink1.hdfs.filePrefix=xrs-SegmentEventData
agent1.sinks.HdfsSink1.hdfs.fileSuffix=.avro
agent1.sinks.HdfsSink1.hdfs.fileType=DataStream
agent1.sinks.HdfsSink1.hdfs.maxOpenFiles=50
##agent1.sinks.HdfsSink1.hdfs.path=/data/%{topic}/%y-%m-%d
agent1.sinks.HdfsSink1.hdfs.path=/tmp/%y-%m-%d
agent1.sinks.HdfsSink1.hdfs.rollCount=1000
agent1.sinks.HdfsSink1.hdfs.rollInterval=60
agent1.sinks.HdfsSink1.hdfs.rollSize=0
agent1.sinks.HdfsSink1.hdfs.rollTimerPoolSize=1
agent1.sinks.HdfsSink1.hdfs.threadsPoolSize=100
agent1.sinks.HdfsSink1.hdfs.txnEventMax=40000
agent1.sinks.HdfsSink1.hdfs.useLocalTimeStamp=true
agent1.sinks.HdfsSink1.hdfs.writeFormat=Text
agent1.sinks.HdfsSink1.type=hdfs Ambari logs for service startup shows that command ran successfully. [..]
2016-11-09 09:25:11,411 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'),
'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-11-09 09:25:11,715 - File['/var/run/flume/ambari-state.txt'] {'content': 'INSTALLED'}
2016-11-09 09:25:11,719 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
2016-11-09 09:25:11,723 - Directory['/var/run/flume'] {'owner': 'flume', 'group': 'hadoop'}
2016-11-09 09:25:11,723 - Directory['/usr/hdp/current/flume-server/conf'] {'owner': 'flume', 'create_parents': True}
2016-11-09 09:25:11,724 - Directory['/var/log/flume'] {'owner': 'flume', 'group': 'hadoop', 'create_parents': True,
'mode': 0755, 'cd_access': 'a'}
2016-11-09 09:25:11,726 - File['/var/run/flume/ambari-state.txt'] {'content': 'STARTED'}
2016-11-09 09:25:11,726 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
Command completed successfully! ROOT CAUSE: This issue occurs because the flume.py script is designed to checks for the agent name by parsing the line containing "sources" definition in flume.conf. And it ignores the other parameters like channels and sinks. # vim /var/lib/ambari-server/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py
[..]
def build_flume_topology(content):
result = {}
agent_names = []
for line in content.split('\n'):
rline = line.strip()
if 0 != len(rline) and not rline.startswith('#'):
pair = rline.split('=')
lhs = pair[0].strip()
# workaround for properties that contain '='
rhs = "=".join(pair[1:]).strip()
part0 = lhs.split('.')[0]
if lhs.endswith(".sources"):
agent_names.append(part0)
if not result.has_key(part0):
result[part0] = {}
result[part0][lhs] = rhs
# trim out non-agents
for k in result.keys():
if not k in agent_names:
del result[k]
return result
[..] SOLUTION Add a dummy source definition in the starting of the flume.conf. This will ensure that Ambari detects the flume agent and adds the same in the array of flume agents.
Sample source definition: # Flume agent config
agent1.sources = dummysource
agent1.sinks = HdfsSink1
agent1.channels = channel1
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- Flume
- Issue Resolution
Labels:
06-30-2017
02:57 PM
PROBLEM DESCRIPTION: Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully.
The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari. # Flume agent config
agent1.sinks = HdfsSink1
agent1.channels = channel1
agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannel
agent1.channels.channel1.brokerList=node11.openstacklocal:6667
agent1.channels.channel1.kafka.topic=test
agent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181
agent1.channels.channel1.capacity=10000
agent1.channels.channel1.transactionCapacity=1000
agent1.channels.channel1.parseAsFlumeEvent=false
agent1.channels.channel1.kafka.consumer.group.id=test.hdfs-c
agent1.sinks.HdfsSink1.channel=channel1
agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000
agent1.sinks.HdfsSink1.hdfs.batchSize=1000
agent1.sinks.HdfsSink1.hdfs.callTimeout=10000
agent1.sinks.HdfsSink1.hdfs.filePrefix=xrs-SegmentEventData
agent1.sinks.HdfsSink1.hdfs.fileSuffix=.avro
agent1.sinks.HdfsSink1.hdfs.fileType=DataStream
agent1.sinks.HdfsSink1.hdfs.maxOpenFiles=50
##agent1.sinks.HdfsSink1.hdfs.path=/data/%{topic}/%y-%m-%d
agent1.sinks.HdfsSink1.hdfs.path=/tmp/%y-%m-%d
agent1.sinks.HdfsSink1.hdfs.rollCount=1000
agent1.sinks.HdfsSink1.hdfs.rollInterval=60
agent1.sinks.HdfsSink1.hdfs.rollSize=0
agent1.sinks.HdfsSink1.hdfs.rollTimerPoolSize=1
agent1.sinks.HdfsSink1.hdfs.threadsPoolSize=100
agent1.sinks.HdfsSink1.hdfs.txnEventMax=40000
agent1.sinks.HdfsSink1.hdfs.useLocalTimeStamp=true
agent1.sinks.HdfsSink1.hdfs.writeFormat=Text
agent1.sinks.HdfsSink1.type=hdfs Ambari logs for service startup shows that command ran successfully [..]
2016-11-09 09:25:11,411 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'),
'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-11-09 09:25:11,715 - File['/var/run/flume/ambari-state.txt'] {'content': 'INSTALLED'}
2016-11-09 09:25:11,719 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
2016-11-09 09:25:11,723 - Directory['/var/run/flume'] {'owner': 'flume', 'group': 'hadoop'}
2016-11-09 09:25:11,723 - Directory['/usr/hdp/current/flume-server/conf'] {'owner': 'flume', 'create_parents': True}
2016-11-09 09:25:11,724 - Directory['/var/log/flume'] {'owner': 'flume', 'group': 'hadoop', 'create_parents': True,
'mode': 0755, 'cd_access': 'a'}
2016-11-09 09:25:11,726 - File['/var/run/flume/ambari-state.txt'] {'content': 'STARTED'}
2016-11-09 09:25:11,726 - Writing File['/var/run/flume/ambari-state.txt'] because contents don't match
Command completed successfully! ROOT CAUSE: This issue occurs because the flume.py script is designed to checks for the agent name by parsing the line containing "sources" definition in flume.conf. And it ignores the other parameters like channels and sinks. # vim /var/lib/ambari-server/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py
[..]
def build_flume_topology(content):
result = {}
agent_names = []
for line in content.split('\n'):
rline = line.strip()
if 0 != len(rline) and not rline.startswith('#'):
pair = rline.split('=')
lhs = pair[0].strip()
# workaround for properties that contain '='
rhs = "=".join(pair[1:]).strip()
part0 = lhs.split('.')[0]
if lhs.endswith(".sources"):
agent_names.append(part0)
if not result.has_key(part0):
result[part0] = {}
result[part0][lhs] = rhs
# trim out non-agents
for k in result.keys():
if not k in agent_names:
del result[k]
return result
[..] SOLUTION/WORKAROUND Add a dummy source definition in the starting of the flume.conf. This will ensure that Ambari detects the flume agent and adds the same in the array of flume agents.
Sample source definition: # Flume agent config
agent1.sources = dummysource
agent1.sinks = HdfsSink1
agent1.channels = channel1
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- Flume
- Issue Resolution
Labels:
06-30-2017
01:37 PM
1 Kudo
@rpulluru This issue occurs when one of the following is true: If a file is written to after being placed into the spooling directory, Flume will print an error to its log file and stop processing. If a file name is reused at a later time, Flume will print an error to its log file and stop processing. If you are copying the files in your /data/src/input directory, change the operation to ‘mv’, Or you can copy the files as .tmp and then 'mv' the '.tmp' file to the same spooling directory with the actual name. Add the following line in flume.conf to ignore .tmp files in SpoolDir: Agent1.sources.spooldir-source.ignorePattern=^.*\.tmp$
... View more
06-30-2017
05:18 AM
1 Kudo
@mliem Did you copy the flow.xml.gz from your old installation to this one after you wiped everything 2.x related and installed 3. All the sensitive properties inside the flow.xml.gz file are encrypted using the sensitive property defined in the nifi.properties file (If blank, NiFi uses an internal default value). If you move your flow.xml.gz file to another NiFi, the sensitive property value used must be the same or NiFi will fail to start because it cannot decrypt the sensitive properties in the file.
... View more
06-29-2017
07:12 PM
@Sandeep Nemuri Awesome. That worked. Thanks.
... View more
06-29-2017
07:03 PM
@Sandeep Nemuri Here are the properties from my cluster. # Generated by Apache Ambari. Mon Jun 12 12:33:51 2017
atlas.authentication.method.kerberos=False
atlas.cluster.name=mycluster
atlas.jaas.KafkaClient.option.renewTicket=true.
atlas.jaas.KafkaClient.option.useTicketCache=true
atlas.kafka.bootstrap.servers=lab1.hwx.com:6667
atlas.kafka.hook.group.id=atlas
atlas.kafka.zookeeper.connect=lab1.hwx.com:2181
atlas.kafka.zookeeper.connection.timeout.ms=30000
atlas.kafka.zookeeper.session.timeout.ms=60000
atlas.kafka.zookeeper.sync.time.ms=20
atlas.notification.create.topics=True
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.rest.address=http://lab1.hwx.com:21000
... View more
06-29-2017
06:53 PM
While running the sqoop command I get the following error message ERROR security.InMemoryJAASConfiguration: Unable to add JAAS configuration for client [KafkaClient] as it is missing param [atlas.jaas.KafkaClient.loginModuleName]. Skipping JAAS config for [KafkaClient] Any pointers are appreciated.
... View more
Labels:
- Labels:
-
Apache Sqoop
06-29-2017
03:15 PM
1 Kudo
The error message points that the permissions on worker-launcher binary are not correct. Please check that the permissions on worker-launcher binary. It should be owned by root:hadoop or the group parameter configured in worker-launcher.cfg. From the log message it appears that the group ownership is root.
Also note that the permissions on the binary should be 6550 else it would fail again after you change the group ownership. Here is the output of the ownership from my test system. # stat /usr/hdp/2.5.0.0-1133/storm/bin/worker-launcher
File: `/usr/hdp/2.5.0.0-1133/storm/bin/worker-launcher'
Size: 56848 Blocks: 112 IO Block: 4096 regular file
Device: fc01h/64513d Inode: 1444319 Links: 1
Access: (6550/-r-sr-s---) Uid: ( 0/ root) Gid: ( 501/ hadoop)
Access: 2016-08-03 13:25:37.000000000 +0000
Modify: 2016-08-03 13:25:37.000000000 +0000
Change: 2016-11-19 13:23:02.764000118 +0000
... View more
06-27-2017
09:26 AM
@mayki wogno What is the nifi version and can you please upload a screenshot of the ExecuteScripct processor config
... View more
06-26-2017
02:36 PM
@Raj B Please check the timestamps of the files remaining in the directory. If they are being added during the process run time. Or if the timestamp is older than the CRON runtime of the processor.
... View more
06-26-2017
10:08 AM
1 Kudo
@Raj B This looks similar to NIFI-4069 As a workaround, please try and change the cron schedule to 0,30 30 0 * *. so that it runs twice in the same minute. Let us know if that helps.
... View more
06-26-2017
09:57 AM
@mayki wogno This looks like a known issue I have seen earlier. Could you please try the following workaround.
Copy hive-site.xml, core-site.xml, hdfs-site.xml to the conf directory of NiFi Clear the Hive Configuration Resources property in PutHiveStreaming processor. Create an ExecuteScript processor on the canvas scheduled to run as often as the ticket needs to be refreshed (every hour should do for most setups) with the following groovy script, replacing nifi@HDF.COM with your principal and /etc/nifi.headless.keytab with your keytab import org.apache.nifi.nar.NarClassLoader
import org.apache.nifi.nar.NarClassLoaders
NarClassLoaders.instance.extensionClassLoaders.each { c ->
if (c instanceof NarClassLoader && c.workingDirectory.absolutePath.contains('nifi-hive')) {
def originalClassloader = Thread.currentThread().getContextClassLoader();
Thread.currentThread().setContextClassLoader(c);
try {
def configClass = c.loadClass('org.apache.hadoop.conf.Configuration', true)
def hiveConfigurator = c.loadClass('org.apache.nifi.util.hive.HiveConfigurator', true).newInstance();
def config = hiveConfigurator.getConfigurationFromFiles('')
hiveConfigurator.preload(config)
c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('setConfiguration', configClass).invoke(null, config)
c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('loginUserFromKeytab', String.class, String.class).invoke(null, 'nifi@HDF.COM', '/etc/nifi.headless.keytab')
log.info('Successfully logged in')
session.transfer(session.create(), REL_SUCCESS)
} catch (Exception e) {
log.error('Unable to login with keytab', e)
session.transfer(session.create(), REL_FAILURE)
} finally {
Thread.currentThread().setContextClassLoader(originalClassloader);
}
}
}
... View more
06-23-2017
01:51 PM
1 Kudo
@AViradia I suspect that Windows locking mechanism restricts the original file to be renamed(Check http://dev.eclipse.org/mhonarc/lists/jetty-users/msg03222.html for details). You can omit the file property in logback.xml, and then the active log file will be computed a new for each period based on the value of fileNamePattern.
A working rollingpolicy for a NiFi node on Windows is as follows: <appender name="APP_FILE">
<!-- <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> -->
<rollingPolicy>
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy>
<maxFileSize>100MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
Ref: https://logback.qos.ch/manual/appenders.html
... View more
06-23-2017
10:59 AM
1 Kudo
@subash sharma The error messages you are seeing shows some issues with ssh configurations.
error : "host key verification failed" This error message shows that the ssh client is not able to verify 'edge-node-hostname'. It is possible that the entry for 'edge-node-hostname' is not available in the known_hosts file located in the home directory of the user you are running NiFi as. If you are running NiFi as root, check the known_hosts file in /root/.ssh/known_hosts. And if NiFi is running as a non-root user, check the '.ssh/known_hosts' files in the home directory of that user. You can also run the following command which would then ask you for a confirmation in yes/no input format to add the entry of the 'edge-node-hostname' in the known_hosts file. sudo -u <NiFi_user> ssh username@edge-node-hostname
error : "Pseudo-Terminal will not be allocated because stdin is not a terminal" This error message is because you are just trying to obtain a remote shell in the executeProcess processor which is not possible. You can either add a command to be run after ssh succeeds or you can add -t -t arguments to ssh command which should help. ssh username@edge-node-hostname date
OR ssh -t -t username@edge-node-hostname
... View more
06-10-2017
03:48 PM
@JT Ng Check if you have oozie sharelib updated. Also, try adding --hive-home in sqoop <arg> <arg>--hive-home</arg>
<arg>path/to/hive_home</arg>
... View more
06-10-2017
02:52 PM
@Anishkumar Valsalam Have you added the policy to grant access for /proxy to all the NiFi Nodes? Check the audit tab in ranger to understand for what resource the access was forbidden. You can also check the following resources to integrate NiFi with Ranger. https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.html https://community.hortonworks.com/articles/57980/hdf-20-apache-nifi-integration-with-apache-ambarir.html http://bryanbende.com/development/2016/08/22/apache-nifi-1.0.0-using-the-apache-ranger-authorizer
... View more
06-08-2017
07:00 PM
@Karan Alang If you are using the new Kafka Consumer, try bootstrap.servers and see if that helps. Check https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html for more details.
... View more
06-08-2017
02:18 PM
@Anishkumar Valsalam You would need to do identity mappings for the users. Set the following parameters in your NiFi conf and restart NiFi. nifi.security.identity.mapping.pattern.dn =^CN=(.*?), OU=(.*?), OU=(.*?), OU=(.*?), DC=(.*?), DC=(.*?), DC=(.*?)$
nifi.security.identity.mapping.value.dn = $1 You can read following Knowledge articles for more details. https://community.hortonworks.com/articles/81184/understanding-the-initial-admin-identity-access-po.html
... View more
06-08-2017
02:18 PM
@Anishkumar Valsalam You would need to do identity mappings for the users.
Set the following parameters in your NiFi conf and restart NiFi. nifi.security.identity.mapping.pattern.dn = ^CN=(.*?), OU=(.*?), OU=(.*?), OU=(.*?), DC=(.*?), DC=(.*?), DC=(.*?)$
nifi.security.identity.mapping.value.dn = $1
You can read following Knowledge articles for more details. https://community.hortonworks.com/articles/81184/understanding-the-initial-admin-identity-access-po.html
... View more
03-19-2017
12:37 PM
@Micaël Dias Error messages in Controller.log appear to be from a different timestamp as compared to the timestamp of your producers and consumers. Even if we ignore the mismatched timestamp, I see Controller1003's connection to broker kafka.host.com:6667 (id: 1003 rack: null) was unsuccessful
so we should check server.log for the same timestamp. Please perform following steps, which might help in debugging the issue. Also, if you do not see any directories created for the topic and partition, that might be an issue. - Restart brokers. - Keep monitoring server.log and controller.log for any error messages. - Create a topic, while monitoring the logs and see if the topic and partitions are created successfully. Additionally check state-change.log - If there are no errors, try producing and consuming now while monitoring the logs.
... View more