Member since
12-09-2015
13
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6014 | 12-27-2017 02:38 PM |
09-12-2018
12:43 AM
@Akhil S Naik Thank you - running "cat id_rsa.pub >> authorized_keys" on Ambari Server did the trick
... View more
09-11-2018
02:13 PM
I've installed Ambari Server and followed all pre-requisite steps, When trying to create HDF 3.2 cluster via Ambari wizard, the Ambari Agent is installed and started but registration step fails: Creating target directory... ========================== Command start time 2018-09-10 23:54:57 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:58 ========================== Copying ambari sudo script... ========================== Command start time 2018-09-10 23:54:58 scp /var/lib/ambari-server/ambari-sudo.sh host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:58 ========================== Copying common functions script... ========================== Command start time 2018-09-10 23:54:58 scp /usr/lib/ambari-server/lib/ambari_commons host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:58 ========================== Copying create-python-wrap script... ========================== Command start time 2018-09-10 23:54:58 scp /var/lib/ambari-server/create-python-wrap.sh host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:59 ========================== Copying OS type check script... ========================== Command start time 2018-09-10 23:54:59 scp
/usr/lib/ambari-server/lib/ambari_server/os_check_type.py host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:59 ========================== Running create-python-wrap script... ========================== Command start time 2018-09-10 23:54:59 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:59 ========================== Running OS type check... ========================== Command start time 2018-09-10 23:54:59 Cluster primary/cluster OS family is redhat7 and
local/current OS family is redhat7 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:54:59 ========================== Checking 'sudo' package on remote host... ========================== Command start time 2018-09-10 23:54:59 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:00 ========================== Copying repo file to 'tmp' folder... ========================== Command start time 2018-09-10 23:55:00 scp /etc/yum.repos.d/ambari.repo host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:00 ========================== Moving file to repo dir... ========================== Command start time 2018-09-10 23:55:00 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:00 ========================== Changing permissions for ambari.repo... ========================== Command start time 2018-09-10 23:55:00 Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:01 ========================== Copying setup script file... ========================== Command start time 2018-09-10 23:55:01 scp /usr/lib/ambari-server/lib/ambari_server/setupAgent.py host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:01 ========================== Running setup agent script... ========================== Command start time 2018-09-10 23:55:01 ("WARNING 2018-09-10 23:55:35,248 shell.py:822 - can
not switch user for RUN_COMMAND. WARNING 2018-09-10 23:55:35,352 shell.py:822 - can not
switch user for RUN_COMMAND. INFO 2018-09-10 23:55:35,456 main.py:311 - Agent not going
to die gracefully, going to execute kill -9 WARNING 2018-09-10 23:55:35,456 shell.py:822 - can not
switch user for RUN_COMMAND. INFO 2018-09-10 23:55:35,460 main.py:322 - Agent stopped
successfully by kill -9, exiting. INFO 2018-09-10 23:55:35,460 ExitHelper.py:57 - Performing
cleanup before exiting... INFO 2018-09-10 23:55:35,461 AlertSchedulerHandler.py:159 -
[AlertScheduler] Stopped the alert scheduler. INFO 2018-09-10 23:55:35,461 AlertSchedulerHandler.py:159 -
[AlertScheduler] Stopped the alert scheduler. INFO 2018-09-10 23:55:35,740 main.py:155 -
loglevel=logging.INFO INFO 2018-09-10 23:55:35,742 Hardware.py:68 - Initializing
host system information. INFO 2018-09-10 23:55:35,746 Hardware.py:188 - Some mount
points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /run/user/1000,
/run/user/0 INFO 2018-09-10 23:55:35,762 Facter.py:202 - Directory:
'/etc/resource_overrides' does not exist - it won't be used for gathering
system resources. INFO 2018-09-10 23:55:35,765 Hardware.py:73 - Host system
information: {'kernel': 'Linux', 'domain': 'ec2.internal',
'physicalprocessorcount': 8, 'kernelrelease': '3.10.0-693.el7.x86_64',
'uptime_days': '0', 'memorytotal': 31962140, 'swapfree': '0.00 GB',
'memorysize': 31962140, 'osfamily': 'redhat', 'swapsize': '0.00 GB', 'processorcount':
8, 'netmask': '255.255.255.128', 'timezone': 'UTC', 'hardwareisa': 'x86_64',
'memoryfree': 31314048, 'operatingsystem': 'redhat', 'kernelmajversion':
'3.10', 'kernelversion': '3.10.0', 'macaddress': '0A:97:71:30:53:26',
'operatingsystemrelease': '7.4', 'ipaddress': '10.40.145.105', 'hostname':
'ip-10-40-145-105', 'uptime_hours': '0', 'fqdn':
'ip-10-40-145-105.ec2.internal', 'id': 'root', 'architecture': 'x86_64',
'selinux': True, 'mounts': [{'available': '19599720', 'used': '1359492', 'percent':
'7%', 'device': '/dev/nvme0n1p2', 'mountpoint': '/', 'type': 'xfs', 'size':
'20959212'}, {'available': '927944', 'used': '2564', 'percent': '1%', 'device':
'/dev/nvme3n1', 'mountpoint': '/db-repo', 'type': 'ext4', 'size': '999320'},
{'available': '24299724', 'used': '45080', 'percent': '1%', 'device':
'/dev/nvme2n1', 'mountpoint': '/provenance-repo', 'type': 'ext4', 'size':
'25671908'}, {'available': '48783816', 'used': '53272', 'percent': '1%',
'device': '/dev/nvme1n1', 'mountpoint': '/nifi-logs', 'type': 'ext4', 'size':
'51474912'}, {'available': '97760160', 'used': '61464', 'percent': '1%',
'device': '/dev/nvme4n1', 'mountpoint': '/content-repo', 'type': 'ext4',
'size': '103080888'}, {'available': '48783816', 'used': '53272', 'percent':
'1%', 'device': '/dev/nvme5n1', 'mountpoint': '/flowfile-repo', 'type': 'ext4',
'size': '51474912'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '1176',
'interfaces': 'eth0,lo'} INFO 2018-09-10 23:55:35,767 DataCleaner.py:39 - Data
cleanup thread started INFO 2018-09-10 23:55:35,768 DataCleaner.py:120 - Data
cleanup started INFO 2018-09-10 23:55:35,768 DataCleaner.py:122 - Data
cleanup finished INFO 2018-09-10 23:55:35,798 hostname.py:67 -
agent:hostname_script configuration not defined thus read hostname 'ip-10-40-145-105.ec2.internal'
using socket.getfqdn(). INFO 2018-09-10 23:55:35,803 PingPortListener.py:50 - Ping
port listener started on port: 8670 INFO 2018-09-10 23:55:35,805 main.py:481 - Connecting to
Ambari server at https://ip-10-40-145-25.ec2.internal:8440 (10.40.145.25) INFO 2018-09-10 23:55:35,806 NetUtil.py:61 - Connecting to
https://ip-10-40-145-25.ec2.internal:8440/ca ", None) ("WARNING 2018-09-10 23:55:35,248 shell.py:822 - can
not switch user for RUN_COMMAND. WARNING 2018-09-10 23:55:35,352 shell.py:822 - can not
switch user for RUN_COMMAND. INFO 2018-09-10 23:55:35,456 main.py:311 - Agent not going
to die gracefully, going to execute kill -9 WARNING 2018-09-10 23:55:35,456 shell.py:822 - can not
switch user for RUN_COMMAND. INFO 2018-09-10 23:55:35,460 main.py:322 - Agent stopped
successfully by kill -9, exiting. INFO 2018-09-10 23:55:35,460 ExitHelper.py:57 - Performing
cleanup before exiting... INFO 2018-09-10 23:55:35,461 AlertSchedulerHandler.py:159 -
[AlertScheduler] Stopped the alert scheduler. INFO 2018-09-10 23:55:35,461 AlertSchedulerHandler.py:159 -
[AlertScheduler] Stopped the alert scheduler. INFO 2018-09-10 23:55:35,740 main.py:155 -
loglevel=logging.INFO INFO 2018-09-10 23:55:35,742 Hardware.py:68 - Initializing
host system information. INFO 2018-09-10 23:55:35,746 Hardware.py:188 - Some mount
points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /run/user/1000,
/run/user/0 INFO 2018-09-10 23:55:35,762 Facter.py:202 - Directory:
'/etc/resource_overrides' does not exist - it won't be used for gathering
system resources. INFO 2018-09-10 23:55:35,765 Hardware.py:73 - Host system
information: {'kernel': 'Linux', 'domain': 'ec2.internal',
'physicalprocessorcount': 8, 'kernelrelease': '3.10.0-693.el7.x86_64',
'uptime_days': '0', 'memorytotal': 31962140, 'swapfree': '0.00 GB',
'memorysize': 31962140, 'osfamily': 'redhat', 'swapsize': '0.00 GB',
'processorcount': 8, 'netmask': '255.255.255.128', 'timezone': 'UTC',
'hardwareisa': 'x86_64', 'memoryfree': 31314048, 'operatingsystem': 'redhat',
'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress':
'0A:97:71:30:53:26', 'operatingsystemrelease': '7.4', 'ipaddress':
'10.40.145.105', 'hostname': 'ip-10-40-145-105', 'uptime_hours': '0', 'fqdn':
'ip-10-40-145-105.ec2.internal', 'id': 'root', 'architecture': 'x86_64',
'selinux': True, 'mounts': [{'available': '19599720', 'used': '1359492',
'percent': '7%', 'device': '/dev/nvme0n1p2', 'mountpoint': '/', 'type': 'xfs',
'size': '20959212'}, {'available': '927944', 'used': '2564', 'percent': '1%',
'device': '/dev/nvme3n1', 'mountpoint': '/db-repo', 'type': 'ext4', 'size':
'999320'}, {'available': '24299724', 'used': '45080', 'percent': '1%',
'device': '/dev/nvme2n1', 'mountpoint': '/provenance-repo', 'type': 'ext4',
'size': '25671908'}, {'available': '48783816', 'used': '53272', 'percent':
'1%', 'device': '/dev/nvme1n1', 'mountpoint': '/nifi-logs', 'type': 'ext4',
'size': '51474912'}, {'available': '97760160', 'used': '61464', 'percent':
'1%', 'device': '/dev/nvme4n1', 'mountpoint': '/content-repo', 'type': 'ext4',
'size': '103080888'}, {'available': '48783816', 'used': '53272', 'percent':
'1%', 'device': '/dev/nvme5n1', 'mountpoint': '/flowfile-repo', 'type': 'ext4',
'size': '51474912'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '1176',
'interfaces': 'eth0,lo'} INFO 2018-09-10 23:55:35,767 DataCleaner.py:39 - Data
cleanup thread started INFO 2018-09-10 23:55:35,768 DataCleaner.py:120 - Data
cleanup started INFO 2018-09-10 23:55:35,768 DataCleaner.py:122 - Data
cleanup finished INFO 2018-09-10 23:55:35,798 hostname.py:67 -
agent:hostname_script configuration not defined thus read hostname
'ip-10-40-145-105.ec2.internal' using socket.getfqdn(). INFO 2018-09-10 23:55:35,803 PingPortListener.py:50 - Ping
port listener started on port: 8670 INFO 2018-09-10 23:55:35,805 main.py:481 - Connecting to
Ambari server at https://ip-10-40-145-25.ec2.internal:8440 (10.40.145.25) INFO 2018-09-10 23:55:35,806 NetUtil.py:61 - Connecting to
https://ip-10-40-145-25.ec2.internal:8440/ca ", None) Connection to ip-10-40-145-105.ec2.internal closed. SSH command execution finished host=ip-10-40-145-105.ec2.internal, exitcode=0 Command end time 2018-09-10 23:55:38 Registering with the server... Registration with the server failed.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Cloudera DataFlow (CDF)
05-15-2018
11:10 AM
Is there a simple way to convert a date string such as "Mar 16 2018 19:18:15 GMT" into date in NiFi?
... View more
Labels:
- Labels:
-
Apache NiFi
02-08-2018
09:31 PM
I am trying to setup a connection to SQL Server database in NiFi using Microsoft's JDBC driver, but getting connection errors. My NiFi cluster runs on RHEL image inside AWS. Same db connection URL, ran on my laptop using db query tools, can connect to the database. Below is the error message, and DBCPConnectionPool properties screenshot is attached. Any suggestions? 2018-02-07 17:43:34,730 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=536ab6b9-0161-1000-f96a-08612add1288] Error during processing: Cannot create PoolableConnectionFactory (Login failed for user 'glo-securonix-sql-svc'. ClientConnectionId:07a7414e-798d-4a88-9f77-3f551673a6aa): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'glo-securonix-sql-svc'. ClientConnectionId:07a7414e-798d-4a88-9f77-3f551673a6aa)
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'glo-securonix-sql-svc'. ClientConnectionId:07a7414e-798d-4a88-9f77-3f551673a6aa)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy79.getConnection(Unknown Source)
at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:270)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'glo-securonix-sql-svc'. ClientConnectionId:07a7414e-798d-4a88-9f77-3f551673a6aa
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:256)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:108)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4290)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3157)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$100(SQLServerConnection.java:82)
at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3121)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2026)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1687)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1528)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:866)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:569)
at org.apache.nifi.dbcp.DriverShim.connect(DriverShim.java:46)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 21 common frames omitted Thank you, Alex
... View more
Labels:
- Labels:
-
Apache NiFi
12-27-2017
02:38 PM
1 Kudo
@Shu Thank you - after some tweaking and tuning of the parameters you mentioned I was able to achieve desired results. Alex
... View more
12-27-2017
01:38 AM
1 Kudo
Attached is the template. Thanks, Alex
... View more
12-26-2017
05:57 PM
1 Kudo
I need to extract data from a relational database and load it into S3 bucket. I have a 5 node cluster, and use "GenerateTableFetch" (Primary node) --> "ExecuteSQL" (All nodes) combination to read the data in parallel. I also need to merge extracted data into a single file before loading it into S3, but my "MergeContent" processor produces multiple files in S3. Is there a way to get this done? The full flow looks like this: "GenerateTableFetch" --> "ExecuteSQL" --> "MergeConent" --> "ConvertAvroToJSON" --> "UpdateAttribute" --> "CompressContent" --> "PutS3Object"
... View more