Hi! I need to execute following awk command on flow file via ExecuteStreamCommand processor awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' When run it throws an error: Executable command awk ended in an error: awk: fatal: cannot open file `print $0}" for reading (No such file or directory). Flow file comes from ListFTP -> FetchFTP -> ExecuteStreamCommand Could you please help to solve this issue!! Thank you!! When running the same awk command on server it works fine: awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' myfile.csv ... View more

Hi! How can I configure InvokeHTTP to return comma separated flow file? At the moment it is returning file in below format (tab separated): RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772103971 RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772104117 RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772101000 Thanks in advance! ... View more

Hi, all! Can anyone give any suggestion in configuring new Ambari server with an existing HDF cluster Previos version of ambari fails in ambari-server start after downgrade [root@bi-nifi1 dmp]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.............ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. [root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 An unexpected error occured during starting Ambari Server. org.apache.ambari.server.AmbariException: An error occured during updating current repository versions with stack repositories. at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:99) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1061) Caused by: java.lang.NullPointerException at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.updateRepoVersion(UpdateActiveRepoVersionOnStartup.java:106) at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:92) ... 2 more Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1075) [root@bi-nifi1 dmp]# ... View more

Hi!! I need to run a Linux command on list of files received from ListSFTP (most probably it should be run on remote FTP server) and pass forward the result of this operation as flow file to putHDFS. Linux command is: cat ${LIST} | xargs -n 1 -I {} stdbuf -o 5GB awk -F'\x1b' '{OFS="|"};{print $1,$2,FILENAME}' {} Can you please advise which processor can be most suitable. Thanks in advance! ... View more

Hi, dear Experts! Could you please help with following issue: I have a processor ExtractText that processes JSON flow file and creates two attributes with large JSON text. mainJSON = ^.*(?=\x1b)|^((?!\x1b).)*$) enrichmentJSON = ((?<=\x1b).*) As an output there were created 3 attributes for mainJSON and enrichmentJSON mainJSON: mainJSON, mainJSON.0, mainJSON.1 --- each containing the same portion of expected result. Isn't it possible to store large text value in an attribute? Is there another way to store and pass large text value as an attribute? As a next step I wanted to combine these two attributes with other attributes in ReplaceText processor and put them into hive table as separate columns of one row. Thanks in advance! ... View more

Hi! My Flow file contains more then one JSON objects { Nested JSON object 1 }{ Nested JSON object 2 }{ Nested JSON object N } Can you please help to split JSON flow file into separate JSON objects using Nifi processors. What should be the JsonPath Expression in SplitJSON processor? thanks in advance! ... View more

Hi! Can you please help to permanently remove a disk from all the datanodes in working cluster. DataNode directories /grid/0,/grid/2,/grid/3,/grid/4,/grid/5,/grid/6,/grid/7,/grid/8,/grid/9,/grid/10,/grid/11,/grid/12,/grid/14,/grid/15 I would like to remove /grid/15. Thanks in advance!! with regards, Gulshan ... View more

Hi, dear Experts! Could you please help with the following issue? Ranger Audit Access tab is failing with the following error: Error Error running solr query, please check solr configs. Could not find a healthy node to handle the request. in /hadoop/log/ambari-infra-solr/solr.log : ERROR [c:audit_logs s:shard0 r:core_node2 x:audit_logs_shard0_replica1] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Invalid Date String:'2017-09-05T02017 with regards, Gulshan ... View more

Hi, dear experts! I'm trying to upgrade my HDP from 2.6.0.3 to 2.6.4 using Ambari. Ambari upgrade was completed successfuly, but while registering and installing of target version process fails with the following error: 2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages retry_count=agent_stack_retry_count File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package return self.install_package(name, use_repos, skip_repos, is_upgrade) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 473, in <module> InstallPackages().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 153, in actionexecute raise Fail("Failed to distribute repositories/install packages") resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages stdout: /var/lib/ambari-agent/data/output-8438.txt 2018-02-14 22:56:17,861 - Installing package tez_hive2_2_6_4_0_91 ('/usr/bin/yum -d 0 -e 0 -y install tez_hive2_2_6_4_0_91') 2018-02-14 22:56:33,825 - No package found for slider_${stack_version}(slider_(\d|_)+$) 2018-02-14 22:56:33,826 - Package[None] {'retry_on_repo_unavailability': False, 'retry_count': 5, 'action': ['upgrade']} 2018-02-14 22:56:33,826 - Installing package None ('/usr/bin/yum -d 0 -e 0 -y install ''') 2018-02-14 22:56:49,179 - Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do 2018-02-14 22:56:49,179 - Failed to install package None. Executing '/usr/bin/yum clean metadata' 2018-02-14 22:56:49,422 - Retrying to install package None after 30 seconds 2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages retry_count=agent_stack_retry_count File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package return self.install_package(name, use_repos, skip_repos, is_upgrade) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do 2018-02-14 22:57:53,898 - Installation of packages failed. Checking if installation was partially complete 2018-02-14 22:57:53,899 - Old versions: ['2.6.0.3-8', '2.6.4.0-91'] 2018-02-14 22:57:53,899 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {} 2018-02-14 22:57:53,925 - call returned (0, '2.6.0.3-8\n2.6.4.0-91') 2018-02-14 22:57:53,925 - New versions: ['2.6.0.3-8', '2.6.4.0-91'] 2018-02-14 22:57:53,925 - Deltas: set([]) Command failed after 1 tries ... View more