Member since
08-16-2017
39
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2112 | 07-12-2018 05:49 AM |
06-20-2019
11:20 AM
Hi! I need to execute following awk command on flow file via ExecuteStreamCommand processor awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' When run it throws an error: Executable command awk ended in an error: awk: fatal: cannot open file `print $0}" for reading (No such file or directory). Flow file comes from ListFTP -> FetchFTP -> ExecuteStreamCommand Could you please help to solve this issue!! Thank you!! When running the same awk command on server it works fine: awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' myfile.csv
... View more
Labels:
- Labels:
-
Apache NiFi
06-13-2019
09:05 AM
Hi! How can I configure InvokeHTTP to return comma separated flow file? At the moment it is returning file in below format (tab separated): RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772103971 RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772104117 RELM Satieva_microloans 6 3369 2019-06-12 11:42:00 772101000 Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
05-03-2019
05:33 AM
Hi! Could you please help with converting CSV file containing JSON values into JSON: 1|portout|Donor Reject Accepted|719dfa55b8dd42be8f64d55ddee0c805|10000001518591|2018-09-26 22:00:00|2018-09-25 17:18:53|0|89011122244|Satish Chippa|200003530|1|{"xxxxxxx98009":[{"subscription_state":{"status":"Terminated","reason_code":4001}},{"subscription_early_portout":{"status":"Early Portout","reason_code":4005}}]}||2|
2|portout|Donor Reject Accepted|554a2faa9bb548dc848915e95938fe22|10000001518592|2018-09-26 22:00:00|2018-09-25 17:30:25|0|67811122233|Satish Chippa|200003528|1|{"xxxxxxx38009":[{"subscription_state":{"status":"Terminated","reason_code":4001}}]}||2|
3|portout|Donor Reject Accepted|3732aa2495ce4f37844716848e88c989|10000001518596|2018-09-28 04:00:00|2018-09-26 17:31:09|0|99001990981|Aman PortOut|200003523|2|{"xxxxxxx69009":[{"subscription_state":{"status":"Terminated","reason_code":4001}},{"subscription_early_portout":{"status":"Early Portout","reason_code":4005}},{"customer_open_invoices":{"status":"Customer has open invoices","reason_code":4002}}]}||2|
I've tried with ConvertRecord and InferAvroSchema processors, but they fail to process the field with json value. Thanks in advance!
... View more
12-18-2018
11:09 AM
Hi! Unable to start ambari-server [root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
Log at /var/log/ambari-server/ambari-server-check-database.log says: 2018-12-18 16:32:33,076 INFO - Checking DB store version
2018-12-18 16:32:33,643 INFO - DB store version is compatible
2018-12-18 16:32:41,042 INFO - ******************************* Check database started *******************************
2018-12-18 16:32:41,042 INFO - Ensuring that the schema set for Postgres is correct
2018-12-18 16:32:41,048 INFO - Checking for configs that are not mapped to any service
2018-12-18 16:32:41,150 INFO - Checking for configs selected more than once
2018-12-18 16:32:41,152 INFO - Checking for hosts without state
2018-12-18 16:32:41,152 INFO - Checking host component states count equals host component desired states count
2018-12-18 16:32:41,153 INFO - Checking services and their configs
2018-12-18 16:32:41,153 INFO - Getting ambari metainfo instance
2018-12-18 16:32:41,153 INFO - Executing query 'GET_SERVICES_WITHOUT_CONFIGS'
2018-12-18 16:32:41,154 INFO - Executing query 'GET_SERVICE_CONFIG_WITHOUT_MAPPING'
2018-12-18 16:32:41,155 INFO - Getting stack info from database
2018-12-18 16:32:41,155 INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
2018-12-18 16:32:41,160 INFO - Comparing service configs from stack with configs that we got from db
2018-12-18 16:32:41,160 INFO - Getting services from metainfo
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / RANGER
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / AMBARI_METRICS
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / AMBARI_INFRA
2018-12-18 16:32:41,160 WARN - Service AMBARI_INFRA is not available for stack HDF-3.0 in cluster hdf_kg
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / LOGSEARCH
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / KAFKA
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / NIFI
2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / ZOOKEEPER
2018-12-18 16:32:41,160 INFO - Comparing required service configs from stack with mapped service configs from db
2018-12-18 16:32:41,160 WARN - Required config(s): atlas-tagsync-ssl is(are) not available for service RANGER with service config version 23 in cluster hdf_kg
2018-12-18 16:32:41,161 INFO - Getting services which has mapped configs which are not selected in clusterconfig
2018-12-18 16:32:41,164 INFO - Checking Topology tables
2018-12-18 16:32:41,166 INFO - Checking for tables with large physical size
2018-12-18 16:32:41,167 INFO - The database table host_role_command is currently 3.641 MB and is within normal limits (3000.000)
2018-12-18 16:32:41,167 INFO - The database table execution_command is currently 2.109 MB and is within normal limits (3000.000)
2018-12-18 16:32:41,168 INFO - The database table stage is currently 0.477 MB and is within normal limits (3000.000)
2018-12-18 16:32:41,168 INFO - The database table request is currently 0.297 MB and is within normal limits (3000.000)
2018-12-18 16:32:41,168 INFO - The database table alert_history is currently 4.758 MB and is within normal limits (3000.000)
2018-12-18 16:32:41,168 INFO - Checking config group host mappings
2018-12-18 16:32:41,169 INFO - ******************************* Check database completed *******************************
We tried manually start the AMBARI_INFRA using the following command given by @scharan, service seems to be started, but it didn't help su - infra-solr -c '/usr/lib/ambari-infra-solr/bin/solr start -cloud -noprompt -s /opt/ambari_infra_solr/data' ambari=# select * from ambari.hostcomponentstate where service_name = 'AMBARI_INFRA';
id | cluster_id | component_name | version | current_state | host_id | service_name | upgrade_state | security_state
-----+------------+-------------------+---------+---------------+---------+--------------+---------------+----------------
105 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 101 | AMBARI_INFRA | NONE | UNKNOWN
4 | 2 | INFRA_SOLR | UNKNOWN | STARTED | 51 | AMBARI_INFRA | NONE | UNKNOWN
104 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 103 | AMBARI_INFRA | NONE | UNKNOWN
106 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 102 | AMBARI_INFRA | NONE | UNKNOWN
20 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 51 | AMBARI_INFRA | NONE | UNKNOWN
21 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 4 | AMBARI_INFRA | NONE | UNKNOWN
24 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 1 | AMBARI_INFRA | NONE | UNKNOWN
(7 rows)
Cluster is not in the cloud, maybe we shouldn't have to stated -cloud option in this command or
ambari=# select * from ambari.servicecomponentdesiredstate where service_name = 'AMBARI_INFRA';
id | component_name | cluster_id | desired_state | service_name | recovery_enabled | desired_repo_version_id | repo_state
----+-------------------+------------+---------------+--------------+------------------+-------------------------+------------
4 | INFRA_SOLR_CLIENT | 2 | INSTALLED | AMBARI_INFRA | 0 | 51 | CURRENT
3 | INFRA_SOLR | 2 | STARTED | AMBARI_INFRA | 1 | 51 | CURRENT
(2 rows)
Should we start INFRA_SOLR_CLIENTs also? If yes how can they be started manually? Could you please give any suggestion? with my best regards Gulshan
... View more
Labels:
- Labels:
-
Apache Ambari
-
Cloudera DataFlow (CDF)
12-03-2018
06:19 AM
@scharan, here is the result of commands you asked. I'm trying to restore my Ambari after unsuccessful upgrade from HDF 3.0 to HDF 3.1 ambari=# select * from ambari.hostcomponentstate where service_name = 'AMBARI_INFRA';
id | cluster_id | component_name | version | current_state | host_id | service_name | upgrade_state | security_state
-----+------------+-------------------+---------+---------------+---------+--------------+---------------+----------------
105 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 101 | AMBARI_INFRA | NONE | UNKNOWN
4 | 2 | INFRA_SOLR | UNKNOWN | STARTED | 51 | AMBARI_INFRA | NONE | UNKNOWN
104 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 103 | AMBARI_INFRA | NONE | UNKNOWN
106 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 102 | AMBARI_INFRA | NONE | UNKNOWN
20 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 51 | AMBARI_INFRA | NONE | UNKNOWN
21 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 4 | AMBARI_INFRA | NONE | UNKNOWN
24 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 1 | AMBARI_INFRA | NONE | UNKNOWN
(7 rows)
ambari=# select * from ambari.servicecomponentdesiredstate where service_name = 'AMBARI_INFRA';
id | component_name | cluster_id | desired_state | service_name | recovery_enabled | desired_repo_version_id | repo_state
----+-------------------+------------+---------------+--------------+------------------+-------------------------+------------
4 | INFRA_SOLR_CLIENT | 2 | INSTALLED | AMBARI_INFRA | 0 | 51 | CURRENT
3 | INFRA_SOLR | 2 | STARTED | AMBARI_INFRA | 1 | 51 | CURRENT
(2 rows)
Can you please help to start the INFRA_SOLR_CLIENT services? With kind regards!
... View more
12-03-2018
03:04 AM
@scharan, thank you for your response!! I've run the command you gave, but still ambari-server start gives the same error. Maybe Infra Solr Clients should also be started? If yes, how can they manually be started, if not, could you please give any suggestion? org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA [root@bi-nifi1 dmp]# su - infra-solr -c '/usr/lib/ambari-infra-solr/bin/solr start -cloud -noprompt -s /opt/ambari_infra_solr/data'
NOTE: Please install lsof as this script needs it to determine if Solr is listening on port 8983.
Started Solr server on port 8983 (pid=8878). Happy searching!
[root@bi-nifi1 dmp]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.................
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
Thanks in advance!
... View more
12-02-2018
12:14 PM
Hi! After ambari-server setup, I recovered the database from backup. And when trying to start ambari-server it gives following error: ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA # ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start................
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 backup]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
[root@bi-nifi1 backup]#
Is it possible to manually start AMBARI_INFRA service? Thanks in advance!
... View more
Labels:
- Labels:
-
Apache Ambari
11-30-2018
11:25 AM
Hi @Jagadeesan A S Thank you for your response! I have followed all the steps you have stated. After ambari-server setup I dropped the new ambari database, and recovered the database from backup. And when trying to start ambari-server it gives following error: ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA # ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start................
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 backup]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
Database consistency check result: DB_CHECK_WARNING
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067)
[root@bi-nifi1 backup]#
Is it possible to manually start AMBARI_INFRA service? Thanks in advance!
... View more
11-21-2018
02:56 AM
ambari-server.zip
... View more
11-21-2018
02:54 AM
1 Kudo
Hi @Jagadeesan A S Thank you for your concern! How can I identify which libraries are from older versions, there is no version on the file name? [dmp@bi-nifi1 ~]$ cd /usr/lib/ambari-server/lib/ambari_commons
[dmp@bi-nifi1 ambari_commons]$ ls
aggregate_functions.py ambari_service.pyc credential_store_helper.pyo inet_utils.py logging_utils.pyc os_check.pyo os_utils.py parallel_processing.pyc str_utils.pyc xml_utils.pyo
aggregate_functions.pyc ambari_service.pyo exceptions.py inet_utils.pyc logging_utils.pyo os_family_impl.py os_utils.pyc parallel_processing.pyo str_utils.pyo yaml_utils.py
aggregate_functions.pyo constants.py exceptions.pyc inet_utils.pyo network.py os_family_impl.pyc os_utils.pyo resources urllib_handlers.py yaml_utils.pyc
ambari_metrics_helper.py constants.pyc exceptions.pyo __init__.py network.pyc os_family_impl.pyo os_windows.py shell.py urllib_handlers.pyc yaml_utils.pyo
ambari_metrics_helper.pyc constants.pyo firewall.py __init__.pyc network.pyo os_linux.py os_windows.pyc shell.pyc urllib_handlers.pyo
ambari_metrics_helper.pyo credential_store_helper.py firewall.pyc __init__.pyo os_check.py os_linux.pyc os_windows.pyo shell.pyo xml_utils.py
ambari_service.py credential_store_helper.pyc firewall.pyo logging_utils.py os_check.pyc os_linux.pyo parallel_processing.py str_utils.py xml_utils.pyc
... View more
11-20-2018
01:29 PM
Hi, all! Can anyone give any suggestion in configuring new Ambari server with an existing HDF cluster Previos version of ambari fails in ambari-server start after downgrade [root@bi-nifi1 dmp]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.............ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
An unexpected error occured during starting Ambari Server.
org.apache.ambari.server.AmbariException: An error occured during updating current repository versions with stack repositories.
at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:99)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1061)
Caused by: java.lang.NullPointerException
at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.updateRepoVersion(UpdateActiveRepoVersionOnStartup.java:106)
at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:92)
... 2 more
Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server
at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1075)
[root@bi-nifi1 dmp]#
... View more
Labels:
- Labels:
-
Apache Ambari
-
Cloudera DataFlow (CDF)
11-16-2018
09:23 AM
@Akhil S Naik As you see in this screen shot, It is not allowing me to install HDF-3.2 version, although the version on ambari is 2.7.0.0 ambari.png What can you suggest?
... View more
11-16-2018
08:42 AM
Hi @Akhil S Naik Thanks for your reply! Initially I was planning to upgrade to HDF-3.2, after successful upgrade of Ambari to 2.7.0, I registered new 3.2 version of HDF and tried to install it. But it didn't allow me, it said that: Directly upgrading to this version is not supported So that's why I uninstalled the mpack for HDF-3.2 and installed HDF-3.1.2 So after that while trying to upgrade to HDF-3.1.2 I'm receiving the above error. Can you please suggest what actions can I take. I want to upgrade to HDF-3.2. Thanks!
... View more
11-16-2018
03:11 AM
I am upgrading HDF 3.0.2 to HDF 3.1.2 . I managed to upgrade ambari from 2.6.0.0 to 2.7.0 But upgrade of HDF fails with the error : Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 99, in <module>
NifiServiceCheck().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 39, in service_check
import params
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/params.py", line 284, in <module>
for host in config['clusterHostInfo']['zookeeper_hosts']:
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'zookeeper_hosts' was not found in configurations dictionary! Could someone give any suggestions?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Cloudera DataFlow (CDF)
11-15-2018
09:57 AM
Hi! While upgrading HDF - 3.0.2 to HDF-3.2.0 I'm facing the same problem. ambari=> select cluster_id, desired_stack_id from clusters; cluster_id | desired_stack_id
- 2 | 51 ambari=> select * from clusterstate ; cluster_id | current_cluster_state | current_stack_id 2 | | 51 ambari=> select stack_id, version from repo_version; stack_id | version 2 | 2.1.4.0-5 51 | 3.0.2.0-76 101 | 3.1.2.0-7 102 | 3.2.0.0 ---------------------- My current version in 3.0.2.0-76, stack_id=51 what can be the reason of why I can't install and upgrade to 3.2.0.0 Thanks in advance! with kind regards, Gulshan
... View more
07-17-2018
05:25 AM
@Matt Clarke Thanks for your recommendation concerning running Linux commands! Is there any processor that will output a list of all files in a specific directory? I would like to pass this list to Linux command.
... View more
07-16-2018
09:14 AM
Hi!! I need to run a Linux command on list of files received from ListSFTP (most probably it should be run on remote FTP server) and pass forward the result of this operation as flow file to putHDFS. Linux command is:
cat ${LIST} | xargs -n 1 -I {} stdbuf -o 5GB awk -F'\x1b' '{OFS="|"};{print $1,$2,FILENAME}' {} Can you please advise which processor can be most suitable. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
07-12-2018
05:49 AM
@Matt Burgess So it is needed to replace ESC (\x1b) with delimiter: '|' using UpdateRecord processor. Could you please help to configure this processor to impement this replacement in flow file records!
... View more
07-12-2018
04:25 AM
Hi, @Matt Burgess! My sample formatted JSON file content (in original flow file JSON objects are separated by new line) {
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:38.652+03:00"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "77ECA557BFA74C98B2792222C9C72CED"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "5A834B1C27DE4FAF9F038B370AA3DDA4"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}{
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}
{
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:39.652+03:10"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "72DEA157BFA74C98B2792222C0C11CBE"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "1E2234B1C27DE4FAF9F038B370AA3DBE4"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}
{
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:40.652+02:20"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "55CBA557BFA74C98B2792222C9A11CDE"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "1E244B1C27DE4FAF9F038B370AA3DDD5"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}{
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}
<br> Important thing to notice here is that some of the JSON objects in the flow file contain extention seperated with ESC (\x1b). {
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}<br> Schema looks like below: {mainJSON}{extentionJSON} {mainJSON} {mainJSON}{extentionJSON}
..... In the output I would like to have the following format: mainJSON | extentionJSON mainJSON | mainJSON | extentionJSON etc... Thank you!
... View more
07-11-2018
12:12 PM
Hi, dear Experts! Could you please help with following issue: I have a processor ExtractText that processes JSON flow file and creates two attributes with large JSON text. mainJSON = ^.*(?=\x1b)|^((?!\x1b).)*$) enrichmentJSON = ((?<=\x1b).*) As an output there were created 3 attributes for mainJSON and enrichmentJSON mainJSON: mainJSON, mainJSON.0, mainJSON.1 --- each containing the same portion of expected result. Isn't it possible to store large text value in an attribute? Is there another way to store and pass large text value as an attribute? As a next step I wanted to combine these two attributes with other attributes in ReplaceText processor and put them into hive table as separate columns of one row. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
07-06-2018
05:06 AM
1 Kudo
Thank you very much @Shu and @Matt Burgess !! @Shu, Split Content processor really does great thing! I have managed to split the nested JSON objects into separate objects using Text as Byte Sequence Format and }{ as Byte Sequence. Thank you very much for your advise!! with all the best wishes, Gulshan
... View more
07-05-2018
10:54 AM
Hi! My Flow file contains more then one JSON objects {
Nested JSON object 1
}{
Nested JSON object 2
}{
Nested JSON object N
} Can you please help to split JSON flow file into separate JSON objects using Nifi processors. What should be the JsonPath Expression in SplitJSON processor? thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
05-16-2018
09:59 AM
Oh, great! Thank you very much Sandeep!!
... View more
05-16-2018
09:14 AM
Hi! Can you please help to permanently remove a disk from all the datanodes in working cluster. DataNode directories /grid/0,/grid/2,/grid/3,/grid/4,/grid/5,/grid/6,/grid/7,/grid/8,/grid/9,/grid/10,/grid/11,/grid/12,/grid/14,/grid/15 I would like to remove /grid/15. Thanks in advance!! with regards, Gulshan
... View more
Labels:
04-23-2018
05:38 AM
Hi, dear Experts! Could you please help with the following issue? Ranger Audit Access tab is failing with the following error: Error Error running solr query, please check solr configs. Could not find a healthy node to handle the request. in /hadoop/log/ambari-infra-solr/solr.log : ERROR [c:audit_logs s:shard0 r:core_node2 x:audit_logs_shard0_replica1] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Invalid Date String:'2017-09-05T02017 with regards, Gulshan
... View more
Labels:
- Labels:
-
Apache Solr
02-21-2018
05:39 AM
hi @Baruch AMOUSSOU DJANGBAN We have the same problem... have you managed to solve the problem, if yes, can you please share thanks in advance
... View more
02-15-2018
06:38 AM
Hi, @Jay Kumar SenSharma! Thank you, for the link you have provided. My problem was also because of "quasi-installed state where its repo is listed as 'installed' even though it is not" I could resolve the problem by removing the package that was installed(partially or somewhat oddly) and relaunching the installation of packages from ambari again Thanks!!
... View more
02-15-2018
06:17 AM
@Jay Kumar SenSharma Hi, Jay Kumar! But I'm using Apache Ambari Version 2.6.1.0. Before starting upgrade of HDP, I 've upgraded Ambari to 2.6.1.0. From the link you gave I couldn't find the workaround.
... View more
02-15-2018
04:14 AM
Hi, dear experts! I'm trying to upgrade my HDP from 2.6.0.3 to 2.6.4 using Ambari. Ambari upgrade was completed successfuly, but while registering and installing of target version process fails with the following error: 2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages
retry_count=agent_stack_retry_count
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
return self.install_package(name, use_repos, skip_repos, is_upgrade)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 473, in <module>
InstallPackages().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 153, in actionexecute
raise Fail("Failed to distribute repositories/install packages")
resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages
stdout: /var/lib/ambari-agent/data/output-8438.txt
2018-02-14 22:56:17,861 - Installing package tez_hive2_2_6_4_0_91 ('/usr/bin/yum -d 0 -e 0 -y install tez_hive2_2_6_4_0_91')
2018-02-14 22:56:33,825 - No package found for slider_${stack_version}(slider_(\d|_)+$)
2018-02-14 22:56:33,826 - Package[None] {'retry_on_repo_unavailability': False, 'retry_count': 5, 'action': ['upgrade']}
2018-02-14 22:56:33,826 - Installing package None ('/usr/bin/yum -d 0 -e 0 -y install ''')
2018-02-14 22:56:49,179 - Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
2018-02-14 22:56:49,179 - Failed to install package None. Executing '/usr/bin/yum clean metadata'
2018-02-14 22:56:49,422 - Retrying to install package None after 30 seconds
2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages
retry_count=agent_stack_retry_count
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
return self.install_package(name, use_repos, skip_repos, is_upgrade)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
2018-02-14 22:57:53,898 - Installation of packages failed. Checking if installation was partially complete
2018-02-14 22:57:53,899 - Old versions: ['2.6.0.3-8', '2.6.4.0-91']
2018-02-14 22:57:53,899 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
2018-02-14 22:57:53,925 - call returned (0, '2.6.0.3-8\n2.6.4.0-91')
2018-02-14 22:57:53,925 - New versions: ['2.6.0.3-8', '2.6.4.0-91']
2018-02-14 22:57:53,925 - Deltas: set([])
Command failed after 1 tries
... View more
Labels: