Hi! Could you please help with converting CSV file containing JSON values into JSON: 1|portout|Donor Reject Accepted|719dfa55b8dd42be8f64d55ddee0c805|10000001518591|2018-09-26 22:00:00|2018-09-25 17:18:53|0|89011122244|Satish Chippa|200003530|1|{"xxxxxxx98009":[{"subscription_state":{"status":"Terminated","reason_code":4001}},{"subscription_early_portout":{"status":"Early Portout","reason_code":4005}}]}||2| 2|portout|Donor Reject Accepted|554a2faa9bb548dc848915e95938fe22|10000001518592|2018-09-26 22:00:00|2018-09-25 17:30:25|0|67811122233|Satish Chippa|200003528|1|{"xxxxxxx38009":[{"subscription_state":{"status":"Terminated","reason_code":4001}}]}||2| 3|portout|Donor Reject Accepted|3732aa2495ce4f37844716848e88c989|10000001518596|2018-09-28 04:00:00|2018-09-26 17:31:09|0|99001990981|Aman PortOut|200003523|2|{"xxxxxxx69009":[{"subscription_state":{"status":"Terminated","reason_code":4001}},{"subscription_early_portout":{"status":"Early Portout","reason_code":4005}},{"customer_open_invoices":{"status":"Customer has open invoices","reason_code":4002}}]}||2| I've tried with ConvertRecord and InferAvroSchema processors, but they fail to process the field with json value. Thanks in advance! ... View more

Hi! Unable to start ambari-server [root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Database consistency check result: DB_CHECK_WARNING An unexpected error occured during starting Ambari Server. org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864) at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067) Log at /var/log/ambari-server/ambari-server-check-database.log says: 2018-12-18 16:32:33,076 INFO - Checking DB store version 2018-12-18 16:32:33,643 INFO - DB store version is compatible 2018-12-18 16:32:41,042 INFO - ******************************* Check database started ******************************* 2018-12-18 16:32:41,042 INFO - Ensuring that the schema set for Postgres is correct 2018-12-18 16:32:41,048 INFO - Checking for configs that are not mapped to any service 2018-12-18 16:32:41,150 INFO - Checking for configs selected more than once 2018-12-18 16:32:41,152 INFO - Checking for hosts without state 2018-12-18 16:32:41,152 INFO - Checking host component states count equals host component desired states count 2018-12-18 16:32:41,153 INFO - Checking services and their configs 2018-12-18 16:32:41,153 INFO - Getting ambari metainfo instance 2018-12-18 16:32:41,153 INFO - Executing query 'GET_SERVICES_WITHOUT_CONFIGS' 2018-12-18 16:32:41,154 INFO - Executing query 'GET_SERVICE_CONFIG_WITHOUT_MAPPING' 2018-12-18 16:32:41,155 INFO - Getting stack info from database 2018-12-18 16:32:41,155 INFO - Executing query 'GET_SERVICES_WITH_CONFIGS' 2018-12-18 16:32:41,160 INFO - Comparing service configs from stack with configs that we got from db 2018-12-18 16:32:41,160 INFO - Getting services from metainfo 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / RANGER 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / AMBARI_METRICS 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / AMBARI_INFRA 2018-12-18 16:32:41,160 WARN - Service AMBARI_INFRA is not available for stack HDF-3.0 in cluster hdf_kg 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / LOGSEARCH 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / KAFKA 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / NIFI 2018-12-18 16:32:41,160 INFO - Processing HDF-3.0 / ZOOKEEPER 2018-12-18 16:32:41,160 INFO - Comparing required service configs from stack with mapped service configs from db 2018-12-18 16:32:41,160 WARN - Required config(s): atlas-tagsync-ssl is(are) not available for service RANGER with service config version 23 in cluster hdf_kg 2018-12-18 16:32:41,161 INFO - Getting services which has mapped configs which are not selected in clusterconfig 2018-12-18 16:32:41,164 INFO - Checking Topology tables 2018-12-18 16:32:41,166 INFO - Checking for tables with large physical size 2018-12-18 16:32:41,167 INFO - The database table host_role_command is currently 3.641 MB and is within normal limits (3000.000) 2018-12-18 16:32:41,167 INFO - The database table execution_command is currently 2.109 MB and is within normal limits (3000.000) 2018-12-18 16:32:41,168 INFO - The database table stage is currently 0.477 MB and is within normal limits (3000.000) 2018-12-18 16:32:41,168 INFO - The database table request is currently 0.297 MB and is within normal limits (3000.000) 2018-12-18 16:32:41,168 INFO - The database table alert_history is currently 4.758 MB and is within normal limits (3000.000) 2018-12-18 16:32:41,168 INFO - Checking config group host mappings 2018-12-18 16:32:41,169 INFO - ******************************* Check database completed ******************************* We tried manually start the AMBARI_INFRA using the following command given by @scharan, service seems to be started, but it didn't help su - infra-solr -c '/usr/lib/ambari-infra-solr/bin/solr start -cloud -noprompt -s /opt/ambari_infra_solr/data' ambari=# select * from ambari.hostcomponentstate where service_name = 'AMBARI_INFRA'; id | cluster_id | component_name | version | current_state | host_id | service_name | upgrade_state | security_state -----+------------+-------------------+---------+---------------+---------+--------------+---------------+---------------- 105 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 101 | AMBARI_INFRA | NONE | UNKNOWN 4 | 2 | INFRA_SOLR | UNKNOWN | STARTED | 51 | AMBARI_INFRA | NONE | UNKNOWN 104 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 103 | AMBARI_INFRA | NONE | UNKNOWN 106 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 102 | AMBARI_INFRA | NONE | UNKNOWN 20 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 51 | AMBARI_INFRA | NONE | UNKNOWN 21 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 4 | AMBARI_INFRA | NONE | UNKNOWN 24 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | INSTALLED | 1 | AMBARI_INFRA | NONE | UNKNOWN (7 rows) Cluster is not in the cloud, maybe we shouldn't have to stated -cloud option in this command or ambari=# select * from ambari.servicecomponentdesiredstate where service_name = 'AMBARI_INFRA'; id | component_name | cluster_id | desired_state | service_name | recovery_enabled | desired_repo_version_id | repo_state ----+-------------------+------------+---------------+--------------+------------------+-------------------------+------------ 4 | INFRA_SOLR_CLIENT | 2 | INSTALLED | AMBARI_INFRA | 0 | 51 | CURRENT 3 | INFRA_SOLR | 2 | STARTED | AMBARI_INFRA | 1 | 51 | CURRENT (2 rows) Should we start INFRA_SOLR_CLIENTs also? If yes how can they be started manually? Could you please give any suggestion? with my best regards Gulshan ... View more

Hi! After ambari-server setup, I recovered the database from backup. And when trying to start ambari-server it gives following error: ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA # ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................ DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details. ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. [root@bi-nifi1 backup]# cat /var/log/ambari-server/ambari-server.out Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Database consistency check result: DB_CHECK_WARNING An unexpected error occured during starting Ambari Server. org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=hdf_kg, serviceName=AMBARI_INFRA at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864) at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067) [root@bi-nifi1 backup]# Is it possible to manually start AMBARI_INFRA service? Thanks in advance! ... View more

Hi, all! Can anyone give any suggestion in configuring new Ambari server with an existing HDF cluster Previos version of ambari fails in ambari-server start after downgrade [root@bi-nifi1 dmp]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.............ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. [root@bi-nifi1 dmp]# cat /var/log/ambari-server/ambari-server.out Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 An unexpected error occured during starting Ambari Server. org.apache.ambari.server.AmbariException: An error occured during updating current repository versions with stack repositories. at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:99) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1061) Caused by: java.lang.NullPointerException at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.updateRepoVersion(UpdateActiveRepoVersionOnStartup.java:106) at org.apache.ambari.server.stack.UpdateActiveRepoVersionOnStartup.process(UpdateActiveRepoVersionOnStartup.java:92) ... 2 more Exception in thread "main" org.apache.ambari.server.AmbariException: Error stopping the server at org.apache.ambari.server.controller.AmbariServer.stop(AmbariServer.java:880) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1075) [root@bi-nifi1 dmp]# ... View more

Hi All! I have a question, if it is possible to restore Ambari server and its agents to its previous version if HDF-3.2.0 cluster upgrade 3.0.2 to 3.2.0 failed with error: Directly upgrading to this version is not supported. Although https://supportmatrix.hortonworks.com/ says that it should be possible. (Though Ambari-2.6.0 was successfully upgraded to 2.7.0) The next attemp was to upgrade HDF to HDF-3.1.2 with Ambari 2.7.0. but it also failed with following error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 99, in <module> NifiServiceCheck().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 39, in service_check import params File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/params.py", line 284, in <module> for host in config['clusterHostInfo']['zookeeper_hosts']: File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'zookeeper_hosts' was not found in configurations dictionary! And supportmatrix says that Ambari 2.7.0 cannot work with HDF-3.1.2 . So, now I have to somehow restore my Ambari server and its agents to its previous version. I have taken following backups before starting upgrade $ /usr/bin/pg_dumpall | gzip > /data/backup/pg_dumpall_15112018.gz cp -r /var/lib/ambari-server/resources /var/lib/ambari-server/resources.backup.15112018 cp /etc/ambari-server/conf/ambari.properties /etc/ambari-server/conf/ambari.properties.backup.15112018 Please help to restore my Ambari server and its agents to its previous version. Thanks in advance! with regards, Gulshan ... View more

I am upgrading HDF 3.0.2 to HDF 3.1.2 . I managed to upgrade ambari from 2.6.0.0 to 2.7.0 But upgrade of HDF fails with the error : Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 99, in <module> NifiServiceCheck().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/service_check.py", line 39, in service_check import params File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/params.py", line 284, in <module> for host in config['clusterHostInfo']['zookeeper_hosts']: File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'zookeeper_hosts' was not found in configurations dictionary! Could someone give any suggestions? ... View more

Hi, dear Experts! Nifi service check fails with the following error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in <module> Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 135, in stop import params File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/params.py", line 284, in <module> for host in config['clusterHostInfo']['zookeeper_hosts']: File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'zookeeper_hosts' was not found in configurations dictionary! ... View more

Hi, dear Experts! Nifi service check fails with the following error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in <module> Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 135, in stop import params File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/params.py", line 284, in <module> for host in config['clusterHostInfo']['zookeeper_hosts']: File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'zookeeper_hosts' was not found in configurations dictionary! ... View more

Dear experts! I have nested JSON files that I want to load into Hive using Nifi. { "schemaNameSpace": "com.racle.cha.event.schema", "schemaName": "ChaUsageChargeEvent", "schemaVersion": "11.4.0", "eventHeader": { "com.racle.cha.event.schema.EventHeader": { "eventCreationTime": { "com.racle.cha.event.schema.Time": { "timestamp": { "string": "2017-09-20T10:04:48Z" }, "timeZoneType": { "string": "SYSTEM_TIME_ZONE" }, "zoneName": null } }, "chargingInstanceHost": { "string": "env1-cha-core1.racle.kg" }, "recordUniqueId": { "string": "aa09ecd4-7679-4d7c-8af8-38c34ddced85" } } }, "genericInterfaceParameters": { "com.racle.cha.event.schema.GenericInterfaceParameters": { "originatingParameterSet": { "com.racle.cha.event.schema.OriginatingParameterSet": { "transactionId": { "string": "f7108ba0-1d8a-4081-89c8-1b4370ff9846" }, "timeStamp": { "string": "2017-09-20T13:04:48+03:00" }, "instanceId": { "string": "env1-cha-access1.racle.kg" }, "userId": { "string": "" }, "reason": { "string": "Online Rate and Charge Ro" } } }, "externalParameterSet": { "com.racle.cha.event.schema.ExternalParameterSet": { "transactionId": null, "timeStamp": null, "instanceId": null, "userId": null, "reason": null, "additionalData": { "map": { "3gppChargingId": "1543508609", "ggsnAddress": "10.0.203.105" } } } }, "passingParameterSet": null } } } The Hive table will have the following structure: create table EVENTPUBLISHING ( schemanamespace VARCHAR(100), schemaname VARCHAR(100), schemaversion VARCHAR(10), originalevent string, filename VARCHAR(255), load_dt string ) Using Nifi processors I want the first 3 columns of table to be filled with values of first 3 JSON attributes and 4th column - originalevent to contain whole JSON object. Can you please advise the best way to implement this. Thanks in advance! ... View more

Hi, dear Experts! Could you please help with the following issue? Ranger Audit Access tab is failing with the following error: Error Error running solr query, please check solr configs. Could not find a healthy node to handle the request. in /hadoop/log/ambari-infra-solr/solr.log : ERROR [c:audit_logs s:shard0 r:core_node2 x:audit_logs_shard0_replica1] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Invalid Date String:'2017-09-05T02017 with regards, Gulshan ... View more