Member since
02-27-2017
9
Posts
1
Kudos Received
0
Solutions
07-03-2017
01:38 PM
I've run into a bug when upgrading the Ambari from 2.4 to 2.5. I followed all the recommended step until I got to upgrading the Ambari Server database schema using the "ambari-server upgrade" command which failed with the following error due to a known issue when the Lucidworks HDPSearch component is previously added to Ambari: $> ambari-server upgrade -v
Using python /usr/bin/python
Upgrading ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: ===========================================================================================
INFO: Executing Mpack Replay Log :
INFO: {'purge': False, 'mpack_command': 'install-mpack', 'mpack_path': '/var/lib/ambari-server/resources/mpacks/cache/solr-service-mpack-5.5.2.2.5.tar.gz', 'force': False, 'verbose': False}
INFO: ===========================================================================================
INFO: Installing management pack /var/lib/ambari-server/resources/mpacks/cache/solr-service-mpack-5.5.2.2.5.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/solr-service-mpack-5.5.2.2.5.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/solr-service
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Stage management pack solr-ambari-mpack-5.5.2.2.5 to staging location /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5
INFO: Force removing previously installed management pack from /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5
INFO: Processing artifact SOLR-common-services of type service-definitions in /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5/common-services
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 941, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 911, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 863, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 78, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 363, in upgrade
replay_mpack_logs()
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 983, in replay_mpack_logs
install_mpack(replay_options, replay_mode=True)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack
(mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 792, in _install_mpack
process_service_definitions_artifact(artifact, artifact_source_dir, options)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 515, in process_service_definitions_artifact
create_symlink(src_service_definitions_dir, dest_service_definitions_dir, file, options.force)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 235, in create_symlink
create_symlink_using_path(src_path, dest_link, force)
File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 247, in create_symlink_using_path
sudo.symlink(src_path, dest_link)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink
os.symlink(source, link_name)
OSError: [Errno 17] File exists
This issue is logged but the fix is only availble in the next release! : https://issues.apache.org/jira/browse/AMBARI-21263 Trying to rectify this, I uninstalled the Lucidworks HDPSearch from the cluster but that didn't help either. Now what should I do? How should I either resolve this upgrade issue or at least rollback to 2.4 without messing up ?
... View more
Labels:
03-22-2017
09:07 AM
I tried that but it doesn't process the flowfile. It also gives the following warning: ConvertJSONToAvro[id=8aec4759-eae0-1dab-ffff-ffff9b831c59] Failed to convert 1/1 records from JSON to Avro
... View more
03-17-2017
08:51 AM
I have a JSON file containing an array of records and I'm trying to convert the entire JSON to Avro file using the ConvertJSONToAvro processor. JSON sample: [
{
"id": 123,
"title": "foo"
},
{
"id": 345,
"title": "bar"
}
]
Avro Schema: {
"name": "test",
"type": "array",
"items": {
"type": "record",
"name": "user",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "title",
"type": "string"
}
]
}
}
However using the above Avro schema the processor throws an exception: java.lang.IllegalArgumentException: Schemas for JSON files should be record
at org.kitesdk.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) ~[kite-data-core-1.0.0.jar:na]
at org.kitesdk.data.spi.filesystem.JSONFileReader.initialize(JSONFileReader.java:84) ~[kite-data-core-1.0.0.jar:na]
at org.apache.nifi.processors.kite.ConvertJSONToAvro$1.process(ConvertJSONToAvro.java:144) ~[nifi-kite-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2578) ~[nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.processors.kite.ConvertJSONToAvro.onTrigger(ConvertJSONToAvro.java:139) ~[nifi-kite-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
Does this mean the ConvertJsonToAvro processor cannot convert an array of records, and I have to split the JSON file before feeding the records to this process? It seems like it doesn't recognise at "type": "array" at the root of the schema.
... View more
Labels:
- Labels:
-
Apache NiFi