HDP-2.6.4.0 - Apache superset start failing with error binascii.Error: Incorrect padding

[#1]

Using HDP-2.6.4.0. Enabled apache superset service. Service start is failing with following error. I am using postgresql as database for storing superset metadata.

2018-08-28 10:05:16,587:INFO:root:Creating database reference
Loaded your LOCAL configuration
Traceback (most recent call last):
  File "/usr/hdp/current/superset/bin/superset", line 85, in <module>
    manager.run()
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/superset/cli.py", line 27, in init
    security.sync_role_definitions()
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/superset/security.py", line 122, in sync_role_definitions
    get_or_create_main_db()
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/superset/security.py", line 96, in get_or_create_main_db
    .filter_by(database_name='main')
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2690, in first
    ret = list(self[0:1])
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2482, in __getitem__
    return list(res)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 90, in instances
    util.raise_from_cause(err)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 75, in instances
    rows = [proc(row) for row in fetch]
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance
    loaded_instance, populate_existing, populators)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full
    dict_[key] = getter(row)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process
    return process_value(impl_processor(value), dialect)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy_utils/types/encrypted.py", line 266, in process_result_value
    decrypted_value = self.engine.decrypt(value)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/sqlalchemy_utils/types/encrypted.py", line 90, in decrypt
    decrypted = base64.b64decode(value)
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/base64.py", line 90, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

Looks like a malformed string getting passed to base64.b64decode() function

[#2]

@Vinuraj M

Can you please check if you have the UTF-8 encoding set for your database?

postgres=# \c superset 
superset=# SHOW SERVER_ENCODING; 

.

Also if it is a fresh Superset installation which is failing so in that case can you try reinstalling the Superset (after dropping/cleaning the superset database Or after taking a DB dump as a backup) ?

Also during Superset installation did you notice any error/warning in Postgres Database log related to encryption related to SECRET_KEY Or did you notice any error/warning in the "/var/log/ambari-server/ambari-server.log" Or on the "/var/log/ambari-agent/ambari-agent.log" on the host where Superset was being installed?

[#3]

See postgres command output as following

superset=# SHOW SERVER_ENCODING;
 server_encoding
-----------------
 UTF8
(1 row)
ambari-agent log has following
INFO 2018-08-28 12:50:19,237 PythonExecutor.py:130 - Command ['/usr/bin/python',
 '/var/lib/ambari-agent/cache/common-services/SUPERSET/0.15.0/package/scripts/superset.py',
 'START',
 '/var/lib/ambari-agent/data/command-1688.json',
 '/var/lib/ambari-agent/cache/common-services/SUPERSET/0.15.0/package',
 '/var/lib/ambari-agent/data/structured-out-1688.json',
 'INFO',
 '/var/lib/ambari-agent/tmp',
 'PROTOCOL_TLSv1',
 ''] failed with exitcode=1

[#4]

I have added some print statement in the code where the exception is happening as i thought the issue was with python base64 encoding problem with padding. But looks like the the string passed to base64.b64decode() itself is malformed. See following log

2018-08-28 12:50:18,780:INFO:root:Syncing role definition
2018-08-28 12:50:18,800:INFO:root:Creating database reference
Loaded your LOCAL configuration
value: b'x393565452f683845463837555032666d5941577164673d3d'
Traceback (most recent call last):
  File "/usr/hdp/current/superset/bin/superset", line 85, in <module>
    manager.run()
  File "/usr/hdp/2.6.4.0-91/superset/lib/python3.4/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])