Hue 3.10 installation : Unable to dump the existing database to a text file for migration to MySql

[#1]

We have deployed HDP 2.4 on aws. MySql is deployed and runs fine. A huedb has been created, permission is granted and hue.ini has been correctly configured to reflect the change in the db. We have also made configured relevant service changes in Ambari .

I am unable to create a dump of the Sqlite 3 db, which is located in "/usr/local/hue/desktop/desktop.db". I have followed the following links as a guide.

https://community.hortonworks.com/questions/399/how-to-move-hue-database-from-default-sqlite-datab.h...

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configu...

Linux is throwing errors on the following irrespective of cl issued with or without 'sudo'.

(1) sudo /usr/local/hue/build/env/bin/hue dumpdata > /tmp/hue_db_dump.json

CommandError: Unable to serialize database: (1146, "Table 'huedb.django_openid_auth_nonce' doesn't exist")

(2)  /usr/local/hue/build/env/bin/hue dumpdata > /tmp/hue_db_dump.json

Traceback (most recent call last):
  File "/usr/local/hue/build/env/bin/hue", line 9, in <module>
    load_entry_point('desktop==3.10.0', 'console_scripts', 'hue')()
  File "/usr/local/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
    execute_from_command_line(sys.argv)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command
    commands = get_commands()
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands
    apps = settings.INSTALLED_APPS
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
    __import__(name)
  File "/usr/local/hue/desktop/core/src/desktop/settings.py", line 63, in <module>
    desktop.log.basic_logging(os.environ[ENV_HUE_PROCESS_NAME])
  File "/usr/local/hue/desktop/core/src/desktop/log/__init__.py", line 149, in basic_logging
    logging.config.fileConfig(log_conf)
  File "/usr/lib64/python2.7/logging/config.py", line 78, in fileConfig
    handlers = _install_handlers(cp, formatters)
  File "/usr/lib64/python2.7/logging/config.py", line 156, in _install_handlers
    h = klass(*args)
  File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/home/ec2-user/logs/dumpdata.log'

(3)  /usr/local/hue/build/env/bin/hue syncdb –noinput

Traceback (most recent call last):
  File "/usr/local/hue/build/env/bin/hue", line 9, in <module>
    load_entry_point('desktop==3.10.0', 'console_scripts', 'hue')()
  File "/usr/local/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
    execute_from_command_line(sys.argv)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command
    commands = get_commands()
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands
    apps = settings.INSTALLED_APPS
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
    __import__(name)
  File "/usr/local/hue/desktop/core/src/desktop/settings.py", line 63, in <module>
    desktop.log.basic_logging(os.environ[ENV_HUE_PROCESS_NAME])
  File "/usr/local/hue/desktop/core/src/desktop/log/__init__.py", line 149, in basic_logging
    logging.config.fileConfig(log_conf)
  File "/usr/lib64/python2.7/logging/config.py", line 78, in fileConfig
    handlers = _install_handlers(cp, formatters)
  File "/usr/lib64/python2.7/logging/config.py", line 156, in _install_handlers
    h = klass(*args)
  File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/home/ec2-user/logs/syncdb.log'
[ec2-user@namenode ~]$

(4) sudo /usr/local/hue/build/env/bin/hue syncdb –noinput
CommandError: Command doesn't accept any arguments

Please advise

Thanks,

Srinivasa

[#2]

@ibhatt:

Many thanks. That worked!

I have a questions as clarifications please.

# hue syncdb gave me these results

Synced:
 > django.contrib.auth
 > django_openid_auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > django.contrib.admin
 > south
 > axes
 > about
 > filebrowser
 > help
 > impala
 > jobbrowser
 > metastore
 > proxy
 > rdbms
 > zookeeper
 > indexer




Not synced (use migrations):
 - django_extensions
 - desktop
 - beeswax
 - hbase
 - jobsub
 - oozie
 - pig
 - search
 - spark
 - sqoop
 - useradmin
 - notebook

Can I ignore the not synced please?

It asked me if I wanted to create a super user for Django's auth system. It created one with a user name, which is different to "hue" as a user in mysql for the huedb.

I assume this does not pose any conflict. If I need to remove this, please advise the steps to correct.

The last command "hue migrate" gave me these messages.

Running migrations for django_extensions:
 - Migrating forwards to 0001_empty.
 > django_extensions:0001_empty
 - Loading initial data for django_extensions.
Installed 0 object(s) from 0 fixture(s)
Running migrations for desktop:
 - Migrating forwards to 0022_auto__del_field_defaultconfiguration_group__del_unique_defaultconfigur.
 > pig:0001_initial
 > oozie:0001_initial
 > oozie:0002_auto__add_hive
 > oozie:0003_auto__add_sqoop
 > oozie:0004_auto__add_ssh
 > oozie:0005_auto__add_shell
 > oozie:0006_auto__chg_field_java_files__chg_field_java_archives__chg_field_sqoop_f
 > oozie:0007_auto__chg_field_sqoop_script_path
 > oozie:0008_auto__add_distcp
 > oozie:0009_auto__add_decision
 > oozie:0010_auto__add_fs
 > oozie:0011_auto__add_email
 > oozie:0012_auto__add_subworkflow__chg_field_email_subject__chg_field_email_body
 > oozie:0013_auto__add_generic
 > oozie:0014_auto__add_decisionend
 > oozie:0015_auto__add_field_dataset_advanced_start_instance__add_field_dataset_ins
 > oozie:0016_auto__add_field_coordinator_job_properties
 > oozie:0017_auto__add_bundledcoordinator__add_bundle
 > oozie:0018_auto__add_field_workflow_managed
 > oozie:0019_auto__add_field_java_capture_output
 > oozie:0020_chg_large_varchars_to_textfields
 > oozie:0021_auto__chg_field_java_args__add_field_job_is_trashed
 > oozie:0022_auto__chg_field_mapreduce_node_ptr__chg_field_start_node_ptr
 > oozie:0022_change_examples_path_format
 - Migration 'oozie:0022_change_examples_path_format' is marked for no-dry-run.
 > oozie:0023_auto__add_field_node_data__add_field_job_data
 > oozie:0024_auto__chg_field_subworkflow_sub_workflow
 > oozie:0025_change_examples_path_format
 - Migration 'oozie:0025_change_examples_path_format' is marked for no-dry-run.
 > desktop:0001_initial
 > desktop:0002_add_groups_and_homedirs
 > desktop:0003_group_permissions
 > desktop:0004_grouprelations
 > desktop:0005_settings
 > desktop:0006_settings_add_tour
 > beeswax:0001_initial
 > beeswax:0002_auto__add_field_queryhistory_notify
 > beeswax:0003_auto__add_field_queryhistory_server_name__add_field_queryhistory_serve
 > beeswax:0004_auto__add_session__add_field_queryhistory_server_type__add_field_query
 > beeswax:0005_auto__add_field_queryhistory_statement_number
 > beeswax:0006_auto__add_field_session_application
 > beeswax:0007_auto__add_field_savedquery_is_trashed
 > beeswax:0008_auto__add_field_queryhistory_query_type
 > desktop:0007_auto__add_documentpermission__add_documenttag__add_document
 > desktop:0008_documentpermission_m2m_tables
 > desktop:0009_auto__chg_field_document_name
 > desktop:0010_auto__add_document2__chg_field_userpreferences_key__chg_field_userpref
 > desktop:0011_auto__chg_field_document2_uuid
 > desktop:0012_auto__chg_field_documentpermission_perms
 > desktop:0013_auto__add_unique_documenttag_owner_tag
 > desktop:0014_auto__add_unique_document_content_type_object_id
 > desktop:0015_auto__add_unique_documentpermission_doc_perms
 > desktop:0016_auto__add_unique_document2_uuid_version_is_history
 > desktop:0017_auto__add_document2permission__add_unique_document2permission_doc_perm
 > desktop:0018_auto__add_field_document2_parent_directory
 > desktop:0019_auto
 > desktop:0020_auto__del_field_document2permission_all
 > desktop:0021_auto__add_defaultconfiguration__add_unique_defaultconfiguration_app_is
 > desktop:0022_auto__del_field_defaultconfiguration_group__del_unique_defaultconfigur
 - Loading initial data for desktop.
Installed 0 object(s) from 0 fixture(s)
Running migrations for beeswax:
 - Migrating forwards to 0014_auto__add_field_queryhistory_is_cleared.
 > beeswax:0009_auto__add_field_savedquery_is_redacted__add_field_queryhistory_is_reda
 > beeswax:0009_auto__chg_field_queryhistory_server_port
 > beeswax:0010_merge_database_state
 > beeswax:0011_auto__chg_field_savedquery_name
 > beeswax:0012_auto__add_field_queryhistory_extra
 > beeswax:0013_auto__add_field_session_properties
 > beeswax:0014_auto__add_field_queryhistory_is_cleared
 - Loading initial data for beeswax.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hbase:
 - Migrating forwards to 0001_initial.
 > hbase:0001_initial
 - Loading initial data for hbase.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jobsub:
 - Migrating forwards to 0006_chg_varchars_to_textfields.
 > jobsub:0001_initial
 > jobsub:0002_auto__add_ooziestreamingaction__add_oozieaction__add_oozieworkflow__ad
 > jobsub:0003_convertCharFieldtoTextField
 > jobsub:0004_hue1_to_hue2
 - Migration 'jobsub:0004_hue1_to_hue2' is marked for no-dry-run.
 > jobsub:0005_unify_with_oozie
 - Migration 'jobsub:0005_unify_with_oozie' is marked for no-dry-run.
 > jobsub:0006_chg_varchars_to_textfields
 - Loading initial data for jobsub.
Installed 0 object(s) from 0 fixture(s)
Running migrations for oozie:
 - Migrating forwards to 0027_auto__chg_field_node_name__chg_field_job_name.
 > oozie:0026_set_default_data_values
 - Migration 'oozie:0026_set_default_data_values' is marked for no-dry-run.
 > oozie:0027_auto__chg_field_node_name__chg_field_job_name
 - Loading initial data for oozie.
Installed 0 object(s) from 0 fixture(s)
Running migrations for pig:
- Nothing to migrate.
 - Loading initial data for pig.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
 - Migrating forwards to 0003_auto__add_field_collection_owner.
 > search:0001_initial
 > search:0002_auto__del_core__add_collection
 > search:0003_auto__add_field_collection_owner
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for spark:
 - Migrating forwards to 0001_initial.
 > spark:0001_initial
 - Loading initial data for spark.
Installed 0 object(s) from 0 fixture(s)
Running migrations for sqoop:
 - Migrating forwards to 0001_initial.
 > sqoop:0001_initial
 - Loading initial data for sqoop.
Installed 0 object(s) from 0 fixture(s)
Running migrations for useradmin:
 - Migrating forwards to 0006_auto__add_index_userprofile_last_activity.
 > useradmin:0001_permissions_and_profiles
 - Migration 'useradmin:0001_permissions_and_profiles' is marked for no-dry-run.
 > useradmin:0002_add_ldap_support
 - Migration 'useradmin:0002_add_ldap_support' is marked for no-dry-run.
 > useradmin:0003_remove_metastore_readonly_huepermission
 - Migration 'useradmin:0003_remove_metastore_readonly_huepermission' is marked for no-dry-run.
 > useradmin:0004_add_field_UserProfile_first_login
 > useradmin:0005_auto__add_field_userprofile_last_activity
 > useradmin:0006_auto__add_index_userprofile_last_activity
 - Loading initial data for useradmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for notebook:
 - Migrating forwards to 0001_initial.
 > notebook:0001_initial
 - Loading initial data for notebook.
Installed 0 object(s) from 0 fixture(s)

Thanks, Srinivasa

[#3]

I have encountered multiple errors/warnings in Hue Browser:

Checking current configuration

Configuration files located in /usr/local/hue/desktop/conf 

Potential misconfiguration detected. Fix and restart Hue.

HBase BrowserThe application won't work without a running HBase Thrift Server v1.ImpalaNo available Impalad to send queries to.SparkThe app won't work without a running Livy Spark Server

I cannot install individual app examples (Impala, Solr and HBase)

Could not install table: Error creating table sample_07: Could not connect to localhost:21050 (code THRIFTTRANSPORT): TTransportException('Could not connect to localhost:21050',).

×Instance directory /usr/local/hue/desktop/libs/indexer/src/indexer/../data/collections/twitter_demo already exists! Please remove it from the file system.

×Could not connect to xx9090 (code THRIFTTRANSPORT): TTransportException('Could not connect to ec2-xxx.compute-1.amazonaws.com:9090',)

Please advise. thanks

[#4 · Accepted solution]

@ibhatt's recommendations helped fix the db migration issue. Strangely, his reply has disappeared from this post.

Many thanks.