Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
kaliyugantagoni
kaliyugantagoni
Expert Contributor

Connection to MetaStore refused

by Expert Contributor kaliyugantagoni in Support Questions
‎12-04-2017 03:48 PM
‎12-04-2017 03:48 PM
HDP sandbox 2.6.3 on my local Windows 7 machine. The Hive services are running fine. Using the shell or via Ambari views, I am able to create and view tables: Hive_services_running.JPG The entry in the hosts file: 127.0.0.1 localhost sandbox.hortonworks.com sandbox-hdp.hortonworks.com sandbox-hdf.hortonworks.com I downloaded the client configs. using Ambari, now, I pass the hive-site.xml to the below code that I am trying to execute: private static void connectHiveMetastore(String clientConfigHiveSiteAbsolutePath) throws MetaException, MalformedURLException { System.setProperty("hadoop.home.dir", "E:\Software\Virtualization"); Configuration configuration = new Configuration(); /*Start : Commented or un-commented, immaterial ...*/ configuration.addResource(clientConfigHiveSiteAbsolutePath); /*End : Commented or un-commented, immaterial ...*/ HiveConf hiveConf = new HiveConf(configuration,Configuration.class); /*Start : Commented or un-commented, immaterial ...*/ URL url = new File(clientConfigHiveSiteAbsolutePath).toURI().toURL(); HiveConf.setHiveSiteLocation(url); hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS,"thrift://sandbox-hdp.hortonworks.com:9083"); /*End : Commented or un-commented, immaterial ...*/ HiveMetaStoreClient hiveMetaStoreClient = new HiveMetaStoreClient(hiveConf); System.out.println("****************************Metastore client : "+hiveMetaStoreClient); System.out.println("****************************Is local metastore ? "+hiveMetaStoreClient.isLocalMetaStore()); System.out.println(hiveMetaStoreClient.getAllDatabases()); hiveMetaStoreClient.close(); } It's unable to connect to the Hive metastore: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/E:/Development/lib/hdp/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/E:/Development/lib/hdp/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2017-12-04 16:36:53,809 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(443)) - Trying to connect to metastore with URI thrift://sandbox-hdp.hortonworks.com:9083 2017-12-04 16:36:54,893 WARN [main] hive.metastore (HiveMetaStoreClient.java:open(495)) - Failed to connect to the MetaStore Server... 2017-12-04 16:36:54,893 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(526)) - Waiting 1 seconds before next connection attempt. 2017-12-04 16:36:55,893 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(443)) - Trying to connect to metastore with URI thrift://sandbox-hdp.hortonworks.com:9083 2017-12-04 16:36:56,894 WARN [main] hive.metastore (HiveMetaStoreClient.java:open(495)) - Failed to connect to the MetaStore Server... 2017-12-04 16:36:56,894 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(526)) - Waiting 1 seconds before next connection attempt. 2017-12-04 16:36:57,894 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(443)) - Trying to connect to metastore with URI thrift://sandbox-hdp.hortonworks.com:9083 2017-12-04 16:36:58,904 WARN [main] hive.metastore (HiveMetaStoreClient.java:open(495)) - Failed to connect to the MetaStore Server... 2017-12-04 16:36:58,904 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(526)) - Waiting 1 seconds before next connection attempt. Exception in thread "main" MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:226) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:188) at com.client.MetaStoreWindowsClient.connectHiveMetastore(MetaStoreWindowsClient.java:38) at com.client.MetaStoreWindowsClient.main(MetaStoreWindowsClient.java:18) Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ... 5 more ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:188) at com.client.MetaStoreWindowsClient.connectHiveMetastore(MetaStoreWindowsClient.java:38) at com.client.MetaStoreWindowsClient.main(MetaStoreWindowsClient.java:18) Process finished with exit code 1 What am I missing? ... View more
Labels:

Create PK failing on sandbox

by Expert Contributor kaliyugantagoni in Support Questions
‎12-04-2017 02:45 PM
‎12-04-2017 02:45 PM
HDP sandbox 2.6.3 on my local Windows 7 machine. From the beeline shell/hive view from Ambari, I am trying to create a table with a primary key: create table department(Id int COMMENT 'Surrogate PK is not fun', Description string, Code string, primary key(Id) disable novalidate); The above query works fine with our dev./prod Hive set-up but when I execute it on the my local sandbox: 0: jdbc:hive2://sandbox-hdp.hortonworks.com:2> create table department(Id int COMMENT 'Surrogate PK is not fun', Description string, Code string, primary key(Id) disable novalidate); Error: Error while compiling statement: FAILED: ParseException line 1:107 cannot recognize input near 'key' '(' 'Id' in column type (state=42000,code=40000) Do I need to change any Hive config(via Ambari) for the PK and FK to work? ... View more
Labels:

Re: Services not starting on HDP sandbox 2.6.3

by Expert Contributor kaliyugantagoni in Support Questions
‎12-04-2017 10:05 AM
‎12-04-2017 10:05 AM
That worked 🙂 Never faced that issue in the previous versions of the sandbox, is this a new post-installation step or a sporadic, package-related error or something else? ... View more

Services not starting on HDP sandbox 2.6.3

by Expert Contributor kaliyugantagoni in Support Questions
‎12-04-2017 08:13 AM
‎12-04-2017 08:13 AM
For the host machine config. and other screenshots, please refer the background thread . The network is 'NAT'. Note that the 'Bridged Adapter' network setting doesn't work - I can neither access Ambari or the VM via putty. I am able to log-in Ambari at http://localhost:8080, also, via PuttY: [root@sandbox-hdp ~]# [root@sandbox-hdp ~]# ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02 inet addr:172.17.0.2 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:741159 errors:0 dropped:0 overruns:0 frame:0 TX packets:535534 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:288135779 (274.7 MiB) TX bytes:351577113 (335.2 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:40787371 errors:0 dropped:0 overruns:0 frame:0 TX packets:40787371 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:27903918067 (25.9 GiB) TX bytes:27903918067 (25.9 GiB) On Ambari log-in, all the services were stopped, including HDFS. I tried starting the HDFS but received errors: stderr: /var/lib/ambari-agent/data/errors-489.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 73, in <module> HdfsClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 35, in install import params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params.py", line 25, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py", line 391, in <module> lzo_packages = get_lzo_packages(stack_version_unformatted) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_lzo_packages.py", line 45, in get_lzo_packages lzo_packages += [script_instance.format_package_name("hadooplzo_${stack_version}"), File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 538, in format_package_name raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name, self.available_packages_in_repos)) resource_management.core.exceptions.Fail: Cannot match package for regexp name hadooplzo_${stack_version}. Available packages: ['atlas-metadata_2_6_3_0_235', 'atlas-metadata_2_6_3_0_235-falcon-plugin', 'atlas-metadata_2_6_3_0_235-hive-plugin', 'atlas-metadata_2_6_3_0_235-sqoop-plugin', 'atlas-metadata_2_6_3_0_235-storm-plugin', 'bigtop-jsvc', 'bigtop-tomcat', 'datafu_2_6_3_0_235', 'falcon_2_6_3_0_235', 'flume_2_6_3_0_235', 'hadoop_2_6_3_0_235', 'hadoop_2_6_3_0_235-client', 'hadoop_2_6_3_0_235-hdfs', 'hadoop_2_6_3_0_235-libhdfs', 'hadoop_2_6_3_0_235-mapreduce', 'hadoop_2_6_3_0_235-yarn', 'hbase_2_6_3_0_235', 'hdp-select', 'hive2_2_6_3_0_235', 'hive2_2_6_3_0_235-jdbc', 'hive_2_6_3_0_235', 'hive_2_6_3_0_235-hcatalog', 'hive_2_6_3_0_235-jdbc', 'hive_2_6_3_0_235-webhcat', 'hue', 'hue-beeswax', 'hue-common', 'hue-hcatalog', 'hue-oozie', 'hue-pig', 'hue-server', 'kafka_2_6_3_0_235', 'knox_2_6_3_0_235', 'livy2_2_6_3_0_235', 'oozie_2_6_3_0_235', 'oozie_2_6_3_0_235-client', 'oozie_2_6_3_0_235-common', 'oozie_2_6_3_0_235-sharelib', 'oozie_2_6_3_0_235-sharelib-distcp', 'oozie_2_6_3_0_235-sharelib-hcatalog', 'oozie_2_6_3_0_235-sharelib-hive', 'oozie_2_6_3_0_235-sharelib-hive2', 'oozie_2_6_3_0_235-sharelib-mapreduce-streaming', 'oozie_2_6_3_0_235-sharelib-pig', 'oozie_2_6_3_0_235-sharelib-spark', 'oozie_2_6_3_0_235-sharelib-sqoop', 'oozie_2_6_3_0_235-webapp', 'phoenix_2_6_3_0_235', 'pig_2_6_3_0_235', 'ranger_2_6_3_0_235-admin', 'ranger_2_6_3_0_235-atlas-plugin', 'ranger_2_6_3_0_235-hbase-plugin', 'ranger_2_6_3_0_235-hdfs-plugin', 'ranger_2_6_3_0_235-hive-plugin', 'ranger_2_6_3_0_235-kafka-plugin', 'ranger_2_6_3_0_235-kms', 'ranger_2_6_3_0_235-knox-plugin', 'ranger_2_6_3_0_235-solr-plugin', 'ranger_2_6_3_0_235-storm-plugin', 'ranger_2_6_3_0_235-tagsync', 'ranger_2_6_3_0_235-usersync', 'ranger_2_6_3_0_235-yarn-plugin', 'shc_2_6_3_0_235', 'slider_2_6_3_0_235', 'spark2_2_6_3_0_235', 'spark2_2_6_3_0_235-python', 'spark2_2_6_3_0_235-yarn-shuffle', 'spark_2_6_3_0_235', 'spark_2_6_3_0_235-python', 'spark_2_6_3_0_235-yarn-shuffle', 'spark_llap_2_6_3_0_235', 'sqoop_2_6_3_0_235', 'storm_2_6_3_0_235', 'storm_2_6_3_0_235-slider-client', 'tez_2_6_3_0_235', 'tez_hive2_2_6_3_0_235', 'zeppelin_2_6_3_0_235', 'zookeeper_2_6_3_0_235', 'zookeeper_2_6_3_0_235-server', 'extjs'] stdout: /var/lib/ambari-agent/data/output-489.txt 2017-12-04 08:01:38,824 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=None -> 2.6 2017-12-04 08:01:38,825 - Using hadoop conf dir: /usr/hdp/2.6.3.0-235/hadoop/conf 2017-12-04 08:01:38,826 - Group['livy'] {} 2017-12-04 08:01:38,829 - Group['spark'] {} 2017-12-04 08:01:38,830 - Group['ranger'] {} 2017-12-04 08:01:38,830 - Group['hdfs'] {} 2017-12-04 08:01:38,830 - Group['zeppelin'] {} 2017-12-04 08:01:38,830 - Group['hadoop'] {} 2017-12-04 08:01:38,830 - Group['users'] {} 2017-12-04 08:01:38,831 - Group['knox'] {} 2017-12-04 08:01:38,831 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,832 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,835 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,835 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,836 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None} 2017-12-04 08:01:38,837 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,839 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None} 2017-12-04 08:01:38,840 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger'], 'uid': None} 2017-12-04 08:01:38,841 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None} 2017-12-04 08:01:38,842 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['zeppelin', 'hadoop'], 'uid': None} 2017-12-04 08:01:38,843 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,844 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,847 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users'], 'uid': None} 2017-12-04 08:01:38,848 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,849 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,850 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs'], 'uid': None} 2017-12-04 08:01:38,852 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,853 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,853 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,857 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,860 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,861 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2017-12-04 08:01:38,864 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-12-04 08:01:38,868 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2017-12-04 08:01:38,891 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2017-12-04 08:01:38,891 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2017-12-04 08:01:38,892 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-12-04 08:01:38,896 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-12-04 08:01:38,897 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2017-12-04 08:01:38,919 - call returned (0, '1002') 2017-12-04 08:01:38,920 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1002'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2017-12-04 08:01:38,940 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1002'] due to not_if 2017-12-04 08:01:38,940 - Group['hdfs'] {} 2017-12-04 08:01:38,941 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hdfs']} 2017-12-04 08:01:38,941 - FS Type: 2017-12-04 08:01:38,941 - Directory['/etc/hadoop'] {'mode': 0755} 2017-12-04 08:01:38,963 - File['/usr/hdp/2.6.3.0-235/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2017-12-04 08:01:38,963 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2017-12-04 08:01:38,985 - Repository['HDP-2.6-repo-1'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None} 2017-12-04 08:01:38,997 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-2.6-repo-1]\nname=HDP-2.6-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0\n\npath=/\nenabled=1\ngpgcheck=0'} 2017-12-04 08:01:38,999 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match 2017-12-04 08:01:39,000 - Repository['HDP-UTILS-1.1.0.21-repo-1'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None} 2017-12-04 08:01:39,006 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-2.6-repo-1]\nname=HDP-2.6-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-1.1.0.21-repo-1]\nname=HDP-UTILS-1.1.0.21-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6\n\npath=/\nenabled=1\ngpgcheck=0'} 2017-12-04 08:01:39,007 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match 2017-12-04 08:01:39,007 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-12-04 08:01:39,200 - Skipping installation of existing package unzip 2017-12-04 08:01:39,200 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-12-04 08:01:39,316 - Skipping installation of existing package curl 2017-12-04 08:01:39,316 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5} 2017-12-04 08:01:39,420 - Skipping installation of existing package hdp-select 2017-12-04 08:01:39,422 - The repository with version 2.6.3.0-235 for this command has been marked as resolved. It will be used to report the version of the component which was installed 2017-12-04 08:01:39,739 - Using hadoop conf dir: /usr/hdp/2.6.3.0-235/hadoop/conf 2017-12-04 08:01:39,750 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=None -> 2.6 2017-12-04 08:01:39,754 - Using hadoop conf dir: /usr/hdp/2.6.3.0-235/hadoop/conf 2017-12-04 08:01:39,760 - Command repositories: HDP-2.6-repo-1, HDP-UTILS-1.1.0.21-repo-1 2017-12-04 08:01:39,760 - Applicable repositories: HDP-2.6-repo-1, HDP-UTILS-1.1.0.21-repo-1 2017-12-04 08:01:39,769 - Looking for matching packages in the following repositories: HDP-2.6-repo-1, HDP-UTILS-1.1.0.21-repo-1 2017-12-04 08:02:41,328 - No package found for hadooplzo_${stack_version}(hadooplzo_(\d|_)+$) 2017-12-04 08:02:41,329 - The repository with version 2.6.3.0-235 for this command has been marked as resolved. It will be used to report the version of the component which was installed Command failed after 1 tries ... View more
Labels:

Re: HDP sandbox 2.6.3 not starting

by Expert Contributor kaliyugantagoni in Support Questions
‎11-29-2017 01:08 PM
‎11-29-2017 01:08 PM
I have already verified the checksum. The settings have more than 14 GB of memory(check virtualbox-sandbox-config.jpg) ... View more

HiveMetaStoreClient fails to connect to a Kerberiz...

by Expert Contributor kaliyugantagoni in Support Questions
‎11-28-2017 02:08 PM
‎11-28-2017 02:08 PM
Kerberized HDP-2.6.3.0. **********Start-Edit-1********** I commented the created an 'uber' jar and executed the above class directly on one of the metastore servers(Linux machines), yet, the error persists. **********End-Edit-1********** I have a test code running on my local Windows 7 machine. Note the commented code, as well, that's not making a difference. <code>private static void connectHiveMetastore() throws MetaException, MalformedURLException { System.setProperty("hadoop.home.dir", "E:\\Development\\Software\\Virtualization"); /*Start : Commented or un-commented, immaterial ...*/ System.setProperty("javax.security.auth.useSubjectCredsOnly","false"); System.setProperty("java.security.auth.login.config","E:\\Development\\lib\\hdp\\loginconf.ini"); System.setProperty("java.security.krb5.conf","E:\\Development\\lib\\hdp\\krb5.conf"); /*End : Commented or un-commented, immaterial ...*/ Configuration configuration = new Configuration(); /*Start : Commented or un-commented, immaterial ...*/ //configuration.addResource("E:\\Development\\lib\\hdp\\client_config\\HDFS_CLIENT\\core-site.xml"); //configuration.addResource("E:\\Development\\lib\\hdp\\client_config\\HDFS_CLIENT\\hdfs-site.xml"); //configuration.addResource("E:\\Development\\lib\\hdp\\client_config\\HIVE_CLIENT\\hive-site.xml"); //configuration.set("hive.server2.authentication","KERBEROS"); //configuration.set("hadoop.security.authentication", "Kerberos"); /*End : Commented or un-commented, immaterial ...*/ HiveConf hiveConf = new HiveConf(configuration,Configuration.class); /*Start : Commented or un-commented, immaterial ...*/ /*URL url = new File("E:\\Development\\lib\\hdp\\client_config\\HIVE_CLIENT\\hive-site.xml").toURI().toURL(); HiveConf.setHiveSiteLocation(url);*/ //hiveConf.setVar(HiveConf.ConfVars.HIVE_SERVER2_AUTHENTICATION,"KERBEROS"); /*End : Commented or un-commented, immaterial ...*/ hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS,"thrift://l4283t.sss.se.com:9083,thrift://l4284t.sss.se.com:9083"); HiveMetaStoreClient hiveMetaStoreClient = new HiveMetaStoreClient(hiveConf); System.out.println("Metastore client : "+hiveMetaStoreClient); System.out.println("Is local metastore ? "+hiveMetaStoreClient.isLocalMetaStore()); System.out.println(hiveMetaStoreClient.getAllDatabases()); hiveMetaStoreClient.close(); } The output I receive on the local machine(Note 'Is local metastore ? false' 😞 <code>SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/E:/Development/lib/hdp/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/E:/Development/lib/hdp/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2017-11-28 14:20:14,631 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(443)) - Trying to connect to metastore with URI thrift://l4284t.sss.se.com:9083 2017-11-28 14:20:14,873 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-11-28 14:20:15,005 WARN [main] security.ShellBasedUnixGroupsMapping (ShellBasedUnixGroupsMapping.java:getUnixGroups(87)) - got exception trying to get groups for user ojoqcu: Incorrect command line arguments. 2017-11-28 14:20:15,042 WARN [main] hive.metastore (HiveMetaStoreClient.java:open(511)) - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3802) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3788) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:503) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:188) at MetaStoreClient.connectHiveMetastore(MetaStoreClient.java:47) at MetaStoreClient.main(MetaStoreClient.java:14) 2017-11-28 14:20:15,045 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(539)) - Connected to metastore. Metastore client : org.apache.hadoop.hive.metastore.HiveMetaStoreClient@5fdcaa40 Is local metastore ? false Exception in thread "main" MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null) at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1256) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1129) at MetaStoreClient.connectHiveMetastore(MetaStoreClient.java:51) at MetaStoreClient.main(MetaStoreClient.java:14) 2017-11-28 14:20:15,057 ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1254)) - Got exception: org.apache.thrift.transport.TTransportException null org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:771) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:759) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1127) at MetaStoreClient.connectHiveMetastore(MetaStoreClient.java:51) at MetaStoreClient.main(MetaStoreClient.java:14) 2017-11-28 14:20:15,058 ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1255)) - Converting exception to MetaException Process finished with exit code 1 The metastore log throws: <code>java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1846) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Invalid status -128 at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) In the client-configs(which I downloaded from the cluster via Ambari) as well as the /etc/hive2/conf/hive-site.xml, /etc/hive/conf/conf.server/hive-site.xml, hive-metastore/conf/hive-site.xml have the same following property: <code><property> <name>hive.metastore.sasl.enabled</name> <value>true</value> </property> <property> <name>hive.server2.thrift.sasl.qop</name> <value>auth</value> </property> For reference, providing relevant parts of the client config hive-site.xml: <code><configuration> <property> <name>ambari.hive.db.schema.name</name> <value>hive_metastore</value> </property> <property> <name>atlas.rest.address</name> <value>https://l4284t.sss.se.com:21443</value> </property> <property> <name>hive.server2.thrift.http.path</name> <value>cliservice</value> </property> <property> <name>hive.server2.thrift.http.port</name> <value>10001</value> </property> <property> <name>hive.server2.thrift.max.worker.threads</name> <value>500</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10000</value> </property> <property> <name>hive.server2.thrift.sasl.qop</name> <value>auth</value> </property> <property> <name>hive.server2.transport.mode</name> <value>http</value> </property> <property> <name>hive.server2.use.SSL</name> <value>false</value> </property> <property> <name>hive.metastore.kerberos.keytab.file</name> <value>/etc/security/keytabs/hive.service.keytab</value> </property> <property> <name>hive.metastore.kerberos.principal</name> <value>hive/_HOST@GLOBAL.SCD.COM</value> </property> <property> <name>hive.metastore.pre.event.listeners</name> <value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value> </property> <property> <name>hive.metastore.sasl.enabled</name> <value>true</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://l4283t.sss.se.com:9083,thrift://l4284t.sss.se.com:9083</value> </property> <property> <name>hive.security.metastore.authenticator.manager</name> <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value> </property> <property> <name>hive.server2.authentication</name> <value>KERBEROS</value> </property> <property> <name>hive.server2.authentication.kerberos.keytab</name> <value>/etc/security/keytabs/hive.service.keytab</value> </property> <property> <name>hive.server2.authentication.kerberos.principal</name> <value>hive/_HOST@GLOBAL.SCD.COM</value> </property> <property> <name>hive.server2.authentication.spnego.keytab</name> <value>/etc/security/keytabs/spnego.service.keytab</value> </property> <property> <name>hive.server2.authentication.spnego.principal</name> <value>HTTP/_HOST@GLOBAL.SCD.COM</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.keystore.path</name> <value>/etc/security/ssl/default-key.jks</value> </property> <property> <name>hive.server2.logging.operation.enabled</name> <value>true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://l4373t.sss.se.com/hive_metastore</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> </configuration> Hive database connection via jdbc using the following lines of code works: <code>System.setProperty("javax.security.auth.useSubjectCredsOnly","false"); System.setProperty("java.security.auth.login.config","E:\\Development\\lib\\hdp\\loginconf.ini"); System.setProperty("java.security.krb5.conf","E:\\Development\\lib\\hdp\\krb5.conf"); ... View more
Labels:

HDP sandbox 2.6.3 not starting

by Expert Contributor kaliyugantagoni in Support Questions
‎11-28-2017 07:58 AM
1 Kudo
‎11-28-2017 07:58 AM
1 Kudo
I tried running sandbox 2.6.3 on both VMware and VirtualBox with a RAM > 14GB, after referring the following threads: https://community.hortonworks.com/questions/103239/hdp-sanbbox-26-not-working-in-windows-10.html https://community.hortonworks.com/questions/103261/unable-to-start-hdp-26-sandbox.html https://community.hortonworks.com/questions/135835/cant-start-sandbox-261.html https://community.hortonworks.com/questions/57757/hdp-25-sandbox-not-starting.htmlIt's stuck at 'Starting Sandbox container'. The VirtualBox log(VBox.log 😞 VirtualBox VM 5.1.30 r118389 win.amd64 (Oct 16 2017 10:47:00) release log 00:00:00.846301 Log opened 2017-11-28T07:40:53.297635200Z 00:00:00.846302 Build Type: release 00:00:00.846304 OS Product: Windows 7 00:00:00.846306 OS Release: 6.1.7601 00:00:00.846307 OS Service Pack: 1 00:00:00.910929 DMI Product Name: Precision 3510 00:00:00.913338 DMI Product Version: 00:00:00.913344 Host RAM: 16046MB (15.6GB) total, 7470MB (7.2GB) available 00:00:00.913348 Executable: C:\Program Files\Oracle\VirtualBox\VirtualBox.exe 00:00:00.913348 Process ID: 9412 00:00:00.913349 Package type: WINDOWS_64BITS_GENERIC 00:00:00.913904 Installed Extension Packs: 00:00:00.913944 None installed! 00:00:00.914658 Console: Machine state changed to 'Starting' 00:00:00.914854 Qt version: 5.6.2 00:00:00.915539 VRDE: VirtualBox Remote Desktop Extension is not available. 00:00:00.916653 GUI: UIMediumEnumerator: Medium-enumeration finished! 00:00:01.125401 SUP: Loaded VMMR0.r0 (C:\Program Files\Oracle\VirtualBox\VMMR0.r0) at 0xXXXXXXXXXXXXXXXX - ModuleInit at XXXXXXXXXXXXXXXX and ModuleTerm at XXXXXXXXXXXXXXXX using the native ring-0 loader 00:00:01.125430 SUP: VMMR0EntryEx located at XXXXXXXXXXXXXXXX and VMMR0EntryFast at XXXXXXXXXXXXXXXX 00:00:01.125435 SUP: windbg> .reload /f C:\Program Files\Oracle\VirtualBox\VMMR0.r0=0xXXXXXXXXXXXXXXXX 00:00:01.128704 Guest OS type: 'RedHat_64' 00:00:01.129367 fHMForced=true - Lots of RAM 00:00:01.129376 fHMForced=true - SMP 00:00:01.129380 fHMForced=true - 64-bit guest 00:00:01.130563 File system of 'E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP\Snapshots' (snapshots) is unknown 00:00:01.130579 File system of 'E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP\Hortonworks Docker Sandbox HDP-disk1.vmdk' is ntfs 00:00:01.159494 Shared clipboard service loaded 00:00:01.159511 Shared clipboard mode: Off 00:00:01.183303 Drag and drop service loaded 00:00:01.183322 Drag and drop mode: Off 00:00:01.230906 Guest Control service loaded 00:00:01.232679 ************************* CFGM dump ************************* 00:00:01.232681 [/] (level 0) 00:00:01.232685 CSAMEnabled <integer> = 0x0000000000000001 (1) 00:00:01.232687 CpuExecutionCap <integer> = 0x0000000000000064 (100) 00:00:01.232688 EnablePAE <integer> = 0x0000000000000001 (1) 00:00:01.232689 HMEnabled <integer> = 0x0000000000000001 (1) 00:00:01.232690 MemBalloonSize <integer> = 0x0000000000000000 (0) 00:00:01.232690 Name <string> = "Hortonworks Docker Sandbox HDP" (cb=31) 00:00:01.232691 NumCPUs <integer> = 0x0000000000000004 (4) 00:00:01.232692 PATMEnabled <integer> = 0x0000000000000001 (1) 00:00:01.232693 PageFusionAllowed <integer> = 0x0000000000000000 (0) 00:00:01.232694 RamHoleSize <integer> = 0x0000000020000000 (536 870 912, 512 MB) 00:00:01.232696 RamSize <integer> = 0x000000037dc00000 (14 994 636 800, 14 300 MB, 13 GB) 00:00:01.232698 RawR0Enabled <integer> = 0x0000000000000001 (1) 00:00:01.232698 RawR3Enabled <integer> = 0x0000000000000001 (1) 00:00:01.232699 TimerMillies <integer> = 0x000000000000000a (10) 00:00:01.232700 UUID <bytes> = "aa 65 5f 19 b4 25 6e 4f 90 dd 62 ec c2 b2 86 78" (cb=16) 00:00:01.232702 00:00:01.232703 [/CPUM/] (level 1) 00:00:01.232704 GuestCpuName <string> = "host" (cb=5) 00:00:01.232705 PortableCpuIdLevel <integer> = 0x0000000000000000 (0) 00:00:01.232705 00:00:01.232705 [/DBGC/] (level 1) 00:00:01.232706 GlobalInitScript <string> = "C:\Users\ojoqcu\.VirtualBox/dbgc-init" (cb=38) 00:00:01.232707 HistoryFile <string> = "C:\Users\ojoqcu\.VirtualBox/dbgc-history" (cb=41) 00:00:01.232708 LocalInitScript <string> = "E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP/dbgc-init" (cb=99) 00:00:01.232709 00:00:01.232709 [/DBGF/] (level 1) 00:00:01.232709 Path <string> = "E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP/debug/;E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP/;C:\Users\ojoqcu\" (cb=203) 00:00:01.232710 00:00:01.232711 [/Devices/] (level 1) 00:00:01.232711 00:00:01.232712 [/Devices/8237A/] (level 2) 00:00:01.232712 00:00:01.232713 [/Devices/8237A/0/] (level 3) 00:00:01.232714 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232714 00:00:01.232715 [/Devices/GIMDev/] (level 2) 00:00:01.232715 00:00:01.232716 [/Devices/GIMDev/0/] (level 3) 00:00:01.232716 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232717 00:00:01.232717 [/Devices/VMMDev/] (level 2) 00:00:01.232718 00:00:01.232718 [/Devices/VMMDev/0/] (level 3) 00:00:01.232719 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.232720 PCIDeviceNo <integer> = 0x0000000000000004 (4) 00:00:01.232721 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.232721 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232722 00:00:01.232722 [/Devices/VMMDev/0/Config/] (level 4) 00:00:01.232723 GuestCoreDumpDir <string> = "E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP\Snapshots" (cb=99) 00:00:01.232724 00:00:01.232724 [/Devices/VMMDev/0/LUN#0/] (level 4) 00:00:01.232725 Driver <string> = "HGCM" (cb=5) 00:00:01.232726 00:00:01.232726 [/Devices/VMMDev/0/LUN#0/Config/] (level 5) 00:00:01.232727 Object <integer> = 0x0000000004273b10 (69 679 888) 00:00:01.232728 00:00:01.232728 [/Devices/VMMDev/0/LUN#999/] (level 4) 00:00:01.232729 Driver <string> = "MainStatus" (cb=11) 00:00:01.232730 00:00:01.232730 [/Devices/VMMDev/0/LUN#999/Config/] (level 5) 00:00:01.232731 First <integer> = 0x0000000000000000 (0) 00:00:01.232732 Last <integer> = 0x0000000000000000 (0) 00:00:01.232732 papLeds <integer> = 0x0000000003fdf4e8 (66 974 952) 00:00:01.232733 00:00:01.232733 [/Devices/acpi/] (level 2) 00:00:01.232734 00:00:01.232734 [/Devices/acpi/0/] (level 3) 00:00:01.232735 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.232736 PCIDeviceNo <integer> = 0x0000000000000007 (7) 00:00:01.232737 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.232737 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232738 00:00:01.232738 [/Devices/acpi/0/Config/] (level 4) 00:00:01.232739 CpuHotPlug <integer> = 0x0000000000000000 (0) 00:00:01.232740 FdcEnabled <integer> = 0x0000000000000000 (0) 00:00:01.232741 HostBusPciAddress <integer> = 0x0000000000000000 (0) 00:00:01.232741 HpetEnabled <integer> = 0x0000000000000000 (0) 00:00:01.232742 IOAPIC <integer> = 0x0000000000000001 (1) 00:00:01.232743 IocPciAddress <integer> = 0x0000000000010000 (65 536) 00:00:01.232744 NumCPUs <integer> = 0x0000000000000004 (4) 00:00:01.232744 Parallel0IoPortBase <integer> = 0x0000000000000000 (0) 00:00:01.232745 Parallel0Irq <integer> = 0x0000000000000000 (0) 00:00:01.232746 Parallel1IoPortBase <integer> = 0x0000000000000000 (0) 00:00:01.232746 Parallel1Irq <integer> = 0x0000000000000000 (0) 00:00:01.232747 Serial0IoPortBase <integer> = 0x0000000000000000 (0) 00:00:01.232748 Serial0Irq <integer> = 0x0000000000000000 (0) 00:00:01.232748 Serial1IoPortBase <integer> = 0x0000000000000000 (0) 00:00:01.232749 Serial1Irq <integer> = 0x0000000000000000 (0) 00:00:01.232749 ShowCpu <integer> = 0x0000000000000001 (1) 00:00:01.232750 ShowRtc <integer> = 0x0000000000000000 (0) 00:00:01.232751 SmcEnabled <integer> = 0x0000000000000000 (0) 00:00:01.232751 00:00:01.232752 [/Devices/acpi/0/LUN#0/] (level 4) 00:00:01.232753 Driver <string> = "ACPIHost" (cb=9) 00:00:01.232753 00:00:01.232753 [/Devices/acpi/0/LUN#0/Config/] (level 5) 00:00:01.232754 00:00:01.232755 [/Devices/acpi/0/LUN#1/] (level 4) 00:00:01.232755 Driver <string> = "ACPICpu" (cb=8) 00:00:01.232756 00:00:01.232756 [/Devices/acpi/0/LUN#1/Config/] (level 5) 00:00:01.232757 00:00:01.232757 [/Devices/acpi/0/LUN#2/] (level 4) 00:00:01.232758 Driver <string> = "ACPICpu" (cb=8) 00:00:01.232759 00:00:01.232759 [/Devices/acpi/0/LUN#2/Config/] (level 5) 00:00:01.232760 00:00:01.232760 [/Devices/acpi/0/LUN#3/] (level 4) 00:00:01.232761 Driver <string> = "ACPICpu" (cb=8) 00:00:01.232761 00:00:01.232761 [/Devices/acpi/0/LUN#3/Config/] (level 5) 00:00:01.232762 00:00:01.232763 [/Devices/apic/] (level 2) 00:00:01.232763 00:00:01.232764 [/Devices/apic/0/] (level 3) 00:00:01.232764 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232765 00:00:01.232765 [/Devices/apic/0/Config/] (level 4) 00:00:01.232766 IOAPIC <integer> = 0x0000000000000001 (1) 00:00:01.232767 Mode <integer> = 0x0000000000000003 (3) 00:00:01.232767 NumCPUs <integer> = 0x0000000000000004 (4) 00:00:01.232768 00:00:01.232768 [/Devices/e1000/] (level 2) 00:00:01.232769 00:00:01.232769 [/Devices/e1000/0/] (level 3) 00:00:01.232770 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.232771 PCIDeviceNo <integer> = 0x0000000000000003 (3) 00:00:01.232772 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.232772 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.232773 00:00:01.232773 [/Devices/e1000/0/Config/] (level 4) 00:00:01.232775 AdapterType <integer> = 0x0000000000000000 (0) 00:00:01.232776 CableConnected <integer> = 0x0000000000000001 (1) 00:00:01.232776 LineSpeed <integer> = 0x0000000000000000 (0) 00:00:01.232777 MAC <bytes> = "08 00 27 21 8e c8" (cb=6) 00:00:01.232778 00:00:01.232778 [/Devices/e1000/0/LUN#0/] (level 4) 00:00:01.232779 Driver <string> = "NAT" (cb=4) 00:00:01.232780 00:00:01.232780 [/Devices/e1000/0/LUN#0/Config/] (level 5) 00:00:01.232781 AliasMode <integer> = 0x0000000000000000 (0) 00:00:01.232782 BootFile <string> = "Hortonworks Docker Sandbox HDP.pxe" (cb=35) 00:00:01.232783 DNSProxy <integer> = 0x0000000000000000 (0) 00:00:01.232783 Network <string> = "10.0.2.0/24" (cb=12) 00:00:01.232784 PassDomain <integer> = 0x0000000000000001 (1) 00:00:01.232785 TFTPPrefix <string> = "C:\Users\ojoqcu\.VirtualBox\TFTP" (cb=33) 00:00:01.232802 UseHostResolver <integer> = 0x0000000000000000 (0) 00:00:01.232802 00:00:01.232803 [/Devices/e1000/0/LUN#0/Config/PortForwarding/] (level 6) 00:00:01.232804 00:00:01.232804 [/Devices/e1000/0/LUN#0/Config/PortForwarding/0/] (level 7) 00:00:01.232806 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232806 GuestPort <integer> = 0x000000000000c3af (50 095) 00:00:01.232807 HostPort <integer> = 0x000000000000c3af (50 095) 00:00:01.232808 Name <string> = "Accumulo" (cb=9) 00:00:01.232809 Protocol <string> = "TCP" (cb=4) 00:00:01.232809 00:00:01.232810 [/Devices/e1000/0/LUN#0/Config/PortForwarding/1/] (level 7) 00:00:01.232811 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232811 GuestPort <integer> = 0x00000000000022b6 (8 886) 00:00:01.232812 HostPort <integer> = 0x00000000000022b6 (8 886) 00:00:01.232829 Name <string> = "AmbariInfra" (cb=12) 00:00:01.232829 Protocol <string> = "TCP" (cb=4) 00:00:01.232830 00:00:01.232830 [/Devices/e1000/0/LUN#0/Config/PortForwarding/10/] (level 7) 00:00:01.232831 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232832 GuestPort <integer> = 0x0000000000003a9a (15 002) 00:00:01.232833 HostPort <integer> = 0x0000000000003a9a (15 002) 00:00:01.232834 Name <string> = "Custom15" (cb=9) 00:00:01.232834 Protocol <string> = "TCP" (cb=4) 00:00:01.232835 00:00:01.232835 [/Devices/e1000/0/LUN#0/Config/PortForwarding/11/] (level 7) 00:00:01.232836 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232837 GuestPort <integer> = 0x000000000000006f (111) 00:00:01.232838 HostPort <integer> = 0x000000000000006f (111) 00:00:01.232838 Name <string> = "Custom16" (cb=9) 00:00:01.232839 Protocol <string> = "TCP" (cb=4) 00:00:01.232839 00:00:01.232840 [/Devices/e1000/0/LUN#0/Config/PortForwarding/12/] (level 7) 00:00:01.232841 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232841 GuestPort <integer> = 0x0000000000000801 (2 049) 00:00:01.232842 HostPort <integer> = 0x0000000000000801 (2 049) 00:00:01.232843 Name <string> = "Custom17" (cb=9) 00:00:01.232843 Protocol <string> = "TCP" (cb=4) 00:00:01.232844 00:00:01.232844 [/Devices/e1000/0/LUN#0/Config/PortForwarding/13/] (level 7) 00:00:01.232845 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232846 GuestPort <integer> = 0x0000000000001092 (4 242) 00:00:01.232846 HostPort <integer> = 0x0000000000001092 (4 242) 00:00:01.232847 Name <string> = "Custom18" (cb=9) 00:00:01.232848 Protocol <string> = "TCP" (cb=4) 00:00:01.232848 00:00:01.232849 [/Devices/e1000/0/LUN#0/Config/PortForwarding/14/] (level 7) 00:00:01.232850 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232850 GuestPort <integer> = 0x000000000000c39f (50 079) 00:00:01.232851 HostPort <integer> = 0x000000000000c39f (50 079) 00:00:01.232852 Name <string> = "Custom19" (cb=9) 00:00:01.232852 Protocol <string> = "TCP" (cb=4) 00:00:01.232853 00:00:01.232853 [/Devices/e1000/0/LUN#0/Config/PortForwarding/15/] (level 7) 00:00:01.232854 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232855 GuestPort <integer> = 0x0000000000000bb8 (3 000) 00:00:01.232855 HostPort <integer> = 0x0000000000000bb8 (3 000) 00:00:01.232856 Name <string> = "Custom20" (cb=9) 00:00:01.232856 Protocol <string> = "TCP" (cb=4) 00:00:01.232857 00:00:01.232857 [/Devices/e1000/0/LUN#0/Config/PortForwarding/16/] (level 7) 00:00:01.232858 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232859 GuestPort <integer> = 0x0000000000003e80 (16 000) 00:00:01.232860 HostPort <integer> = 0x0000000000003e80 (16 000) 00:00:01.232860 Name <string> = "Custom21" (cb=9) 00:00:01.232861 Protocol <string> = "TCP" (cb=4) 00:00:01.232861 00:00:01.232862 [/Devices/e1000/0/LUN#0/Config/PortForwarding/17/] (level 7) 00:00:01.232863 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232863 GuestPort <integer> = 0x0000000000003e94 (16 020) 00:00:01.232864 HostPort <integer> = 0x0000000000003e94 (16 020) 00:00:01.232865 Name <string> = "Custom22" (cb=9) 00:00:01.232865 Protocol <string> = "TCP" (cb=4) 00:00:01.232866 00:00:01.232866 [/Devices/e1000/0/LUN#0/Config/PortForwarding/18/] (level 7) 00:00:01.232867 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232867 GuestPort <integer> = 0x0000000000003c8c (15 500) 00:00:01.232868 HostPort <integer> = 0x0000000000003c8c (15 500) 00:00:01.232869 Name <string> = "Custom23" (cb=9) 00:00:01.232869 Protocol <string> = "TCP" (cb=4) 00:00:01.232870 00:00:01.232870 [/Devices/e1000/0/LUN#0/Config/PortForwarding/19/] (level 7) 00:00:01.232871 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232871 GuestPort <integer> = 0x0000000000003c8d (15 501) 00:00:01.232872 HostPort <integer> = 0x0000000000003c8d (15 501) 00:00:01.232873 Name <string> = "Custom24" (cb=9) 00:00:01.232873 Protocol <string> = "TCP" (cb=4) 00:00:01.232874 00:00:01.232874 [/Devices/e1000/0/LUN#0/Config/PortForwarding/2/] (level 7) 00:00:01.232875 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232876 GuestPort <integer> = 0x0000000000001068 (4 200) 00:00:01.232876 HostPort <integer> = 0x0000000000001068 (4 200) 00:00:01.232877 Name <string> = "AmbariShell" (cb=12) 00:00:01.232878 Protocol <string> = "TCP" (cb=4) 00:00:01.232878 00:00:01.232879 [/Devices/e1000/0/LUN#0/Config/PortForwarding/20/] (level 7) 00:00:01.232880 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232880 GuestPort <integer> = 0x0000000000003c8e (15 502) 00:00:01.232881 HostPort <integer> = 0x0000000000003c8e (15 502) 00:00:01.232882 Name <string> = "Custom25" (cb=9) 00:00:01.232882 Protocol <string> = "TCP" (cb=4) 00:00:01.232883 00:00:01.232883 [/Devices/e1000/0/LUN#0/Config/PortForwarding/21/] (level 7) 00:00:01.232884 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232885 GuestPort <integer> = 0x0000000000003c8f (15 503) 00:00:01.232885 HostPort <integer> = 0x0000000000003c8f (15 503) 00:00:01.232886 Name <string> = "Custom26" (cb=9) 00:00:01.232887 Protocol <string> = "TCP" (cb=4) 00:00:01.232887 00:00:01.232887 [/Devices/e1000/0/LUN#0/Config/PortForwarding/22/] (level 7) 00:00:01.232888 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232889 GuestPort <integer> = 0x0000000000003c90 (15 504) 00:00:01.232889 HostPort <integer> = 0x0000000000003c90 (15 504) 00:00:01.232890 Name <string> = "Custom27" (cb=9) 00:00:01.232891 Protocol <string> = "TCP" (cb=4) 00:00:01.232891 00:00:01.232891 [/Devices/e1000/0/LUN#0/Config/PortForwarding/23/] (level 7) 00:00:01.232893 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232893 GuestPort <integer> = 0x0000000000003c91 (15 505) 00:00:01.232894 HostPort <integer> = 0x0000000000003c91 (15 505) 00:00:01.232895 Name <string> = "Custom28" (cb=9) 00:00:01.232895 Protocol <string> = "TCP" (cb=4) 00:00:01.232895 00:00:01.232896 [/Devices/e1000/0/LUN#0/Config/PortForwarding/24/] (level 7) 00:00:01.232897 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232897 GuestPort <integer> = 0x000000000000223d (8 765) 00:00:01.232898 HostPort <integer> = 0x000000000000223d (8 765) 00:00:01.232899 Name <string> = "Custom3" (cb=8) 00:00:01.232899 Protocol <string> = "TCP" (cb=4) 00:00:01.232900 00:00:01.232900 [/Devices/e1000/0/LUN#0/Config/PortForwarding/25/] (level 7) 00:00:01.232901 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232902 GuestPort <integer> = 0x0000000000001f9a (8 090) 00:00:01.232902 HostPort <integer> = 0x0000000000001f9a (8 090) 00:00:01.232903 Name <string> = "Custom4" (cb=8) 00:00:01.232904 Protocol <string> = "TCP" (cb=4) 00:00:01.232904 00:00:01.232904 [/Devices/e1000/0/LUN#0/Config/PortForwarding/26/] (level 7) 00:00:01.232906 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232906 GuestPort <integer> = 0x0000000000001f9b (8 091) 00:00:01.232907 HostPort <integer> = 0x0000000000001f9b (8 091) 00:00:01.232907 Name <string> = "Custom5" (cb=8) 00:00:01.232908 Protocol <string> = "TCP" (cb=4) 00:00:01.232908 00:00:01.232909 [/Devices/e1000/0/LUN#0/Config/PortForwarding/27/] (level 7) 00:00:01.232910 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232910 GuestPort <integer> = 0x0000000000001f45 (8 005) 00:00:01.232911 HostPort <integer> = 0x0000000000001f45 (8 005) 00:00:01.232912 Name <string> = "Custom6" (cb=8) 00:00:01.232912 Protocol <string> = "TCP" (cb=4) 00:00:01.232913 00:00:01.232913 [/Devices/e1000/0/LUN#0/Config/PortForwarding/28/] (level 7) 00:00:01.232914 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232915 GuestPort <integer> = 0x0000000000001f96 (8 086) 00:00:01.232916 HostPort <integer> = 0x0000000000001f96 (8 086) 00:00:01.232916 Name <string> = "Custom7" (cb=8) 00:00:01.232917 Protocol <string> = "TCP" (cb=4) 00:00:01.232917 00:00:01.232918 [/Devices/e1000/0/LUN#0/Config/PortForwarding/29/] (level 7) 00:00:01.232919 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232919 GuestPort <integer> = 0x0000000000001f92 (8 082) 00:00:01.232920 HostPort <integer> = 0x0000000000001f92 (8 082) 00:00:01.232921 Name <string> = "Custom8" (cb=8) 00:00:01.232921 Protocol <string> = "TCP" (cb=4) 00:00:01.232922 00:00:01.232922 [/Devices/e1000/0/LUN#0/Config/PortForwarding/3/] (level 7) 00:00:01.232923 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232924 GuestPort <integer> = 0x0000000000005208 (21 000) 00:00:01.232924 HostPort <integer> = 0x0000000000005208 (21 000) 00:00:01.232925 Name <string> = "Atlas" (cb=6) 00:00:01.232926 Protocol <string> = "TCP" (cb=4) 00:00:01.232926 00:00:01.232926 [/Devices/e1000/0/LUN#0/Config/PortForwarding/30/] (level 7) 00:00:01.232927 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232928 GuestPort <integer> = 0x00000000000046a1 (18 081) 00:00:01.232929 HostPort <integer> = 0x00000000000046a1 (18 081) 00:00:01.232929 Name <string> = "Custom9" (cb=8) 00:00:01.232930 Protocol <string> = "TCP" (cb=4) 00:00:01.232930 00:00:01.232931 [/Devices/e1000/0/LUN#0/Config/PortForwarding/31/] (level 7) 00:00:01.232932 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232932 GuestPort <integer> = 0x000000000000c39b (50 075) 00:00:01.232933 HostPort <integer> = 0x000000000000c39b (50 075) 00:00:01.232934 Name <string> = "Datanode" (cb=9) 00:00:01.232934 Protocol <string> = "TCP" (cb=4) 00:00:01.232935 00:00:01.232935 [/Devices/e1000/0/LUN#0/Config/PortForwarding/32/] (level 7) 00:00:01.232936 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232936 GuestPort <integer> = 0x00000000000008ae (2 222) 00:00:01.232937 HostPort <integer> = 0x00000000000008ae (2 222) 00:00:01.232938 Name <string> = "DockerSSH" (cb=10) 00:00:01.232938 Protocol <string> = "TCP" (cb=4) 00:00:01.232939 00:00:01.232939 [/Devices/e1000/0/LUN#0/Config/PortForwarding/33/] (level 7) 00:00:01.232940 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232940 GuestPort <integer> = 0x0000000000003a98 (15 000) 00:00:01.232941 HostPort <integer> = 0x0000000000003a98 (15 000) 00:00:01.232942 Name <string> = "Falcon" (cb=7) 00:00:01.232942 Protocol <string> = "TCP" (cb=4) 00:00:01.232943 00:00:01.232943 [/Devices/e1000/0/LUN#0/Config/PortForwarding/34/] (level 7) 00:00:01.232944 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232945 GuestPort <integer> = 0x0000000000003e8a (16 010) 00:00:01.232945 HostPort <integer> = 0x0000000000003e8a (16 010) 00:00:01.232946 Name <string> = "HBaseMaster" (cb=12) 00:00:01.232947 Protocol <string> = "TCP" (cb=4) 00:00:01.232947 00:00:01.232947 [/Devices/e1000/0/LUN#0/Config/PortForwarding/35/] (level 7) 00:00:01.232948 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232949 GuestPort <integer> = 0x0000000000003e9e (16 030) 00:00:01.232950 HostPort <integer> = 0x0000000000003e9e (16 030) 00:00:01.232950 Name <string> = "HBaseRegion" (cb=12) 00:00:01.232951 Protocol <string> = "TCP" (cb=4) 00:00:01.232951 00:00:01.232952 [/Devices/e1000/0/LUN#0/Config/PortForwarding/36/] (level 7) 00:00:01.232953 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232953 GuestPort <integer> = 0x0000000000002710 (10 000) 00:00:01.232954 HostPort <integer> = 0x0000000000002710 (10 000) 00:00:01.232955 Name <string> = "HS2" (cb=4) 00:00:01.232955 Protocol <string> = "TCP" (cb=4) 00:00:01.232956 00:00:01.232956 [/Devices/e1000/0/LUN#0/Config/PortForwarding/37/] (level 7) 00:00:01.232957 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232957 GuestPort <integer> = 0x0000000000002711 (10 001) 00:00:01.232958 HostPort <integer> = 0x0000000000002711 (10 001) 00:00:01.232959 Name <string> = "HS2Http" (cb=8) 00:00:01.232959 Protocol <string> = "TCP" (cb=4) 00:00:01.232960 00:00:01.232960 [/Devices/e1000/0/LUN#0/Config/PortForwarding/38/] (level 7) 00:00:01.232961 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232962 GuestPort <integer> = 0x0000000000002904 (10 500) 00:00:01.232962 HostPort <integer> = 0x0000000000002904 (10 500) 00:00:01.232963 Name <string> = "HS2v2" (cb=6) 00:00:01.232963 Protocol <string> = "TCP" (cb=4) 00:00:01.232964 00:00:01.232964 [/Devices/e1000/0/LUN#0/Config/PortForwarding/39/] (level 7) 00:00:01.232965 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232966 GuestPort <integer> = 0x0000000000002328 (9 000) 00:00:01.232967 HostPort <integer> = 0x0000000000002328 (9 000) 00:00:01.232967 Name <string> = "HST" (cb=4) 00:00:01.232968 Protocol <string> = "TCP" (cb=4) 00:00:01.232968 00:00:01.232969 [/Devices/e1000/0/LUN#0/Config/PortForwarding/4/] (level 7) 00:00:01.232970 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232970 GuestPort <integer> = 0x000000000000ea60 (60 000) 00:00:01.232971 HostPort <integer> = 0x000000000000ea60 (60 000) 00:00:01.232972 Name <string> = "Custom1" (cb=8) 00:00:01.232972 Protocol <string> = "TCP" (cb=4) 00:00:01.232973 00:00:01.232973 [/Devices/e1000/0/LUN#0/Config/PortForwarding/40/] (level 7) 00:00:01.232974 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232975 GuestPort <integer> = 0x0000000000000016 (22) 00:00:01.232975 HostPort <integer> = 0x000000000000084a (2 122) 00:00:01.232976 Name <string> = "HostSSH" (cb=8) 00:00:01.232976 Protocol <string> = "TCP" (cb=4) 00:00:01.232977 00:00:01.232977 [/Devices/e1000/0/LUN#0/Config/PortForwarding/41/] (level 7) 00:00:01.232978 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232979 GuestPort <integer> = 0x0000000000004db0 (19 888) 00:00:01.232979 HostPort <integer> = 0x0000000000004db0 (19 888) 00:00:01.232980 Name <string> = "JobHistory" (cb=11) 00:00:01.232981 Protocol <string> = "TCP" (cb=4) 00:00:01.232981 00:00:01.232981 [/Devices/e1000/0/LUN#0/Config/PortForwarding/42/] (level 7) 00:00:01.232983 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232983 GuestPort <integer> = 0x00000000000022b9 (8 889) 00:00:01.232984 HostPort <integer> = 0x00000000000022b9 (8 889) 00:00:01.232984 Name <string> = "Jupyter" (cb=8) 00:00:01.232985 Protocol <string> = "TCP" (cb=4) 00:00:01.232985 00:00:01.232986 [/Devices/e1000/0/LUN#0/Config/PortForwarding/43/] (level 7) 00:00:01.232987 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232987 GuestPort <integer> = 0x00000000000020fb (8 443) 00:00:01.232988 HostPort <integer> = 0x00000000000020fb (8 443) 00:00:01.232989 Name <string> = "Knox" (cb=5) 00:00:01.232989 Protocol <string> = "TCP" (cb=4) 00:00:01.232990 00:00:01.232990 [/Devices/e1000/0/LUN#0/Config/PortForwarding/44/] (level 7) 00:00:01.232991 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232992 GuestPort <integer> = 0x0000000000002382 (9 090) 00:00:01.232992 HostPort <integer> = 0x0000000000002382 (9 090) 00:00:01.232993 Name <string> = "Nifi" (cb=5) 00:00:01.232994 Protocol <string> = "TCP" (cb=4) 00:00:01.232994 00:00:01.232994 [/Devices/e1000/0/LUN#0/Config/PortForwarding/45/] (level 7) 00:00:01.232995 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.232996 GuestPort <integer> = 0x0000000000001f6a (8 042) 00:00:01.232997 HostPort <integer> = 0x0000000000001f6a (8 042) 00:00:01.232997 Name <string> = "NodeManager" (cb=12) 00:00:01.232998 Protocol <string> = "TCP" (cb=4) 00:00:01.232998 00:00:01.232999 [/Devices/e1000/0/LUN#0/Config/PortForwarding/46/] (level 7) 00:00:01.233000 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233000 GuestPort <integer> = 0x0000000000002af8 (11 000) 00:00:01.233001 HostPort <integer> = 0x0000000000002af8 (11 000) 00:00:01.233002 Name <string> = "Oozie" (cb=6) 00:00:01.233002 Protocol <string> = "TCP" (cb=4) 00:00:01.233002 00:00:01.233003 [/Devices/e1000/0/LUN#0/Config/PortForwarding/47/] (level 7) 00:00:01.233004 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233004 GuestPort <integer> = 0x0000000000001f60 (8 032) 00:00:01.233005 HostPort <integer> = 0x0000000000001f60 (8 032) 00:00:01.233006 Name <string> = "RM" (cb=3) 00:00:01.233006 Protocol <string> = "TCP" (cb=4) 00:00:01.233007 00:00:01.233007 [/Devices/e1000/0/LUN#0/Config/PortForwarding/48/] (level 7) 00:00:01.233008 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233008 GuestPort <integer> = 0x0000000000002321 (8 993) 00:00:01.233009 HostPort <integer> = 0x0000000000002321 (8 993) 00:00:01.233010 Name <string> = "Solr" (cb=5) 00:00:01.233010 Protocol <string> = "TCP" (cb=4) 00:00:01.233011 00:00:01.233011 [/Devices/e1000/0/LUN#0/Config/PortForwarding/49/] (level 7) 00:00:01.233012 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233013 GuestPort <integer> = 0x0000000000002317 (8 983) 00:00:01.233013 HostPort <integer> = 0x0000000000002317 (8 983) 00:00:01.233014 Name <string> = "SolrAdmin" (cb=10) 00:00:01.233015 Protocol <string> = "TCP" (cb=4) 00:00:01.233015 00:00:01.233015 [/Devices/e1000/0/LUN#0/Config/PortForwarding/5/] (level 7) 00:00:01.233017 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233017 GuestPort <integer> = 0x000000000000271f (10 015) 00:00:01.233018 HostPort <integer> = 0x000000000000271f (10 015) 00:00:01.233018 Name <string> = "Custom10" (cb=9) 00:00:01.233019 Protocol <string> = "TCP" (cb=4) 00:00:01.233019 00:00:01.233020 [/Devices/e1000/0/LUN#0/Config/PortForwarding/50/] (level 7) 00:00:01.233021 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233021 GuestPort <integer> = 0x0000000000000fc8 (4 040) 00:00:01.233022 HostPort <integer> = 0x0000000000000fc8 (4 040) 00:00:01.233023 Name <string> = "Spark" (cb=6) 00:00:01.233023 Protocol <string> = "TCP" (cb=4) 00:00:01.233024 00:00:01.233024 [/Devices/e1000/0/LUN#0/Config/PortForwarding/51/] (level 7) 00:00:01.233025 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233026 GuestPort <integer> = 0x00000000000046a0 (18 080) 00:00:01.233026 HostPort <integer> = 0x00000000000046a0 (18 080) 00:00:01.233027 Name <string> = "SparkHistoryServer" (cb=19) 00:00:01.233028 Protocol <string> = "TCP" (cb=4) 00:00:01.233028 00:00:01.233028 [/Devices/e1000/0/LUN#0/Config/PortForwarding/52/] (level 7) 00:00:01.233029 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233030 GuestPort <integer> = 0x0000000000002228 (8 744) 00:00:01.233031 HostPort <integer> = 0x0000000000002228 (8 744) 00:00:01.233031 Name <string> = "StormUI" (cb=8) 00:00:01.233032 Protocol <string> = "TCP" (cb=4) 00:00:01.233032 00:00:01.233032 [/Devices/e1000/0/LUN#0/Config/PortForwarding/53/] (level 7) 00:00:01.233034 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233034 GuestPort <integer> = 0x00000000000022b8 (8 888) 00:00:01.233035 HostPort <integer> = 0x00000000000022b8 (8 888) 00:00:01.233035 Name <string> = "Tutorials" (cb=10) 00:00:01.233036 Protocol <string> = "TCP" (cb=4) 00:00:01.233036 00:00:01.233037 [/Devices/e1000/0/LUN#0/Config/PortForwarding/54/] (level 7) 00:00:01.233038 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233038 GuestPort <integer> = 0x000000000000eab0 (60 080) 00:00:01.233039 HostPort <integer> = 0x000000000000eab0 (60 080) 00:00:01.233040 Name <string> = "WebHBase" (cb=9) 00:00:01.233040 Protocol <string> = "TCP" (cb=4) 00:00:01.233041 00:00:01.233041 [/Devices/e1000/0/LUN#0/Config/PortForwarding/55/] (level 7) 00:00:01.233042 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233043 GuestPort <integer> = 0x000000000000c3bf (50 111) 00:00:01.233043 HostPort <integer> = 0x000000000000c3bf (50 111) 00:00:01.233044 Name <string> = "WebHcat" (cb=8) 00:00:01.233045 Protocol <string> = "TCP" (cb=4) 00:00:01.233045 00:00:01.233045 [/Devices/e1000/0/LUN#0/Config/PortForwarding/56/] (level 7) 00:00:01.233046 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233047 GuestPort <integer> = 0x000000000000c396 (50 070) 00:00:01.233048 HostPort <integer> = 0x000000000000c396 (50 070) 00:00:01.233048 Name <string> = "WebHdfs" (cb=8) 00:00:01.233049 Protocol <string> = "TCP" (cb=4) 00:00:01.233049 00:00:01.233050 [/Devices/e1000/0/LUN#0/Config/PortForwarding/57/] (level 7) 00:00:01.233051 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233051 GuestPort <integer> = 0x00000000000017c0 (6 080) 00:00:01.233052 HostPort <integer> = 0x00000000000017c0 (6 080) 00:00:01.233053 Name <string> = "XASecure" (cb=9) 00:00:01.233053 Protocol <string> = "TCP" (cb=4) 00:00:01.233054 00:00:01.233054 [/Devices/e1000/0/LUN#0/Config/PortForwarding/58/] (level 7) 00:00:01.233055 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233055 GuestPort <integer> = 0x0000000000001ffc (8 188) 00:00:01.233056 HostPort <integer> = 0x0000000000001ffc (8 188) 00:00:01.233057 Name <string> = "YarnATS" (cb=8) 00:00:01.233057 Protocol <string> = "TCP" (cb=4) 00:00:01.233058 00:00:01.233058 [/Devices/e1000/0/LUN#0/Config/PortForwarding/59/] (level 7) 00:00:01.233059 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233060 GuestPort <integer> = 0x0000000000001f98 (8 088) 00:00:01.233060 HostPort <integer> = 0x0000000000001f98 (8 088) 00:00:01.233061 Name <string> = "YarnRM" (cb=7) 00:00:01.233062 Protocol <string> = "TCP" (cb=4) 00:00:01.233062 00:00:01.233062 [/Devices/e1000/0/LUN#0/Config/PortForwarding/6/] (level 7) 00:00:01.233063 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233064 GuestPort <integer> = 0x0000000000002720 (10 016) 00:00:01.233065 HostPort <integer> = 0x0000000000002720 (10 016) 00:00:01.233065 Name <string> = "Custom11" (cb=9) 00:00:01.233066 Protocol <string> = "TCP" (cb=4) 00:00:01.233066 00:00:01.233067 [/Devices/e1000/0/LUN#0/Config/PortForwarding/60/] (level 7) 00:00:01.233068 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233068 GuestPort <integer> = 0x000000000000270b (9 995) 00:00:01.233069 HostPort <integer> = 0x000000000000270b (9 995) 00:00:01.233069 Name <string> = "Zeppelin1" (cb=10) 00:00:01.233070 Protocol <string> = "TCP" (cb=4) 00:00:01.233070 00:00:01.233071 [/Devices/e1000/0/LUN#0/Config/PortForwarding/61/] (level 7) 00:00:01.233072 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233072 GuestPort <integer> = 0x000000000000270c (9 996) 00:00:01.233073 HostPort <integer> = 0x000000000000270c (9 996) 00:00:01.233073 Name <string> = "Zeppelin2" (cb=10) 00:00:01.233074 Protocol <string> = "TCP" (cb=4) 00:00:01.233074 00:00:01.233075 [/Devices/e1000/0/LUN#0/Config/PortForwarding/62/] (level 7) 00:00:01.233076 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233076 GuestPort <integer> = 0x0000000000000885 (2 181) 00:00:01.233077 HostPort <integer> = 0x0000000000000885 (2 181) 00:00:01.233078 Name <string> = "Zookeeper" (cb=10) 00:00:01.233078 Protocol <string> = "TCP" (cb=4) 00:00:01.233078 00:00:01.233079 [/Devices/e1000/0/LUN#0/Config/PortForwarding/63/] (level 7) 00:00:01.233080 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233080 GuestPort <integer> = 0x0000000000001f90 (8 080) 00:00:01.233081 HostPort <integer> = 0x0000000000001f90 (8 080) 00:00:01.233082 Name <string> = "ambari" (cb=7) 00:00:01.233082 Protocol <string> = "TCP" (cb=4) 00:00:01.233083 00:00:01.233083 [/Devices/e1000/0/LUN#0/Config/PortForwarding/64/] (level 7) 00:00:01.233084 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233084 GuestPort <integer> = 0x0000000000000050 (80) 00:00:01.233085 HostPort <integer> = 0x000000000000a460 (42 080) 00:00:01.233086 Name <string> = "apache" (cb=7) 00:00:01.233086 Protocol <string> = "TCP" (cb=4) 00:00:01.233087 00:00:01.233087 [/Devices/e1000/0/LUN#0/Config/PortForwarding/65/] (level 7) 00:00:01.233088 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233089 GuestPort <integer> = 0x0000000000001f54 (8 020) 00:00:01.233089 HostPort <integer> = 0x0000000000001f54 (8 020) 00:00:01.233090 Name <string> = "hdfs" (cb=5) 00:00:01.233090 Protocol <string> = "TCP" (cb=4) 00:00:01.233091 00:00:01.233091 [/Devices/e1000/0/LUN#0/Config/PortForwarding/66/] (level 7) 00:00:01.233092 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233093 GuestPort <integer> = 0x000000000000a47f (42 111) 00:00:01.233093 HostPort <integer> = 0x000000000000a47f (42 111) 00:00:01.233094 Name <string> = "nfs" (cb=4) 00:00:01.233095 Protocol <string> = "TCP" (cb=4) 00:00:01.233095 00:00:01.233095 [/Devices/e1000/0/LUN#0/Config/PortForwarding/67/] (level 7) 00:00:01.233096 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233097 GuestPort <integer> = 0x0000000000001f68 (8 040) 00:00:01.233098 HostPort <integer> = 0x0000000000001f68 (8 040) 00:00:01.233098 Name <string> = "nodemanager" (cb=12) 00:00:01.233099 Protocol <string> = "TCP" (cb=4) 00:00:01.233099 00:00:01.233099 [/Devices/e1000/0/LUN#0/Config/PortForwarding/7/] (level 7) 00:00:01.233101 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233101 GuestPort <integer> = 0x0000000000002906 (10 502) 00:00:01.233102 HostPort <integer> = 0x0000000000002906 (10 502) 00:00:01.233102 Name <string> = "Custom12" (cb=9) 00:00:01.233103 Protocol <string> = "TCP" (cb=4) 00:00:01.233103 00:00:01.233104 [/Devices/e1000/0/LUN#0/Config/PortForwarding/8/] (level 7) 00:00:01.233105 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233105 GuestPort <integer> = 0x0000000000008311 (33 553) 00:00:01.233106 HostPort <integer> = 0x0000000000008311 (33 553) 00:00:01.233107 Name <string> = "Custom13" (cb=9) 00:00:01.233107 Protocol <string> = "TCP" (cb=4) 00:00:01.233108 00:00:01.233108 [/Devices/e1000/0/LUN#0/Config/PortForwarding/9/] (level 7) 00:00:01.233109 BindIP <string> = "127.0.0.1" (cb=10) 00:00:01.233110 GuestPort <integer> = 0x00000000000099fb (39 419) 00:00:01.233111 HostPort <integer> = 0x00000000000099fb (39 419) 00:00:01.233111 Name <string> = "Custom14" (cb=9) 00:00:01.233112 Protocol <string> = "TCP" (cb=4) 00:00:01.233112 00:00:01.233113 [/Devices/e1000/0/LUN#999/] (level 4) 00:00:01.233113 Driver <string> = "MainStatus" (cb=11) 00:00:01.233114 00:00:01.233114 [/Devices/e1000/0/LUN#999/Config/] (level 5) 00:00:01.233115 First <integer> = 0x0000000000000000 (0) 00:00:01.233116 Last <integer> = 0x0000000000000000 (0) 00:00:01.233117 papLeds <integer> = 0x0000000003fdf3c8 (66 974 664) 00:00:01.233118 00:00:01.233118 [/Devices/i8254/] (level 2) 00:00:01.233119 00:00:01.233119 [/Devices/i8254/0/] (level 3) 00:00:01.233120 00:00:01.233120 [/Devices/i8254/0/Config/] (level 4) 00:00:01.233121 00:00:01.233121 [/Devices/i8259/] (level 2) 00:00:01.233122 00:00:01.233122 [/Devices/i8259/0/] (level 3) 00:00:01.233123 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233123 00:00:01.233124 [/Devices/i8259/0/Config/] (level 4) 00:00:01.233124 00:00:01.233125 [/Devices/ichac97/] (level 2) 00:00:01.233125 00:00:01.233126 [/Devices/ichac97/0/] (level 3) 00:00:01.233126 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.233127 PCIDeviceNo <integer> = 0x0000000000000005 (5) 00:00:01.233128 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.233128 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233129 00:00:01.233129 [/Devices/ichac97/0/AudioConfig/] (level 4) 00:00:01.233130 00:00:01.233130 [/Devices/ichac97/0/Config/] (level 4) 00:00:01.233131 Codec <string> = "STAC9700" (cb=9) 00:00:01.233132 00:00:01.233132 [/Devices/ichac97/0/LUN#0/] (level 4) 00:00:01.233133 Driver <string> = "AUDIO" (cb=6) 00:00:01.233133 00:00:01.233134 [/Devices/ichac97/0/LUN#0/AttachedDriver/] (level 5) 00:00:01.233135 Driver <string> = "DSoundAudio" (cb=12) 00:00:01.233135 00:00:01.233135 [/Devices/ichac97/0/LUN#0/AttachedDriver/Config/] (level 6) 00:00:01.233136 StreamName <string> = "Hortonworks Docker Sandbox HDP" (cb=31) 00:00:01.233137 00:00:01.233137 [/Devices/ichac97/0/LUN#0/Config/] (level 5) 00:00:01.233138 00:00:01.233139 [/Devices/ichac97/0/LUN#1/] (level 4) 00:00:01.233139 Driver <string> = "AUDIO" (cb=6) 00:00:01.233140 00:00:01.233140 [/Devices/ichac97/0/LUN#1/AttachedDriver/] (level 5) 00:00:01.233141 Driver <string> = "AudioVRDE" (cb=10) 00:00:01.233142 00:00:01.233142 [/Devices/ichac97/0/LUN#1/AttachedDriver/Config/] (level 6) 00:00:01.233143 AudioDriver <string> = "AudioVRDE" (cb=10) 00:00:01.233143 Object <integer> = 0x000000000402e820 (67 299 360) 00:00:01.233144 ObjectVRDPServer <integer> = 0x000000000402fc40 (67 304 512) 00:00:01.233145 StreamName <string> = "Hortonworks Docker Sandbox HDP" (cb=31) 00:00:01.233146 00:00:01.233146 [/Devices/ioapic/] (level 2) 00:00:01.233147 00:00:01.233147 [/Devices/ioapic/0/] (level 3) 00:00:01.233148 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233149 00:00:01.233149 [/Devices/ioapic/0/Config/] (level 4) 00:00:01.233150 NumCPUs <integer> = 0x0000000000000004 (4) 00:00:01.233150 00:00:01.233150 [/Devices/mc146818/] (level 2) 00:00:01.233151 00:00:01.233152 [/Devices/mc146818/0/] (level 3) 00:00:01.233152 00:00:01.233153 [/Devices/mc146818/0/Config/] (level 4) 00:00:01.233154 UseUTC <integer> = 0x0000000000000001 (1) 00:00:01.233154 00:00:01.233154 [/Devices/parallel/] (level 2) 00:00:01.233155 00:00:01.233155 [/Devices/pcarch/] (level 2) 00:00:01.233156 00:00:01.233156 [/Devices/pcarch/0/] (level 3) 00:00:01.233157 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233158 00:00:01.233158 [/Devices/pcarch/0/Config/] (level 4) 00:00:01.233159 00:00:01.233159 [/Devices/pcbios/] (level 2) 00:00:01.233160 00:00:01.233160 [/Devices/pcbios/0/] (level 3) 00:00:01.233161 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233161 00:00:01.233162 [/Devices/pcbios/0/Config/] (level 4) 00:00:01.233163 APIC <integer> = 0x0000000000000001 (1) 00:00:01.233164 BootDevice0 <string> = "IDE" (cb=4) 00:00:01.233164 BootDevice1 <string> = "DVD" (cb=4) 00:00:01.233165 BootDevice2 <string> = "NONE" (cb=5) 00:00:01.233165 BootDevice3 <string> = "NONE" (cb=5) 00:00:01.233166 FloppyDevice <string> = "i82078" (cb=7) 00:00:01.233166 HardDiskDevice <string> = "piix3ide" (cb=9) 00:00:01.233167 IOAPIC <integer> = 0x0000000000000001 (1) 00:00:01.233168 McfgBase <integer> = 0x0000000000000000 (0) 00:00:01.233168 McfgLength <integer> = 0x0000000000000000 (0) 00:00:01.233169 NumCPUs <integer> = 0x0000000000000004 (4) 00:00:01.233169 PXEDebug <integer> = 0x0000000000000000 (0) 00:00:01.233170 UUID <bytes> = "aa 65 5f 19 b4 25 6e 4f 90 dd 62 ec c2 b2 86 78" (cb=16) 00:00:01.233172 00:00:01.233172 [/Devices/pcbios/0/Config/NetBoot/] (level 5) 00:00:01.233173 00:00:01.233173 [/Devices/pcbios/0/Config/NetBoot/0/] (level 6) 00:00:01.233174 NIC <integer> = 0x0000000000000000 (0) 00:00:01.233175 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.233176 PCIDeviceNo <integer> = 0x0000000000000003 (3) 00:00:01.233176 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.233177 00:00:01.233177 [/Devices/pci/] (level 2) 00:00:01.233178 00:00:01.233178 [/Devices/pci/0/] (level 3) 00:00:01.233179 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233179 00:00:01.233180 [/Devices/pci/0/Config/] (level 4) 00:00:01.233180 IOAPIC <integer> = 0x0000000000000001 (1) 00:00:01.233181 00:00:01.233181 [/Devices/pckbd/] (level 2) 00:00:01.233182 00:00:01.233182 [/Devices/pckbd/0/] (level 3) 00:00:01.233183 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233183 00:00:01.233184 [/Devices/pckbd/0/Config/] (level 4) 00:00:01.233184 00:00:01.233185 [/Devices/pckbd/0/LUN#0/] (level 4) 00:00:01.233185 Driver <string> = "KeyboardQueue" (cb=14) 00:00:01.233186 00:00:01.233186 [/Devices/pckbd/0/LUN#0/AttachedDriver/] (level 5) 00:00:01.233187 Driver <string> = "MainKeyboard" (cb=13) 00:00:01.233188 00:00:01.233188 [/Devices/pckbd/0/LUN#0/AttachedDriver/Config/] (level 6) 00:00:01.233189 Object <integer> = 0x0000000003b87d80 (62 422 400) 00:00:01.233190 00:00:01.233190 [/Devices/pckbd/0/LUN#0/Config/] (level 5) 00:00:01.233191 QueueSize <integer> = 0x0000000000000040 (64) 00:00:01.233192 00:00:01.233192 [/Devices/pckbd/0/LUN#1/] (level 4) 00:00:01.233193 Driver <string> = "MouseQueue" (cb=11) 00:00:01.233194 00:00:01.233194 [/Devices/pckbd/0/LUN#1/AttachedDriver/] (level 5) 00:00:01.233195 Driver <string> = "MainMouse" (cb=10) 00:00:01.233195 00:00:01.233195 [/Devices/pckbd/0/LUN#1/AttachedDriver/Config/] (level 6) 00:00:01.233197 Object <integer> = 0x000000000402f690 (67 303 056) 00:00:01.233197 00:00:01.233198 [/Devices/pckbd/0/LUN#1/Config/] (level 5) 00:00:01.233199 QueueSize <integer> = 0x0000000000000080 (128) 00:00:01.233199 00:00:01.233200 [/Devices/pcnet/] (level 2) 00:00:01.233200 00:00:01.233201 [/Devices/piix3ide/] (level 2) 00:00:01.233201 00:00:01.233202 [/Devices/piix3ide/0/] (level 3) 00:00:01.233203 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.233203 PCIDeviceNo <integer> = 0x0000000000000001 (1) 00:00:01.233204 PCIFunctionNo <integer> = 0x0000000000000001 (1) 00:00:01.233204 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233205 00:00:01.233205 [/Devices/piix3ide/0/Config/] (level 4) 00:00:01.233206 Type <string> = "PIIX4" (cb=6) 00:00:01.233207 00:00:01.233207 [/Devices/piix3ide/0/Config/PrimaryMaster/] (level 5) 00:00:01.233208 NonRotationalMedium <integer> = 0x0000000000000000 (0) 00:00:01.233209 00:00:01.233209 [/Devices/piix3ide/0/LUN#0/] (level 4) 00:00:01.233210 Driver <string> = "VD" (cb=3) 00:00:01.233211 00:00:01.233211 [/Devices/piix3ide/0/LUN#0/Config/] (level 5) 00:00:01.233212 Format <string> = "VMDK" (cb=5) 00:00:01.233213 Mountable <integer> = 0x0000000000000000 (0) 00:00:01.233213 Path <string> = "E:\Omkar\Development\Software\Virtualization\VirtualBoxVM\Hortonworks Docker Sandbox HDP\Hortonworks Docker Sandbox HDP-disk1.vmdk" (cb=131) 00:00:01.233214 Type <string> = "HardDisk" (cb=9) 00:00:01.233215 00:00:01.233215 [/Devices/piix3ide/0/LUN#999/] (level 4) 00:00:01.233216 Driver <string> = "MainStatus" (cb=11) 00:00:01.233216 00:00:01.233217 [/Devices/piix3ide/0/LUN#999/Config/] (level 5) 00:00:01.233218 DeviceInstance <string> = "piix3ide/0" (cb=11) 00:00:01.233218 First <integer> = 0x0000000000000000 (0) 00:00:01.233219 Last <integer> = 0x0000000000000003 (3) 00:00:01.233220 pConsole <integer> = 0x0000000003fdece0 (66 972 896) 00:00:01.233221 papLeds <integer> = 0x0000000003fdf0c8 (66 973 896) 00:00:01.233222 pmapMediumAttachments <integer> = 0x0000000003fdf508 (66 974 984) 00:00:01.233223 00:00:01.233223 [/Devices/serial/] (level 2) 00:00:01.233224 00:00:01.233224 [/Devices/usb-ohci/] (level 2) 00:00:01.233225 00:00:01.233225 [/Devices/usb-ohci/0/] (level 3) 00:00:01.233226 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.233226 PCIDeviceNo <integer> = 0x0000000000000006 (6) 00:00:01.233227 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.233228 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233228 00:00:01.233228 [/Devices/usb-ohci/0/Config/] (level 4) 00:00:01.233229 00:00:01.233229 [/Devices/usb-ohci/0/LUN#0/] (level 4) 00:00:01.233230 Driver <string> = "VUSBRootHub" (cb=12) 00:00:01.233231 00:00:01.233231 [/Devices/usb-ohci/0/LUN#0/Config/] (level 5) 00:00:01.233232 00:00:01.233232 [/Devices/usb-ohci/0/LUN#999/] (level 4) 00:00:01.233233 Driver <string> = "MainStatus" (cb=11) 00:00:01.233234 00:00:01.233234 [/Devices/usb-ohci/0/LUN#999/Config/] (level 5) 00:00:01.233235 First <integer> = 0x0000000000000000 (0) 00:00:01.233236 Last <integer> = 0x0000000000000000 (0) 00:00:01.233236 papLeds <integer> = 0x0000000003fdf4f0 (66 974 960) 00:00:01.233237 00:00:01.233237 [/Devices/vga/] (level 2) 00:00:01.233238 00:00:01.233238 [/Devices/vga/0/] (level 3) 00:00:01.233239 PCIBusNo <integer> = 0x0000000000000000 (0) 00:00:01.233240 PCIDeviceNo <integer> = 0x0000000000000002 (2) 00:00:01.233240 PCIFunctionNo <integer> = 0x0000000000000000 (0) 00:00:01.233241 Trusted <integer> = 0x0000000000000001 (1) 00:00:01.233241 00:00:01.233242 [/Devices/vga/0/Config/] (level 4) 00:00:01.233243 CustomVideoModes <integer> = 0x0000000000000000 (0) 00:00:01.233244 FadeIn <integer> = 0x0000000000000001 (1) 00:00:01.233244 FadeOut <integer> = 0x0000000000000001 (1) 00:00:01.233245 HeightReduction <integer> = 0x0000000000000000 (0) 00:00:01.233245 LogoFile <string> = "" (cb=1) 00:00:01.233246 LogoTime <integer> = 0x0000000000000000 (0) 00:00:01.233247 MonitorCount <integer> = 0x0000000000000001 (1) 00:00:01.233247 ShowBootMenu <integer> = 0x0000000000000002 (2) 00:00:01.233248 VRamSize <integer> = 0x0000000000800000 (8 388 608, 8 MB) 00:00:01.233249 00:00:01.233250 [/Devices/vga/0/LUN#0/] (level 4) 00:00:01.233250 Driver <string> = "MainDisplay" (cb=12) 00:00:01.233251 00:00:01.233251 [/Devices/vga/0/LUN#0/Config/] (level 5) 00:00:01.233252 Object <integer> = 0x0000000003fe2490 (66 987 152) 00:00:01.233253 00:00:01.233253 [/Devices/vga/0/LUN#999/] (level 4) 00:00:01.233254 Driver <string> = "MainStatus" (cb=11) 00:00:01.233255 00:00:01.233255 [/Devices/vga/0/LUN#999/Config/] (level 5) 00:00:01.233256 First <integer> = 0x0000000000000000 (0) 00:00:01.233256 Last <integer> = 0x0000000000000000 (0) 00:00:01.233257 papLeds <integer> = 0x0000000003fdf500 (66 974 976) 00:00:01.233258 00:00:01.233258 [/Devices/virtio-net/] (level 2) 00:00:01.233259 00:00:01.233259 [/EM/] (level 1) 00:00:01.233260 TripleFaultReset <integer> = 0x0000000000000000 (0) 00:00:01.233260 00:00:01.233261 [/GIM/] (level 1) 00:00:01.233261 Provider <string> = "KVM" (cb=4) 00:00:01.233262 00:00:01.233262 [/HM/] (level 1) 00:00:01.233263 64bitEnabled <integer> = 0x0000000000000001 (1) 00:00:01.233263 EnableLargePages <integer> = 0x0000000000000000 (0) 00:00:01.233264 EnableNestedPaging <integer> = 0x0000000000000001 (1) 00:00:01.233264 EnableUX <integer> = 0x0000000000000001 (1) 00:00:01.233265 EnableVPID <integer> = 0x0000000000000001 (1) 00:00:01.233266 Exclusive <integer> = 0x0000000000000000 (0) 00:00:01.233325 HMForced <integer> = 0x0000000000000001 (1) 00:00:01.233326 00:00:01.233327 [/MM/] (level 1) 00:00:01.233327 CanUseLargerHeap <integer> = 0x0000000000000000 (0) 00:00:01.233328 00:00:01.233328 [/PDM/] (level 1) 00:00:01.233329 00:00:01.233329 [/PDM/AsyncCompletion/] (level 2) 00:00:01.233330 00:00:01.233330 [/PDM/AsyncCompletion/File/] (level 3) 00:00:01.233331 00:00:01.233331 [/PDM/AsyncCompletion/File/BwGroups/] (level 4) 00:00:01.233332 00:00:01.233332 [/PDM/BlkCache/] (level 2) 00:00:01.233333 CacheSize <integer> = 0x0000000000500000 (5 242 880, 5 MB) 00:00:01.233334 00:00:01.233334 [/PDM/Devices/] (level 2) 00:00:01.233335 00:00:01.233335 [/PDM/Drivers/] (level 2) 00:00:01.233336 00:00:01.233336 [/PDM/Drivers/VBoxC/] (level 3) 00:00:01.233337 Path <string> = "VBoxC" (cb=6) 00:00:01.233338 00:00:01.233338 [/PDM/NetworkShaper/] (level 2) 00:00:01.233339 00:00:01.233339 [/PDM/NetworkShaper/BwGroups/] (level 3) 00:00:01.233340 00:00:01.233340 [/TM/] (level 1) 00:00:01.233341 UTCOffset <integer> = 0x0000000000000000 (0) 00:00:01.233342 00:00:01.233342 [/USB/] (level 1) 00:00:01.233343 00:00:01.233343 [/USB/HidMouse/] (level 2) 00:00:01.233344 00:00:01.233344 [/USB/HidMouse/0/] (level 3) 00:00:01.233345 00:00:01.233345 [/USB/HidMouse/0/Config/] (level 4) 00:00:01.233346 Mode <string> = "absolute" (cb=9) 00:00:01.233346 00:00:01.233347 [/USB/HidMouse/0/LUN#0/] (level 4) 00:00:01.233347 Driver <string> = "MouseQueue" (cb=11) 00:00:01.233348 00:00:01.233348 [/USB/HidMouse/0/LUN#0/AttachedDriver/] (level 5) 00:00:01.233349 Driver <string> = "MainMouse" (cb=10) 00:00:01.233350 00:00:01.233350 [/USB/HidMouse/0/LUN#0/AttachedDriver/Config/] (level 6) 00:00:01.233351 Object <integer> = 0x000000000402f690 (67 303 056) 00:00:01.233367 00:00:01.233368 [/USB/HidMouse/0/LUN#0/Config/] (level 5) 00:00:01.233369 QueueSize <integer> = 0x0000000000000080 (128) 00:00:01.233369 00:00:01.233370 [/USB/USBProxy/] (level 2) 00:00:01.233370 00:00:01.233370 [/USB/USBProxy/GlobalConfig/] (level 3) 00:00:01.233371 00:00:01.233372 ********************* End of CFGM dump ********************** 00:00:01.233431 VM: fHMEnabled=true (configured) fRecompileUser=false fRecompileSupervisor=false 00:00:01.233432 VM: fRawRing1Enabled=false CSAM=true PATM=true 00:00:01.233644 HM: HMR3Init: VT-x w/ nested paging and unrestricted guest execution hw support 00:00:01.233779 MM: cbHyperHeap=0x140000 (1310720) 00:00:01.235117 CPUM: fXStateHostMask=0x7; initial: 0x7; host XCR0=0x7 00:00:01.238015 CPUM: Matched host CPU INTEL 0x6/0x5e/0x3 Intel_Core7_Skylake with CPU DB entry 'Intel Core i7-6700K' (INTEL 0x6/0x5e/0x3 Intel_Core7_Skylake) 00:00:01.238234 CPUM: MXCSR_MASK=0xffff (host: 0xffff) 00:00:01.238277 CPUM: Microcode revision 0x0000009E 00:00:01.238356 CPUM: SetGuestCpuIdFeature: Enabled PAE 00:00:01.238376 CPUM: VCPU 0: Cached APIC base MSR = 0x0 00:00:01.238382 CPUM: VCPU 1: Cached APIC base MSR = 0x0 00:00:01.238387 CPUM: VCPU 2: Cached APIC base MSR = 0x0 00:00:01.238391 CPUM: VCPU 3: Cached APIC base MSR = 0x0 00:00:01.239786 PGM: HCPhysInterPD=000000008cc22000 HCPhysInterPaePDPT=000000008cc1d000 HCPhysInterPaePML4=000000008cc1b000 00:00:01.239811 PGM: apInterPTs={000000008cc21000,000000008cc20000} apInterPaePTs={000000003bbed000,00000002119f9000} apInterPaePDs={00000001643fa000,00000000610fb000,00000001ce40d000,000000012eccb000} pInterPaePDPT64=000000008cc1c000 00:00:01.239869 PGM: Host paging mode: AMD64+PGE+NX 00:00:01.239942 PGM: PGMPool: cMaxPages=7200 (u64MaxPages=7195) 00:00:01.239942 PGM: pgmR3PoolInit: cMaxPages=0x1c20 cMaxUsers=0x3840 cMaxPhysExts=0x2000 fCacheEnable=true 00:00:01.326850 TM: GIP - u32Mode=3 (Invariant) u32UpdateHz=100 u32UpdateIntervalNS=10000000 enmUseTscDelta=2 (Pratically Zero) fGetGipCpu=0x1 cCpus=8 00:00:01.326892 TM: GIP - u64CpuHz=2 718 944 806 (0xa20fce26) SUPGetCpuHzFromGip => 2 718 944 806 00:00:01.326899 TM: GIP - CPU: iCpuSet=0x0 idCpu=0x0 idApic=0x0 iGipCpu=0x0 i64TSCDelta=0 enmState=3 u64CpuHz=2718944806(*) cErrors=0 00:00:01.326905 TM: GIP - CPU: iCpuSet=0x1 idCpu=0x1 idApic=0x1 iGipCpu=0x1 i64TSCDelta=0 enmState=3 u64CpuHz=2392559689(*) cErrors=0 00:00:01.326909 TM: GIP - CPU: iCpuSet=0x2 idCpu=0x2 idApic=0x2 iGipCpu=0x7 i64TSCDelta=0 enmState=3 u64CpuHz=2728334143(*) cErrors=0 00:00:01.326914 TM: GIP - CPU: iCpuSet=0x3 idCpu=0x3 idApic=0x3 iGipCpu=0x2 i64TSCDelta=0 enmState=3 u64CpuHz=2712037041(*) cErrors=0 00:00:01.326918 TM: GIP - CPU: iCpuSet=0x4 idCpu=0x4 idApic=0x4 iGipCpu=0x5 i64TSCDelta=0 enmState=3 u64CpuHz=2749400563(*) cErrors=0 00:00:01.326922 TM: GIP - CPU: iCpuSet=0x5 idCpu=0x5 idApic=0x5 iGipCpu=0x3 i64TSCDelta=0 enmState=3 u64CpuHz=2712010054(*) cErrors=0 00:00:01.326927 TM: GIP - CPU: iCpuSet=0x6 idCpu=0x6 idApic=0x6 iGipCpu=0x6 i64TSCDelta=0 enmState=3 u64CpuHz=2732084806(*) cErrors=0 00:00:01.326931 TM: GIP - CPU: iCpuSet=0x7 idCpu=0x7 idApic=0x7 iGipCpu=0x4 i64TSCDelta=0 enmState=3 u64CpuHz=2750459509(*) cErrors=0 00:00:01.327018 TM: cTSCTicksPerSecond=2 718 944 806 (0xa20fce26) enmTSCMode=1 (VirtTscEmulated) 00:00:01.327020 TM: TSCTiedToExecution=false TSCNotTiedToHalt=false 00:00:01.328766 VMM: CoreCode: R3=00000000039f0000 R0=XXXXXXXXXXXXXXXX RC=a10c2000 Phys=000000008cbfe000 cb=0x1000 00:00:01.329154 IEM: TargetCpu=CURRENT, Microarch=Intel_Core7_Skylake 00:00:01.329365 GIM: Using provider 'KVM' (Implementation version: 0) 00:00:01.329387 CPUM: SetGuestCpuIdFeature: Enabled Hypervisor Present bit 00:00:01.329548 AIOMgr: Default manager type is 'Async' 00:00:01.329555 AIOMgr: Default file backend is 'NonBuffered' 00:00:01.329942 BlkCache: Cache successfully initialized. Cache size is 5242880 bytes 00:00:01.329942 BlkCache: Cache commit interval is 10000 ms 00:00:01.329942 BlkCache: Cache commit threshold is 2621440 bytes 00:00:01.573442 PcBios: [SMP] BIOS with 4 CPUs 00:00:01.573545 PcBios: Using the 386+ BIOS image. 00:00:01.573876 PcBios: MPS table at 000e1300 00:00:01.576482 PcBios: fCheckShutdownStatusForSoftReset=true fClearShutdownStatusOnHardReset=true 00:00:01.606964 SUP: Loaded VBoxDDR0.r0 (C:\Program Files\Oracle\VirtualBox\VBoxDDR0.r0) at 0xXXXXXXXXXXXXXXXX - ModuleInit at XXXXXXXXXXXXXXXX and ModuleTerm at XXXXXXXXXXXXXXXX using the native ring-0 loader 00:00:01.606993 SUP: windbg> .reload /f C:\Program Files\Oracle\VirtualBox\VBoxDDR0.r0=0xXXXXXXXXXXXXXXXX 00:00:01.607831 CPUM: SetGuestCpuIdFeature: Enabled xAPIC 00:00:01.607842 CPUM: SetGuestCpuIdFeature: Enabled x2APIC 00:00:01.608396 IOAPIC: Using implementation 2.0! 00:00:01.608548 PIT: mode=3 count=0x10000 (65536) - 18.20 Hz (ch=0) 00:00:01.635548 Shared Folders service loaded 00:00:01.642595 VGA: Using the 386+ BIOS image. 00:00:01.647569 DrvVD: Flushes will be ignored 00:00:01.647590 DrvVD: Async flushes will be passed to the disk 00:00:01.647861 VD: VDInit finished 00:00:01.652981 VD: Opening the disk took 5365477 ns 00:00:01.653219 PIIX3 ATA: LUN#0: disk, PCHS=16383/16/63, total number of sectors 102400000 00:00:01.653233 PIIX3 ATA: LUN#1: no unit 00:00:01.653605 PIIX3 ATA: LUN#2: no unit 00:00:01.653620 PIIX3 ATA: LUN#3: no unit 00:00:01.653671 PIIX3 ATA: Ctl#0: finished processing RESET 00:00:01.653764 PIIX3 ATA: Ctl#1: finished processing RESET 00:00:01.653889 E1000#0 Chip=82540EM LinkUpDelay=5000ms EthernetCRC=on GSO=enabled Itr=enabled ItrRx=enabled R0=enabled GC=enabled 00:00:01.661390 NAT: Guest address guess set to 10.0.2.15 by initialization 00:00:01.689942 NAT: DNS#0: 138.106.13.128 00:00:01.689942 NAT: DNS#1: 138.106.14.37 00:00:01.689942 NAT: DNS#2: 192.168.43.1 00:00:01.689942 NAT: DNS#3: 192.168.18.2 00:00:01.712037 NAT: Set redirect TCP 127.0.0.1:50095 -> 0.0.0.0:50095 00:00:01.712294 NAT: Set redirect TCP 127.0.0.1:8886 -> 0.0.0.0:8886 00:00:01.712541 NAT: Set redirect TCP 127.0.0.1:15002 -> 0.0.0.0:15002 00:00:01.712737 NAT: Set redirect TCP 127.0.0.1:111 -> 0.0.0.0:111 00:00:01.712898 NAT: Set redirect TCP 127.0.0.1:2049 -> 0.0.0.0:2049 00:00:01.713061 NAT: Set redirect TCP 127.0.0.1:4242 -> 0.0.0.0:4242 00:00:01.713164 NAT: Set redirect TCP 127.0.0.1:50079 -> 0.0.0.0:50079 00:00:01.713242 NAT: Set redirect TCP 127.0.0.1:3000 -> 0.0.0.0:3000 00:00:01.713320 NAT: Set redirect TCP 127.0.0.1:16000 -> 0.0.0.0:16000 00:00:01.713407 NAT: Set redirect TCP 127.0.0.1:16020 -> 0.0.0.0:16020 00:00:01.713485 NAT: Set redirect TCP 127.0.0.1:15500 -> 0.0.0.0:15500 00:00:01.713561 NAT: Set redirect TCP 127.0.0.1:15501 -> 0.0.0.0:15501 00:00:01.713636 NAT: Set redirect TCP 127.0.0.1:4200 -> 0.0.0.0:4200 00:00:01.713711 NAT: Set redirect TCP 127.0.0.1:15502 -> 0.0.0.0:15502 00:00:01.713788 NAT: Set redirect TCP 127.0.0.1:15503 -> 0.0.0.0:15503 00:00:01.713864 NAT: Set redirect TCP 127.0.0.1:15504 -> 0.0.0.0:15504 00:00:01.713942 NAT: Set redirect TCP 127.0.0.1:15505 -> 0.0.0.0:15505 00:00:01.714354 NAT: Set redirect TCP 127.0.0.1:8765 -> 0.0.0.0:8765 00:00:01.714445 NAT: Set redirect TCP 127.0.0.1:8090 -> 0.0.0.0:8090 00:00:01.714524 NAT: Set redirect TCP 127.0.0.1:8091 -> 0.0.0.0:8091 00:00:01.714599 NAT: Set redirect TCP 127.0.0.1:8005 -> 0.0.0.0:8005 00:00:01.714674 NAT: Set redirect TCP 127.0.0.1:8086 -> 0.0.0.0:8086 00:00:01.714748 NAT: Set redirect TCP 127.0.0.1:8082 -> 0.0.0.0:8082 00:00:01.714824 NAT: Set redirect TCP 127.0.0.1:21000 -> 0.0.0.0:21000 00:00:01.714900 NAT: Set redirect TCP 127.0.0.1:18081 -> 0.0.0.0:18081 00:00:01.714975 NAT: Set redirect TCP 127.0.0.1:50075 -> 0.0.0.0:50075 00:00:01.715083 NAT: Set redirect TCP 127.0.0.1:2222 -> 0.0.0.0:2222 00:00:01.715189 NAT: Set redirect TCP 127.0.0.1:15000 -> 0.0.0.0:15000 00:00:01.715265 NAT: Set redirect TCP 127.0.0.1:16010 -> 0.0.0.0:16010 00:00:01.715374 NAT: Set redirect TCP 127.0.0.1:16030 -> 0.0.0.0:16030 00:00:01.715489 NAT: Set redirect TCP 127.0.0.1:10000 -> 0.0.0.0:10000 00:00:01.715582 NAT: Set redirect TCP 127.0.0.1:10001 -> 0.0.0.0:10001 00:00:01.715663 NAT: Set redirect TCP 127.0.0.1:10500 -> 0.0.0.0:10500 00:00:01.715741 NAT: Set redirect TCP 127.0.0.1:9000 -> 0.0.0.0:9000 00:00:01.715816 NAT: Set redirect TCP 127.0.0.1:60000 -> 0.0.0.0:60000 00:00:01.715889 NAT: Set redirect TCP 127.0.0.1:2122 -> 0.0.0.0:22 00:00:01.715964 NAT: Set redirect TCP 127.0.0.1:19888 -> 0.0.0.0:19888 00:00:01.716039 NAT: Set redirect TCP 127.0.0.1:8889 -> 0.0.0.0:8889 00:00:01.716118 NAT: Set redirect TCP 127.0.0.1:8443 -> 0.0.0.0:8443 00:00:01.716195 NAT: Set redirect TCP 127.0.0.1:9090 -> 0.0.0.0:9090 00:00:01.716270 NAT: Set redirect TCP 127.0.0.1:8042 -> 0.0.0.0:8042 00:00:01.716345 NAT: Set redirect TCP 127.0.0.1:11000 -> 0.0.0.0:11000 00:00:01.716421 NAT: Set redirect TCP 127.0.0.1:8032 -> 0.0.0.0:8032 00:00:01.716503 NAT: Set redirect TCP 127.0.0.1:8993 -> 0.0.0.0:8993 00:00:01.716579 NAT: Set redirect TCP 127.0.0.1:8983 -> 0.0.0.0:8983 00:00:01.716654 NAT: Set redirect TCP 127.0.0.1:10015 -> 0.0.0.0:10015 00:00:01.716730 NAT: Set redirect TCP 127.0.0.1:4040 -> 0.0.0.0:4040 00:00:01.716806 NAT: Set redirect TCP 127.0.0.1:18080 -> 0.0.0.0:18080 00:00:01.716881 NAT: Set redirect TCP 127.0.0.1:8744 -> 0.0.0.0:8744 00:00:01.716955 NAT: Set redirect TCP 127.0.0.1:8888 -> 0.0.0.0:8888 00:00:01.717029 NAT: Set redirect TCP 127.0.0.1:60080 -> 0.0.0.0:60080 00:00:01.717104 NAT: Set redirect TCP 127.0.0.1:50111 -> 0.0.0.0:50111 00:00:01.717179 NAT: Set redirect TCP 127.0.0.1:50070 -> 0.0.0.0:50070 00:00:01.717258 NAT: Set redirect TCP 127.0.0.1:6080 -> 0.0.0.0:6080 00:00:01.717335 NAT: Set redirect TCP 127.0.0.1:8188 -> 0.0.0.0:8188 00:00:01.717409 NAT: Set redirect TCP 127.0.0.1:8088 -> 0.0.0.0:8088 00:00:01.717484 NAT: Set redirect TCP 127.0.0.1:10016 -> 0.0.0.0:10016 00:00:01.717575 NAT: Set redirect TCP 127.0.0.1:9995 -> 0.0.0.0:9995 00:00:01.717665 NAT: Set redirect TCP 127.0.0.1:9996 -> 0.0.0.0:9996 00:00:01.717739 NAT: Set redirect TCP 127.0.0.1:2181 -> 0.0.0.0:2181 00:00:01.717813 NAT: Set redirect TCP 127.0.0.1:8080 -> 0.0.0.0:8080 00:00:01.717889 NAT: Set redirect TCP 127.0.0.1:42080 -> 0.0.0.0:80 00:00:01.717963 NAT: Set redirect TCP 127.0.0.1:8020 -> 0.0.0.0:8020 00:00:01.718038 NAT: Set redirect TCP 127.0.0.1:42111 -> 0.0.0.0:42111 00:00:01.718112 NAT: Set redirect TCP 127.0.0.1:8040 -> 0.0.0.0:8040 00:00:01.718187 NAT: Set redirect TCP 127.0.0.1:10502 -> 0.0.0.0:10502 00:00:01.718263 NAT: Set redirect TCP 127.0.0.1:33553 -> 0.0.0.0:33553 00:00:01.718338 NAT: Set redirect TCP 127.0.0.1:39419 -> 0.0.0.0:39419 00:00:01.719725 Audio: Initializing DirectSound audio driver 00:00:01.822624 DSound: Output: GUID: {9F8EA98D-B246-4B07-BB17-69BDD99DF445} [Headset Earphone (2- Microsoft LifeChat LX-6000)] (Module: {0.0.0.00000000}.{9f8ea98d-b246-4b07-bb17-69bdd99df445}) 00:00:01.822646 DSound: Output: GUID: {EECC4F2B-7ED9-4CF8-AFFB-71A70944DF20} [Speakers / Headphones (Realtek High Definition Audio)] (Module: {0.0.0.00000000}.{eecc4f2b-7ed9-4cf8-affb-71a70944df20}) 00:00:01.949942 DSound: Input: GUID: {613F31C7-684F-4F46-954F-8F495AB29314} [Microphone (HD Pro Webcam C920)] (Module: {0.0.1.00000000}.{613f31c7-684f-4f46-954f-8f495ab29314}) 00:00:01.949942 DSound: Input: GUID: {454316F8-E0A8-436B-836D-7E149B151774} [Microphone Array (Realtek High Definition Audio)] (Module: {0.0.1.00000000}.{454316f8-e0a8-436b-836d-7e149b151774}) 00:00:01.949942 DSound: Input: GUID: {E29E26D9-DB66-4905-8ED7-B32C30E8B594} [Headset Microphone (2- Microsoft LifeChat LX-6000)] (Module: {0.0.1.00000000}.{e29e26d9-db66-4905-8ed7-b32c30e8b594}) 00:00:01.949942 DSound: Found 2 host playback devices 00:00:01.949942 DSound: Found 3 host capturing devices 00:00:01.951284 Audio: Host audio backend supports 2 output streams and 3 input streams at once 00:00:01.951308 Audio: Initializing VRDE driver 00:00:01.951317 Audio: Host audio backend supports 1 output streams and 2 input streams at once 00:00:01.951362 AC97: Reset 00:00:01.951896 DSound: Guest "Microphone In" is using host device with GUID: {613F31C7-684F-4F46-954F-8F495AB29314} 00:00:01.967020 VUSB: Attached 'HidMouse' to port 1 00:00:01.967297 PGM: The CPU physical address width is 39 bits 00:00:01.967309 PGM: PGMR3InitFinalize: 4 MB PSE mask 0000007fffffffff 00:00:01.967493 TM: TMR3InitFinalize: fTSCModeSwitchAllowed=true 00:00:02.007550 VMM: Thread-context hooks unavailable 00:00:02.013188 HM: Using VT-x implementation 2.0 00:00:02.013189 HM: Host CR4 = 0x406f8 00:00:02.013191 HM: Host EFER = 0xd01 00:00:02.013192 HM: MSR_IA32_SMM_MONITOR_CTL = 0x0 00:00:02.013192 HM: MSR_IA32_FEATURE_CONTROL = 0x5 00:00:02.013193 HM: MSR_IA32_VMX_BASIC_INFO = 0xda040000000004 00:00:02.013194 HM: VMCS id = 0x4 00:00:02.013194 HM: VMCS size = 1024 bytes 00:00:02.013195 HM: VMCS physical address limit = None 00:00:02.013196 HM: VMCS memory type = 0x6 00:00:02.013196 HM: Dual-monitor treatment support = true 00:00:02.013197 HM: OUTS & INS instruction-info = true 00:00:02.013197 HM: Max resume loops = 1024 00:00:02.013198 HM: MSR_IA32_VMX_PINBASED_CTLS = 0x7f00000016 00:00:02.013199 HM: EXT_INT_EXIT 00:00:02.013199 HM: NMI_EXIT 00:00:02.013199 HM: VIRTUAL_NMI 00:00:02.013200 HM: PREEMPT_TIMER 00:00:02.013200 HM: POSTED_INTR (must be cleared) 00:00:02.013200 HM: MSR_IA32_VMX_PROCBASED_CTLS = 0xfff9fffe0401e172 00:00:02.013201 HM: INT_WINDOW_EXIT 00:00:02.013202 HM: USE_TSC_OFFSETTING 00:00:02.013202 HM: HLT_EXIT 00:00:02.013202 HM: INVLPG_EXIT 00:00:02.013229 HM: MWAIT_EXIT 00:00:02.013230 HM: RDPMC_EXIT 00:00:02.013230 HM: RDTSC_EXIT 00:00:02.013230 HM: CR3_LOAD_EXIT (must be set) 00:00:02.013231 HM: CR3_STORE_EXIT (must be set) 00:00:02.013231 HM: CR8_LOAD_EXIT 00:00:02.013231 HM: CR8_STORE_EXIT 00:00:02.013232 HM: USE_TPR_SHADOW 00:00:02.013232 HM: NMI_WINDOW_EXIT 00:00:02.013232 HM: MOV_DR_EXIT 00:00:02.013232 HM: UNCOND_IO_EXIT 00:00:02.013233 HM: USE_IO_BITMAPS 00:00:02.013233 HM: MONITOR_TRAP_FLAG 00:00:02.013233 HM: USE_MSR_BITMAPS 00:00:02.013234 HM: MONITOR_EXIT 00:00:02.013234 HM: PAUSE_EXIT 00:00:02.013234 HM: USE_SECONDARY_EXEC_CTRL 00:00:02.013235 HM: MSR_IA32_VMX_PROCBASED_CTLS2 = 0x1ffcff00000000 00:00:02.013236 HM: VIRT_APIC 00:00:02.013236 HM: EPT 00:00:02.013236 HM: DESCRIPTOR_TABLE_EXIT 00:00:02.013236 HM: RDTSCP 00:00:02.013237 HM: VIRT_X2APIC 00:00:02.013237 HM: VPID 00:00:02.013237 HM: WBINVD_EXIT 00:00:02.013238 HM: UNRESTRICTED_GUEST 00:00:02.013238 HM: APIC_REG_VIRT (must be cleared) 00:00:02.013238 HM: VIRT_INTR_DELIVERY (must be cleared) 00:00:02.013239 HM: PAUSE_LOOP_EXIT 00:00:02.013239 HM: RDRAND_EXIT 00:00:02.013239 HM: INVPCID 00:00:02.013240 HM: VMFUNC 00:00:02.013240 HM: VMCS_SHADOWING 00:00:02.013240 HM: ENCLS_EXIT 00:00:02.013240 HM: RDSEED_EXIT 00:00:02.013241 HM: PML 00:00:02.013241 HM: EPT_VE 00:00:02.013241 HM: CONCEAL_FROM_PT 00:00:02.013242 HM: XSAVES_XRSTORS 00:00:02.013242 HM: TSC_SCALING (must be cleared) 00:00:02.013255 HM: MSR_IA32_VMX_ENTRY_CTLS = 0x3ffff000011ff 00:00:02.013256 HM: LOAD_DEBUG (must be set) 00:00:02.013256 HM: IA32E_MODE_GUEST 00:00:02.013257 HM: ENTRY_SMM 00:00:02.013257 HM: DEACTIVATE_DUALMON 00:00:02.013257 HM: LOAD_GUEST_PERF_MSR 00:00:02.013257 HM: LOAD_GUEST_PAT_MSR 00:00:02.013258 HM: LOAD_GUEST_EFER_MSR 00:00:02.013258 HM: MSR_IA32_VMX_EXIT_CTLS = 0x1ffffff00036dff 00:00:02.013259 HM: SAVE_DEBUG (must be set) 00:00:02.013272 HM: HOST_ADDR_SPACE_SIZE 00:00:02.013273 HM: LOAD_PERF_MSR 00:00:02.013273 HM: ACK_EXT_INT 00:00:02.013273 HM: SAVE_GUEST_PAT_MSR 00:00:02.013273 HM: LOAD_HOST_PAT_MSR 00:00:02.013274 HM: SAVE_GUEST_EFER_MSR 00:00:02.013274 HM: LOAD_HOST_EFER_MSR 00:00:02.013274 HM: SAVE_VMX_PREEMPT_TIMER 00:00:02.013275 HM: MSR_IA32_VMX_EPT_VPID_CAP = 0xf0106334141 00:00:02.013275 HM: RWX_X_ONLY 00:00:02.013276 HM: PAGE_WALK_LENGTH_4 00:00:02.013276 HM: EMT_UC 00:00:02.013276 HM: EMT_WB 00:00:02.013277 HM: PDE_2M 00:00:02.013277 HM: PDPTE_1G 00:00:02.013277 HM: INVEPT 00:00:02.013277 HM: EPT_ACCESS_DIRTY 00:00:02.013278 HM: INVEPT_SINGLE_CONTEXT 00:00:02.013278 HM: INVEPT_ALL_CONTEXTS 00:00:02.013278 HM: INVVPID 00:00:02.013279 HM: INVVPID_INDIV_ADDR 00:00:02.013279 HM: INVVPID_SINGLE_CONTEXT 00:00:02.013279 HM: INVVPID_ALL_CONTEXTS 00:00:02.013280 HM: INVVPID_SINGLE_CONTEXT_RETAIN_GLOBALS 00:00:02.013280 HM: MSR_IA32_VMX_MISC = 0x7004c1e7 00:00:02.013281 HM: PREEMPT_TSC_BIT = 0x7 00:00:02.013281 HM: STORE_EFERLMA_VMEXIT = true 00:00:02.013282 HM: ACTIVITY_STATES = 0x7 00:00:02.013282 HM: CR3_TARGET = 0x4 00:00:02.013282 HM: MAX_MSR = 512 00:00:02.013283 HM: RDMSR_SMBASE_MSR_SMM = true 00:00:02.013283 HM: SMM_MONITOR_CTL_B2 = true 00:00:02.013284 HM: VMWRITE_VMEXIT_INFO = true 00:00:02.013284 HM: MSEG_ID = 0x0 00:00:02.013285 HM: MSR_IA32_VMX_CR0_FIXED0 = 0x80000021 00:00:02.013285 HM: MSR_IA32_VMX_CR0_FIXED1 = 0xffffffff 00:00:02.013286 HM: MSR_IA32_VMX_CR4_FIXED0 = 0x2000 00:00:02.013287 HM: MSR_IA32_VMX_CR4_FIXED1 = 0x3767ff 00:00:02.013287 HM: MSR_IA32_VMX_VMCS_ENUM = 0x2e 00:00:02.013288 HM: HIGHEST_INDEX = 0x17 00:00:02.013288 HM: MSR_IA32_VMX_VMFUNC = 0x1 00:00:02.013289 HM: EPTP_SWITCHING 00:00:02.013289 HM: APIC-access page physaddr = 0x000000008cbfd000 00:00:02.013290 HM: VCPU 0: MSR bitmap physaddr = 0x000000008cbfa000 00:00:02.013291 HM: VCPU 0: VMCS physaddr = 0x000000008cbfc000 00:00:02.013292 HM: VCPU 1: MSR bitmap physaddr = 0x000000008cbf4000 00:00:02.013293 HM: VCPU 1: VMCS physaddr = 0x000000008cbf7000 00:00:02.013293 HM: VCPU 2: MSR bitmap physaddr = 0x000000008cbee000 00:00:02.013294 HM: VCPU 2: VMCS physaddr = 0x000000008cbf0000 00:00:02.013295 HM: VCPU 3: MSR bitmap physaddr = 0x000000008cbe7000 00:00:02.013295 HM: VCPU 3: VMCS physaddr = 0x000000008cbea000 00:00:02.013296 HM: Guest support: 32-bit and 64-bit 00:00:02.013302 HM: Supports VMCS EFER fields = true 00:00:02.013303 HM: Enabled VMX 00:00:02.013305 CPUM: SetGuestCpuIdFeature: Enabled SYSENTER/EXIT 00:00:02.013306 CPUM: SetGuestCpuIdFeature: Enabled PAE 00:00:02.013306 CPUM: SetGuestCpuIdFeature: Enabled LONG MODE 00:00:02.013306 CPUM: SetGuestCpuIdFeature: Enabled SYSCALL/RET 00:00:02.013307 CPUM: SetGuestCpuIdFeature: Enabled LAHF/SAHF 00:00:02.013307 CPUM: SetGuestCpuIdFeature: Enabled NX 00:00:02.013308 HM: Enabled nested paging 00:00:02.013308 HM: EPT flush type = VMXFLUSHEPT_SINGLE_CONTEXT 00:00:02.013308 HM: Enabled unrestricted guest execution 00:00:02.013309 HM: Enabled VPID 00:00:02.013309 HM: VPID flush type = VMXFLUSHVPID_SINGLE_CONTEXT 00:00:02.013309 HM: Enabled VMX-preemption timer (cPreemptTimerShift=7) 00:00:02.013310 HM: VT-x/AMD-V init method: LOCAL 00:00:02.013325 CPUM: VCPU 0: Cached APIC base MSR = 0xfee00900 00:00:02.013374 CPUM: VCPU 1: Cached APIC base MSR = 0xfee00800 00:00:02.013382 CPUM: VCPU 2: Cached APIC base MSR = 0xfee00800 00:00:02.013386 CPUM: VCPU 3: Cached APIC base MSR = 0xfee00800 00:00:02.013390 VMM: fUsePeriodicPreemptionTimers=false 00:00:02.013440 CPUM: Logical host processors: 8 present, 8 max, 8 online, online mask: 00000000000000ff 00:00:02.013442 CPUM: Physical host cores: 4 00:00:02.013442 ************************* CPUID dump ************************ 00:00:02.013472 Raw Standard CPUID Leaves 00:00:02.013472 Leaf/sub-leaf eax ebx ecx edx 00:00:02.013492 Gst: 00000000/0000 00000016 756e6547 6c65746e 49656e69 00:00:02.013493 Hst: 00000016 756e6547 6c65746e 49656e69 00:00:02.013494 Gst: 00000001/0000 000506e3 00040800 d6f82203 178bfbff 00:00:02.013495 Hst: 000506e3 00100800 7ffafbff bfebfbff 00:00:02.013496 Gst: 00000002/0000 76036301 00f0b5ff 00000000 00c30000 00:00:02.013497 Hst: 76036301 00f0b5ff 00000000 00c30000 00:00:02.013498 Gst: 00000003/0000 00000000 00000000 00000000 00000000 00:00:02.013499 Hst: 00000000 00000000 00000000 00000000 00:00:02.013500 Gst: 00000004/0000 0c000121 01c0003f 0000003f 00000000 00:00:02.013501 Hst: 1c004121 01c0003f 0000003f 00000000 00:00:02.013502 Gst: 00000004/0001 0c000122 01c0003f 0000003f 00000000 00:00:02.013503 Hst: 1c004122 01c0003f 0000003f 00000000 00:00:02.013503 Gst: 00000004/0002 0c000143 00c0003f 000003ff 00000000 00:00:02.013504 Hst: 1c004143 00c0003f 000003ff 00000000 00:00:02.013505 Gst: 00000004/0003 0c000163 03c0003f 00001fff 00000006 00:00:02.013506 Hst: 1c03c163 03c0003f 00001fff 00000006 00:00:02.013507 Gst: 00000004/0004 0c000000 00000000 00000000 00000000 00:00:02.013508 Hst: 00000000 00000000 00000000 00000000 00:00:02.013508 Gst: 00000005/0000 00000000 00000000 00000000 00000000 00:00:02.013509 Hst: 00000040 00000040 00000003 11142120 00:00:02.013510 Gst: 00000006/0000 00000000 00000000 00000000 00000000 00:00:02.013511 Hst: 000027f7 00000002 00000009 00000000 00:00:02.013511 Gst: 00000007/0000 00000000 00842000 00000000 00000000 00:00:02.013512 Hst: 00000000 029c6fbf 00000000 00000000 00:00:02.013513 Gst: 00000007/0001 00000000 00000000 00000000 00000000 00:00:02.013513 Hst: 00000000 00000000 00000000 00000000 00:00:02.013514 Gst: 00000008/0000 00000000 00000000 00000000 00000000 00:00:02.013515 Hst: 00000000 00000000 00000000 00000000 00:00:02.013515 Gst: 00000009/0000 00000000 00000000 00000000 00000000 00:00:02.013516 Hst: 00000000 00000000 00000000 00000000 00:00:02.013516 Gst: 0000000a/0000 00000000 00000000 00000000 00000000 00:00:02.013517 Hst: 07300404 00000000 00000000 00000603 00:00:02.013518 Gst: 0000000b/0000 00000000 00000001 00000100 00000000 00:00:02.013518 Hst: 00000001 00000002 00000100 00000000 00:00:02.013519 Gst: 0000000b/0001 00000002 00000004 00000201 00000000 00:00:02.013520 Hst: 00000004 00000008 00000201 00000000 00:00:02.013520 Gst: 0000000b/0002 00000000 00000000 00000002 00000000 00:00:02.013521 Hst: 00000000 00000000 00000002 00000000 00:00:02.013522 Gst: 0000000c/0000 00000000 00000000 00000000 00000000 00:00:02.013522 Hst: 00000000 00000000 00000000 00000000 00:00:02.013523 Gst: 0000000d/0000 00000007 00000340 00000440 00000000 00:00:02.013523 Hst: 0000001f 00000340 00000440 00000000 00:00:02.013524 Gst: 0000000d/0001 00000000 00000340 00000000 00000000 00:00:02.013525 Hst: 0000000f 00000340 00000100 00000000 00:00:02.013526 Gst: 0000000d/0002 00000100 00000240 00000000 00000000 00:00:02.013526 Hst: 00000100 00000240 00000000 00000000 00:00:02.013527 Gst: 0000000d/0003 00000000 00000000 00000000 00000000 00:00:02.013527 Hst: 00000040 000003c0 00000000 00000000 00:00:02.013528 Gst: 0000000d/0004 00000000 00000000 00000000 00000000 00:00:02.013529 Hst: 00000040 00000400 00000000 00000000 00:00:02.013529 Gst: 0000000d/0005 00000000 00000000 00000000 00000000 00:00:02.013530 Hst: 00000000 00000000 00000000 00000000 00:00:02.013531 Gst: 0000000d/0006 00000000 00000000 00000000 00000000 00:00:02.013531 Hst: 00000000 00000000 00000000 00000000 00:00:02.013532 Gst: 0000000d/0007 00000000 00000000 00000000 00000000 00:00:02.013532 Hst: 00000000 00000000 00000000 00000000 00:00:02.013533 Gst: 0000000d/0008 00000000 00000000 00000000 00000000 00:00:02.013534 Hst: 00000080 00000000 00000001 00000000 00:00:02.013534 Gst: 0000000d/0009 00000000 00000000 00000000 00000000 00:00:02.013535 Hst: 00000000 00000000 00000000 00000000 00:00:02.013555 Gst: 0000000e/0000 00000000 00000000 00000000 00000000 00:00:02.013556 Hst: 00000000 00000000 00000000 00000000 00:00:02.013556 Gst: 0000000f/0000 00000000 00000000 00000000 00000000 00:00:02.013557 Hst: 00000000 00000000 00000000 00000000 00:00:02.013557 Gst: 00000010/0000 00000000 00000000 00000000 00000000 00:00:02.013558 Hst: 00000000 00000000 00000000 00000000 00:00:02.013559 Gst: 00000011/0000 00000000 00000000 00000000 00000000 00:00:02.013559 Hst: 00000000 00000000 00000000 00000000 00:00:02.013560 Gst: 00000012/0000 00000000 00000000 00000000 00000000 00:00:02.013561 Hst: 00000000 00000000 00000000 00000000 00:00:02.013561 Gst: 00000013/0000 00000000 00000000 00000000 00000000 00:00:02.013562 Hst: 00000000 00000000 00000000 00000000 00:00:02.013562 Gst: 00000014/0000 00000000 00000000 00000000 00000000 00:00:02.013563 Hst: 00000001 0000000f 00000007 00000000 00:00:02.013564 Hst: 00000015/0000 00000002 000000e2 00000000 00000000 00:00:02.013565 Hst: 00000016/0000 00000a8c 00000e10 00000064 00000000 00:00:02.013566 Name: GenuineIntel 00:00:02.013567 Supports: 0x00000000-0x00000016 00:00:02.013593 Family: 6 Extended: 0 Effective: 6 00:00:02.013595 Model: 14 Extended: 5 Effective: 94 00:00:02.013596 Stepping: 3 00:00:02.013596 Type: 0 (primary) 00:00:02.013597 APIC ID: 0x00 00:00:02.013598 Logical CPUs: 4 00:00:02.013598 CLFLUSH Size: 8 00:00:02.013599 Brand ID: 0x00 00:00:02.013600 Features 00:00:02.013600 Mnemonic - Description = guest (host) 00:00:02.013601 FPU - x87 FPU on Chip = 1 (1) 00:00:02.013602 VME - Virtual 8086 Mode Enhancements = 1 (1) 00:00:02.013603 DE - Debugging extensions = 1 (1) 00:00:02.013604 PSE - Page Size Extension = 1 (1) 00:00:02.013605 TSC - Time Stamp Counter = 1 (1) 00:00:02.013606 MSR - Model Specific Registers = 1 (1) 00:00:02.013607 PAE - Physical Address Extension = 1 (1) 00:00:02.013608 MCE - Machine Check Exception = 1 (1) 00:00:02.013608 CX8 - CMPXCHG8B instruction = 1 (1) 00:00:02.013609 APIC - APIC On-Chip = 1 (1) 00:00:02.013610 SEP - SYSENTER and SYSEXIT Present = 1 (1) 00:00:02.013611 MTRR - Memory Type Range Registers = 1 (1) 00:00:02.013612 PGE - PTE Global Bit = 1 (1) 00:00:02.013613 MCA - Machine Check Architecture = 1 (1) 00:00:02.013614 CMOV - Conditional Move instructions = 1 (1) 00:00:02.013614 PAT - Page Attribute Table = 1 (1) 00:00:02.013615 PSE-36 - 36-bit Page Size Extension = 1 (1) 00:00:02.013616 PSN - Processor Serial Number = 0 (0) 00:00:02.013617 CLFSH - CLFLUSH instruction = 1 (1) 00:00:02.013618 DS - Debug Store = 0 (1) 00:00:02.013619 ACPI - Thermal Mon. & Soft. Clock Ctrl. = 0 (1) 00:00:02.013620 MMX - Intel MMX Technology = 1 (1) 00:00:02.013621 FXSR - FXSAVE and FXRSTOR instructions = 1 (1) 00:00:02.013621 SSE - SSE support = 1 (1) 00:00:02.013622 SSE2 - SSE2 support = 1 (1) 00:00:02.013623 SS - Self Snoop = 0 (1) 00:00:02.013624 HTT - Hyper-Threading Technology = 1 (1) 00:00:02.013625 TM - Therm. Monitor = 0 (1) 00:00:02.013626 PBE - Pending Break Enabled = 0 (1) 00:00:02.013627 SSE3 - SSE3 support = 1 (1) 00:00:02.013628 PCLMUL - PCLMULQDQ support (for AES-GCM) = 1 (1) 00:00:02.013629 DTES64 - DS Area 64-bit Layout = 0 (1) 00:00:02.013629 MONITOR - MONITOR/MWAIT instructions = 0 (1) 00:00:02.013630 CPL-DS - CPL Qualified Debug Store = 0 (1) 00:00:02.013631 VMX - Virtual Machine Extensions = 0 (1) 00:00:02.013632 SMX - Safer Mode Extensions = 0 (1) 00:00:02.013633 EST - Enhanced SpeedStep Technology = 0 (1) 00:00:02.013633 TM2 - Terminal Monitor 2 = 0 (1) 00:00:02.013634 SSSE3 - Supplemental Streaming SIMD Extensions 3 = 1 (1) 00:00:02.013635 CNTX-ID - L1 Context ID = 0 (0) 00:00:02.013636 SDBG - Silicon Debug interface = 0 (1) 00:00:02.013637 FMA - Fused Multiply Add extensions = 0 (1) 00:00:02.013638 CX16 - CMPXCHG16B instruction = 1 (1) 00:00:02.013638 TPRUPDATE - xTPR Update Control = 0 (1) 00:00:02.013639 PDCM - Perf/Debug Capability MSR = 0 (1) 00:00:02.013640 PCID - Process Context Identifiers = 0 (1) 00:00:02.013641 DCA - Direct Cache Access = 0 (0) 00:00:02.013642 SSE4_1 - SSE4_1 support = 1 (1) 00:00:02.013643 SSE4_2 - SSE4_2 support = 1 (1) 00:00:02.013644 X2APIC - x2APIC support = 1 (1) 00:00:02.013644 MOVBE - MOVBE instruction = 1 (1) 00:00:02.013645 POPCNT - POPCNT instruction = 1 (1) 00:00:02.013646 TSCDEADL - Time Stamp Counter Deadline = 0 (1) 00:00:02.013647 AES - AES instructions = 1 (1) 00:00:02.013648 XSAVE - XSAVE instruction = 1 (1) 00:00:02.013649 OSXSAVE - OSXSAVE instruction = 0 (1) 00:00:02.013649 AVX - AVX support = 1 (1) 00:00:02.013650 F16C - 16-bit floating point conversion instructions = 0 (1) 00:00:02.013665 RDRAND - RDRAND instruction = 1 (1) 00:00:02.013666 HVP - Hypervisor Present (we're a guest) = 1 (0) 00:00:02.013667 Structured Extended Feature Flags Enumeration (leaf 7): 00:00:02.013667 Mnemonic - Description = guest (host) 00:00:02.013668 FSGSBASE - RDFSBASE/RDGSBASE/WRFSBASE/WRGSBASE instr. = 0 (1) 00:00:02.013668 TSCADJUST - Supports MSR_IA32_TSC_ADJUST = 0 (1) 00:00:02.013669 SGX - Supports Software Guard Extensions = 0 (1) 00:00:02.013670 BMI1 - Advanced Bit Manipulation extension 1 = 0 (1) 00:00:02.013671 HLE - Hardware Lock Elision = 0 (1) 00:00:02.013672 AVX2 - Advanced Vector Extensions 2 = 0 (1) 00:00:02.013672 FDP_EXCPTN_ONLY - FPU DP only updated on exceptions = 0 (0) 00:00:02.013673 SMEP - Supervisor Mode Execution Prevention = 0 (1) 00:00:02.013674 BMI2 - Advanced Bit Manipulation extension 2 = 0 (1) 00:00:02.013674 ERMS - Enhanced REP MOVSB/STOSB instructions = 0 (1) 00:00:02.013675 INVPCID - INVPCID instruction = 0 (1) 00:00:02.013676 RTM - Restricted Transactional Memory = 0 (1) 00:00:02.013677 PQM - Platform Quality of Service Monitoring = 0 (0) 00:00:02.013677 DEPFPU_CS_DS - Deprecates FPU CS, FPU DS values if set = 1 (1) 00:00:02.013678 MPE - Intel Memory Protection Extensions = 0 (1) 00:00:02.013679 PQE - Platform Quality of Service Enforcement = 0 (0) 00:00:02.013680 AVX512F - AVX512 Foundation instructions = 0 (0) 00:00:02.013680 RDSEED - RDSEED instruction = 1 (1) 00:00:02.013681 ADX - ADCX/ADOX instructions = 0 (1) 00:00:02.013682 SMAP - Supervisor Mode Access Prevention = 0 (1) 00:00:02.013683 CLFLUSHOPT - CLFLUSHOPT (Cache Line Flush) instruction = 1 (1) 00:00:02.013683 INTEL_PT - Intel Processor Trace = 0 (1) 00:00:02.013684 AVX512PF - AVX512 Prefetch instructions = 0 (0) 00:00:02.013685 AVX512ER - AVX512 Exponential & Reciprocal instructions = 0 (0) 00:00:02.013686 AVX512CD - AVX512 Conflict Detection instructions = 0 (0) 00:00:02.013686 SHA - Secure Hash Algorithm extensions = 0 (0) 00:00:02.013687 PREFETCHWT1 - PREFETCHWT1 instruction = 0 (0) 00:00:02.013688 PKU - Protection Key for Usermode pages = 0 (0) 00:00:02.013689 OSPKU - CR4.PKU mirror = 0 (0) 00:00:02.013704 Processor Extended State Enumeration (leaf 0xd): 00:00:02.013705 XSAVE area cur/max size by XCR0, guest: 0x340/0x440 00:00:02.013705 XSAVE area cur/max size by XCR0, host: 0x340/0x440 00:00:02.013706 Valid XCR0 bits, guest: 0x00000000`00000007 ( x87 SSE YMM_Hi128 ) 00:00:02.013708 Valid XCR0 bits, host: 0x00000000`0000001f ( x87 SSE YMM_Hi128 BNDREGS BNDCSR ) 00:00:02.013710 XSAVE features, guest: 00:00:02.013710 XSAVE features, host: XSAVEOPT XSAVEC XGETBC1 XSAVES 00:00:02.013712 XSAVE area cur size XCR0|XSS, guest: 0x340 00:00:02.013712 XSAVE area cur size XCR0|XSS, host: 0x340 00:00:02.013713 Valid IA32_XSS bits, guest: 0x00000000`00000000 00:00:02.013713 Valid IA32_XSS bits, host: 0x00000100`00000000 ( 40 ) 00:00:02.013715 State #2, guest: off=0x0240, cb=0x0100 IA32_XSS-bit -- YMM_Hi128 00:00:02.013716 State #2, host: off=0x0240, cb=0x0100 IA32_XSS-bit -- YMM_Hi128 00:00:02.013718 State #3, host: off=0x03c0, cb=0x0040 IA32_XSS-bit -- BNDREGS 00:00:02.013719 State #4, host: off=0x0400, cb=0x0040 IA32_XSS-bit -- BNDCSR 00:00:02.013720 State #8, host: off=0x0000, cb=0x0080 XCR0-bit -- 8 00:00:02.013730 Unknown CPUID Leaves 00:00:02.013731 Leaf/sub-leaf eax ebx ecx edx 00:00:02.013731 Gst: 00000014/0001 00000000 00000000 00000000 00000000 00:00:02.013732 Hst: 02490002 003f3fff 00000000 00000000 00:00:02.013733 Gst: 00000014/0002 00000000 00000000 00000000 00000000 00:00:02.013734 Hst: 00000000 00000000 00000000 00000000 00:00:02.013734 Gst: 00000015/0000 00000000 00000000 00000000 00000000 00:00:02.013735 Hst: 00000002 000000e2 00000000 00000000 00:00:02.013736 Gst: 00000016/0000 00000000 00000000 00000000 00000000 00:00:02.013737 Hst: 00000a8c 00000e10 00000064 00000000 00:00:02.013738 Raw Hypervisor CPUID Leaves 00:00:02.013738 Leaf/sub-leaf eax ebx ecx edx 00:00:02.013739 Gst: 40000000/0000 40000001 4b4d564b 564b4d56 0000004d 00:00:02.013740 Hst: 00000a8c 00000e10 00000064 00000000 00:00:02.013741 Gst: 40000001/0000 01000089 00000000 00000000 00000000 00:00:02.013742 Hst: 00000a8c 00000e10 00000064 00000000 00:00:02.013743 Raw Extended CPUID Leaves 00:00:02.013743 Leaf/sub-leaf eax ebx ecx edx 00:00:02.013743 Gst: 80000000/0000 80000008 00000000 00000000 00000000 00:00:02.013744 Hst: 80000008 00000000 00000000 00000000 00:00:02.013745 Gst: 80000001/0000 00000000 00000000 00000121 28100800 00:00:02.013746 Hst: 00000000 00000000 00000121 2c100800 00:00:02.013747 Gst: 80000002/0000 65746e49 2952286c 726f4320 4d542865 00:00:02.013748 Hst: 65746e49 2952286c 726f4320 4d542865 00:00:02.013750 Gst: 80000003/0000 37692029 3238362d 20514830 20555043 00:00:02.013751 Hst: 37692029 3238362d 20514830 20555043 00:00:02.013752 Gst: 80000004/0000 2e322040 48473037 0000007a 00000000 00:00:02.013753 Hst: 2e322040 48473037 0000007a 00000000 00:00:02.013754 Gst: 80000005/0000 00000000 00000000 00000000 00000000 00:00:02.013754 Hst: 00000000 00000000 00000000 00000000 00:00:02.013755 Gst: 80000006/0000 00000000 00000000 01006040 00000000 00:00:02.013756 Hst: 00000000 00000000 01006040 00000000 00:00:02.013757 Gst: 80000007/0000 00000000 00000000 00000000 00000100 00:00:02.013757 Hst: 00000000 00000000 00000000 00000100 00:00:02.013758 Gst: 80000008/0000 00003027 00000000 00000000 00000000 00:00:02.013759 Hst: 00003027 00000000 00000000 00000000 00:00:02.013759 Ext Name: 00:00:02.013760 Ext Supports: 0x80000000-0x80000008 00:00:02.013761 Family: 0 Extended: 0 Effective: 0 00:00:02.013761 Model: 0 Extended: 0 Effective: 0 00:00:02.013762 Stepping: 0 00:00:02.013762 Brand ID: 0x000 00:00:02.013763 Ext Features 00:00:02.013763 Mnemonic - Description = guest (host) 00:00:02.013763 FPU - x87 FPU on Chip = 0 (0) 00:00:02.013764 VME - Virtual 8086 Mode Enhancements = 0 (0) 00:00:02.013765 DE - Debugging extensions = 0 (0) 00:00:02.013766 PSE - Page Size Extension = 0 (0) 00:00:02.013767 TSC - Time Stamp Counter = 0 (0) 00:00:02.013768 MSR - K86 Model Specific Registers = 0 (0) 00:00:02.013769 PAE - Physical Address Extension = 0 (0) 00:00:02.013769 MCE - Machine Check Exception = 0 (0) 00:00:02.013770 CX8 - CMPXCHG8B instruction = 0 (0) 00:00:02.013771 APIC - APIC On-Chip = 0 (0) 00:00:02.013772 SEP - SYSCALL/SYSRET = 1 (1) 00:00:02.013773 MTRR - Memory Type Range Registers = 0 (0) 00:00:02.013774 PGE - PTE Global Bit = 0 (0) 00:00:02.013775 MCA - Machine Check Architecture = 0 (0) 00:00:02.013775 CMOV - Conditional Move instructions = 0 (0) 00:00:02.013776 PAT - Page Attribute Table = 0 (0) 00:00:02.013777 PSE-36 - 36-bit Page Size Extension = 0 (0) 00:00:02.013778 NX - No-Execute/Execute-Disable = 1 (1) 00:00:02.013779 AXMMX - AMD Extensions to MMX instructions = 0 (0) 00:00:02.013779 MMX - Intel MMX Technology = 0 (0) 00:00:02.013780 FXSR - FXSAVE and FXRSTOR Instructions = 0 (0) 00:00:02.013781 FFXSR - AMD fast FXSAVE and FXRSTOR instructions = 0 (0) 00:00:02.013782 Page1GB - 1 GB large page = 0 (1) 00:00:02.013782 RDTSCP - RDTSCP instruction = 1 (1) 00:00:02.013783 LM - AMD64 Long Mode = 1 (1) 00:00:02.013784 3DNOWEXT - AMD Extensions to 3DNow = 0 (0) 00:00:02.013785 3DNOW - AMD 3DNow = 0 (0) 00:00:02.013786 LahfSahf - LAHF/SAHF support in 64-bit mode = 1 (1) 00:00:02.013787 CmpLegacy - Core multi-processing legacy mode = 0 (0) 00:00:02.013787 SVM - AMD Secure Virtual Machine extensions = 0 (0) 00:00:02.013788 EXTAPIC - AMD Extended APIC registers = 0 (0) 00:00:02.013789 CR8L - AMD LOCK MOV CR0 means MOV CR8 = 0 (0) 00:00:02.013790 ABM - AMD Advanced Bit Manipulation = 1 (1) 00:00:02.013790 SSE4A - SSE4A instructions = 0 (0) 00:00:02.013791 MISALIGNSSE - AMD Misaligned SSE mode = 0 (0) 00:00:02.013792 3DNOWPRF - AMD PREFETCH and PREFETCHW instructions = 1 (1) 00:00:02.013793 OSVW - AMD OS Visible Workaround = 0 (0) 00:00:02.013793 IBS - Instruct Based Sampling = 0 (0) 00:00:02.013794 XOP - Extended Operation support = 0 (0) 00:00:02.013795 SKINIT - SKINIT, STGI, and DEV support = 0 (0) 00:00:02.013796 WDT - AMD Watchdog Timer support = 0 (0) 00:00:02.013797 LWP - Lightweight Profiling support = 0 (0) 00:00:02.013797 FMA4 - Four operand FMA instruction support = 0 (0) 00:00:02.013798 NodeId - NodeId in MSR C001_100C = 0 (0) 00:00:02.013799 TBM - Trailing Bit Manipulation instructions = 0 (0) 00:00:02.013800 TOPOEXT - Topology Extensions = 0 (0) 00:00:02.013801 Full Name: "Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz" 00:00:02.013801 TLB 2/4M Instr/Uni: res0 0 entries 00:00:02.013802 TLB 2/4M Data: res0 0 entries 00:00:02.013803 TLB 4K Instr/Uni: res0 0 entries 00:00:02.013803 TLB 4K Data: res0 0 entries 00:00:02.013804 L1 Instr Cache Line Size: 0 bytes 00:00:02.013804 L1 Instr Cache Lines Per Tag: 0 00:00:02.013804 L1 Instr Cache Associativity: res0 00:00:02.013805 L1 Instr Cache Size: 0 KB 00:00:02.013805 L1 Data Cache Line Size: 0 bytes 00:00:02.013805 L1 Data Cache Lines Per Tag: 0 00:00:02.013806 L1 Data Cache Associativity: res0 00:00:02.013806 L1 Data Cache Size: 0 KB 00:00:02.013807 L2 TLB 2/4M Instr/Uni: off 0 entries 00:00:02.013807 L2 TLB 2/4M Data: off 0 entries 00:00:02.013808 L2 TLB 4K Instr/Uni: off 0 entries 00:00:02.013808 L2 TLB 4K Data: off 0 entries 00:00:02.013809 L2 Cache Line Size: 0 bytes 00:00:02.013809 L2 Cache Lines Per Tag: 0 00:00:02.013809 L2 Cache Associativity: off 00:00:02.013810 L2 Cache Size: 0 KB 00:00:02.013810 APM Features: TscInvariant 00:00:02.013811 Host Invariant-TSC support: true 00:00:02.013812 Physical Address Width: 39 bits 00:00:02.013812 Virtual Address Width: 48 bits 00:00:02.013812 Guest Physical Address Width: 0 bits 00:00:02.013813 Physical Core Count: 1 00:00:02.013814 00:00:02.013814 ******************** End of CPUID dump ********************** 00:00:02.035410 PcBios: ATA LUN#0 LCHS=1024/255/63 00:00:02.035476 APIC: fPostedIntrsEnabled=false fVirtApicRegsEnabled=false fSupportsTscDeadline=false 00:00:02.035552 VMEmt: Halt method global1 (5) 00:00:02.035627 VMEmt: HaltedGlobal1 config: cNsSpinBlockThresholdCfg=125000 00:00:02.035664 Changing the VM state from 'CREATING' to 'CREATED' 00:00:02.036708 Changing the VM state from 'CREATED' to 'POWERING_ON' 00:00:02.036857 Changing the VM state from 'POWERING_ON' to 'RUNNING' 00:00:02.036878 Console: Machine state changed to 'Running' 00:00:02.042232 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={02326f63-bcb3-4481-96e0-30d1c2ee97f6} aComponent={DisplayWrap} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false aResultDetail=0 00:00:02.045068 VMMDev: Guest Log: BIOS: VirtualBox 5.1.30 00:00:02.046284 PIT: mode=2 count=0x10000 (65536) - 18.20 Hz (ch=0) 00:00:02.052853 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={02326f63-bcb3-4481-96e0-30d1c2ee97f6} aComponent={DisplayWrap} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false aResultDetail=0 00:00:02.057378 Display::handleDisplayResize: uScreenId=0 pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0 flags=0x0 00:00:02.057478 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=720x400, Sending to async-handler 00:00:02.057628 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=720x400 00:00:02.057690 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=720x400 00:00:02.057700 GUI: UIFrameBufferPrivate::performResize: Size=720x400, Directly using source bitmap content 00:00:02.058379 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0x00 (-1 usec ago) CmdIf1=0x00 (-1 usec ago) 00:00:02.058413 PIIX3 ATA: Ctl#0: finished processing RESET 00:00:02.058738 VMMDev: Guest Log: BIOS: ata0-0: PCHS=16383/16/63 LCHS=1024/255/63 00:00:02.059770 PIIX3 ATA: Ctl#0: RESET, DevSel=1 AIOIf=0 CmdIf0=0xec (-1 usec ago) CmdIf1=0x00 (-1 usec ago) 00:00:02.059942 PIIX3 ATA: Ctl#0: finished processing RESET 00:00:02.067062 PIT: mode=2 count=0x48d3 (18643) - 64.00 Hz (ch=0) 00:00:02.081192 Display::handleDisplayResize: uScreenId=0 pvVRAM=0000000012c60000 w=640 h=480 bpp=32 cbLine=0xA00 flags=0x0 00:00:02.081270 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=640x480, Sending to async-handler 00:00:02.081403 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=640x480 00:00:02.081421 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=640x480 00:00:02.081518 GUI: UIFrameBufferPrivate::performResize: Size=640x480, Directly using source bitmap content 00:00:02.346525 GUI: UIMachineViewNormal::resendSizeHint: Restoring guest size-hint for screen 0 to 800x600 00:00:02.346646 VMMDev: SetVideoModeHint: Got a video mode hint (800x600x32)@(0x0),(1;0) at 0 00:00:02.349942 GUI: 2D video acceleration is disabled 00:00:02.349942 GUI: HID LEDs sync is enabled 00:00:02.349942 GUI: UIMachineLogicNormal::sltCheckForRequestedVisualStateType: Requested-state=0, Machine-state=5 00:00:04.557413 Display::handleDisplayResize: uScreenId=0 pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0 flags=0x0 00:00:04.557603 GUI: UIFrameBufferPrivate::NotifyChange: Screen=0, Origin=0x0, Size=720x400, Sending to async-handler 00:00:04.557708 GUI: UIMachineView::sltHandleNotifyChange: Screen=0, Size=720x400 00:00:04.557747 GUI: UIFrameBufferPrivate::handleNotifyChange: Size=720x400 00:00:04.557776 GUI: UIFrameBufferPrivate::performResize: Size=720x400, Directly using source bitmap content 00:00:04.563415 PIT: mode=2 count=0x10000 (65536) - 18.20 Hz (ch=0) 00:00:04.563826 VMMDev: Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032 00:00:04.568770 VMMDev: Guest Log: BIOS: Booting from Hard Disk... 00:00:04.569711 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xc4 (-1 usec ago) CmdIf1=0x00 (-1 usec ago) 00:00:04.569785 PIIX3 ATA: Ctl#0: finished processing RESET 00:00:09.912603 VMMDev: Guest Log: BIOS: KBD: unsupported int 16h function 03 00:00:09.912885 VMMDev: Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000 00:00:09.913346 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=81 00:00:09.913677 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=81 00:00:09.914024 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=82 00:00:09.914351 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=82 00:00:09.914702 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=83 00:00:09.914996 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=83 00:00:09.915415 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=84 00:00:09.915835 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=84 00:00:09.916140 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=85 00:00:09.916461 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=85 00:00:09.916759 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=86 00:00:09.917078 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=86 00:00:09.917417 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=87 00:00:09.917695 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=87 00:00:09.918074 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=88 00:00:09.918356 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=88 00:00:09.918699 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=89 00:00:09.918978 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=89 00:00:09.919318 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8a 00:00:09.919596 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8a 00:00:09.919926 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8b 00:00:09.919942 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8b 00:00:09.919942 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8c 00:00:09.919942 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8c 00:00:09.919942 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8d 00:00:09.920050 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8d 00:00:09.920348 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8e 00:00:09.920631 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8e 00:00:09.920929 VMMDev: Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8f 00:00:09.921207 VMMDev: Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8f 00:00:10.100413 GIM: KVM: VCPU 0: Enabled system-time struct. at 0x000000039db52000 - u32TscScale=0xbc4eefc7 i8TscShift=-1 uVersion=2 fFlags=0x1 uTsc=0x51abec127 uVirtNanoTS=0x1e09b82e1 00:00:10.100554 TM: Switching TSC mode from 'VirtTscEmulated' to 'RealTscOffset' 00:00:10.457505 GIM: KVM: Enabled wall-clock struct. at 0x00000000020c66e8 - u32Sec=1511854862 u32Nano=908505120 uVersion=2 00:00:10.468042 PIT: mode=2 count=0x4a9 (1193) - 1000.15 Hz (ch=0) 00:00:10.499942 APIC0: Switched mode to x2APIC 00:00:10.604577 PIT: mode=0 count=0x10000 (65536) - 18.20 Hz (ch=0) 00:00:10.605408 CPUM: VCPU 1: Cached APIC base MSR = 0xfee00800 00:00:10.605502 APIC1: Switched mode to x2APIC 00:00:10.605537 GIM: KVM: VCPU 1: Enabled system-time struct. at 0x000000039db52040 - u32TscScale=0xbc4eefc7 i8TscShift=-1 uVersion=2 fFlags=0x1 uTsc=0x51abec127 uVirtNanoTS=0x1e09b82e1 00:00:10.608234 CPUM: VCPU 2: Cached APIC base MSR = 0xfee00800 00:00:10.608324 APIC2: Switched mode to x2APIC 00:00:10.608357 GIM: KVM: VCPU 2: Enabled system-time struct. at 0x000000039db52080 - u32TscScale=0xbc4eefc7 i8TscShift=-1 uVersion=2 fFlags=0x1 uTsc=0x51abec127 uVirtNanoTS=0x1e09b82e1 00:00:10.609943 CPUM: VCPU 3: Cached APIC base MSR = 0xfee00800 00:00:10.610015 APIC3: Switched mode to x2APIC 00:00:10.610047 GIM: KVM: VCPU 3: Enabled system-time struct. at 0x000000039db520c0 - u32TscScale=0xbc4eefc7 i8TscShift=-1 uVersion=2 fFlags=0x1 uTsc=0x51abec127 uVirtNanoTS=0x1e09b82e1 00:00:10.811411 OHCI: Software reset 00:00:11.319059 OHCI: USB Reset 00:00:11.370332 OHCI: Software reset 00:00:11.370898 OHCI: USB Operational 00:00:12.000874 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0xc4 (-1 usec ago) CmdIf1=0x00 (-1 usec ago) 00:00:12.000955 PIIX3 ATA: Ctl#0: finished processing RESET 00:00:12.001289 PIIX3 ATA: Ctl#1: RESET, DevSel=0 AIOIf=0 CmdIf0=0x00 (-1 usec ago) CmdIf1=0x00 (-1 usec ago) 00:00:12.001375 PIIX3 ATA: Ctl#1: finished processing RESET 00:00:13.652074 AC97: Reset 00:00:13.652446 AC97: Reset 00:00:13.999528 NAT: Link up 00:00:15.879441 NAT: IPv6 not supported 00:00:15.962698 NAT: DHCP offered IP address 10.0.2.15 The VirtualBox sandbox configuration, the Windows machine config. and the actual memory utilization are attached ... View more

Any way to retrieve the constraints of a Hive tabl...

by Expert Contributor kaliyugantagoni in Support Questions
‎11-24-2017 02:08 PM
‎11-24-2017 02:08 PM
Kerberized HDP-2.6.3.0. The table has 'Id' as it's primary key and 'DepartmentId' as the foreign key: +---------------------------------------------------------+-------------------------------------------------------------------------+-----------------------------+--+ | col_name | data_type | comment | +---------------------------------------------------------+-------------------------------------------------------------------------+-----------------------------+--+ | # col_name | data_type | comment | | | NULL | NULL | | id | int | Surrogate PK is not fun | | firstname | string | | | lastname | string | | | dob | date | | | departmentid | int | | | | NULL | NULL | | # Detailed Table Information | NULL | NULL | | Database: | group_hadoopdeveloper | NULL | | Owner: | ojoqcu | NULL | | CreateTime: | Fri Nov 24 13:39:02 UTC 2017 | NULL | | LastAccessTime: | UNKNOWN | NULL | | Retention: | 0 | NULL | | Location: | hdfs://devhadoop/apps/hive/warehouse/group_hadoopdeveloper.db/employee | NULL | | Table Type: | MANAGED_TABLE | NULL | | Table Parameters: | NULL | NULL | | | COLUMN_STATS_ACCURATE | {\"BASIC_STATS\":\"true\"} | | | numFiles | 0 | | | numRows | 0 | | | rawDataSize | 0 | | | totalSize | 0 | | | transient_lastDdlTime | 1511530742 | | | NULL | NULL | | # Storage Information | NULL | NULL | | SerDe Library: | org.apache.hadoop.hive.ql.io.orc.OrcSerde | NULL | | InputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL | | OutputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | | Compressed: | No | NULL | | Num Buckets: | -1 | NULL | | Bucket Columns: | [] | NULL | | Sort Columns: | [] | NULL | | Storage Desc Params: | NULL | NULL | | | serialization.format | 1 | | | NULL | NULL | | # Constraints | NULL | NULL | | | NULL | NULL | | # Primary Key | NULL | NULL | | Table: | group_hadoopdeveloper.employee | NULL | | Constraint Name: | pk_203923149_1511530742932_0 | NULL | | Column Names: | id | | | | NULL | NULL | | # Foreign Keys | NULL | NULL | | Table: | group_hadoopdeveloper.employee | NULL | | Constraint Name: | fk_employee_department | NULL | | Parent Column Name:group_hadoopdeveloper.department.id | Column Name:departmentid | Key Sequence:1 | | | NULL | NULL | +---------------------------------------------------------+-------------------------------------------------------------------------+-----------------------------+--+ I tried the following ways to retrieve the constraints: Hive JDBC - can retrieve the primary keys but the foreign keys are not implemented in the driver WebHCat: http://l4283t.sss.se.com:50111/templeton/v1/ddl/database/group_hadoopdeveloper/table/employee {"columns":[{"name":"id","type":"int","comment":"Surrogate PK is not fun"},{"name":"firstname","type":"string"},{"name":"lastname","type":"string"},{"name":"dob","type":"date"},{"name":"departmentid","type":"int"}],"database":"group_hadoopdeveloper","table":"employee"} Still struggling with the HiveMetaStoreClient implementation to retrieve the foreign keys I have the following questions: Where /how are these constraints stored in the Hive metastore Is there any way to retrieve(NOT from the beeline but those need to be available to external programs) these constraints ... View more
Labels:

Use of HiveMetaStoreClient(thereby, HiveConf) to r...

by Expert Contributor kaliyugantagoni in Support Questions
‎11-23-2017 02:58 PM
‎11-23-2017 02:58 PM
Kerberized HDP-2.6.3.0. I am able to connect to Hive from my Windows machine using the Hive JDBC driver, however, I need to use some methods of the HiveMetaStoreClient . I flipped through the api and wrote a test code which I am executing from an IDE. private static void connectHiveMetastore() throws MetaException, MalformedURLException { //System.setProperty("javax.security.auth.useSubjectCredsOnly","false"); //System.setProperty("java.security.krb5.conf","C:\\kerb5.conf"); Configuration configuration = new Configuration(); //configuration.addResource("E:\\hdp\\client_config\\HDFS_CLIENT\\core-site.xml"); //configuration.addResource("E:\\hdp\\client_config\\HDFS_CLIENT\\hdfs-site.xml"); HiveConf hiveConf = new HiveConf(configuration,Configuration.class); //URL url = new File("E:\\hdp\\client_config\\HDFS_CLIENT\\hive-site.xml").toURI().toURL(); //hiveConf.setHiveSiteLocation(url); //hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS,"thrift://l4283t.sss.com:9083,thrift://l4284t.sss.com:9083"); HiveMetaStoreClient hiveMetaStoreClient = new HiveMetaStoreClient(hiveConf); } The dependencies in the pom file: </dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-metastore --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-metastore</artifactId> <version>2.3.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>2.3.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.9.0</version> </dependency> </dependencies> Irrespective of whether I comment or uncomment the lines pertaining to the config and Kerberos, I receive the following exception which is explained on the Hive wiki: 15:35:27.139 [main] ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler - MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7564) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) at com.sun.proxy.$Proxy8.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:129) at com.my.App.connectHiveMetastore(App.java:58) at com.my.App.main(App.java:37) 15:35:27.141 [main] ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler - HMSHandler Fatal error: MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7564) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) at com.sun.proxy.$Proxy8.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:129) at com.my.App.connectHiveMetastore(App.java:58) at com.my.App.main(App.java:37) Exception in thread "main" MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:83) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:129) at com.my.App.connectHiveMetastore(App.java:58) at com.my.App.main(App.java:37) Caused by: MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7564) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) at com.sun.proxy.$Proxy8.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79) ... 6 more Process finished with exit code 1 I have the following questions/concerns: Is the way I am thinking about and connecting the HiveMetaStoreClient correct? If not, how do I retrieve the metadata information provided by the methods of HiveMetaStoreClient ? The code certainly isn't 'reaching' the cluster. Is the above exception pertaining to the dependency versions? If not, what can be the root cause? ... View more
Labels:

Re: JDBC from Oracle Data Integrator to Kerberized...

by Expert Contributor kaliyugantagoni in Support Questions
‎11-10-2017 04:53 PM
‎11-10-2017 04:53 PM
Yeah, I have tried that approach as well. The ODI doc. mentions about using it's weblogic hive jdbc driver but one can use other drivers as well. The question that I have mentioned here is around the standard(Apache)jdbc driver. ... View more

JDBC from Oracle Data Integrator to Kerberized HDP...

by Expert Contributor kaliyugantagoni in Support Questions
‎11-10-2017 01:51 PM
‎11-10-2017 01:51 PM
ODI studio 12.2.1.2.6 Kerberized HDP 2.6.3.0 Windows 7 hive-jdbc-2.1.0.2.6.3.0-235-standalone.jar(that is available under /usr/hdp/2.6.3.0-235/hive2/jdbc) I simply get a 'Connection refused', also, I didn't find any answers to the following Exception: Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider The complete error: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: Could not establish connection to jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: org.apache.hive.org.apache.http.client.ClientProtocolException at oracle.odi.jdbc.datasource.LoginTimeoutDatasourceAdapter.doGetConnection(LoginTimeoutDatasourceAdapter.java:144) at oracle.odi.jdbc.datasource.LoginTimeoutDatasourceAdapter.getConnection(LoginTimeoutDatasourceAdapter.java:73) at com.sunopsis.sql.SnpsConnection.testConnection(SnpsConnection.java:1258) at com.sunopsis.graphical.dialog.SnpsDialogTestConnet.getLocalConnect(SnpsDialogTestConnet.java:204) at com.sunopsis.graphical.dialog.SnpsDialogTestConnet.access$500(SnpsDialogTestConnet.java:62) at com.sunopsis.graphical.dialog.SnpsDialogTestConnet$6.doInBackground(SnpsDialogTestConnet.java:402) at com.sunopsis.graphical.dialog.SnpsDialogTestConnet$6.doInBackground(SnpsDialogTestConnet.java:398) at oracle.odi.ui.framework.AbsUIRunnableTask.run(AbsUIRunnableTask.java:258) at oracle.ide.dialogs.ProgressBar.run(ProgressBar.java:961) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: Could not establish connection to jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: org.apache.hive.org.apache.http.client.ClientProtocolException at oracle.odi.jdbc.datasource.LoginTimeoutDatasourceAdapter.doGetConnection(LoginTimeoutDatasourceAdapter.java:144) at oracle.odi.jdbc.datasource.LoginTimeoutDatasourceAdapter.getConnection(LoginTimeoutDatasourceAdapter.java:73) at oracle.odi.core.datasource.dwgobject.support.OnConnectOnDisconnectDataSourceAdapter.getConnection(OnConnectOnDisconnectDataSourceAdapter.java:87) at oracle.odi.jdbc.datasource.LoginTimeoutDatasourceAdapter$ConnectionProcessor.run(LoginTimeoutDatasourceAdapter.java:228) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: Could not establish connection to jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: org.apache.hive.org.apache.http.client.ClientProtocolException at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:211) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at oracle.odi.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:412) at oracle.odi.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:385) at oracle.odi.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:352) at oracle.odi.jdbc.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:331) ... 6 more Caused by: java.sql.SQLException: Could not establish connection to jdbc:hive2://l4284t.sss.se.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.COM;httpPath=cliservice: org.apache.hive.org.apache.http.client.ClientProtocolException at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:589) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:188) ... 11 more Caused by: org.apache.hive.org.apache.thrift.transport.TTransportException: org.apache.hive.org.apache.http.client.ClientProtocolException at org.apache.hive.org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297) at org.apache.hive.org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313) at org.apache.hive.org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) at org.apache.hive.org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:162) at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:154) at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:578) ... 12 more Caused by: org.apache.hive.org.apache.http.client.ClientProtocolException at org.apache.hive.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.hive.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117) at org.apache.hive.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.hive.org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251) ... 18 more Caused by: org.apache.hive.org.apache.http.HttpException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86) at org.apache.hive.org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) at org.apache.hive.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182) at org.apache.hive.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.hive.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.hive.org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84) at org.apache.hive.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ... 21 more Caused by: org.apache.hive.org.apache.http.HttpException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68) at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74) ... 27 more Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2273) at org.apache.hadoop.security.Groups.<init>(Groups.java:99) at org.apache.hadoop.security.Groups.<init>(Groups.java:95) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:420) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:324) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:291) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:846) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:816) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.getCurrentUGIWithConf(HadoopThriftAuthBridge.java:122) at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:81) at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62) ... 28 more Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267) ... 39 more<br> I have added the necessary config. in the odi.conf file I have added the necessary configs. in the odi.conf AddVMOption -Djava.security.krb5.conf=C:\Oracle\Middleware\Oracle_Home\oracle_common\modules\datadirect\kerb5.conf AddVMOption -Djavax.security.auth.useSubjectCredsOnly=false I am sure that the JDBC driver + my kerberos ticket cache etc. are in place, the below, stand-alone Java class is working fine: package com.my; import java.sql.*; /** * Hello world! * */public class App { public static void main( String[] args ) throws ClassNotFoundException, SQLException { System.out.println( "Hello World!" ); System.setProperty("javax.security.auth.useSubjectCredsOnly","false"); System.setProperty("java.security.krb5.conf","C:\Oracle\Middleware\Oracle_Home\oracle_common\modules\datadirect\kerb5.conf"); //System.setProperty("sun.security.krb5.debug","true");Class.forName("org.apache.hive.jdbc.HiveDriver"); System.out.println("getting connection"); Connection con = DriverManager.getConnection("jdbc:hive2://l4284t.sss.se.scania.com:10501/;transportMode=http;principal=hive/_HOST@GLOBAL.SCD.SCANIA.COM;httpPath=cliservice"); System.out.println("Connected"); Statement stmt = con.createStatement(); ResultSet rs = stmt.executeQuery("show tables"); while (rs.next()){ System.out.println("table name : "+rs.getString(1)); } stmt.close(); con.close(); } } ... View more
Labels:

Sure-shot ways to put postpone a processor's execu...

by Expert Contributor kaliyugantagoni in Support Questions
‎08-16-2017 08:49 AM
1 Kudo
‎08-16-2017 08:49 AM
1 Kudo
NiFi 1.2.0. There is a custom processor that reads data from SQL Server and creates a flow file per row. This causes creation of millions of flow-files and the destination queue often gets full. The objective is to prevent triggering of the processor if the destination queue is full. The background thread about the backpressuring & throttling . In our dev. env., in spite of adding 'Back Pressure Object Threshold' and 'Back Pressure Data Size Threshold', the processor executed every (scheduled)5 minutes even when the destination queue had far more number of files than configured. I am a bit confused about the following said in the documentation : Several factors exist that will contribute to when a Processor’s onTrigger method is invoked. First, the Processor will not be triggered unless a user has configured the Processor to run. If a Processor is scheduled to run, the Framework periodically (the period is configured by users in the User Interface) checks if there is work for the Processor to do, as described above. If so, the Framework will check downstream destinations of the Processor. If any of the Processor’s outbound Connections is full, by default, the Processor will not be scheduled to run. Can any of the below ways prevent processor triggering the moment : Usage of TriggerWhenAnyDestinationAvailable . It's documentation says : By default, NiFi will not schedule a Processor to run if any of its outbound queues is full. This allows back-pressure to be applied all the way a chain of Processors. However, some Processors may need to run even if one of the outbound queues is full. This annotations indicates that the Processor should run if any Relationship is "available." A Relationship is said to be "available" if none of the connections that use that Relationship is full. For example, the DistributeLoad Processor makes use of this annotation. If the "round robin" scheduling strategy is used, the Processor will not run if any outbound queue is full. However, if the "next available" scheduling strategy is used, the Processor will run if any Relationship at all is available and will route FlowFiles only to those relationships that are available. It's a horrible thought but can, within the processor's code, ProcessSession.getQueueSize() be used to detect the destination queue size and sleep the processor ? ... View more
Labels:

Custom processor and backpressuring, throttling

by Expert Contributor kaliyugantagoni in Support Questions
‎07-03-2017 12:38 PM
‎07-03-2017 12:38 PM
NiFi 1.2.0 There is a custom processor that reads data from db and passes it further. In a recent stress testing, the 'success' relationship queue was clogged and also the later flow as the processor dumped hundred thousands of flow files of several GBs. Obviously, the backpressuring was not implemented. I also read an informative post about throttling and backpressuring. What I have figured out is that backpressuring is something we configure in the relationship queue and standard processors like ControlRate can help to regulate the data flow. Question : Is additional coding required(e.g: Some interface to be implemented) in the processor to enable it to 'sleep/stop consuming data' for backpressuring or does the NiFi framework handle that, once the 'success' relationship of the processor is configured for backpressuring ... View more
Labels:

Re: NiFi custom processor custom log, logging in t...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-28-2017 12:24 PM
‎06-28-2017 12:24 PM
additivity="false" is essential Complete answer on StackOverflow. ... View more

StateMap keys across different instances of the sa...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-28-2017 12:10 PM
‎06-28-2017 12:10 PM
Nifi 1.2.0. In a custom processor, an LSN is used to fetch data from a SQL Server db table. Following are the snippets of the code used for: Storing a key-value pair final StateManager stateManager = context.getStateManager(); try { StateMap stateMap = stateManager.getState(Scope.CLUSTER); final Map<String, String> newStateMapProperties = new HashMap<>(); String lsnUsedDuringLastLoadStr = Base64.getEncoder().encodeToString(lsnUsedDuringLastLoad); //Just a constant String used as key newStateMapProperties.put(ProcessorConstants.LAST_MAX_LSN, lsnUsedDuringLastLoadStr); if (stateMap.getVersion() == -1) { stateManager.setState(newStateMapProperties, Scope.CLUSTER); } else { stateManager.replace(stateMap, newStateMapProperties, Scope.CLUSTER); } } Retrieving the key-value pair final StateManager stateManager = context.getStateManager(); final StateMap stateMap; final Map<String, String> stateMapProperties; byte[] lastMaxLSN = null; try { stateMap = stateManager.getState(Scope.CLUSTER); stateMapProperties = new HashMap<>(stateMap.toMap()); lastMaxLSN = (stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN) == null || stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).isEmpty()) ? null : Base64.getDecoder() .decode(stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).getBytes()); } When a single instance of this processor is running, the LSN is stored and retrieved properly and the logic of fetching data from SQL Server tables works fine. As per the NiFi doc. about state management: Storing and Retrieving State State is stored using the StateManager’s getState , setState , replace , and clear methods. All of these methods require that a Scope be provided. It should be noted that the state that is stored with the Local scope is entirely different than state stored with a Cluster scope. If a Processor stores a value with the key of My Key using the Scope.CLUSTER scope, and then attempts to retrieve the value using the Scope.LOCAL scope, the value retrieved will be null (unless a value was also stored with the same key using the Scope.CLUSTER scope). Each Processor’s state, is stored in isolation from other Processors' state. When two instances of this processor are running, only one is able to fetch the data. This has led to the following question: Is the StateMap a 'global map' which must have unique keys across the instances of the same processor and also the instances of different processors? In simple words, whenever a processor puts a key in the statemap, the key should be unique across the NiFi processors(and other services, if any, that use the State API) ? If yes, can anyone suggest what unique key should I use in my case? Note: I quickly glanced at the standard MySQL CDC processor code class(CaptureChangeMySQL.java) and it has a similar logic to store and retrieve the state but then am I overlooking something ? ... View more
Labels:

NiFi custom processor custom log, logging in two p...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-22-2017 12:47 PM
‎06-22-2017 12:47 PM
NiFi 1.2.0 I am having a custom processor and I wish to have a dedicated log file for the same. Accordingly, I have configured the com.datalake.processors.SQLServerCDCProcessor class to use an appender named 'SQLSERVER-CDC' Following is the logback.xml : <?xml version="1.0" encoding="UTF-8"?> <configuration scan="true" scanPeriod="30 seconds"> <contextListener> <resetJUL>true</resetJUL> </contextListener> <appender name="APP_FILE"> <file>/var/log/nifi/nifi-app.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy> <maxFileSize>100MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> <immediateFlush>true</immediateFlush> </appender> <appender name="USER_FILE"> <file>/var/log/nifi/nifi-user.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-user_%d.log</fileNamePattern> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="BOOTSTRAP_FILE"> <file>/var/log/nifi/nifi-bootstrap.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-bootstrap_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="CONSOLE"> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <!-- Start : Added for log for custom processor --> <appender name="SQLSERVER-CDC"> <file>/var/log/nifi/sqlserver-cdc.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/sqlserver-cdc_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy> <maxFileSize>25MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> <immediateFlush>true</immediateFlush> </encoder> </appender> <!-- End : Added for log for custom processor --> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR --> <logger name="org.apache.nifi" level="INFO"/> <logger name="org.apache.nifi.processors" level="WARN"/> <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/> <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" /> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" /> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" /> <logger name="org.apache.curator.ConnectionState" level="OFF" /> <!-- Logger for managing logging statements for nifi clusters. --> <logger name="org.apache.nifi.cluster" level="INFO"/> <!-- Logger for logging HTTP requests received by the web server. --> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> <!-- Logger for managing logging statements for jetty --> <logger name="org.eclipse.jetty" level="INFO"/> <!-- Suppress non-error messages due to excessive logging by class or library --> <logger name="com.sun.jersey.spi.container.servlet.WebComponent" level="ERROR"/> <logger name="com.sun.jersey.spi.spring" level="ERROR"/> <logger name="org.springframework" level="ERROR"/> <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) --> <logger name="com.sun.jersey.spi.inject.Errors" level="ERROR"/> <!-- Logger for capturing user events. We do not want to propagate these log events to the root logger. These messages are only sent to the user-log appender. --> <logger name="org.apache.nifi.web.security" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <!-- Logger for capturing Bootstrap logs and NiFi's standard error and standard out. --> <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false"> <appender-ref ref="CONSOLE" /> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level --> <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level --> <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Start : Added for log for custom processor --> <logger name="com.datalake.processors.SQLServerCDCProcessor" level="DEBUG" > <appender-ref ref="SQLSERVER-CDC"/> </logger> <!-- End : Added for log for custom processor --> <root level="INFO"> <appender-ref ref="APP_FILE"/> </root> </configuration> The strange fact is that the custom processor debug statements are written to both 'nifi-app.log' and the 'sqlserver-cdc.log' but I want these statements to be written only in the latter('sqlserver-cdc.log'). What am I missing ? ... View more
Labels:

Maven config. for custom processor to Read/Write H...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-16-2017 02:10 PM
‎06-16-2017 02:10 PM
NiFi 1.2.0 I have a custom processor who pom files look as the following. processor/pom.xml <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>com.datalake</groupId> <artifactId>CDCNiFi</artifactId> <version>1.0-SNAPSHOT</version> </parent> <artifactId>nifi-NiFiCDCPoC-processors</artifactId> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-utils</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-dbcp-service-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-processor-utils</artifactId> </dependency> <!-- Third-party --> <!-- <dependency> <groupId>com.microsoft.sqlserver</groupId> <artifactId>mssql-jdbc</artifactId> <version>6.1.0.jre8</version> </dependency> --> <dependency> <groupId>org.codehaus.jackson</groupId> <artifactId>jackson-mapper-asl</artifactId> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.8.7</version> </dependency> <!-- Testing & Cross-cutting concerns --> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-mock</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <scope>test</scope> </dependency> </dependencies> </project> nar/pom.xml <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>com.datalake</groupId> <artifactId>CDCNiFi</artifactId> <version>1.0-SNAPSHOT</version> </parent> <artifactId>nifi-NiFiCDCPoC-nar</artifactId> <version>1.0-SNAPSHOT</version> <packaging>nar</packaging> <properties> <maven.javadoc.skip>true</maven.javadoc.skip> <source.skip>true</source.skip> </properties> <dependencies> <dependency> <groupId>com..datalake</groupId> <artifactId>nifi-NiFiCDCPoC-processors</artifactId> <version>1.0-SNAPSHOT</version> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-services-api-nar</artifactId> <type>nar</type> </dependency> <!-- <dependency> <groupId>com.microsoft.sqlserver</groupId> <artifactId>mssql-jdbc</artifactId> <version>6.1.0.jre8</version> <scope>runtime</scope> </dependency>--> </dependencies> </project> Now, I wish to do some HDFS file operations from the processor like read/write a SequenceFile, retrieve files from a HDFS directory and so on. I had a look at the existing processors like PutHDFS which has some pom entries as follows : <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-hadoop-utils</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-flowfile-packager</artifactId> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <scope>provided</scope> </dependency> In the lib dir. of NiFi, I could see files like nifi-hadoop-libraries-nar-1.2.0.nar, nifi-hadoop-nar-1.2.0.nar The inclusion of above entries in my pom.xml didn't work, what shall I do to use the HDFS file system api within my custom processor ? ... View more
Labels:

Re: NiFi state mgt., ambiguous behaviour across di...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-15-2017 11:43 AM
‎06-15-2017 11:43 AM
Any suggestions to check if the LSN is persisted and retrieved correctly? ... View more

Re: NiFi state mgt., ambiguous behaviour across di...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-14-2017 06:43 AM
‎06-14-2017 06:43 AM
The point is that error occurs only on the Linux machines, that too after the state is stored. With state cleared or on the Windows machine, the JDBC error doesn't occur and the data gets inserted. Attaching the sample processor code ... View more

NiFi state mgt., ambiguous behaviour across differ...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-13-2017 12:45 PM
‎06-13-2017 12:45 PM
NiFi 1.2.0, two nodes viz: l4513t.sss.se.com & l4514t.sss.se.com, both are RHEL 7 nodes. I have a local(Windows 7) NiFi 1.2.0 where I develop and test my custom processor. Following is a state management code snippet that stores an LSN, a SQL Server binary data type(retrieved and used as byte array in Java) : final StateManager stateManager = context.getStateManager(); try { StateMap stateMap = stateManager.getState(Scope.CLUSTER); final Map<String, String> newStateMapProperties = new HashMap<>(); newStateMapProperties.put(ProcessorConstants.LAST_MAX_LSN, new String(lsnUsedDuringLastLoad)); logger.debug("Persisting stateMap : " + newStateMapProperties); if (stateMap.getVersion() == -1) { stateManager.setState(newStateMapProperties, Scope.CLUSTER); } else { stateManager.replace(stateMap, newStateMapProperties, Scope.CLUSTER); } } catch (IOException ioException) { logger.error("Error while persisting the state to NiFi", ioException); throw new ProcessException("The state(LSN) couldn't be persisted", ioException); } The below code is used to retrieve the LSN : final StateManager stateManager = context.getStateManager(); final StateMap stateMap; final Map<String, String> stateMapProperties; byte[] lastMaxLSN = null; try { stateMap = stateManager.getState(Scope.CLUSTER); stateMapProperties = new HashMap<>(stateMap.toMap()); logger.debug("Retrieved the statemap : " + stateMapProperties); lastMaxLSN = (stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN) == null || stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).isEmpty()) ? null : stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).getBytes(); } catch (IOException ioe) { logger.error("Couldn't load the state map", ioe); throw new ProcessException(ioe); } The processor works smoothly on the local, Windows 7 machine and the LSN is stored and retrieved several times, without any errors. The problem surfaced when I deployed the processor on the dev. env. which has the above mentioned two RHEL 7 nodes. The processor executed successfully ONLY ONCE, later, it didn't throw any error but simply didn't get expected LSN, thus, not doing any work. To get a complete idea of the situation, I created a test table which would store every query that would execute with the LSN. Again, this worked fine on the local, Windows 7 machine, following are some rows inserted in the test table. Note : Just focus on the LSN field which is a binary(10) field, rest all can be ignored. On dev., the processor executes EXACTLY ONCE, either when it was deployed the first time or after clearing its state. On the second execution, it retrieves the LSN and tries to store it back in the test table, only to get a JDBC exception: com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated. com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated. at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217) at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1655) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:440) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2445) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:328) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at com.datalake.processors.SQLServerCDCProcessorCCC.logCTQueryWithParams(SQLServerCDCProcessorCCC.java:62) at com.datalake.processors.SQLServerCDCProcessor.writeDataFromChangeTablesToFlowFiles(SQLServerCDCProcessor.java:575) at com.datalake.processors.SQLServerCDCProcessor.onTrigger(SQLServerCDCProcessor.java:193) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I believe that the underlying ZK is the one which stores this metadata. I am unable to pinpoint the root cause and the location. ***********Edit-1*********** Out of curiosity/desperation, I tried using the UTF-8 charset : while storing : newStateMapProperties.put(ProcessorConstants.LAST_MAX_LSN, new String(lsnUsedDuringLastLoad, StandardCharsets.UTF_8)); and retrieving the bytes : lastMaxLSN = (stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN) == null || stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).isEmpty()) ? null : `stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).getBytes(StandardCharsets.UTF_8);` Now, I get the same exception even on the Windows machine, the RHEL error remains the same. ... View more
Labels:

Re: Separate log file for custom processor

by Expert Contributor kaliyugantagoni in Support Questions
‎06-09-2017 07:20 AM
‎06-09-2017 07:20 AM
I got the answer at StackOverflow. I am able to create a logfile on local, Windows machine but the same config. is not working on the Linux env., maybe I have missed something. Below is the logback.xml used on my local machine : <?xml version="1.0" encoding="UTF-8"?> <configuration scan="true" scanPeriod="30 seconds"> <contextListener> <resetJUL>true</resetJUL> </contextListener> <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="USER_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-user_%d.log</fileNamePattern> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="BOOTSTRAP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-bootstrap_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="CONSOLE"> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <!-- Start : Added for log for custom processor --> <appender name="SQLSERVER-CDC"> <file>${org.apache.nifi.bootstrap.config.log.dir}/sqlserver-cdc.log</file> <rollingPolicy> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/sqlserver-cdc_%d.log</fileNamePattern> <maxHistory>30</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <!-- End : Added for log for custom processor --> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR --> <logger name="org.apache.nifi" level="INFO"/> <logger name="org.apache.nifi.processors" level="WARN"/> <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/> <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" /> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" /> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" /> <logger name="org.apache.curator.ConnectionState" level="OFF" /> <!-- Logger for managing logging statements for nifi clusters. --> <logger name="org.apache.nifi.cluster" level="INFO"/> <!-- Logger for logging HTTP requests received by the web server. --> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> <!-- Logger for managing logging statements for jetty --> <logger name="org.eclipse.jetty" level="INFO"/> <!-- Suppress non-error messages due to excessive logging by class or library --> <logger name="com.sun.jersey.spi.container.servlet.WebComponent" level="ERROR"/> <logger name="com.sun.jersey.spi.spring" level="ERROR"/> <logger name="org.springframework" level="ERROR"/> <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) --> <logger name="com.sun.jersey.spi.inject.Errors" level="ERROR"/> <!-- Logger for capturing user events. We do not want to propagate these log events to the root logger. These messages are only sent to the user-log appender. --> <logger name="org.apache.nifi.web.security" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <!-- Logger for capturing Bootstrap logs and NiFi's standard error and standard out. --> <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false"> <appender-ref ref="CONSOLE" /> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level --> <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level --> <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Start : Added for log for custom processor --> <logger name="com.datalake.processors.SQLServerCDCProcessor" level="DEBUG" > <appender-ref ref="SQLSERVER-CDC"/> </logger> <!-- End : Added for log for custom processor --> <root level="info"> <appender-ref ref="APP_FILE"/> </root> </configuration> ... View more

Separate log file for custom processor

by Expert Contributor kaliyugantagoni in Support Questions
‎06-07-2017 06:29 AM
‎06-07-2017 06:29 AM
NiFi 1.2.0 Need to create a separate log file, say, customprocessor.log besides the app.log file created by NiFi. I went through some interesting, existing threads like this, however, I am unable to figure out how to make it working in the code. Following is the existing logback.xml : <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration scan="true" scanPeriod="30 seconds"> <contextListener> <resetJUL>true</resetJUL> </contextListener> <appender name="APP_FILE"> <file>/var/log/nifi/nifi-app.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy> <maxFileSize>100MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> <immediateFlush>true</immediateFlush> </appender> <appender name="USER_FILE"> <file>/var/log/nifi/nifi-user.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-user_%d.log</fileNamePattern> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="BOOTSTRAP_FILE"> <file>/var/log/nifi/nifi-bootstrap.log</file> <rollingPolicy> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/nifi-bootstrap_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <appender name="CONSOLE"> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR --> <logger name="org.apache.nifi" level="INFO"/> <logger name="org.apache.nifi.processors" level="WARN"/> <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/> <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" /> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" /> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" /> <logger name="org.apache.curator.ConnectionState" level="OFF" /> <!-- Logger for managing logging statements for nifi clusters. --> <logger name="org.apache.nifi.cluster" level="INFO"/> <!-- Logger for logging HTTP requests received by the web server. --> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> <!-- Logger for managing logging statements for jetty --> <logger name="org.eclipse.jetty" level="INFO"/> <!-- Suppress non-error messages due to excessive logging by class or library --> <logger name="com.sun.jersey.spi.container.servlet.WebComponent" level="ERROR"/> <logger name="com.sun.jersey.spi.spring" level="ERROR"/> <logger name="org.springframework" level="ERROR"/> <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) --> <logger name="com.sun.jersey.spi.inject.Errors" level="ERROR"/> <!-- Logger for capturing user events. We do not want to propagate these log events to the root logger. These messages are only sent to the user-log appender. --> <logger name="org.apache.nifi.web.security" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <logger name="org.apache.nifi.web.filter.RequestLogger" level="INFO" additivity="false"> <appender-ref ref="USER_FILE"/> </logger> <!-- Logger for capturing Bootstrap logs and NiFi's standard error and standard out. --> <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false"> <appender-ref ref="CONSOLE" /> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level --> <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level --> <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false"> <appender-ref ref="BOOTSTRAP_FILE" /> </logger> <root level="DEBUG"> <appender-ref ref="APP_FILE"/> </root> </configuration> Now, I can add a new appender for the custom log file : <!-- Start : Separate log file for custom processor --> <appender name="CUSTOM_FILE"> <file>/var/log/nifi/custom-processor.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>/var/log/nifi/archive/custom-processor_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy> <maxFileSize>100MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> <!-- keep 30 log files worth of history --> <maxHistory>3</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> <immediateFlush>true</immediateFlush> </appender> <!-- End : Separate log file for custom processor --> <!-- Start : Separate log file for custom processor --> <logger name="com.nifi.CustomLog" level="DEBUG" additivity="false"> <appender-ref ref="CUSTOM_FILE" /> </logger> <!-- End : Separate log file for custom processor --> I have the following questions : Are the entries that I am adding correct In the code, I use the following snippet to get the root logger, however, I didn't find a method/constructor to get my custom logger in the code, how shall I do that ? import org.apache.nifi.logging.ComponentLog;..final ComponentLog logger = getLogger();logger.debug("..."); ... View more
Labels:

Re: Erroneous persistence of data via State Manage...

by Expert Contributor kaliyugantagoni in Support Questions
‎06-02-2017 07:10 AM
‎06-02-2017 07:10 AM
Added the state-management.xml from both the nodes, can you check ? There is no 'LOCAL' word in the processor code 🙂 ... View more

Erroneous persistence of data via State Manager

by Expert Contributor kaliyugantagoni in Support Questions
‎05-31-2017 11:20 AM
‎05-31-2017 11:20 AM
NiFi 1.2.0, two nodes. There is a custom processor which persists some bytes via the State api, following is the relevant code snippet : @Override public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { final StateManager stateManager = context.getStateManager(); try { StateMap stateMap = stateManager.getState(Scope.CLUSTER); final Map<String, String> newStateMapProperties = new HashMap<>(); newStateMapProperties.put(ProcessorConstants.LAST_MAX_LSN, new String(lsnUsedDuringLastLoad)); logger.debug("Persisting stateMap : " + newStateMapProperties); if (stateMap.getVersion() == -1) { stateManager.setState(newStateMapProperties, Scope.CLUSTER); } else { stateManager.replace(stateMap, newStateMapProperties, Scope.CLUSTER); } } catch (IOException ioException) { logger.error("Error while persisting the state to NiFi", ioException); throw new ProcessException("The state(LSN) couldn't be persisted", ioException); } . . . } private boolean writeDataFromChangeTablesToFlowFiles(ProcessContext context, ProcessSession session) { . . . final StateManager stateManager = context.getStateManager(); final StateMap stateMap; final Map<String, String> stateMapProperties; . . . stateMap = stateManager.getState(Scope.CLUSTER); stateMapProperties = new HashMap<>(stateMap.toMap()); logger.debug("Retrieved the statemap : " + stateMapProperties); lastMaxLSN = (stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN) == null || stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).isEmpty()) ? null : stateMapProperties.get(ProcessorConstants.LAST_MAX_LSN).getBytes(); } On my local machine, the processor runs fine for even days, the data is picked up whenever it's inserted/updated/deleted. When the nar was first deployed, the processor worked fine and fetched the data it was supposed to. After some time(probably, the second execution itself), it simply stops pulling data. I checked the state of the processor and I am unsure if looks correct - the lsn bytes should be stored as a single, uniform value accessible to all nodes : Now, if I start-stop the processor and clear the state, it starts fetching the data ? What is the logical mistake that I have committed ? **********Edit-1********* I configured the processor to run only on the primary node, yet I have the same issue. **********Edit-2********** The state-management.xml on l4513t : <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <stateManagement> <!-- State Provider that stores state locally in a configurable directory. This Provider requires the following properties: Directory - the directory to store components' state in. If the directory being used is a sub-directory of the NiFi installation, it is important that the directory be copied over to the new version when upgrading NiFi. Always Sync - If set to true, any change to the repository will be synchronized to the disk, meaning that NiFi will ask the operating system not to cache the information. This is very expensive and can significantly reduce NiFi performance. However, if it is false, there could be the potential for data loss if either there is a sudden power loss or the operating system crashes. The default value is false. Partitions - The number of partitions. Checkpoint Interval - The amount of time between checkpoints. --> <local-provider> <id>local-provider</id> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class> <property name="Directory">./state/local</property> <property name="Always Sync">false</property> <property name="Partitions">16</property> <property name="Checkpoint Interval">2 mins</property> </local-provider> <!-- State Provider that is used to store state in ZooKeeper. This Provider requires the following properties: Root Node - the root node in ZooKeeper where state should be stored. The default is '/nifi', but it is advisable to change this to a different value if not using the embedded ZooKeeper server and if multiple NiFi instances may all be using the same ZooKeeper Server. Connect String - A comma-separated list of host:port pairs to connect to ZooKeeper. For example, myhost.mydomain:2181,host2.mydomain:5555,host3:6666 Session Timeout - Specifies how long this instance of NiFi is allowed to be disconnected from ZooKeeper before creating a new ZooKeeper Session. Default value is "30 seconds" Access Control - Specifies which Access Controls will be applied to the ZooKeeper ZNodes that are created by this State Provider. This value must be set to one of: - Open : ZNodes will be open to any ZooKeeper client. - CreatorOnly : ZNodes will be accessible only by the creator. The creator will have full access to create children, read, write, delete, and administer the ZNodes. This option is available only if access to ZooKeeper is secured via Kerberos or if a Username and Password are set. <property name="Connect String">l4513t.sss.se.com:2181,l4514t.sss.se.com:2181</property> --> <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">l4373t.sss.se.com:2181,l4283t.sss.se.com:2181,l4284t.sss.se.com:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> </stateManagement> The state-management.xml on l4514t : <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <!-- This file provides a mechanism for defining and configuring the State Providers that should be used for storing state locally and across a NiFi cluster. In order to use a specific provider, it must be configured here and its identifier must be specified in the nifi.properties file. --> <stateManagement> <!-- State Provider that stores state locally in a configurable directory. This Provider requires the following properties: Directory - the directory to store components' state in. If the directory being used is a sub-directory of the NiFi installation, it is important that the directory be copied over to the new version when upgrading NiFi. Always Sync - If set to true, any change to the repository will be synchronized to the disk, meaning that NiFi will ask the operating system not to cache the information. This is very expensive and can significantly reduce NiFi performance. However, if it is false, there could be the potential for data loss if either there is a sudden power loss or the operating system crashes. The default value is false. Partitions - The number of partitions. Checkpoint Interval - The amount of time between checkpoints. --> <local-provider> <id>local-provider</id> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class> <property name="Directory">./state/local</property> <property name="Always Sync">false</property> <property name="Partitions">16</property> <property name="Checkpoint Interval">2 mins</property> </local-provider> <!-- State Provider that is used to store state in ZooKeeper. This Provider requires the following properties: Root Node - the root node in ZooKeeper where state should be stored. The default is '/nifi', but it is advisable to change this to a different value if not using the embedded ZooKeeper server and if multiple NiFi instances may all be using the same ZooKeeper Server. Connect String - A comma-separated list of host:port pairs to connect to ZooKeeper. For example, myhost.mydomain:2181,host2.mydomain:5555,host3:6666 Session Timeout - Specifies how long this instance of NiFi is allowed to be disconnected from ZooKeeper before creating a new ZooKeeper Session. Default value is "30 seconds" Access Control - Specifies which Access Controls will be applied to the ZooKeeper ZNodes that are created by this State Provider. This value must be set to one of: - Open : ZNodes will be open to any ZooKeeper client. - CreatorOnly : ZNodes will be accessible only by the creator. The creator will have full access to create children, read, write, delete, and administer the ZNodes. This option is available only if access to ZooKeeper is secured via Kerberos or if a Username and Password are set. <property name="Connect String">l4513t.sss.se.com:2181,l4514t.sss.se.com:2181</property> --> <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">l4373t.sss.se.com:2181,l4283t.sss.se.com:2181,l4284t.sss.se.com:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> </stateManagement> ... View more
Labels:

Re: New, custom processor causing NiFi to fail

by Expert Contributor kaliyugantagoni in Support Questions
‎05-26-2017 02:02 PM
‎05-26-2017 02:02 PM
Yes, your guess is right ! There was an upgrade from 1.1 The tip seems crucial, will try that ... ... View more

New, custom processor causing NiFi to fail

by Expert Contributor kaliyugantagoni in Support Questions
‎05-26-2017 09:04 AM
1 Kudo
‎05-26-2017 09:04 AM
1 Kudo
NiFi 1.2.0, two nodes, kerberized. In the previous version, the custom processor executed properly. The process I follow for deployment : Place the nar file in lib of both the nodes(l4513t.sss.se.com and l4514t.sss.se.com, NiFi installations) Restart one NiFi instance, at a time I am getting issues when I begin with one of the NiFi nodes(l4513t.sss.se.com) First, I got the following error when I placed the nar file in lib and tried to restart NiFi : Failed to connect node to cluster because local flow is different than cluster flow. org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:934) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:790) at org.apache.nifi.NiFi.<init>(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows: Local Fingerprint: 91910-015b-1000-9677-68b017463306com.datalake.processors.SQLServerCDCProcessorNO_VALUEdefaultnifi-NiFiCDCPoC-narunversionedDatabase Connection Pooling Service=fe153649-5193-1a68-ffff-ffffc37686 Cluster Fingerprint: 91910-015b-1000-9677-68b017463306com.datalake.processors.SQLServerCDCProcessorNO_VALUEdefaultunknownunversionedDatabase Connection Pooling Service=fe153649-5193-1a68-ffff-ffffc37686c6containerD at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:259) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1544) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:720) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:909) ... 4 common frames omitted 2017-05-26 09:05:06,199 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator l4513t.sss.se.com:9443 requested disconnection from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. 2017-05-26 09:05:06,199 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Status of l4513t.sss.se.com:9443 changed from NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=CONNECTING, updateId=368] to NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=368] 2017-05-26 09:05:06,395 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for l4513t.sss.se.com:9443 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. 2017-05-26 09:05:06,395 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Primary Node' becuase that role is not registered 2017-05-26 09:05:06,395 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.IllegalStateException: Already closed or has not been started at com.google.common.base.Preconditions.checkState(Preconditions.java:173) at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270) at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.unregister(CuratorLeaderElectionManager.java:151) at org.apache.nifi.controller.FlowController.setClustered(FlowController.java:3667) at org.apache.nifi.controller.StandardFlowService.handleConnectionFailure(StandardFlowService.java:554) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:518) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:790) at org.apache.nifi.NiFi.<init>(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) I flipped through several existing threads and deleted the flow.xml.gz from a node and attempted to restart, now I am getting the following : 2017-05-26 09:20:30,803 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Status of l4513t.sss.se.com:9443 changed from null to NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=CONNECTING, updateId=370] 2017-05-26 09:20:30,811 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 25e40b0f-b0ab-4f92-9e7f-440abb116999 (type=NODE_STATUS_CHANGE, length=1071 bytes) from l4514t.sss.se.com in 117 millis 2017-05-26 09:20:30,914 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {l4513t.sss.se.com:9443=NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=CONNECTING, updateId=370]} to {l4514t.sss.se.com:9443=NodeConnectionStatus[nodeId=l4514t.sss.se.com:9443, state=CONNECTED, updateId=360], l4513t.sss.se.com:9443=NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=CONNECTING, updateId=370]} 2017-05-26 09:20:31,348 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.MissingBundleException: Failed to connect node to cluster because cluster flow contains bundles that do not exist on the current node org.apache.nifi.controller.MissingBundleException: Failed to connect node to cluster because cluster flow contains bundles that do not exist on the current node at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:936) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:790) at org.apache.nifi.NiFi.<init>(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) Caused by: org.apache.nifi.controller.MissingBundleException: com.datalake.processors.SQLServerCDCProcessor from default:unknown:unversioned is not known to this NiFi instance. at org.apache.nifi.controller.StandardFlowSynchronizer.checkBundleCompatibility(StandardFlowSynchronizer.java:445) at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:253) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1544) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:720) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:909) ... 4 common frames omitted Caused by: java.lang.IllegalStateException: com.datalake.processors.SQLServerCDCProcessor from default:unknown:unversioned is not known to this NiFi instance. at org.apache.nifi.util.BundleUtils.findCompatibleBundle(BundleUtils.java:55) at org.apache.nifi.util.BundleUtils.getBundle(BundleUtils.java:98) at org.apache.nifi.controller.StandardFlowSynchronizer.checkBundleCompatibility(StandardFlowSynchronizer.java:443) ... 9 common frames omitted 2017-05-26 09:20:31,348 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator l4513t.sss.se.com:9443 requested disconnection from cluster due to org.apache.nifi.controller.MissingBundleException: Failed to connect node to cluster because cluster flow contains bundles that do not exist on the current node 2017-05-26 09:20:31,348 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Status of l4513t.sss.se.com:9443 changed from NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=CONNECTING, updateId=370] to NodeConnectionStatus[nodeId=l4513t.sss.se.com:9443, state=DISCONNECTED, Disconnect Code=Node was missing bundle used by Cluster Flow, Disconnect Reason=org.apache.nifi.controller.MissingBundleException: Failed to connect node to cluster because cluster flow contains bundles that do not exist on the current node, updateId=370] 2017-05-26 09:20:31,448 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for l4513t.sss.se.com:9443 -- Node disconnected from cluster due to org.apache.nifi.controller.MissingBundleException: Failed to connect node to cluster because cluster flow contains bundles that do not exist on the current node 2017-05-26 09:20:31,448 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Primary Node' becuase that role is not registered 2017-05-26 09:20:31,448 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.IllegalStateException: Already closed or has not been started at com.google.common.base.Preconditions.checkState(Preconditions.java:173) at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270) at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.unregister(CuratorLeaderElectionManager.java:151) at org.apache.nifi.controller.FlowController.setClustered(FlowController.java:3667) at org.apache.nifi.controller.StandardFlowService.handleConnectionFailure(StandardFlowService.java:554) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:518) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:790) at org.apache.nifi.NiFi.<init>(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) 2017-05-26 09:20:31,450 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2017-05-26 09:20:31,456 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@73bd146c{SSL,[ssl, http/1.1]}{l4513t.sss.se.com:9443} 2017-05-26 09:20:31,456 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging 2017-05-26 09:20:31,457 DEBUG [Thread-1] org.apache.jasper.servlet.JspServlet JspServlet.destroy() What is that I am missing ? ... View more
Labels:

Error retrieving state map in kerberized cluster

by Expert Contributor kaliyugantagoni in Support Questions
‎04-26-2017 12:26 PM
‎04-26-2017 12:26 PM
HDP-2.5.3.0. A custom processor uses the State api to persist some data, this thread has all the details. The code snippet that uses the State api : try { stateMap = stateManager.getState(Scope.CLUSTER); stateMapProperties = new HashMap<>(stateMap.toMap()); logger.debug("Retrieved the statemap : " + stateMapProperties); ... ... ... } catch (IOException ioe) { logger.error("Couldn't load the state map", ioe); throw new ProcessException(ioe); } The processor works fine on my local machine's NiFi but when I put it on our (kerberized)dev cluster which has 2 NiFi nodes, it fails with the following error : java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:420) ~[na:na] at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63) ~[na:na] at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:480) [nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at com.datalake.processors.SQLServerCDCProcessor.onTrigger(SQLServerCDCProcessor.java:191) [nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /nifi/components/d7fff389-015a-1000-ffff-ffffd04d1279 at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[na:na] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[na:na] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[na:na] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) ~[na:na] at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403) ~[na:na] . . . . . . . . . org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:493) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at com.datalake.processors.SQLServerCDCProcessor.onTrigger(SQLServerCDCProcessor.java:191) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_112] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: java.io.IOException: Failed to obtain value from ZooKeeper for component with ID d7fff389-015a-1000-ffff-ffffd04d1279 with exception code NOAUTH at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:420) ~[na:na] at org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63) ~[na:na] at com.datalake.processors.SQLServerCDCProcessor.getDataFromChangeTables(SQLServerCDCProcessor.java:480) ~[nifi-NiFiCDCPoC-processors-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] ... 13 common frames omitted Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /nifi/components/d7fff389-015a-1000-ffff-ffffd04d1279 at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[na:na] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[na:na] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[na:na] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) ~[na:na] at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403) ~[na:na] ... 15 common frames omitted Attached is a bit more of the NiFi app log statemap-zk-error.txt Following are the entries in the state-management.xml <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">l4373t.sss.se.com:2181,l4283t.sss.se.com:2181,l4284t.sss.se.com:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">CreatorOnly</property> </cluster-provider> </stateManagement> Any ideas ? *****Edit-2***** Providing the existing kafka-jaas.conf bash-4.2$ cat kafka-jaas.conf KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true renewTicket=true useTicketCache=true serviceName="kafka" keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" principal="nifi/l4513t.sss.se.com@GLOBAL.SCD.COM"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=true renewTicket=true serviceName="kafka" keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" principal="nifi/l4513t.sss.se.com@GLOBAL.SCD.COM"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=true serviceName="kafka" keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" principal="nifi/l4513t.sss.se.com@GLOBAL.SCD.COM"; }; *****Edit-1***** As per the NiFi state management doc., added the zk jaas configuration, still the issue persists. <code>bash-4.2$ cat zookeeper-jaas.conf Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/usr/local/nifi/keys/nifi_l4513t.sss.se.com.keytab" storeKey=true useTicketCache=true principal="nifi/l4513t.sss.se.com@GLOBAL.SCD.COM"; }; The entry(as 'java.arg.16') in the bootstrap.conf file : <code>bash-4.2$ vi bootstrap.conf # # Java command to use when running NiFi java=java # Username to use when running NiFi. This value will be ignored on Windows. run.as= # Configure where NiFi's lib and conf directories live lib.dir=./lib conf.dir=./conf # How long to wait after telling NiFi to shutdown before explicitly killing the Process graceful.shutdown.seconds=20 # Disable JSR 199 so that we can use JSP's without running a JDK java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true # JVM memory settings java.arg.2=-Xms1024m java.arg.3=-Xmx2048m # Enable Remote Debugging #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 java.arg.4=-Djava.net.preferIPv4Stack=true # allowRestrictedHeaders is required for Cluster/Node communications to work properly java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol java.arg.7=-Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi # The G1GC is still considered experimental but has proven to be very advantageous in providing great # performance without significant "stop-the-world" delays. java.arg.13=-XX:+UseG1GC #Set headless mode by default java.arg.14=-Djava.awt.headless=true java.arg.15=-Djava.security.auth.login.config=/usr/local/nifi/conf/kafka-jaas.conf java.arg.16=-Djava.security.auth.login.config=/usr/local/nifi/conf/zookeeper-jaas.conf # Master key in hexadecimal format for encrypted sensitive configuration values nifi.bootstrap.sensitive.key= ### # Notification Services for notifying interested parties when NiFi is stopped, started, dies ### ... View more
Labels:

Re: DBCPConnectionPool controller service for SQL ...

by Expert Contributor kaliyugantagoni in Support Questions
‎03-23-2017 08:13 AM
‎03-23-2017 08:13 AM
Did what - the configuration or ... ? The original question has link to another question that has the config details. ... View more

State Manager not persisting/retrieving data

by Expert Contributor kaliyugantagoni in Support Questions
‎03-20-2017 11:41 AM
‎03-20-2017 11:41 AM
NiFi 1.1.1 I am trying to persist a byte [] using the State Manager. private byte[] lsnUsedDuringLastLoad; @Override public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { ... ... ... final StateManager stateManager = context.getStateManager(); try { StateMap stateMap = stateManager.getState(Scope.CLUSTER); final Map<String, String> newStateMapProperties = new HashMap<>(); newStateMapProperties.put(ProcessorConstants.LAST_MAX_LSN, new String(lsnUsedDuringLastLoad)); logger.debug("Persisting stateMap : " + newStateMapProperties); stateManager.replace(stateMap, newStateMapProperties, Scope.CLUSTER); } catch (IOException ioException) { logger.error("Error while persisting the state to NiFi", ioException); throw new ProcessException( "The state(LSN) couldn't be persisted", ioException); } ... ... ... } I don't get any exception or even a log error entry, the processor continues to run. The following load code always returns a null value(Retrieved the statemap : {})for the persisted field : try { stateMap = stateManager.getState(Scope.CLUSTER); stateMapProperties = new HashMap<>(stateMap.toMap()); logger.debug("Retrieved the statemap : "+stateMapProperties); lastMaxLSN = (stateMapProperties .get(ProcessorConstants.LAST_MAX_LSN) == null || stateMapProperties .get(ProcessorConstants.LAST_MAX_LSN).isEmpty()) ? null : stateMapProperties.get( ProcessorConstants.LAST_MAX_LSN).getBytes(); logger.debug("Attempted to load the previous lsn from NiFi state : " + lastMaxLSN); } catch (IOException ioe) { logger.error("Couldn't load the state map", ioe); throw new ProcessException(ioe); } I am wondering if the ZK is at fault or have I missed something while using the State Map ! ... View more
Labels:

Re: DBCPConnectionPool controller service for SQL ...

by Expert Contributor kaliyugantagoni in Support Questions
‎03-20-2017 10:56 AM
‎03-20-2017 10:56 AM
I have circumvented the issue by using a select instead of a CallableStatement 😞 ... View more
Contact Me
Online
Offline
Last Visited
‎12-04-2017 03:48 PM
Public Statistics
Date Registered ‎04-11-2016 02:31 PM
Last Visited ‎12-04-2017 03:48 PM
Total Messages Posted 174
Total Kudos Received 29