Problem: How to enable LDAP for users to view Activity Explorer Dashboard. We may have shiro.ini enabled for LDAP for Zepplin but it is different for Activity Explorer Dashboard. Solution: On SmartSense Activity Explorer node, under [main] section cd /etc/smartsense-activity/conf/ vi shiro.ini [main] sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager securityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login ldapRealm = org.apache.zeppelin.realm.LdapRealm ldapRealm.contextFactory.systemUsername = <fill your details> ldapRealm.contextFactory.systemPassword = <fill your details> ldapRealm.contextFactory.authenticationMechanism = SIMPLE ldapRealm.contextFactory.url = <fill your details> ldapRealm.pagingSize = 200 ldapRealm.authorizationEnabled = true ldapRealm.searchBase = <fill your details> ldapRealm.memberAttributeValueTemplate = <fill your details> ldapRealm.userSearchAttributeName = <fill your details> ldapRealm.userLowerCase = true ... View more

Short Description Migrate Hive saved queries from Hue 3.9.0 of CDH 5.7.2 into Hive Views of Ambari 2.6.2.2 on HDP 2.6.5 Environment: CentOS 7, CDH 5.7.2, Hue 3.9.0, Ambari 2.6.2, HDP 2.6.2, Hue 2.6.2, mySQL5.6 Problem: There are over 40,000 saved / history queries in CDH-Hue. During CDH to HDP migration, Hive saved queries from Hue needs to be migrated to Ambari Hive View saved queries. Assumptions: You have Hue of CDH configured with MySQL and CDH to HDP migration is successful. Solution Step 1: Install Hue 2.6.2 on HDP cluster pointing to a new database called hue_new262 follow the link HDP-2.6.5 - installing_hue Step 2: Insert records from hue_cdh (database of Hue CDH) to hue_new262 (database of Hue HDP) Impacted tables auth_user, auth_user_groups, beeswax_savedquery, beeswax_queryhistory insert into hue_new262.auth_user (id, username, first_name, last_name, email, password, is_staff, is_active, is_superuser, last_login, date_joined) select id, username, first_name, last_name, email, password, is_staff, is_active, is_superuser, last_login, date_joined from hue_cdh.auth_user; insert into hue_new262.auth_user_groups (id, user_id, group_id) select id, user_id, group_id from hue_cdh.auth_user_groups; insert into hue_new262.beeswax_savedquery (name, type, is_auto, mtime, owner_id, data, id, `desc`, is_trashed) select name, type, is_auto, mtime, owner_id, data, id, 'desc', is_trashed from hue_cdh.beeswax_savedquery; insert into hue_new262.beeswax_queryhistory (submission_date, last_state, server_id, log_context, design_id, owner_id, query, has_results, id, notify, server_name, server_host, server_port, server_type, server_guid, operation_type, modified_row_count, statement_number, query_type, result_rows, result_size, query_stats, session_id, cache_tag) select submission_date, last_state, server_id, log_context, design_id, owner_id, query, has_results, id, notify, server_name, server_host, server_port, server_type, server_guid, operation_type, modified_row_count, statement_number, query_type, NULL, NULL, NULL, NULL, NULL from hue_cdh.beeswax_queryhistory; Step 3: Migrating Hue Artifacts to Ambari Views Follow instructions migrating_hue_to_ambari_views to configure HueToAmbariMigration Issues faced 1. If we want to create new Hive View (ex: HueView) then make sure you open the HueView at least once before starting the migration. Otherwise DS_SAVEDQUERY_* table not found error would occur. 2. If we try to open Hue UI, the login will fail as the stored password in table auth_user in Hue 3.9.0 is different from Hue 2.6.2. So do not try to logon to Hue or do not insert the auth_user and auth_user_groups. Create the users as new users in Hue 2.6.2 UI but make sure the order (column id is auto increment) of user creation is same as CDH Hue user creation order. ... View more

Goal: Ingest Geospatial data (ais data in CSV format) by using GeoMesa HBase after GeoMesa HBase is installed and configured. Follow install-and-configure-geomesa-hbase-and-geoserver if GeoMesa HBase is not installed 151001000015660,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:33,1 151001000015661,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:34,1 151002008706375,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:46,1 151002008706376,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:47,1 Create GeoMesa SimpleFeatureType (sft) configuration file as below in ais_data_conf.sft geomesa.sfts.ais-data = { attributes = [ { name = "arkposid", type = "Long", index = true } { name = "mmsi", type = "Integer", index = false } { name = "navigationalstatus", type = "Integer", index = false } { name = "coords", type = "Point", index = true, srid = 4326, default = true } { name = "sog", type = "Double", index = false } { name = "cog", type = "Double", index = false } { name = "rot", type = "Double", index = false } { name = "heading", type = "Integer", index = false } { name = "acquisitiontime", type = "String", index = false } { name = "iptype", type = "Integer", index = false } ] } Create GeoMesa converter configuration file as below in ais_data_feature_conf.convert geomesa.converters.ais-data = { type = "delimited-text", format = "CSV", id-field = "toString($arkposid)", fields = [ { name = "arkposid", transform = "$1::long" } { name = "mmsi", transform = "$2::int" } { name = "navigationalstatus", transform = "$3::int" } { name = "lon", transform = "$4::double" } { name = "lat", transform = "$5::double" } { name = "coords", transform = "point($lon, $lat)" } { name = "sog", transform = "$6::double" } { name = "cog", transform = "$7::double" } { name = "rot", transform = "$8::double" } { name = "heading", transform = "$9::int" } { name = "acquisitiontime", transform = "toString($10)" } { name = "iptype", transform = "$11::int" } ] } Run the geomesa-hbase command to insert the CSV data into HBase geomesa-hbase ingest \ -c ais \ -s /tmp/ais_data_conf.sft \ -C /tmp/ais_data_feature_conf.convert \ /tmp/sample.csv Should received successful message as below <code>INFO Creating schema 'ais-data' INFO Running ingestion in local mode INFO Ingesting 1 file with 1 thread [============================================================] 100% complete 4 ingested 0 failed in 00:00:01 INFO Local ingestion complete in 00:00:01 INFO Ingested 4 features and failed to ingest 0 feature. Refer install-and-configure-geomesa-hbase-and-geoserver to view the ingested data from GeoServer ... View more

Introduction Installing GeoMesa HBase and GeoServer on HDP Prerequisites HDP 2..6.4 on CentOS7.2 Java Development Kit 1.8, Apache Maven, a GitHub client, and JAVA_HOME is configured below commands can be used yum install git -y # configure JAVA_HOME sed -i '$ a export JAVA_HOME=/usr/jdk64/jdk1.8.0_112' /etc/bashrc sed -i '$ a export PATH=$PATH:$JAVA_HOME/bin' /etc/bashrc source /etc/bashrc java -version #install Maven: cd /tmp wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz tar -zxvf apache-maven-3.5.3-bin.tar.gz sudo mv /tmp/apache-maven-3.5.3 /opt sudo chown -R root:root /opt/apache-maven-3.5.3 sudo ln -s /opt/apache-maven-3.5.3 /opt/apache-maven echo 'export PATH=$PATH:/opt/apache-maven/bin' | sudo tee -a /etc/bashrc source /etc/bashrc mvn --version Download source code cd /tmp git clone https://github.com/locationtech/geomesa.git cd geomesa # Build geomesa-hbase from source code mvn clean install -pl geomesa-hbase/geomesa-hbase-dist -am cd /tmp/geomesa/geomesa-hbase/geomesa-hbase-dist/target cp geomesa-hbase_2.11-2.0.0-SNAPSHOT-bin.tar.gz /tmp cd /tmp tar -xvf geomesa-hbase_2.11-2.0.0-SNAPSHOT-bin.tar.gz cp -R geomesa-hbase_2.11-2.0.0-SNAPSHOT /opt ln -s /opt/geomesa-hbase_2.11-2.0.0-SNAPSHOT /opt/geomesa-hbase Configure the environment to use an HDP install Add the below into /etc/profile sed -i '$ a export HADOOP_HOME=/usr/hdp/current/hadoop-client/' /etc/bashrc sed -i '$ a export HBASE_HOME=/usr/hdp/current/hbase-client/' /etc/bashrc sed -i '$ a export GEOMESA_HBASE_HOME=/opt/geomesa-hbase' /etc/bashrc sed -i '$ a export PATH=$PATH:$GEOMESA_HBASE_HOME/bin' /etc/bashrc source /etc/bashrc Due to licensing restrictions, dependencies for shape file support must be separately installed. Do this with the following commands: cd /opt/geomesa-hbase bin/install-jai.sh bin/install-jline.sh bin/install-saxon.sh Deploying the GeoMesa HBase Distributed Runtime JAR Setting the hbase.dynamic.jars.dir property in hbase-site.xml as custom property. Make sure /apps/hbase/lib is already created in HDFS and change owner to hbase:hdfs Copy GeoMesa-Hbase jar to HBase all Nodes cp ${GEOMESA_HBASE_HOME}/dist/hbase/geomesa-hbase-distributed-runtime_2.11-2.0.0-SNAPSHOT.jar /usr/hdp/current/hbase-master/lib chmod 777 /usr/hdp/current/hbase-master/lib/geomesa-hbase-distributed-runtime_2.11-2.0.0-SNAPSHOT.jar Add proprety in Hbase-site.xml via Ambari geomesa.hbase.remote.filtering=false hbase.coprocessor.user.region.classes=org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor Modify zookeeper.znode.parent value in HBase-Site.xml (org: /hbase-unsecure --> New: /hbase) zookeeper.znode.parent=/hbase *Note: Make sure the HBase services are restarted * Test the command that invokes the GeoMesa Tools: $ bin/geomesa-hbase INFO Usage: geomesa-hbase [command] [command options] Commands: ... Download and Build the Tutorial cd ~/ git clone https://github.com/geomesa/geomesa-tutorials.git cd geomesa-tutorials mvn clean install -pl geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart -am Running the Tutorial java -cp geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart/target/geomesa-tutorials-hbase-quickstart-$VERSION.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers <zookeepers> --hbase.catalog <table> # example: java -cp geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart/target/geomesa-tutorials-hbase-quickstart-2.0.0-SNAPSHOT.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers pchalla2.field.hortonworks.com:2181,pchalla0.field.hortonworks.com:2181,pchalla1.field.hortonworks.com:2181 --hbase.catalog qs1 Verify the HBase $ hbase shell hbase(main):001:0> list TABLE qs1 qs1_gdelt_2dquickstart_attr_v4 qs1_gdelt_2dquickstart_id qs1_gdelt_2dquickstart_z2_v2 qs1_gdelt_2dquickstart_z3_v2 5 row(s) in 0.3610 seconds => ["qs1", "qs1_gdelt_2dquickstart_attr_v4", "qs1_gdelt_2dquickstart_id", "qs1_gdelt_2dquickstart_z2_v2", "qs1_gdelt_2dquickstart_z3_v2"] References: http://www.geomesa.org/documentation/user/hbase/install.html https://dev.locationtech.org/mhonarc/lists/geomesa-users/msg02028.html https://viztales.wordpress.com/my-geomesa-experience/ Install GeoServer wget http://sourceforge.net/projects/geoserver/files/GeoServer/2.12.2/geoserver-2.12.2-bin.zip unzip geoserver-2.12.2-bin.zip mv geoserver-2.12.2 /usr/share cd /usr/share ln -s geoserver-2.12.2 geoserver echo "export GEOSERVER_HOME=/usr/share/geoserver" >> /etc/bashrc source /etc/bashrc # Check if GeoServer can be started successfully cd geoserver/bin sh startup.sh open the GeoServer from post 8080/geoserver like below http://pchalla2.field.hortonworks.com:8080/geoserver Installing GeoMesa HBase in GeoServer The following JARs should be copied from the lib directory of your HBase or Hadoop installations into GeoServer’s WEB-INF/lib cd /usr/share/geoserver/webapps/geoserver/WEB-INF/lib/ cp /opt/geomesa-hbase/dist/gs-plugins/geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz . tar -xvf geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz rm -f geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-annotations-2.7.3.2.6.4.0-91.jar . cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-auth-2.7.3.2.6.4.0-91.jar . cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-common-2.7.3.2.6.4.0-91.jar . cp /usr/hdp/2.6.4.0-91/hadoop/lib/protobuf-java-2.5.0.jar . cp /usr/hdp/2.6.4.0-91/hadoop/lib/commons-io-2.4.jar . cp /usr/hdp/2.6.4.0-91/hbase/lib/hbase-server-1.1.2.2.6.4.0-91.jar . cp /usr/hdp/2.6.4.0-91/hadoop/lib/zookeeper-3.4.6.2.6.4.0-91.jar . cp /usr/hdp/2.6.4.0-91/hadoop/lib/commons-configuration-1.6.jar . ln -s /usr/hdp/current/hbase-client/conf/hbase-site.xml ../classes/hbase-site.xml Stop and Start the GeoServer to reflect the GeoMesa HBase plugin Register the GeoMesa Store with GeoServer Log into GeoServer using your user and password credentials. The default administration credentials are: User name: admin; Password: geoserver Click “Stores” and “Add new Store”. Select the HBase (GeoMesa) vector data source, and fill in the required parameters. Basic store info: workspace this is dependent upon your GeoServer installation data source name pick a sensible name, such as geomesa_quick_start description this is strictly decorative; GeoMesa quick start Connection parameters: these are the same parameter values that you supplied on the command line when you ran the tutorial; they describe how to connect to the HBase instance where your data reside Click “Save”, and GeoServer will search your HBase table for any GeoMesa-managed feature types. Publish the Layer GeoServer should recognize the gdelt-quickstart feature type, and should present that as a layer that can be published. Click on the “Publish” link. You will be taken to the “Edit Layer” screen. You will need to enter values for the data bounding boxes. In this case, you can click on the link to compute these values from the data. Bounding Boxes can be updated by clicking on "compute from data" or "compute from native bounds" Click on the “Save” button when you are done. Take a Look Click on the “Layer Preview” link in the left-hand gutter. If you don’t see the quick-start layer on the first page of results, enter the name of the layer you just created into the search box, and press . Once you see your layer, click on the “OpenLayers” link, which will open a new tab. You should see a collection of red dots similar to the following image: refer http://www.geomesa.org/documentation/tutorials/geomesa-quickstart-hbase.html ref: http://docs.geoserver.org/2.12.x/en/user/installation/linux.html https://github.com/geoserver/geoserver/tree/2.12.2 ... View more