Member since
05-16-2019
12
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8142 | 10-23-2017 10:56 AM |
10-26-2019
04:07 AM
I have faced similar issue. When I try to logon to KnoxSSO, it is authenticated correctly and redirected to SSO logon page. After few hours investigation, I have realised that I need to setup ldap configs for my Advanced topology in Knox too. Not just KnoxSSO. I made below changes and I am able to resolve the issue. Under Advanced Topology <param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},cn=users,cn=accounts,dc=us-west-1,dc=compute,dc=internal</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://xxxxxxxxxxx.us-west-1.compute.internal:389</value>
</param> Under Advanced knoxsso-topology <param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},cn=users,cn=accounts,dc=us-west-1,dc=compute,dc=internal</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://xxxxxxxxxxx.us-west-1.compute.internal:389</value>
</param>
<param>
<name>knoxsso.redirect.whitelist.regex</name>
<value>.*</value>
</param> And I have ensured the Ambari LDAP setup has the same basedn
... View more
10-22-2018
03:16 PM
Problem: How to enable LDAP for users to view Activity Explorer Dashboard. We may have shiro.ini enabled for LDAP for Zepplin but it is different for Activity Explorer Dashboard. Solution: On SmartSense Activity Explorer node, under [main] section cd /etc/smartsense-activity/conf/
vi shiro.ini
[main]
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
securityManager.sessionManager = $sessionManager
securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login
ldapRealm = org.apache.zeppelin.realm.LdapRealm
ldapRealm.contextFactory.systemUsername = <fill your details>
ldapRealm.contextFactory.systemPassword = <fill your details>
ldapRealm.contextFactory.authenticationMechanism = SIMPLE
ldapRealm.contextFactory.url = <fill your details>
ldapRealm.pagingSize = 200
ldapRealm.authorizationEnabled = true
ldapRealm.searchBase = <fill your details>
ldapRealm.memberAttributeValueTemplate = <fill your details>
ldapRealm.userSearchAttributeName = <fill your details>
ldapRealm.userLowerCase = true
... View more
06-14-2018
03:05 PM
If we don't want to install Hue on HDP then we can insert the queries manually to Ambari Hive Views by following the below. INSERT INTO ambari_db.DS_SAVEDQUERY_9 VALUES ("4", "default", "admin", "/user/admin/hive/scripts/hive-query-4.hql", "select * from sample_07 limit 15;", "sample_07_manual2" );
echo "select * from sample_07 limit 15;" >> hive-query-4.hql
su - hdfs
hdfs dfs -put hive-query-4.hql /user/admin/hive/scripts/hive-query-4.hql
hdfs dfs -chown admin:hadoop /user/admin/hive/scripts/hive-query-4.hql
... View more
06-14-2018
01:47 PM
2 Kudos
Short Description
Migrate Hive saved queries from Hue 3.9.0 of CDH 5.7.2 into Hive Views of Ambari 2.6.2.2 on HDP 2.6.5
Environment: CentOS 7, CDH 5.7.2, Hue 3.9.0, Ambari 2.6.2, HDP 2.6.2, Hue 2.6.2, mySQL5.6
Problem: There are over 40,000 saved / history queries in CDH-Hue. During CDH to HDP migration, Hive saved queries from Hue needs to be migrated to Ambari Hive View saved queries.
Assumptions: You have Hue of CDH configured with MySQL and CDH to HDP migration is successful. Solution
Step 1: Install Hue 2.6.2 on HDP cluster pointing to a new database called hue_new262
follow the link HDP-2.6.5 - installing_hue
Step 2: Insert records from hue_cdh (database of Hue CDH) to hue_new262 (database of Hue HDP)
Impacted tables auth_user, auth_user_groups, beeswax_savedquery, beeswax_queryhistory
insert into hue_new262.auth_user (id, username, first_name, last_name, email, password, is_staff, is_active, is_superuser, last_login, date_joined) select id, username, first_name, last_name, email, password, is_staff, is_active, is_superuser, last_login, date_joined from hue_cdh.auth_user;
insert into hue_new262.auth_user_groups (id, user_id, group_id) select id, user_id, group_id from hue_cdh.auth_user_groups;
insert into hue_new262.beeswax_savedquery (name, type, is_auto, mtime, owner_id, data, id, `desc`, is_trashed) select name, type, is_auto, mtime, owner_id, data, id, 'desc', is_trashed from hue_cdh.beeswax_savedquery;
insert into hue_new262.beeswax_queryhistory (submission_date, last_state, server_id, log_context, design_id, owner_id, query, has_results, id, notify, server_name, server_host, server_port, server_type, server_guid, operation_type, modified_row_count, statement_number, query_type, result_rows, result_size, query_stats, session_id, cache_tag) select submission_date, last_state, server_id, log_context, design_id, owner_id, query, has_results, id, notify, server_name, server_host, server_port, server_type, server_guid, operation_type, modified_row_count, statement_number, query_type, NULL, NULL, NULL, NULL, NULL from hue_cdh.beeswax_queryhistory;
Step 3: Migrating Hue Artifacts to Ambari Views
Follow instructions migrating_hue_to_ambari_views to configure HueToAmbariMigration
Issues faced
1. If we want to create new Hive View (ex: HueView) then make sure you open the HueView at least once before starting the migration. Otherwise DS_SAVEDQUERY_* table not found error would occur.
2. If we try to open Hue UI, the login will fail as the stored password in table auth_user in Hue 3.9.0 is different from Hue 2.6.2. So do not try to logon to Hue or do not insert the auth_user and auth_user_groups. Create the users as new users in Hue 2.6.2 UI but make sure the order (column id is auto increment) of user creation is same as CDH Hue user creation order.
... View more
Labels:
03-29-2018
02:41 PM
1 Kudo
Goal: Ingest Geospatial data (ais data in CSV format) by using GeoMesa HBase after GeoMesa HBase is installed and configured. Follow install-and-configure-geomesa-hbase-and-geoserver if GeoMesa HBase is not installed 151001000015660,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:33,1
151001000015661,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:34,1
151002008706375,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:46,1
151002008706376,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:47,1 Create GeoMesa SimpleFeatureType (sft) configuration file as below in ais_data_conf.sft geomesa.sfts.ais-data = {
attributes = [
{ name = "arkposid", type = "Long", index = true }
{ name = "mmsi", type = "Integer", index = false }
{ name = "navigationalstatus", type = "Integer", index = false }
{ name = "coords", type = "Point", index = true, srid = 4326, default = true }
{ name = "sog", type = "Double", index = false }
{ name = "cog", type = "Double", index = false }
{ name = "rot", type = "Double", index = false }
{ name = "heading", type = "Integer", index = false }
{ name = "acquisitiontime", type = "String", index = false }
{ name = "iptype", type = "Integer", index = false }
]
}
Create GeoMesa converter configuration file as below in ais_data_feature_conf.convert geomesa.converters.ais-data = {
type = "delimited-text",
format = "CSV",
id-field = "toString($arkposid)",
fields = [
{ name = "arkposid", transform = "$1::long" }
{ name = "mmsi", transform = "$2::int" }
{ name = "navigationalstatus", transform = "$3::int" }
{ name = "lon", transform = "$4::double" }
{ name = "lat", transform = "$5::double" }
{ name = "coords", transform = "point($lon, $lat)" }
{ name = "sog", transform = "$6::double" }
{ name = "cog", transform = "$7::double" }
{ name = "rot", transform = "$8::double" }
{ name = "heading", transform = "$9::int" }
{ name = "acquisitiontime", transform = "toString($10)" }
{ name = "iptype", transform = "$11::int" }
]
}
Run the geomesa-hbase command to insert the CSV data into HBase geomesa-hbase ingest \
-c ais \
-s /tmp/ais_data_conf.sft \
-C /tmp/ais_data_feature_conf.convert \
/tmp/sample.csv Should received successful message as below <code>INFO Creating schema 'ais-data'
INFO Running ingestion in local mode
INFO Ingesting 1 file with 1 thread
[============================================================] 100% complete 4 ingested 0 failed in 00:00:01
INFO Local ingestion complete in 00:00:01
INFO Ingested 4 features and failed to ingest 0 feature. Refer install-and-configure-geomesa-hbase-and-geoserver to view the ingested data from GeoServer
... View more
03-27-2018
01:21 PM
1 Kudo
Introduction Installing GeoMesa HBase and GeoServer on HDP Prerequisites HDP 2..6.4 on CentOS7.2 Java Development Kit 1.8, Apache Maven, a GitHub client, and JAVA_HOME is configured below commands can be used yum install git -y
# configure JAVA_HOME
sed -i '$ a export JAVA_HOME=/usr/jdk64/jdk1.8.0_112' /etc/bashrc
sed -i '$ a export PATH=$PATH:$JAVA_HOME/bin' /etc/bashrc
source /etc/bashrc
java -version
#install Maven:
cd /tmp
wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz
tar -zxvf apache-maven-3.5.3-bin.tar.gz
sudo mv /tmp/apache-maven-3.5.3 /opt
sudo chown -R root:root /opt/apache-maven-3.5.3
sudo ln -s /opt/apache-maven-3.5.3 /opt/apache-maven
echo 'export PATH=$PATH:/opt/apache-maven/bin' | sudo tee -a /etc/bashrc
source /etc/bashrc
mvn --version Download source code cd /tmp
git clone https://github.com/locationtech/geomesa.git
cd geomesa
# Build geomesa-hbase from source code
mvn clean install -pl geomesa-hbase/geomesa-hbase-dist -am
cd /tmp/geomesa/geomesa-hbase/geomesa-hbase-dist/target
cp geomesa-hbase_2.11-2.0.0-SNAPSHOT-bin.tar.gz /tmp
cd /tmp
tar -xvf geomesa-hbase_2.11-2.0.0-SNAPSHOT-bin.tar.gz
cp -R geomesa-hbase_2.11-2.0.0-SNAPSHOT /opt
ln -s /opt/geomesa-hbase_2.11-2.0.0-SNAPSHOT /opt/geomesa-hbase Configure the environment to use an HDP install Add the below into /etc/profile sed -i '$ a export HADOOP_HOME=/usr/hdp/current/hadoop-client/' /etc/bashrc
sed -i '$ a export HBASE_HOME=/usr/hdp/current/hbase-client/' /etc/bashrc
sed -i '$ a export GEOMESA_HBASE_HOME=/opt/geomesa-hbase' /etc/bashrc
sed -i '$ a export PATH=$PATH:$GEOMESA_HBASE_HOME/bin' /etc/bashrc
source /etc/bashrc Due to licensing restrictions, dependencies for shape file support must be separately installed. Do this with the following commands: cd /opt/geomesa-hbase
bin/install-jai.sh
bin/install-jline.sh
bin/install-saxon.sh Deploying the GeoMesa HBase Distributed Runtime JAR Setting the hbase.dynamic.jars.dir property in hbase-site.xml as custom property. Make sure /apps/hbase/lib is already created in HDFS and change owner to hbase:hdfs Copy GeoMesa-Hbase jar to HBase all Nodes cp ${GEOMESA_HBASE_HOME}/dist/hbase/geomesa-hbase-distributed-runtime_2.11-2.0.0-SNAPSHOT.jar /usr/hdp/current/hbase-master/lib
chmod 777 /usr/hdp/current/hbase-master/lib/geomesa-hbase-distributed-runtime_2.11-2.0.0-SNAPSHOT.jar Add proprety in Hbase-site.xml via Ambari geomesa.hbase.remote.filtering=false
hbase.coprocessor.user.region.classes=org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor Modify zookeeper.znode.parent value in HBase-Site.xml (org: /hbase-unsecure --> New: /hbase) zookeeper.znode.parent=/hbase
*Note: Make sure the HBase services are restarted * Test the command that invokes the GeoMesa Tools: $ bin/geomesa-hbase
INFO Usage: geomesa-hbase [command] [command options]
Commands:
... Download and Build the Tutorial cd ~/
git clone https://github.com/geomesa/geomesa-tutorials.git
cd geomesa-tutorials
mvn clean install -pl geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart -am Running the Tutorial java -cp geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart/target/geomesa-tutorials-hbase-quickstart-$VERSION.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers <zookeepers> --hbase.catalog <table>
# example:
java -cp geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart/target/geomesa-tutorials-hbase-quickstart-2.0.0-SNAPSHOT.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers pchalla2.field.hortonworks.com:2181,pchalla0.field.hortonworks.com:2181,pchalla1.field.hortonworks.com:2181 --hbase.catalog qs1 Verify the HBase $ hbase shell
hbase(main):001:0> list
TABLE
qs1
qs1_gdelt_2dquickstart_attr_v4
qs1_gdelt_2dquickstart_id
qs1_gdelt_2dquickstart_z2_v2
qs1_gdelt_2dquickstart_z3_v2
5 row(s) in 0.3610 seconds
=> ["qs1", "qs1_gdelt_2dquickstart_attr_v4", "qs1_gdelt_2dquickstart_id", "qs1_gdelt_2dquickstart_z2_v2", "qs1_gdelt_2dquickstart_z3_v2"]
References: http://www.geomesa.org/documentation/user/hbase/install.html https://dev.locationtech.org/mhonarc/lists/geomesa-users/msg02028.html https://viztales.wordpress.com/my-geomesa-experience/ Install GeoServer wget http://sourceforge.net/projects/geoserver/files/GeoServer/2.12.2/geoserver-2.12.2-bin.zip
unzip geoserver-2.12.2-bin.zip
mv geoserver-2.12.2 /usr/share
cd /usr/share
ln -s geoserver-2.12.2 geoserver
echo "export GEOSERVER_HOME=/usr/share/geoserver" >> /etc/bashrc
source /etc/bashrc
# Check if GeoServer can be started successfully
cd geoserver/bin
sh startup.sh open the GeoServer from post 8080/geoserver like below http://pchalla2.field.hortonworks.com:8080/geoserver Installing GeoMesa HBase in GeoServer The following JARs should be copied from the lib directory of your HBase or Hadoop installations into GeoServer’s WEB-INF/lib cd /usr/share/geoserver/webapps/geoserver/WEB-INF/lib/
cp /opt/geomesa-hbase/dist/gs-plugins/geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz .
tar -xvf geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz
rm -f geomesa-hbase-gs-plugin_2.11-2.0.0-SNAPSHOT-install.tar.gz
cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-annotations-2.7.3.2.6.4.0-91.jar .
cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-auth-2.7.3.2.6.4.0-91.jar .
cp /usr/hdp/2.6.4.0-91/hadoop//hadoop-common-2.7.3.2.6.4.0-91.jar .
cp /usr/hdp/2.6.4.0-91/hadoop/lib/protobuf-java-2.5.0.jar .
cp /usr/hdp/2.6.4.0-91/hadoop/lib/commons-io-2.4.jar .
cp /usr/hdp/2.6.4.0-91/hbase/lib/hbase-server-1.1.2.2.6.4.0-91.jar .
cp /usr/hdp/2.6.4.0-91/hadoop/lib/zookeeper-3.4.6.2.6.4.0-91.jar .
cp /usr/hdp/2.6.4.0-91/hadoop/lib/commons-configuration-1.6.jar .
ln -s /usr/hdp/current/hbase-client/conf/hbase-site.xml ../classes/hbase-site.xml Stop and Start the GeoServer to reflect the GeoMesa HBase plugin Register the GeoMesa Store with GeoServer Log into GeoServer using your user and password credentials. The default administration credentials are: User name: admin; Password: geoserver Click “Stores” and “Add new Store”. Select the HBase (GeoMesa) vector data source, and fill in the required parameters. Basic store info: workspace this is dependent upon your GeoServer installation data source name pick a sensible name, such as geomesa_quick_start description this is strictly decorative; GeoMesa quick start Connection parameters: these are the same parameter values that you supplied on the command line when you ran the tutorial; they describe how to connect to the HBase instance where your data reside Click “Save”, and GeoServer will search your HBase table for any GeoMesa-managed feature types. Publish the Layer GeoServer should recognize the gdelt-quickstart feature type, and should present that as a layer that can be published. Click on the “Publish” link. You will be taken to the “Edit Layer” screen. You will need to enter values for the data bounding boxes. In this case, you can click on the link to compute these values from the data. Bounding Boxes can be updated by clicking on "compute from data" or "compute from native bounds" Click on the “Save” button when you are done. Take a Look Click on the “Layer Preview” link in the left-hand gutter. If you don’t see the quick-start layer on the first page of results, enter the name of the layer you just created into the search box, and press . Once you see your layer, click on the “OpenLayers” link, which will open a new tab. You should see a collection of red dots similar to the following image: refer http://www.geomesa.org/documentation/tutorials/geomesa-quickstart-hbase.html ref: http://docs.geoserver.org/2.12.x/en/user/installation/linux.html https://github.com/geoserver/geoserver/tree/2.12.2
... View more
03-05-2018
02:48 PM
Have you tried with --package option (like --packages com.databricks:spark-xml_2.11:0.4.1) spark-submit \
--packages com.databricks:spark-xml_2.11:0.4.1 \
--class csvdf \
/CSVDF/target/scala-2.11/misc-test_2.11-1.0.jar
... View more
10-23-2017
01:03 PM
you can try to include updateAttribute and modify particular variable in runtime. If that needs to be permanent for all flowfiles, update in nifi.properties--> nifi.variable.registry.properties (example: create custom.properties and add custom variables) and restart as suggested by @Mattclarke
... View more
10-23-2017
10:56 AM
you can try below script: for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done
... View more