About nismaily

nismaily · ‎08-19-2019

This article describes the configuration of the authorizers.xml for using the Composite User Group Provider for both LDAP and File based authentication in Cloudera Manager for CDF 1.0 This article assumes TLS has already been configured for NiFi using either the NiFi CA or your own certs. Here is the Active Directory Structure that I will use for my sync: Here are the groups in the Groups OU: Here are the users in the Users OU: Here are the service users in the ServiceUsers OU: First, I will import my Active Directory root certificate into the NiFi and Java Truststores: keytool -import -file adcert.pem -alias ad -keystore /var/lib/nifi/cert/truststore.jks keytool -import -file adcert.pem -alias ad -keystore /etc/pki/java/cacerts Now login to Cloudera Manager and proceed to the NiFi Configuration. Update the configuration to use the "composite-user-group-provider" as follows: xml.authorizers.accessPolicyProvider.file-access-policy-provider.property.User Group Provider=composite-user-group-provider Next, use the "NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/authorizers.xml" and add the following configurations using the XML view: <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.class</name> <value>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Manager DN</name> <value>CN=ldapbind-svc,OU=ServiceUsers,OU=CLDR,DC=nismaily,DC=com</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Manager Password</name> <value>hadoop</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Referral Strategy</name> <value>FOLLOW</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Connect Timeout</name> <value>10 secs</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Read Timeout</name> <value>10 secs</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Url</name> <value>ldaps://win-ltfjo4jgo4r.nismaily.com:636</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Sync Interval</name> <value>15 mins</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Search Base</name> <value>OU=Users,OU=CLDR,DC=nismaily,DC=com</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Object Class</name> <value>user</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Search Scope</name> <value>SUBTREE</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Identity Attribute</name> <value>sAMAccountName</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.User Group Name Attribute</name> <value>memberof</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Base</name> <value>OU=Groups,OU=CLDR,DC=nismaily,DC=com</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Object Class</name> <value>group</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Search Scope</name> <value>SUBTREE</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Name Attribute</name> <value>cn</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Group Member Attribute</name> <value>member</value> </property> <property> <name>xml.authorizers.userGroupProvider.composite-user-group-provider.class</name> <value>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</value> </property> <property> <name>xml.authorizers.userGroupProvider.composite-user-group-provider.property.Configurable User Group Provider</name> <value>file-user-group-provider</value> </property> <property> <name>xml.authorizers.userGroupProvider.composite-user-group-provider.property.User Group Provider 1</name> <value>ldap-user-group-provider</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.Authentication Strategy</name> <value>LDAPS</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore</name> <value>/var/lib/nifi/cert/keystore.jks</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Password</name> <value>hadoop</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Keystore Type</name> <value>jks</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore</name> <value>/var/lib/nifi/cert/truststore.jks</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Password</name> <value>hadoop</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Truststore Type</name> <value>jks</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Client Auth</name> <value>WANT</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Protocol</name> <value>TLSv1.2</value> </property> <property> <name>xml.authorizers.userGroupProvider.ldap-user-group-provider.property.TLS - Shutdown Gracefully</name> <value>false</value> </property> Restart NiFi and login to the NiFi UI. Proceed to the Users Tab: You will see the users synced from Active Directory:

nismaily · ‎06-24-2017

Try this: sqoop import \ --username username \ --P \ --connect jdbc:sap://hostname:portnumber/ \ --driver com.sap.db.jdbc.Driver \ --query "select * from \"schema\".\"/path/to/hana/calculation\" where \$CONDITIONS" \ --hcatalog-database hivedatabase \ --hcatalog-table hivetable \ --create-hcatalog-table \ --hcatalog-storage-stanza "stored as orc"

nismaily · ‎05-26-2017

This Is Amazing, thanks for posting this !

nismaily · ‎04-08-2017

I'm in a situation where the team doing the data ingest cannot have command line access to the servers. This is the customer policy and there is no exception that can be made. Hue has a Sqoop UI that the developers can use to do the ingest. Will Ambari have a similar 'sqoop view'? Are there any alternatives? The %sh interpreter needs to be disabled in Zeppelin so that won't work either

nismaily · ‎03-25-2017

Did you copy the ojdbc jar to the nifi lib directory and restart nifi?

nismaily · ‎12-21-2016

What is the correct process in moving the Journal Node Directories to a different disk? Lets say I have 3 servers, each running a Journal Node Service: server1 server2 server3 The Journal Nodes are currently writing to /var/lib/hadoop/hdfs/journal I want to change this to /data1/hadoop/hdfs/journal for each server I've tried: 1) Stopping Namenodes (HA) 2) Copying data from /var/lib/hadoop/hdfs/journal to /data1/hadoop/hdfs/journal 3) Changing the dir property for the Journal Nodes in Ambari 4) Starting the Namenodes This did not seem to work

nismaily · ‎12-21-2016

Edit the following file: /etc/ambari-server/conf/ambari.properties Increase the following parameters: views.request.connect.timeout.millis views.request.read.timeout.millis Restart Ambari Server

nismaily · ‎11-08-2016

I increased the capacity and it worked, thanks!

nismaily · ‎11-03-2016

I'm having trouble getting Hive Interactive (LLAP) to work I'm working on a kerberized HDP 2.5 cluster. It seems like only 1 daemon starts, here's the error: 2016-11-03 15:50:50,084 - Marker index for start of JSON data for 'llapsrtatus' comamnd : 0 2016-11-03 15:50:50,085 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live Instances : '1'. Desired Instances : '4' after 221.781697989 secs. 2016-11-03 15:50:50,085 - LLAP app 'llap0' did not come up after a wait of 221.782001972 seconds. 2016-11-03 15:50:50,087 - LLAP app 'llap0' deployment unsuccessful. What is the cause for this?

nismaily · ‎08-18-2016

When I try to enable the Ranger - Knox Plugin and test the connection, I get the following message: Connection Failed. Unable to retrieve any topologies/services using given parameters. You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check xa_portal.log for more info. In the logs I see: Got invalid REST response from: https://<hostname>:8443/gateway/admin/api/v1/topologies, responsStatus: 401 Did I miss something in the configuration?

nismaily · ‎06-14-2016

How do I specify the yarn queue when writing a map reduce job in RStudio?

nismaily · ‎06-01-2016

Lets say I have multiple kerberized clusters running on openstack. I want to share data between them and avoid using an NFS mount. What is the best way to do this? Possibly mirror data between the clusters using Falcon?

nismaily · ‎06-01-2016

Thanks Ravi, I had to: 1) Copy spark shuffle jars to nodemanager classpaths on all nodes 2) add spark_shuffle to yarn.nodemanager.aux-services, set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService in yarn-site.xml (via Ambari) 3) Restart all nodemanagers 4) Add the following to spark-defaults.conf spark.dynamicAllocation.enabled true spark.shuffle.service.enabled true 5) Set these parameters per job basis spark.dynamicAllocation.initialExecutors=# spark.dynamicAllocation.minExecutors=#

nismaily · ‎05-25-2016

Thanks Ravi, this is very close to what I need. Question, spark.dynamicAllocation.minExecutors seems to be a global property in spark-defautls Is there a way to set this property on a job by job basis? Spark job1 -> min executors 8 Spark job2 -> min executors 5

nismaily · ‎05-25-2016

Thank you ,I'll try this out

nismaily · ‎05-25-2016

Thank You!

nismaily · ‎05-25-2016

Can we configure the capacity scheduler in such a way that a spark job only runs when it can procure enough resources? In the current FIFO setup a spark job will start running if it can get a few of the required executors, but the job will fail because it couldn't get enough resources. I would like the spark job to only start when it can procure all the required resources.

nismaily · ‎05-25-2016

Is there any documentation available on integrating FreeIPA with HDP?

nismaily · ‎05-19-2016

Thanks Sridhar, but what if job3 runs after job1, and then job4 runs, and then job2 runs?

nismaily · ‎05-18-2016

Has anyone come across the following scenario: I launch 5 YARN jobs (requiring variable resources) in the same queue in this order: job1, job2, job3, job4, job5 I’ve configured a Capacity Scheduler with FIFO ordering. Observed behavior: job 1 runs job2 - job5 in waiting state once job1 completes, job2-job5 runs in random order (job4, job2, job3, job5) Is this expected?

nismaily · ‎05-18-2016

Thanks Avinash, They are using HDP 2.3.2 with Ambari 2.1.2

nismaily · ‎05-16-2016

Thanks Sagar, This error appears in the ambari-server logs. HDP Version: 2.3.2 Ambari Version: 2.1.2 Java Version: 1.8.x I'll take a look at the links you posted

nismaily · ‎05-16-2016

Thanks Ravi, I've attached the full stack trace. JCE has been installed on all nodes. ambari-server-stack-trace.txt

nismaily · ‎05-16-2016

Kerberizing an HDP 2.x cluster with Active Directory and Ambari 2.x succeeds with JDK 1.7. When we upgrade to JDK 1.8, it fails with the following error: javax.naming.CommunicationException: simple bind failed:x.y.z:389 [Root exception is javax.net.ssl.SSLException: java.net.SocketException: Connection reset] Anyone experienced this before? Is there a specific release of JDK 1.8.x that we know works?

nismaily · ‎05-12-2016

Alon, what are you attempting to do? Is fs.hdfs.impl the correct property you are looking to change?

nismaily · ‎05-12-2016

Thanks Ravi!

nismaily · ‎05-11-2016

Can you attach your logs for hive ? Also change the log level to debug ($HIVE_HOME/bin/hive --hiveconf hive.root.logger=DEBUG,DRFA)

nismaily · ‎05-11-2016

Are there any available metrics for the overhead associated with enabling the various types of encryption (RPC, HTTPS, etc...) across the cluster?

nismaily · ‎05-11-2016

If we look at the configuration for an older version of Hadoop: name|value|description fs.hdfs.impl org.apache.hadoop.dfs.DistributedFileSystem The FileSystem for hdfs: uris. If we look at the configuration of Hadoop 2.7.1 name|value|description fs.AbstractFileSystem.hdfs.impl org.apache.hadoop.fs.Hdfs The FileSystem for hdfs: uris. For the second part of your question: core-default.xml just gives you the default implementation for core-site.xml You can add the properties to core-site.xml to overwrite the default values

nismaily · ‎05-11-2016

Try using the API command: /api/v1/clusters/<cluster_name>?fields=Clusters/health_report

Online	Offline
Last Visited	‎08-20-2019 05:38 PM

Date Registered	‎04-11-2016 05:39 PM
Last Visited	‎08-20-2019 05:38 PM
Total Messages Posted	32
Total Kudos Received	21

Re: Ingesting calculation views from Hana into Hiv...

Re: Nifi Oracle JDBC Connection Issue

Re: tez view timeout

Re: Spark - YARN Capacity Scheduler

Re: Storm Bolt throws an error when trying to read...

NiFi Composite Authorization - LDAP and File - Usi...

Re: Ingesting calculation views from Hana into Hiv...

Re: Using a local instance of Cloudbreak with Vagr...

Will there be a Ambari - Sqoop View to replace Hue...

Re: Nifi Oracle JDBC Connection Issue

Moving Journal Node Directories

Re: tez view timeout

Re: Hive LLAP fails to start with RUNNING_PARTIAL ...

Hive LLAP fails to start with RUNNING_PARTIAL stat...

Ranger-Knox Plugin Connection Failed

Specify Yarn Queues in RStudio

Best Practice for Sharing Data Between Multiple Ke...

Re: Spark - YARN Capacity Scheduler

Re: Spark - YARN Capacity Scheduler

Re: Spark - YARN Capacity Scheduler

Re: HDP - FreeIPA integration

Spark - YARN Capacity Scheduler

HDP - FreeIPA integration

Re: Yarn Queue Capacity Scheduling

Yarn Queue Capacity Scheduling

Re: Kerberizing HDP 2.x with JDK 1.8 fails

Re: Kerberizing HDP 2.x with JDK 1.8 fails

Re: Kerberizing HDP 2.x with JDK 1.8 fails

Kerberizing HDP 2.x with JDK 1.8 fails

Re: where can I find fs.hdfs.impl property

Re: Encryption Overhead Metrics

Re: INSERT COMMAND HANGING

Encryption Overhead Metrics

Re: where can I find fs.hdfs.impl property

Re: HDP Cluster Health Report