Member since
03-24-2015
15
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 228 | 05-23-2022 08:05 PM | |
| 15247 | 01-04-2016 03:21 AM |
11-01-2022
12:01 AM
Hi @Siddu198 Add this config to your job: set("mapreduce.fileoutputcommitter.algorithm.version","2")
... View more
05-23-2022
08:05 PM
I answer my own question. This is due to wrong user login format. It should be in email format. Change this efm.security.user.auth.adminIdentities=admin to efm.security.user.auth.adminIdentities=admin@domain.com Thank you.
... View more
05-23-2022
02:12 AM
Hi, I've been successfully secure EFM to Keycloak server with oidc auth. But Minifi agent wont show up in EFM Dashboard. I've check both EFM and minifi log but not found any clues. Need your help. Here my conf/efm.properties config: # Web Server TLS Properties efm.server.ssl.enabled=true efm.server.ssl.keyStore=/home/efm/certs/keystore.jks efm.server.ssl.keyStoreType=jks efm.server.ssl.keyStorePassword=ksPasswd efm.server.ssl.keyPassword=ksPasswd efm.server.ssl.trustStore=/home/efm/certs/truststore.jks efm.server.ssl.trustStoreType=jks efm.server.ssl.trustStorePassword=changeit efm.server.ssl.clientAuth=WANT # User Authentication Properties efm.security.user.auth.enabled=true efm.security.user.auth.adminIdentities=admin efm.security.user.auth.autoRegisterNewUsers=true efm.security.user.auth.authTokenExpiration=12h efm.security.user.certificate.enabled=true efm.security.user.oidc.enabled=true efm.security.user.oidc.issuerUri=https://keycloak.domain.com:8443/realms/efm efm.security.user.oidc.clientId=efm efm.security.user.oidc.clientSecret=gW23NlKxOfdsFmJMiarFNcXs454g1Zk4ZTew4 efm.security.user.oidc.scopes=profile,email efm.security.user.oidc.usernameAttribute=email efm.security.user.oidc.displayNameAttribute=name efm.security.user.oidc.staticConfig.enabled=false efm.security.user.oidc.staticConfig.authorizationUri= efm.security.user.oidc.staticConfig.tokenUri= efm.security.user.oidc.staticConfig.userInfoUri= efm.security.user.oidc.staticConfig.jwkSetUri= Minifi conf/bootstrap.conf: # Security Properties # # These properties take precedence over any equivalent properties specified in config.yml # nifi.minifi.security.keystore=/home/minifi/certs/keystore.jks nifi.minifi.security.keystoreType=jks nifi.minifi.security.keystorePasswd=ksPasswd nifi.minifi.security.keyPasswd=ksPasswd nifi.minifi.security.truststore=/home/minifi/certs/truststore.jks nifi.minifi.security.truststoreType=jks nifi.minifi.security.truststorePasswd=changeit nifi.minifi.security.ssl.protocol=TLSv1.2 nifi.minifi.sensitive.props.key=myEfmPassword123456 nifi.minifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL nifi.minifi.sensitive.props.provider=BC # MiNiFi Command & Control Configuration # C2 Properties # Enabling C2 Uncomment each of the following options # define those with missing options nifi.c2.enable=true ## define protocol parameters nifi.c2.rest.url=https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat nifi.c2.rest.url.ack=https://efm.domain.com:10090/efm/api/c2-protocol/acknowledge ## heartbeat in milliseconds. defaults to once a second nifi.c2.agent.heartbeat.period=1000 ## define parameters about your agent nifi.c2.agent.class=java-linux # Optional. Defaults to a hardware based unique identifier nifi.c2.agent.identifier=ip221 ## Define TLS security properties for C2 communications nifi.c2.security.truststore.location=/home/minifi/certs/truststore.jks nifi.c2.security.truststore.password=changeit nifi.c2.security.truststore.type=JKS nifi.c2.security.keystore.location=/home/minifi/certs/keystore.jks nifi.c2.security.keystore.password=ksPasswd nifi.c2.security.keystore.type=JKS nifi.c2.security.need.client.auth=true Minifi Logs: $ tail -f logs/minifi-bootstrap.log 2022-05-23 15:15:24,241 INFO [MiNiFi Bootstrap Command Listener] o.apache.nifi.minifi.bootstrap.RunMiNiFi The thread to run Apache MiNiFi is now running and listening for Bootstrap requests on port 37443 2022-05-23 15:15:29,119 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:29,813 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:30,803 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:31,784 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:32,778 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:33,782 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:34,779 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:35,773 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:36,778 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:37,776 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat 2022-05-23 15:15:38,771 INFO [pool-2-thread-1] o.a.n.m.b.s.r.RestHeartbeatReporter Performing request to https://efm.domain.com:10090/efm/api/c2-protocol/heartbeat EFM version efm-1.4.0.0-125 MiNiFi version minifi-0.6.0.1.3.1.0-68 References: - Agent authentication (cloudera.com) - https://nizan-shookroun.medium.com/install-and-configure-minifi-agents-f22a0cc09622
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Security
05-23-2022
12:39 AM
You need to add CA cert to $JAVA_HOME/jre/lib/security/jssecacerts file in your JDK used by minifi.
... View more
03-03-2020
07:05 PM
I face this error when build with "mvn clean package -P mpack": [WARNING] The requested profile "mpack" could not be activated because it does not exist.
[ERROR] Failed to execute goal on project das-commons: Could not resolve dependencies for project com.hortonworks.das:das-commons:jar:1.2.0-SNAPSHOT: Failure to find org.apache.hadoop:hadoop-common:jar:3.1.0.3.0.0.0-1634 in <a href="https://repo.maven.apache.org/maven2" target="_blank">https://repo.maven.apache.org/maven2</a> was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
... View more
10-27-2017
12:58 AM
Hi @andrzej_jedrzej, can you explain how can you solve this problem? thank you.
... View more
01-04-2016
03:39 AM
I fix this by adding following line: export CDH_MR2_HOME=$HADOOP_HOME in .bash_profile
... View more
01-04-2016
03:21 AM
Hi Harsh, @Harsh J wrote: Could you re-run the command also with the below env set? $ export HADOOP_ROOT_LOGGER=TRACE,console $ export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Djavax.net.debug=ssl" $ hadoop fs -ls / Here is the result: 16/01/04 17:42:07 DEBUG util.Shell: setsid exited with exit code 0
16/01/04 17:42:07 DEBUG conf.Configuration: parsing URL jar:file:/app/hadoop-2.6.0-cdh5.4.5/share/hadoop/common/hadoop-common-2.6.0-cdh5.4.5.jar!/core-default.xml
16/01/04 17:42:07 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4ae69619
16/01/04 17:42:07 DEBUG conf.Configuration: parsing URL file:/home/user01/yarn-conf/core-site.xml
16/01/04 17:42:07 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@30317bdd
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
16/01/04 17:42:08 DEBUG security.Groups: Creating new Groups object
16/01/04 17:42:08 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
>>>KinitOptions cache name is /tmp/krb5cc_501
>>>DEBUG <CCacheInputStream> client principal is user01@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> key type: 23
>>>DEBUG <CCacheInputStream> auth time: Mon Jan 04 17:41:23 WIB 2016
>>>DEBUG <CCacheInputStream> start time: Mon Jan 04 17:41:06 WIB 2016
>>>DEBUG <CCacheInputStream> end time: Tue Jan 05 03:41:23 WIB 2016
>>>DEBUG <CCacheInputStream> renew_till time: Mon Jan 11 17:41:06 WIB 2016
>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
16/01/04 17:42:08 DEBUG security.UserGroupInformation: hadoop login
16/01/04 17:42:08 DEBUG security.UserGroupInformation: hadoop login commit
16/01/04 17:42:08 DEBUG security.UserGroupInformation: using kerberos user:user01@DEVELOPMENT.COM
16/01/04 17:42:08 DEBUG security.UserGroupInformation: Using user: "user01@DEVELOPMENT.COM" with name user01@DEVELOPMENT.COM
16/01/04 17:42:08 DEBUG security.UserGroupInformation: failure to login
javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name user01@DEVELOPMENT.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:596)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:812)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:100)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.lang.IllegalArgumentException: Illegal principal name user01@DEVELOPMENT.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
... 30 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 32 more
ls: failure to login From logs above shows that kerberos client config is still pointed to default /etc/krb5.conf. I use different path by exporting env variable KRB5_CONFIG. After edit /etc/krb5.conf to the proper value, its now works properly. I can browse HDFS and submit job to YARN. @Harsh J wrote: Is this remote host also carrying the Unlimited JCE policy jars under its JDK, so it may use AES-256 if that is in use? I use JDK from cloudera: jdk1.7.0_67-cloudera Thank you very much Harsh.
... View more
12-22-2015
02:18 AM
Hi, It is possible to access secure cluster from host that not part of the cluster as service (hdfs/yarn/etc) gateway? I've download client configuration from cluster and configure krb5.conf. kinit is succeeded but still unable to connect to hdfs. $ klist
Ticket cache: FILE:/tmp/krb5cc_501
Default principal: user01@DEVELOPMENT.COM
Valid starting Expires Service principal
12/22/15 14:57:07 12/23/15 00:57:11 krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
renew until 12/29/15 14:57:07
$ export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Djavax.net.debug=ssl"
$ hadoop fs -ls /
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_501
>>>DEBUG <CCacheInputStream> client principal is user01@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> key type: 23
>>>DEBUG <CCacheInputStream> auth time: Tue Dec 22 14:57:11 WIB 2015
>>>DEBUG <CCacheInputStream> start time: Tue Dec 22 14:57:07 WIB 2015
>>>DEBUG <CCacheInputStream> end time: Wed Dec 23 00:57:11 WIB 2015
>>>DEBUG <CCacheInputStream> renew_till time: Tue Dec 29 14:57:07 WIB 2015
>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
ls: failure to login
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN
-
Gateway
-
HDFS
-
Security
06-16-2015
05:03 AM
Hi, I applied point 1 with following syntax, and still not work: $ hadoop jar JarName.jar ClassName -D oozie.launcher.mapreduce.task.classpath.user.prece dence=true param1 param2 In my case, I have fat-jar with modified parquet lib in it. But cluster always picking default parquet lib in CDH. In standalone mode, my program is running fine.
... View more
