Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

ehsan125 — Sat, 02 Nov 2024 05:09:41 GMT

I am usinng Apache Nifi 1.28 and Hortonwrks HDP 2.5.0, while putting the data in hdfs using puthdfs processsor , i am able to get the following error:

`2024-11-02 09:56:58,321 ERROR [Timer-Driven Process Thread-10] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=eb392407-0192-1000-f474-559950bfef43] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:588)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:338)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:309)
at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:188)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:552)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:287)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.lambda$getFileSystemAsUser$0(AbstractHadoopProcessor.java:565)
at java.base/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
at java.base/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
at java.base/java.lang.ScopedValue$Carrier.call(ScopedValue.java:420)
at java.base/java.lang.ScopedValue.callWhere(ScopedValue.java:568)
at java.base/javax.security.auth.Subject.callAs(Subject.java:439)
at java.base/javax.security.auth.Subject.doAs(Subject.java:614)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getFileSystemAsUser(AbstractHadoopProcessor.java:565)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:498)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:387)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1765)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)
2024-11-02 09:56:58,324 WARN [Timer-Driven Process Thread-10] o.a.n.p.hadoop.HDFSResourceHelper Error stopping FileSystem statistics thread: null
2024-11-02 09:56:58,324 INFO [Timer-Driven Process Thread-10] o.a.n.controller.StandardProcessorNode PutHDFS[id=eb392407-0192-1000-f474-559950bfef43] has completely stopped. Completing any associated Futures.`

my nifi.properties are as follows:

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements. See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License. You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Core Properties #

nifi.flow.configuration.file=./conf/flow.xml.gz

nifi.flow.configuration.json.file=./conf/flow.json.gz

nifi.flow.configuration.archive.enabled=true

nifi.flow.configuration.archive.dir=./conf/archive/

nifi.flow.configuration.archive.max.time=30 days

nifi.flow.configuration.archive.max.storage=500 MB

nifi.flow.configuration.archive.max.count=

nifi.flowcontroller.autoResumeState=true

nifi.flowcontroller.graceful.shutdown.period=10 sec

nifi.flowservice.writedelay.interval=500 ms

nifi.administrative.yield.duration=30 sec

# If a component has no work to do (is "bored"), how long should we wait before checking again for work?

nifi.bored.yield.duration=10 millis

nifi.queue.backpressure.count=10000

nifi.queue.backpressure.size=1 GB

nifi.authorizer.configuration.file=./conf/authorizers.xml

nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml

nifi.templates.directory=./conf/templates

nifi.ui.banner.text=

nifi.ui.autorefresh.interval=30 sec

nifi.nar.library.directory=./lib

nifi.nar.library.autoload.directory=./extensions

nifi.nar.working.directory=./work/nar/

nifi.documentation.working.directory=./work/docs/components

nifi.nar.unpack.uber.jar=false

####################

# State Management #

####################

nifi.state.management.configuration.file=./conf/state-management.xml

# The ID of the local state provider

nifi.state.management.provider.local=local-provider

# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.

nifi.state.management.provider.cluster=zk-provider

# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server

nifi.state.management.embedded.zookeeper.start=false

# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true

nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# Database Settings

nifi.database.directory=./database_repository

# Repository Encryption properties override individual repository implementation properties

nifi.repository.encryption.protocol.version=

nifi.repository.encryption.key.id=

nifi.repository.encryption.key.provider=

nifi.repository.encryption.key.provider.keystore.location=

nifi.repository.encryption.key.provider.keystore.password=

# FlowFile Repository

nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository

nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog

nifi.flowfile.repository.directory=./flowfile_repository

nifi.flowfile.repository.checkpoint.interval=20 secs

nifi.flowfile.repository.always.sync=false

nifi.flowfile.repository.retain.orphaned.flowfiles=true

nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager

nifi.queue.swap.threshold=20000

# Content Repository

nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository

nifi.content.claim.max.appendable.size=50 KB

nifi.content.repository.directory.default=./content_repository

nifi.content.repository.archive.max.retention.period=7 days

nifi.content.repository.archive.max.usage.percentage=50%

nifi.content.repository.archive.enabled=true

nifi.content.repository.always.sync=false

nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties

nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

# Persistent Provenance Repository Properties

nifi.provenance.repository.directory.default=./provenance_repository

nifi.provenance.repository.max.storage.time=30 days

nifi.provenance.repository.max.storage.size=10 GB

nifi.provenance.repository.rollover.time=10 mins

nifi.provenance.repository.rollover.size=100 MB

nifi.provenance.repository.query.threads=2

nifi.provenance.repository.index.threads=2

nifi.provenance.repository.compress.on.rollover=true

nifi.provenance.repository.always.sync=false

# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:

# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details

nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship

# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type

nifi.provenance.repository.indexed.attributes=

# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository

# but should provide better performance

nifi.provenance.repository.index.shard.size=500 MB

# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from

# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.

nifi.provenance.repository.max.attribute.length=65536

nifi.provenance.repository.concurrent.merge.threads=2

# Volatile Provenance Respository Properties

nifi.provenance.repository.buffer.size=100000

# Component and Node Status History Repository

nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository

# Volatile Status History Repository Properties

nifi.components.status.repository.buffer.size=1440

nifi.components.status.snapshot.frequency=1 min

# QuestDB Status History Repository Properties

nifi.status.repository.questdb.persist.node.days=14

nifi.status.repository.questdb.persist.component.days=3

nifi.status.repository.questdb.persist.location=./status_repository

# Site to Site properties

nifi.remote.input.host=

nifi.remote.input.secure=true

nifi.remote.input.socket.port=

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

nifi.remote.contents.cache.expiration=30 secs

# web properties #

#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.

# Be aware that changing these properties may affect how your instance can be accessed without any restriction.

# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=

nifi.web.http.port=

nifi.web.http.network.interface.default=

#############################################

nifi.web.https.host=127.0.0.1

nifi.web.https.port=8443

nifi.web.https.network.interface.default=

nifi.web.https.application.protocols=http/1.1

nifi.web.jetty.working.directory=./work/jetty

nifi.web.jetty.threads=200

nifi.web.max.header.size=16 KB

nifi.web.proxy.context.path=

nifi.web.proxy.host=

nifi.web.max.content.size=

nifi.web.max.requests.per.second=30000

nifi.web.max.access.token.requests.per.second=25

nifi.web.request.timeout=60 secs

nifi.web.request.ip.whitelist=

nifi.web.should.send.server.version=true

nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

# Filter JMX MBeans available through the System Diagnostics REST API

nifi.web.jmx.metrics.allowed.filter.pattern=

# Include or Exclude TLS Cipher Suites for HTTPS

nifi.web.https.ciphersuites.include=

nifi.web.https.ciphersuites.exclude=

# security properties #

nifi.sensitive.props.key=UPb36kUIQ3yfIxWqEhMvyPv0AF5WAW3P

nifi.sensitive.props.key.protected=

nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256

nifi.sensitive.props.additional.keys=

nifi.security.autoreload.enabled=false

nifi.security.autoreload.interval=10 secs

nifi.security.keystore=./conf/keystore.p12

nifi.security.keystoreType=PKCS12

nifi.security.keystorePasswd=3a4e313b59186b5b1259bfc743c8d68c

nifi.security.keyPasswd=3a4e313b59186b5b1259bfc743c8d68c

nifi.security.truststore=./conf/truststore.p12

nifi.security.truststoreType=PKCS12

nifi.security.truststorePasswd=50ace3bdcdd2557ca2dc682f6bacce93

nifi.security.user.authorizer=single-user-authorizer

nifi.security.allow.anonymous.authentication=false

nifi.security.user.login.identity.provider=single-user-provider

nifi.security.user.jws.key.rotation.period=PT1H

nifi.security.ocsp.responder.url=

nifi.security.ocsp.responder.certificate=

# OpenId Connect SSO Properties #

nifi.security.user.oidc.discovery.url=

nifi.security.user.oidc.connect.timeout=5 secs

nifi.security.user.oidc.read.timeout=5 secs

nifi.security.user.oidc.client.id=

nifi.security.user.oidc.client.secret=

nifi.security.user.oidc.preferred.jwsalgorithm=

nifi.security.user.oidc.additional.scopes=offline_access

nifi.security.user.oidc.claim.identifying.user=

nifi.security.user.oidc.fallback.claims.identifying.user=

nifi.security.user.oidc.claim.groups=groups

nifi.security.user.oidc.truststore.strategy=JDK

nifi.security.user.oidc.token.refresh.window=60 secs

# Apache Knox SSO Properties #

nifi.security.user.knox.url=

nifi.security.user.knox.publicKey=

nifi.security.user.knox.cookieName=hadoop-jwt

nifi.security.user.knox.audiences=

# SAML Properties #

nifi.security.user.saml.idp.metadata.url=

nifi.security.user.saml.sp.entity.id=

nifi.security.user.saml.identity.attribute.name=

nifi.security.user.saml.group.attribute.name=

nifi.security.user.saml.request.signing.enabled=false

nifi.security.user.saml.want.assertions.signed=true

nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256

nifi.security.user.saml.authentication.expiration=12 hours

nifi.security.user.saml.single.logout.enabled=false

nifi.security.user.saml.http.client.truststore.strategy=JDK

nifi.security.user.saml.http.client.connect.timeout=30 secs

nifi.security.user.saml.http.client.read.timeout=30 secs

# Identity Mapping Properties #

# These properties allow normalizing user identities such that identities coming from different identity providers

# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing

# DNs from certificates and principals from Kerberos into a common identity string:

# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$

# nifi.security.identity.mapping.value.dn=$1@$2

# nifi.security.identity.mapping.transform.dn=NONE

# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$

# nifi.security.identity.mapping.value.kerb=$1@$2

# nifi.security.identity.mapping.transform.kerb=UPPER

# Group Mapping Properties #

# These properties allow normalizing group names coming from external sources like LDAP. The following example

# lowercases any group name.

# nifi.security.group.mapping.pattern.anygroup=^(.*)$

# nifi.security.group.mapping.value.anygroup=$1

# nifi.security.group.mapping.transform.anygroup=LOWER

# Listener Bootstrap properties #

# This property defines the port used to listen for communications from NiFi Bootstrap. If this property

# is missing, empty, or 0, a random ephemeral port is used.

nifi.listener.bootstrap.port=0

# cluster common properties (all nodes must have same values) #

nifi.cluster.protocol.heartbeat.interval=5 sec

nifi.cluster.protocol.heartbeat.missable.max=8

nifi.cluster.protocol.is.secure=false

# cluster node properties (only configure for cluster nodes) #

nifi.cluster.is.node=false

nifi.cluster.node.address=

nifi.cluster.node.protocol.port=

nifi.cluster.node.protocol.max.threads=50

nifi.cluster.node.event.history.size=25

nifi.cluster.node.connection.timeout=5 sec

nifi.cluster.node.read.timeout=5 sec

nifi.cluster.node.max.concurrent.requests=100

nifi.cluster.firewall.file=

nifi.cluster.flow.election.max.wait.time=5 mins

nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #

nifi.cluster.load.balance.host=

nifi.cluster.load.balance.port=6342

nifi.cluster.load.balance.connections.per.node=1

nifi.cluster.load.balance.max.thread.count=8

nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #

nifi.zookeeper.connect.string=

nifi.zookeeper.connect.timeout=10 secs

nifi.zookeeper.session.timeout=10 secs

nifi.zookeeper.root.node=/nifi

nifi.zookeeper.client.secure=false

nifi.zookeeper.security.keystore=

nifi.zookeeper.security.keystoreType=

nifi.zookeeper.security.keystorePasswd=

nifi.zookeeper.security.truststore=

nifi.zookeeper.security.truststoreType=

nifi.zookeeper.security.truststorePasswd=

nifi.zookeeper.jute.maxbuffer=

# Zookeeper properties for the authentication scheme used when creating acls on znodes used for cluster management

# Values supported for nifi.zookeeper.auth.type are "default", which will apply world/anyone rights on znodes

# and "sasl" which will give rights to the sasl/kerberos identity used to authenticate the nifi node

# The identity is determined using the value in nifi.kerberos.service.principal and the removeHostFromPrincipal

# and removeRealmFromPrincipal values (which should align with the kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal

# values configured on the zookeeper server).

nifi.zookeeper.auth.type=

nifi.zookeeper.kerberos.removeHostFromPrincipal=

nifi.zookeeper.kerberos.removeRealmFromPrincipal=

# kerberos #

nifi.kerberos.krb5.file=

# kerberos service principal #

nifi.kerberos.service.principal=

nifi.kerberos.service.keytab.location=

# kerberos spnego principal #

nifi.kerberos.spnego.principal=

nifi.kerberos.spnego.keytab.location=

nifi.kerberos.spnego.authentication.expiration=12 hours

# external properties files for variable registry

# supports a comma delimited list of file locations

nifi.variable.registry.properties=

# analytics properties #

nifi.analytics.predict.enabled=false

nifi.analytics.predict.interval=3 mins

nifi.analytics.query.interval=5 mins

nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares

nifi.analytics.connection.model.score.name=rSquared

nifi.analytics.connection.model.score.threshold=.90

# runtime monitoring properties

nifi.monitor.long.running.task.schedule=

nifi.monitor.long.running.task.threshold=

# Enable automatic diagnostic at shutdown.

nifi.diagnostics.on.shutdown.enabled=false

# Include verbose diagnostic information.

nifi.diagnostics.on.shutdown.verbose=false

# The location of the diagnostics folder.

nifi.diagnostics.on.shutdown.directory=./diagnostics

# The maximum number of files permitted in the directory. If the limit is exceeded, the oldest files are deleted.

nifi.diagnostics.on.shutdown.max.filecount=10

# The diagnostics folder's maximum permitted size in bytes. If the limit is exceeded, the oldest files are deleted.

nifi.diagnostics.on.shutdown.max.directory.size=10 MB

# Performance tracking properties

## Specifies what percentage of the time we should track the amount of time processors are using CPU, reading from/writing to content repo, etc.

## This can be useful to understand which components are the most expensive and to understand where system bottlenecks may be occurring.

## The value must be in the range of 0 (inclusive) to 100 (inclusive). A larger value will produce more accurate results, while a smaller value may be

## less expensive to compute.

## Results can be obtained by running "nifi.sh diagnostics <filename>" and then inspecting the produced file.

nifi.performance.tracking.percentage=0

# NAR Provider Properties #

# These properties allow configuring one or more NAR providers. A NAR provider retrieves NARs from an external source

# and copies them to the directory specified by nifi.nar.library.autoload.directory.

# Each NAR provider property follows the format:

# nifi.nar.library.provider.<identifier>.<property-name>

# Each NAR provider must have at least one property named "implementation".

# Example HDFS NAR Provider:

# nifi.nar.library.provider.hdfs.implementation=org.apache.nifi.flow.resource.hadoop.HDFSExternalResourceProvider

# nifi.nar.library.provider.hdfs.resources=/path/to/core-site.xml,/path/to/hdfs-site.xml

# nifi.nar.library.provider.hdfs.storage.location=hdfs://hdfs-location

# nifi.nar.library.provider.hdfs.source.directory=/nars

# nifi.nar.library.provider.hdfs.kerberos.principal=nifi@NIFI.COM

# nifi.nar.library.provider.hdfs.kerberos.keytab=/path/to/nifi.keytab

# nifi.nar.library.provider.hdfs.kerberos.password=

# Example NiFi Registry NAR Provider:

# nifi.nar.library.provider.nifi-registry.implementation=org.apache.nifi.registry.extension.NiFiRegistryNarProvider

# nifi.nar.library.provider.nifi-registry.url=http://localhost:18080``

Re: Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

MattWho — Mon, 04 Nov 2024 14:26:49 GMT

@ehsan125

What version of Java is you NiFi using?

This may be related to:
https://issues.apache.org/jira/browse/HADOOP-19212

You could try adding a new java.arg to the nifi bootstrap.conf file as below to see if it helps:

java.arg.manager=-Djava.security.manager=allow

Any modifications to bootstrap.conf file will require a NiFi restart to take affect.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

question Re: Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed in Support Questions

Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

Re: Failure is due to java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed