Member since
03-17-2020
3
Posts
0
Kudos Received
0
Solutions
05-15-2020
05:12 AM
I need a Help i have two question please 1-how I can transform the data from apache kafka to hdfs???? 2-how I can transform the data from apache kafka to SparkStreming???? thank you
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
-
HDFS
04-23-2020
11:36 AM
this is my error
stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurations
support_encrypt_authorizers=params.stack_support_encrypt_authorizers
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_properties
Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 9. 2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2020/04/23 18:28:30 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2020/04/23 18:28:30 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2020/04/23 18:28:30 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: bootstrap.conf: /usr/hdf/current/nifi/conf/bootstrap.conf
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) nifi.properties: /usr/hdf/current/nifi/conf/nifi.properties
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) login-identity-providers.xml: null
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) login-identity-providers.xml: null
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) authorizers.xml: /usr/hdf/current/nifi/conf/authorizers.xml
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (src) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: (dest) flow.xml.gz: /var/lib/nifi/conf/flow.xml.gz
2020/04/23 18:28:30 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 153 properties from /usr/hdf/current/nifi/conf/nifi.properties
2020/04/23 18:28:31 INFO [main] org.apache.nifi.properties.NiFiPropertiesLoader: Loaded 153 properties from /usr/hdf/current/nifi/conf/nifi.properties
2020/04/23 18:28:31 INFO [main] org.apache.nifi.properties.ProtectedNiFiProperties: There are 1 protected properties of 5 sensitive properties (100%)
2020/04/23 18:28:31 ERROR [main] org.apache.nifi.properties.AESSensitivePropertyProvider: Error decrypting a protected value
javax.crypto.AEADBadTagException: mac check in GCM failed
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$AEADGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.nifi.properties.AESSensitivePropertyProvider.unprotect(AESSensitivePropertyProvider.java:240)
at org.apache.nifi.properties.ProtectedNiFiProperties.unprotectValue(ProtectedNiFiProperties.java:524)
at org.apache.nifi.properties.ProtectedNiFiProperties.getUnprotectedProperties(ProtectedNiFiProperties.java:343)
at org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:252)
at org.apache.nifi.properties.NiFiPropertiesLoader$load$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.loadNiFiProperties(ConfigEncryptionTool.groovy:585)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1511)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
2020/04/23 18:28:31 WARN [main] org.apache.nifi.properties.ProtectedNiFiProperties: Failed to unprotect 'nifi.sensitive.props.key'
SensitivePropertyProtectionException: Error unprotecting value for nifi.sensitive.props.key
at org.apache.nifi.properties.ProtectedNiFiProperties.unprotectValue(ProtectedNiFiProperties.java:526)
at org.apache.nifi.properties.ProtectedNiFiProperties.getUnprotectedProperties(ProtectedNiFiProperties.java:343)
at org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:252)
at org.apache.nifi.properties.NiFiPropertiesLoader$load$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.loadNiFiProperties(ConfigEncryptionTool.groovy:585)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1511)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
Caused by: javax.crypto.AEADBadTagException: mac check in GCM failed
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$AEADGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.nifi.properties.AESSensitivePropertyProvider.unprotect(AESSensitivePropertyProvider.java:240)
at org.apache.nifi.properties.ProtectedNiFiProperties.unprotectValue(ProtectedNiFiProperties.java:524)
... 32 more
2020/04/23 18:28:31 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error
SensitivePropertyProtectionException: Failed to unprotect key nifi.sensitive.props.key
at org.apache.nifi.properties.ProtectedNiFiProperties.getUnprotectedProperties(ProtectedNiFiProperties.java:358)
at org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:252)
at org.apache.nifi.properties.NiFiPropertiesLoader$load$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.loadNiFiProperties(ConfigEncryptionTool.groovy:585)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1511)
at org.apache.nifi.properties.ConfigEncryptionTool$main.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.LegacyMode.run(LegacyMode.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain.main(EncryptConfigMain.groovy:109)
Cannot migrate key if no previous encryption occurred
usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n ] [-o ] [-l ] [-i ] [-a ] [-u ] [-f ] [-g ]
[-b ] [-k ] [-e ] [-p ] [-w ] [-r] [-m] [-x] [-s <password|keyhex>] [-A ] [-P ] [-c]
This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a master key,
and encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.
-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is
specified)
-i,--outputLoginIdentityProviders The destination login-identity-providers.xml file containing protected config values (will not modify input
login-identity-providers.xml)
-a,--authorizers The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowXml The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowXml The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)
-b,--bootstrapConf The bootstrap.conf file to persist master key
-k,--key The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey The old raw hexadecimal key to use during key migration
-p,--password The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword The old password from which to derive the key during migration
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with
a new key
-x,--encryptFlowXmlOnly If provided, the properties in flow.xml.gz will be re-encrypted with a new key but the nifi.properties and/or
login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.xml.gz
-A,--newFlowAlgorithm The algorithm to use to encrypt the sensitive processor properties in flow.xml.gz
-P,--newFlowProvider The security provider to use to encrypt the sensitive processor properties in flow.xml.gz
-c,--translateCli Translates the nifi.properties file to a format suitable for the NiFi CLI tool
Java home: /usr/jdk64/jdk1.8.0_112
NiFi Toolkit home: /usr/hdf/current/nifi-toolkit</password|keyhex></password|keyhex>
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi