Member since
05-03-2017
73
Posts
4
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1079 | 07-15-2020 06:25 AM | |
| 2236 | 04-21-2020 06:25 AM |
03-09-2021
09:43 AM
Hi Team , Can you suggest me documentation around on how can i use Mirror Maker 2.0 with kerberized Kafka cluster . I tried below configs but its not working below is the properties file # Kafka datacenters. clusters = source, target source.bootstrap.servers = yy:9092,yy:9092 target.bootstrap.servers = xx:9092,xx:9092 source.security.protocol= SASL_PLAINTEXT source.sasl.mechanism= GSSAPI #sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required source.sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab= true \ storeKey= true \ keyTab= xx.keytab \ principal= xx ; source.sasl.kerberos.service.name= kafka # Source and target cluster configurations. source.config.storage.replication.factor = 1 target.config.storage.replication.factor = 1 source.offset.storage.replication.factor = 1 target.offset.storage.replication.factor = 1 source.status.storage.replication.factor = 1 target.status.storage.replication.factor = 1 source->target.enabled = true target->source.enabled = false # Mirror maker configurations. offset-syncs.topic.replication.factor = 1 heartbeats.topic.replication.factor = 1 checkpoints.topic.replication.factor = 1 topics = xxx groups = .* tasks.max = 1 replication.factor = 1 refresh.topics.enabled = false sync.topic.configs.enabled = true refresh.topics.interval.seconds = 30 topics.blacklist = .*[\-\.]internal, .*\.replica, __consumer_offsets groups.blacklist = console-consumer-.*, connect-.*, __.* Error i am getting :- [2021-03-09 12:31:36,042] ERROR Stopping due to error (org.apache.kafka.connect.mirror.MirrorMaker:304) org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51) at org.apache.kafka.connect.mirror.MirrorMaker.addHerder(MirrorMaker.java:235) at org.apache.kafka.connect.mirror.MirrorMaker.lambda$new$1(MirrorMaker.java:136) at java.lang.Iterable.forEach(Iterable.java:75) at org.apache.kafka.connect.mirror.MirrorMaker.<init>(MirrorMaker.java:136) at org.apache.kafka.connect.mirror.MirrorMaker.<init>(MirrorMaker.java:148) at org.apache.kafka.connect.mirror.MirrorMaker.main(MirrorMaker.java:291) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1615311096033, tries=1, nextAllowedTryMs=1615311096134) timed out at 1615311096034 after 1 attempt(s) at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64) ... 7 more
... View more
Labels:
- Labels:
-
Apache Kafka
07-23-2020
05:23 AM
@MattWho Thank you Sir .It worked perfectly . Regards Bharad
... View more
07-15-2020
06:58 AM
Team , We have upgraded NIFI from 1.5 to 1.11 and trying to configure ranger authorization at group level instead of user level . But its not working . Below are the properties i have given in authorizers.xml <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">LDAPS</property> <property name="Manager DN">CN=xxxx,OU=Users,OU=Standard Users,OU=Users and Computers,OU=Atlanta Corporate,OU=CCI,DC=CORP,DC=COX,DC=com</property> <property name="Manager Password">xxxxx</property> <property name="TLS - Keystore">/etc/nifi/conf/signed-keystore.jks</property> <property name="TLS - Keystore Password">xxx</property> <property name="TLS - Keystore Type">JKS</property> <property name="TLS - Truststore">/etc/nifi/conf/coxca-truststore.jks</property> <property name="TLS - Truststore Password">xxx</property> <property name="TLS - Truststore Type">JKS</property> <property name="TLS - Client Auth">WANT</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully">TRUE</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldaps://xxxxx:636</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="User Search Base">ou=cci,dc=corp,dc=cox,dc=com</property> <property name="User Object Class">user</property> <property name="User Search Scope">sub</property> <property name="User Search Filter">(|(memberof=CN=CATL0BDDEVSA,OU=groups,OU=Atlanta Corporate,OU=CCI,DC=CORP,DC=COX,DC=com)(memberof=CN=CATL0BDDEVADMINS,OU=groups,OU=Atlanta Corporate,OU=CCI,DC=CORP,DC=COX,DC=com)(memberof=CN=CATL0BDDEVDEVELOPERS,OU=groups,OU=Atlanta Corporate,OU=CCI,DC=CORP,DC=COX,DC=com))</property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberof, ismemberof</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=cci,dc=corp,dc=cox,dc=com</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">sub</property> <property name="Group Search Filter">(|(cn=CATL0BDDEVSA)(cn=CATL0BDDEVADMINS)(cn=CATL0BDDEVDEVELOPERS))</property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute">uid</property> </userGroupProvider> <authorizer> <identifier>ranger-authorizer</identifier> <class>org.apache.nifi.ranger.authorization.ManagedRangerAuthorizer</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Ranger Audit Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-audit.xml</property> <property name="Ranger Security Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-security.xml</property> <property name="Ranger Service Type">nifi</property> <property name="Ranger Application Id">nifi</property> <property name="Allow Anonymous">true</property> <property name="Ranger Admin Identity"></property> <property name="Ranger Kerberos Enabled">false</property> </authorizer> when i add AD group in ranger for access i get this error 2020-07-13 17:13:33,737 INFO [NiFi Web Server-31] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[bbhimava], groups[none] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden response. 2020-07-13 17:13:45,698 INFO [NiFi Web Server-41] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=xxxx, OU=NIFI) GET https://xxxxxx/nifi-api/site-to-site (source ip: 10.220.3.176) @MattWhorequesting your help .
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Ranger
07-15-2020
06:25 AM
1 Kudo
@Govins28Thanks for he reply .i Figured out the issue was with process group UUID .i have fixed it and it is working as expected . Regards Bharad
... View more
07-14-2020
02:50 PM
Hi Team, I have updated nifi to 1.11.4 from 1.5.0 and the ranger policy is not working as expected .I have set up policy with /process-groups/<process group uuid> and it opens up access to all the process groups which are present in canvas . example :- /process-groups/992fc902-0172-1000-348e-b0bae84f520f. - i gave read and write to this process group .But i am able to go into all the process groups which are available . regards Bharad
... View more
Labels:
- Labels:
-
Apache NiFi
04-29-2020
01:47 PM
Hi @stevenmatison i am trying to get the backup of sqllite database to import into oracle database as i dont have database dump from ambari . Regards Bharad
... View more
04-29-2020
01:15 PM
Hi @stevenmatison , I was able to run the same query from hive ,but from hue i see database lock . When i did some research i got to know that hue stores the user requests in metadata database . so when i checked repository data base i dont see any tables created . so i tried configuring metadata repository by configuring to oracle database and i am running into issues (in the above post ) . Regards Bharad
... View more
04-29-2020
01:09 PM
Hi @lwang I was able to get the details . But when i configure my oracle database i am seeing below error .I dont see any tables were also create in oracle metadata db . [29/Apr/2020 12:36:36 +0000] exception ERROR Internal Server Error: / Traceback (most recent call last): File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/exception.py", line 41, in inner response = get_response(request) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 244, in _legacy_get_response response = middleware_method(request) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/middleware/locale.py", line 25, in process_request language = translation.get_language_from_request(request, check_path=i18n_patterns_used) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/utils/translation/__init__.py", line 211, in get_language_from_request return _trans.get_language_from_request(request, check_path) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/utils/translation/trans_real.py", line 519, in get_language_from_request lang_code = request.session.get(LANGUAGE_SESSION_KEY) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/contrib/sessions/backends/base.py", line 68, in get return self._session.get(key, default) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/contrib/sessions/backends/base.py", line 207, in _get_session self._session_cache = self.load() File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/contrib/sessions/backends/db.py", line 35, in load expire_date__gt=timezone.now() File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 374, in get num = len(clone) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 232, in __len__ self._fetch_all() File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 1121, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 53, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql raise original_exception DatabaseError: ORA-00942: table or view does not exist Below are my databse detaisl i put in hue.ini [[database]] # Database engine is typically one of: # postgresql_psycopg2, mysql, sqlite3 or oracle. # # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes. # Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=(host):(port)/(service_name)". # Note for MariaDB use the 'mysql' engine. engine=oracle host= port=0 user=hue password=xxxxx # conn_max_age option to make database connection persistent value in seconds # https://docs.djangoproject.com/en/1.11/ref/databases/#persistent-connections ## conn_max_age=0 # Execute this script to produce the database password. This will be used when 'password' is not set. ## password_script=/path/script name= options={"threaded":true} # Database schema, to be used only when public schema is revoked in postgres ## schema=public and i see the below coming up when i open hue UI with error page . DATABASES
... View more
04-29-2020
10:15 AM
Hi @lwang , I have installed HUE through Ambari by following below document with the help of @stevenmatison https://github.com/steven-dfheinz/HDP2-Hue4-Service I was able to successfully install and bring up the services . I chose my metadata database as oracle as attached . But when i look into oracle database i don't see and metadata tables created . How do i check where Hue desktop is having its repository . Regards Bharad
... View more
04-29-2020
07:20 AM
Team , We are seeing below errors when querying hive from HUE . Error 1 :- [29/Apr/2020 07:07:22 -0700] middleware ERROR Error saving profile information Traceback (most recent call last): File "/usr/local/hue/apps/useradmin/src/useradmin/middleware.py", line 104, in process_request profile.save() File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/base.py", line 808, in save force_update=force_update, update_fields=update_fields) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/base.py", line 838, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/base.py", line 905, in _save_table forced_update) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/base.py", line 955, in _do_update return filtered._update(values) > 0 File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 667, in _update return query.get_compiler(self.db).execute_sql(CURSOR) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/sql/compiler.py", line 1204, in execute_sql cursor = super(SQLUpdateCompiler, self).execute_sql(result_type) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql raise original_exception OperationalError: database is locked ERROR 2:- [29/Apr/2020 07:06:51 -0700] exception ERROR Internal Server Error: /hue/accounts/login Traceback (most recent call last): File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/exception.py", line 41, in inner response = get_response(request) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_response response = self._get_response(request) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 187, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 185, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/utils/decorators.py", line 185, in inner return func(*args, **kwargs) File "/usr/local/hue/build/env/lib/python2.7/site-packages/django_axes-2.2.0-py2.7.egg/axes/decorators.py", line 271, in decorated_login response = func(request, *args, **kwargs) File "/usr/local/hue/desktop/core/src/desktop/auth/views.py", line 206, in dt_login 'user': request.user File "/usr/local/hue/desktop/core/src/desktop/lib/django_util.py", line 241, in render **kwargs File "/usr/local/hue/desktop/core/src/desktop/lib/django_util.py", line 154, in _render_to_response return django_mako.render_to_response(template, *args, **kwargs) File "/usr/local/hue/desktop/core/src/desktop/lib/django_mako.py", line 127, in render_to_response return HttpResponse(render_to_string(template_name, data_dictionary), **kwargs) File "/usr/local/hue/desktop/core/src/desktop/lib/django_mako.py", line 116, in render_to_string_normal result = template.render(**data_dict) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/template.py", line 462, in render return runtime._render(self, self.callable_, args, data) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py", line 838, in _render **_kwargs_for_callable(callable_, data)) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py", line 873, in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py", line 899, in _exec_template callable_(context, *args, **kwargs) File "/tmp/tmpVKDTTV/desktop/login.mako.py", line 176, in render_body __M_writer(unicode( commonfooter(None, messages) )) File "/usr/local/hue/desktop/core/src/desktop/views.py", line 559, in commonfooter hue_settings = Settings.get_settings() File "/usr/local/hue/desktop/core/src/desktop/models.py", line 130, in get_settings settings, created = Settings.objects.get_or_create(id=1) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 464, in get_or_create return self.get(**lookup), False File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 374, in get num = len(clone) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 232, in __len__ self._fetch_all() File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 1121, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/query.py", line 53, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch) File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql raise original_exception TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block. Any help is appreciated . Regards Bharad
... View more
Labels:
- Labels:
-
Cloudera Hue