Hi, I'm using HDF 3.0.1 and have configured Schema Registry with an AVRO schema to use with Kafka. Reading through the docs it makes sense how I can publish and consumer using NiFi and a Java client but I have a bit of a problem. Some of the systems we use are .NET and I can't find KafkaAvroDeserializer or Serializer for .NET. The Confluent Schema Registry seems to provide a .NET sdk but I believe I can't use that Schema registry or Serializers with Hortonworks. Any recommendations how I can get our .NET clients to publish to a Kafka topic and taking advantage of the HWX schema registry in HDF? Kind Regards, Geouffrey ... View more

Hi, I tried upgrading my HDF from 2.1.4 to 3.0.1.1. The installation failed so I thought I would roll back to the woking state before I try again. But now I am unable to roll back to a working state on my environment and Ambari Service will not start. I performed these steps to roll back: https://community.hortonworks.com/content/supportkb/48757/how-to-perform-a-roll-back-on-ambari-server-after.html The error I'm seeing is similar to what is described here: https://issues.apache.org/jira/browse/AMBARI-18467 Is it possible to roll back major version and, if it is, is there a way to get past this issue without installing my entire stack? Regards, Geouffrey ambari-server.tar.gz ... View more

Hi, I'm running HDF 2.1.4 and have the following components working: 2 Nodes NiFi Cluster (SSL Enabled using NiFi CA) Ranger (integrated with NiFi and I have Initial Admin Identity working from browser) I am now trying to configure AD on my setup. I managed to get Ranger Usync to work and I can see my AD users in Ranger but when I make the changes for NiFi the servicde won't start. I made the following changes in Ambari under Advanced nifi-properties: nifi.security.user.login.identity.provider ldap-provider nifi.security.identity.mapping.pattern.dn ^CN=(.*?), OU=(.*?)$ nifi.security.identity.mapping.value.dn $1@$2 And this is what my Template for login-identity-providers.xml looks like <loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=xxx,OU=Administrator & Service Accounts,OU=Administration,OU=NOC,DC=xxx,DC=xxx</property> <property name="Manager Password">xxx_password</property> <property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property> <property name="TLS - Keystore Password">keystore_password</property> <property name="TLS - Keystore Type">jks</property> <property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property> <property name="TLS - Truststore Password">truststore_password</property> <property name="TLS - Truststore Type">jks</property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ad_controller:389</property> <property name="User Search Base">OU=Administration,OU=NOC,DC=xxx,DC=xxx</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> </loginIdentityProviders> After saving these changes and restarting NiFi I get the following errors: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 360, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 164, in start self.configure(env, is_starting = True) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 145, in configure params.nifi_flow_config_dir, params.nifi_sensitive_props_key, is_starting) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 286, in encrypt_sensitive_properties if nifi_toolkit_util.contains_providers(nifi_config_dir+'/login-identity-providers.xml'): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 257, in contains_providers dom = xml.dom.minidom.parseString(content) File "/usr/lib64/python2.7/xml/dom/minidom.py", line 1931, in parseString return expatbuilder.parseString(string) File "/usr/lib64/python2.7/xml/dom/expatbuilder.py", line 940, in parseString return builder.parseString(string) File "/usr/lib64/python2.7/xml/dom/expatbuilder.py", line 223, in parseString parser.Parse(string, True) xml.parsers.expat.ExpatError: not well-formed (invalid token): line 7, column 74 The login provider configuration looks ok to me but I've never configured NiFi with AD so I could well be missing something. Any ideas? Regards, Geouffrey ... View more

Hi, I'm new here and trying to complete the cloudera live guide. I'm trying to follow the steps required to load the data from access.log.2 but the task doens't complete. It's not giving me any errors and I'm not sure where to look for log files. The code I'm using looks like this: CREATE EXTERNAL TABLE intermediate_access_logs (     ip STRING,     date STRING,     method STRING,     url STRING,     http_version STRING,     code1 STRING,     code2 STRING,     dash STRING,     user_agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES (     'input.regex' = '([^ ]*) - - \\[([^\\]]*)\\] "([^\ ]*) ([^\ ]*) ([^\ ]*)" (\\d*) (\\d*) "([^"]*)" "([^"]*)"',     'output.format.string' = "%1$$s %2$$s %3$$s %4$$s %5$$s %6$$s %7$$s %8$$s %9$$s") LOCATION '/user/hive/warehouse/original_access_logs'; CREATE EXTERNAL TABLE tokenized_access_logs (     ip STRING,     date STRING,     method STRING,     url STRING,     http_version STRING,     code1 STRING,     code2 STRING,     dash STRING,     user_agent STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/hive/warehouse/tokenized_access_logs'; ADD JAR /usr/lib/hive/lib/hive-contrib.jar; INSERT OVERWRITE TABLE tokenized_access_logs SELECT * FROM intermediate_access_logs; I'm sure I'm missing something silly here. Kind Regards, Gee ... View more