Member since
02-05-2018
25
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1058 | 08-17-2018 04:00 PM | |
721 | 07-31-2018 08:07 PM |
09-20-2018
02:37 PM
1 Kudo
It's totally possible with Nifi! There are a number of ways you could go about it, and it depends on if you think the dates will have the same format, or if they will vary. (ie: mm/dd/yyyy or dd-mm-yyyy) I found another useful conversation here that covers using the routeOnContent or routeOnText processors to evaluate a regex expression, checking if the value matches what you expect. I would recommend looking at the documentation on those processors, as well as testing out your regex expression on a site like Regexr. Hope that gets you started!
... View more
09-07-2018
03:18 PM
I'm not seeing Solr as an available service to add onto my HDP 3 cluster. I checked the Ambari upgrade documents and didn't see Solr as one of the services that was being removed. Anyone have news on where Solr is at? I read in this post that the Ambari Infra Solr is for internal use only, so that won't suit my needs.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Solr
09-06-2018
08:30 PM
I am also having a similar issue with hive import on HDP3. Has anyone been successful yet? My sqoop command successfully pulls the database to HDFS, but hangs on the zookeeper connect as well.
... View more
08-30-2018
04:41 PM
I'll have to start there, thanks.
... View more
08-29-2018
09:26 PM
Hey all, I am experimenting with the Sparkling water library to see if it plays nicely with Zeppelin. While looking through the spark application log to check that my cluster was being fully utilized, I noticed a new tab had been added for sparkling water. Unfortunately, I get a 500 error when trying to access it. Does anyone know what configurations I need to set to allow this new view in the Spark History Server UI? The tab is at http://cluster:8088/proxy/app-id/sparkling-water
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
08-22-2018
02:36 PM
Thanks for the Zeppelin articles Franklin! Could you elaborate on the Execution mode 'master' value, and what options there are/when to use each? I'm having trouble finding a good resource on them.
... View more
08-20-2018
06:21 PM
That solved my problem! I went to Hortonworks downloads and grabbed the correct versions of Nifi toolkit. I am now onto the next error: resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.1.0-35/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 1. Error: Could not find or load main class org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain
But I'll save that for another post once I've investigated some more, thanks!
... View more
08-20-2018
04:27 PM
Hey all, I am having issues getting Nifi and Nifi Registry to start so that they may be updated to 1.7 with the release of HDF 3.2. I'm not seeing many other topics covering this issue. I have checked that the file exists that is being referenced here, there is a directory for both Nifi 1.5 and 1.7 toolkit. I have tried moving the 1.7 directory out of the tmp folder in case that was creating confusion but that didn't work either. Any ideas? stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 280, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 128, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 98, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 200, in write_configurations
is_starting, params.toolkit_tmp_dir, params.stack_version_buildnum)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 342, in encrypt_sensitive_properties
encrypt_config_script = get_toolkit_script('encrypt-config.sh',toolkit_tmp_dir, stack_version_buildnum)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 61, in get_toolkit_script
raise Exception("Couldn't find nifi toolkit directory in " + scriptDir)
Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp
stdout:
2018-08-20 16:11:12,671 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,849 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,852 - Group['nifiregistry'] {}
2018-08-20 16:11:12,853 - Group['hadoop'] {}
2018-08-20 16:11:12,853 - Group['nifi'] {}
2018-08-20 16:11:12,854 - User['streamline'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['registry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,856 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,858 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,859 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2018-08-20 16:11:12,861 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2018-08-20 16:11:12,862 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-08-20 16:11:12,863 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-08-20 16:11:12,868 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-08-20 16:11:12,883 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-08-20 16:11:12,889 - Skipping Execute[('setenforce', '0')] due to not_if
2018-08-20 16:11:12,889 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2018-08-20 16:11:13,068 - The unlimited key JCE policy is required, and appears to have been installed.
2018-08-20 16:11:13,313 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:13,325 - Toolkit files dir is /var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/files
2018-08-20 16:11:13,325 - Toolkit tmp dir is /var/lib/ambari-agent/tmp
2018-08-20 16:11:13,376 - Directory['/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520'] {'group': 'nifiregistry', 'cd_access': 'a', 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,378 - Changing owner for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,378 - Changing group for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,388 - Directory['/var/run/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,388 - Directory['/var/lib/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,389 - Directory['/var/lib/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,390 - Directory['/var/lib/nifi-registry/database'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/bin'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,392 - Directory['/usr/hdf/current/nifi-registry/lib'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,394 - Directory['/usr/hdf/current/nifi-registry/docs'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,396 - PropertiesFile['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'mode': 0600, 'properties': ...}
2018-08-20 16:11:13,402 - Generating properties file: /usr/hdf/current/nifi-registry/conf/nifi-registry.properties
2018-08-20 16:11:13,402 - File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600, 'encoding': 'UTF-8'}
2018-08-20 16:11:13,422 - Writing File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] because contents don't match
2018-08-20 16:11:13,425 - File['/usr/hdf/current/nifi-registry/conf/bootstrap.conf'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,428 - File['/usr/hdf/current/nifi-registry/conf/logback.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,436 - File['/usr/hdf/current/nifi-registry/conf/authorizers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,440 - File['/usr/hdf/current/nifi-registry/conf/identity-providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,442 - File['/usr/hdf/current/nifi-registry/conf/providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,444 - File['/usr/hdf/current/nifi-registry/bin/nifi-registry-env.sh'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,445 - Encrypting NiFi Registry sensitive configuration properties
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache NiFi
08-17-2018
04:00 PM
As I was perusing the updated documentation for HDF 3.2 today I found what the documentation was referencing. The parent to the page I was stuck on has an image with steps written out. Here it is.
... View more
08-10-2018
07:25 PM
Hmm I am not familiar with this set of services. Does it have any API endpoints? Are you trying to collect metric data on the services performance to hadoop? Or do you mean to send the output of a virtual device (temperature from a virtual thermometer) to hadoop?
... View more
08-10-2018
04:07 PM
I haven't set anything up using the druid docker image before. Are you using an HDF cluster? It sounds like you already have a working Kafka topic if you have used it to put data to MySQL. Your configuration problem with druid may lay with "druid.discovery.curator.path" : "/druid/discovery", "druid.selectors.indexing.serviceName" : "druid/overlord", I am not certain that these values would be the case with the docker image.
... View more
08-10-2018
01:48 PM
It may be in your best interest to close this issue and open up a new one with a more focused question. That will make it easier for others to find it when searching for answers.
... View more
08-10-2018
01:47 PM
I would need more details. Where exactly are you stuck or having problems? Are you asking how to configure the DruidTranquilityController processor?
... View more
08-10-2018
01:12 PM
For your error with the JsonPathReader, you are not defining any paths for it to read. Here is the documentation. The relevant section says "User-defined properties define the fields that should be extracted from the JSON in order to form the fields of a Record." So if my JSON record was structured {
"data": 1,
"name": "Bob"
} and I wanted my flowfile to contain the "name" field, I would add a custom attribute to the processor. It would have property: name, value: $.name If you are new to JSON Path, I would suggest trying out your expressions here. That should set you on the right track!
... View more
08-10-2018
12:51 PM
Are you referring to AWS Device Farm? It has an API which can be viewed here. You could use Nifi's getHTTP processor to call an endpoint on that API and create flowfiles from it.
... View more
08-01-2018
04:32 PM
No problem on the tags, I just wanted to ensure you got some traffic on this! I had never looked into JOLT before now, this is some pretty cool stuff! I'll look into it and maybe I'll be able to come up with something for you. Sounds like you have got your minimum viable product for now though:P
... View more
07-31-2018
09:34 PM
I assume this is being done in Nifi? (I would add the Apache-Nifi tag if so) If you know the structure of the JSON object that you will be receiving, you can use an evaluateJSONPath processor to pull the JSON key's value into a flowfile attribute. Then an updateAttribute processor where you use Nifi expression language's 'equals' to determine if the value is as you expected. https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#equals The output of updateAttribute will be true or false and you can pass that to your routeOnAttribute processor. example.jpg I hope this helps or gives you some ideas!
... View more
07-31-2018
08:07 PM
If you check the documentation for the Kafka Source, you will see that the 'kafka topic' field needs to pull the schema from the Schema Registry. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_streaming-analytics-manager-user-guide/content/source-configurations.html Meanwhile, the Schema Registry documentation notes that AVRO is the only supported type of schema at this time. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_schema-registry-user-guide/content/create-schema-metadata.html So it's possible in the future more formats will be allowed, but not at the moment. Good luck!
... View more
07-27-2018
01:45 PM
@Sarah Olson @Dominika Bialek Any luck on this?
... View more
07-12-2018
05:57 PM
1 Kudo
Was digging through the HDF 3.1.2 documentation and couldn't find the referenced steps on this page: It says, "In steps 1-4 with HDP, we were able to build a logistical regression model. The model was then exported into PMML." I have looked through some of the recent HDP documentation and have struggled to find anything on this topic. I'm trying to get a full understanding of this demo and would like to see this process. If anyone knows what guide this documentation is referring to I'd be much obliged!
... View more
Labels:
- Labels:
-
Cloudera DataFlow (CDF)