Member since
02-05-2018
25
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1792 | 08-17-2018 04:00 PM | |
1381 | 07-31-2018 08:07 PM |
09-20-2018
02:37 PM
1 Kudo
It's totally possible with Nifi! There are a number of ways you could go about it, and it depends on if you think the dates will have the same format, or if they will vary. (ie: mm/dd/yyyy or dd-mm-yyyy) I found another useful conversation here that covers using the routeOnContent or routeOnText processors to evaluate a regex expression, checking if the value matches what you expect. I would recommend looking at the documentation on those processors, as well as testing out your regex expression on a site like Regexr. Hope that gets you started!
... View more
09-07-2018
03:18 PM
I'm not seeing Solr as an available service to add onto my HDP 3 cluster. I checked the Ambari upgrade documents and didn't see Solr as one of the services that was being removed. Anyone have news on where Solr is at? I read in this post that the Ambari Infra Solr is for internal use only, so that won't suit my needs.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Solr
09-06-2018
08:30 PM
I am also having a similar issue with hive import on HDP3. Has anyone been successful yet? My sqoop command successfully pulls the database to HDFS, but hangs on the zookeeper connect as well.
... View more
08-22-2018
02:36 PM
Thanks for the Zeppelin articles Franklin! Could you elaborate on the Execution mode 'master' value, and what options there are/when to use each? I'm having trouble finding a good resource on them.
... View more
08-20-2018
06:21 PM
That solved my problem! I went to Hortonworks downloads and grabbed the correct versions of Nifi toolkit. I am now onto the next error: resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.1.0-35/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 1. Error: Could not find or load main class org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain
But I'll save that for another post once I've investigated some more, thanks!
... View more
08-20-2018
04:27 PM
Hey all, I am having issues getting Nifi and Nifi Registry to start so that they may be updated to 1.7 with the release of HDF 3.2. I'm not seeing many other topics covering this issue. I have checked that the file exists that is being referenced here, there is a directory for both Nifi 1.5 and 1.7 toolkit. I have tried moving the 1.7 directory out of the tmp folder in case that was creating confusion but that didn't work either. Any ideas? stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 280, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 128, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 98, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 200, in write_configurations
is_starting, params.toolkit_tmp_dir, params.stack_version_buildnum)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 342, in encrypt_sensitive_properties
encrypt_config_script = get_toolkit_script('encrypt-config.sh',toolkit_tmp_dir, stack_version_buildnum)
File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 61, in get_toolkit_script
raise Exception("Couldn't find nifi toolkit directory in " + scriptDir)
Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp
stdout:
2018-08-20 16:11:12,671 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,849 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,852 - Group['nifiregistry'] {}
2018-08-20 16:11:12,853 - Group['hadoop'] {}
2018-08-20 16:11:12,853 - Group['nifi'] {}
2018-08-20 16:11:12,854 - User['streamline'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['registry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,856 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,858 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,859 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2018-08-20 16:11:12,861 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2018-08-20 16:11:12,862 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-08-20 16:11:12,863 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-08-20 16:11:12,868 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-08-20 16:11:12,883 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-08-20 16:11:12,889 - Skipping Execute[('setenforce', '0')] due to not_if
2018-08-20 16:11:12,889 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2018-08-20 16:11:13,068 - The unlimited key JCE policy is required, and appears to have been installed.
2018-08-20 16:11:13,313 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:13,325 - Toolkit files dir is /var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/files
2018-08-20 16:11:13,325 - Toolkit tmp dir is /var/lib/ambari-agent/tmp
2018-08-20 16:11:13,376 - Directory['/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520'] {'group': 'nifiregistry', 'cd_access': 'a', 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,378 - Changing owner for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,378 - Changing group for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,388 - Directory['/var/run/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,388 - Directory['/var/lib/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,389 - Directory['/var/lib/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,390 - Directory['/var/lib/nifi-registry/database'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/bin'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,392 - Directory['/usr/hdf/current/nifi-registry/lib'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,394 - Directory['/usr/hdf/current/nifi-registry/docs'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,396 - PropertiesFile['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'mode': 0600, 'properties': ...}
2018-08-20 16:11:13,402 - Generating properties file: /usr/hdf/current/nifi-registry/conf/nifi-registry.properties
2018-08-20 16:11:13,402 - File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600, 'encoding': 'UTF-8'}
2018-08-20 16:11:13,422 - Writing File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] because contents don't match
2018-08-20 16:11:13,425 - File['/usr/hdf/current/nifi-registry/conf/bootstrap.conf'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,428 - File['/usr/hdf/current/nifi-registry/conf/logback.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,436 - File['/usr/hdf/current/nifi-registry/conf/authorizers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,440 - File['/usr/hdf/current/nifi-registry/conf/identity-providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,442 - File['/usr/hdf/current/nifi-registry/conf/providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,444 - File['/usr/hdf/current/nifi-registry/bin/nifi-registry-env.sh'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,445 - Encrypting NiFi Registry sensitive configuration properties
Command failed after 1 tries
... View more
Labels:
- Labels:
-
Apache NiFi
08-17-2018
04:00 PM
As I was perusing the updated documentation for HDF 3.2 today I found what the documentation was referencing. The parent to the page I was stuck on has an image with steps written out. Here it is.
... View more
08-01-2018
04:32 PM
No problem on the tags, I just wanted to ensure you got some traffic on this! I had never looked into JOLT before now, this is some pretty cool stuff! I'll look into it and maybe I'll be able to come up with something for you. Sounds like you have got your minimum viable product for now though:P
... View more
07-31-2018
09:34 PM
I assume this is being done in Nifi? (I would add the Apache-Nifi tag if so) If you know the structure of the JSON object that you will be receiving, you can use an evaluateJSONPath processor to pull the JSON key's value into a flowfile attribute. Then an updateAttribute processor where you use Nifi expression language's 'equals' to determine if the value is as you expected. https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#equals The output of updateAttribute will be true or false and you can pass that to your routeOnAttribute processor. example.jpg I hope this helps or gives you some ideas!
... View more