apt-get  installation doesn't seem to install any bitcoin package same thing via python package manager (pip ...)   it's probably a mistake in the dockerfile   anyway the docker image is old, and the github repo doesn't seem to exist any more   ... View more

cloudera CDP is based on a Cloudera Runtime Version + a Cloudera Manager version that is compatible (with that cloudera runtime)   https://docs.cloudera.com/cdpdc/7.0/release-guide/topics/cdpdc-release-notes-links.html   at time of writing: CDP DC 1.0 uses Cloudera runtime 7.0.3 and cloudera manager 7.0.3   the Cloudera Runtime Component Version is aimed to keep a set of consistent hadoop components versions that can work together. it will also make more easy to migrate from CDH/HDP if services/components versions are the same or close to the runtime components versions of CDP   if i'm not wrong, there is actually only one CDP DC (1.0) version with minor updates of CM and cloudera components runtime versions   https://docs.cloudera.com/cloudera-manager/7.0.3/release-notes/topics/cm-release-notes.html https://docs.cloudera.com/runtime/7.0.3/release-notes/topics/rt-runtime-component-versions.html       ... View more

in your nifi app log if you have "out of memory" errors it may indicate that you need to add more memory (or a possible memory leak)   increasing heap too much will cause long GC (especially with Old Generation Objects) you may need to use a more efficient GC if you use large heap size in bootstrap.conf uncomment line : #java.arg.13=-XX:+UseG1GC   as you know, some processors are memory intensive   queues between processors use memory, ... until nifi.queue.swap.threshold is reached (default 20000 -> if reached nifi will push to disk remaining DFs  ) -> too much swapping in queues will realy affect performances. (memory + disk)   there are many points that can help you to determine where the PB may be   for ex. (processor 1) --> queue --> (processor 2) - check queue occupation : if always full for some processors parallelize more in following processors (especially if processor a re not memory intensive, and for ex if the processor is cpu intenssive and look likes it's not ingestings quickly from queue adding more threads will help to relieve the queue ). also ading back-pressus/control-rate before.  in tha case just checking "in" and "out" of processor 2 will confirm that  (high IN and low OUT), increase nb of threads -> (threadsScheduling>Concurrent Tasks), sometimes tuning "Run Schedule" and "Run Duration" may also help depending of ingres DFs nature/size and processor type and how parallelization is impacted take a look at "Total queued data" in the status bar (under components bar on top of the UI)   keep in mind that processors may be I/O or CPU or memory intensive or all, and parallelizing more to reduce backlogged data may solve memory PBs but it will cause more threads overuse. adding more NIFI nodes will remain the ultimate solution to increase resources   to increase performances, you may also want to take a look at nifi.provenance.repository.index.threads nifi.provenance.repository.query.threads nifi.provenance.repository.index.shard.size using different filesystems (with good IOPS) for directories for provenance repository, content repository, flowfile repository and more          ... View more

@Mondi  just update dfs.cluster.administrators with the admin username you want in hdfs config (and restart hdfs,yarn,MR2,... services) ex: dfs.cluster.administrators = hdfs,ops   you can also use a HDFS administrators group (only one administrator group) using dfs.permissions.superusergroup ex : dfs.permissions.superusergroup = operations   to verify config has been updated, once services restarted hdfs getconf -confKey dfs.cluster.administrators or hdfs getconf -confKey dfs.permissions.superusergroup   ... View more