About cstanca

cstanca · ‎03-12-2018

@Jane Becker I believe that your views got corrupted. This could have happened due metadata corruption. You may want to check the recent period events that may have lead to this situation. To delete and recreate new instances of Ambari Views, go to "Manage Ambari". Let us know if that addressed your issue.

cstanca · ‎03-12-2018

For a multi-node production cluster, it is sufficient to update hosts files and restart agents in each of the nodes showing the wrong IP.

cstanca · ‎03-09-2018

@Andi Sonde Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

cstanca · ‎03-05-2018

@Gaurang Shah Shu’s proposed approach should work, however, it assumes no activity means zero records. Is that always true? You could take a similar approach using UpdateAttribute to store number of records, preset the attribute to be zero. Then use RouteOnAttribute based on a value condition on the attribute, which you can handle to go to your step to create a file no matter what.

cstanca · ‎03-03-2018

@Connor O'Neal Yes, you can. I assume that you are taking this approach for a development environment. For production, you would need to implement HA at least for MySQL and Schema Registry, assuming they are core components for your applications. Check this: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_planning-your-deployment/content/ch_production-cluster-guidelines.html

cstanca · ‎03-03-2018

@Amira khalifa There is no standard processor capable to address your requirement. You would have to build a custom processor to generate the DDL or use ExecuteSQL or ExecuteScript processor. Anyhow, keep in mind that avro data types are not an exact match with Postgresql data types to not mention that your avro may be hierarchical. For that situation I suggest you to follow the steps: ConvertAvroToJson -> FlattenJson -> ConvertJsonToSQL -> ExecuteSQL. This is just one approach, you can do it in several ways including using Record processors.

cstanca · ‎03-02-2018

@Jeremy Hansen Are you talking about HDP 2.6.4.0? There is no HDF 2.6.4.0. Assuming that you had an existent HDP 2.6.4.0 cluster and you wish to use the same instance of Ambari to install HDF 3.1.1.0 services, you need to install first the management pack for HDF 3.1.1.0. Here are the steps in the documentation: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-on-hdp/content/upgrading_ambari.html Validate that you followed the steps appropriately. Be aware that you will be able to install only NiFi and NiFi Registry. All other services cannot be installed with Ambari 2.6.1.3 and current HDF 3.1.1.1. You would have to wait until HDF 3.2 to be able to add all other services from HDF 3.2 stack using the same Ambari instance for HDP and HDF. If this response helped, please vote and accept the response.

cstanca · ‎03-02-2018

@Jane Becker True. The connector is not currently bundled or supported. I installed it manually and my preliminary tests were successful when using it with Spark, but I did not anything complicated or at scale. I checked recently with Engineering and there is a good chance that it will be supported in the second part of 2018. As this connector gets more attention and importance from the users community, its priority will increase and there will be a better chance that it will be supported sooner. As you may know, this connector does not seem supported even by Google.

cstanca · ‎02-05-2018

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. I find myself quite often puzzled when required to provide differences between releases and just reading the release notes history at https://cwiki.apache.org/confluence/display/NIFI/Release+Notes and looking at the latest list of NiFi processors is not trivial to determine which new processors were added. I put together matrix which I hope will help developers to take advantage of new processor to improve old and develop new flows. In a nutshell, main functionality added is around: AzureEventHub Kafka 0.11 and 1.0 processors Record processors RethinkDB Flatten Json Execute Spark Interactive Execute Groovy Script My favorite improvements are around record processors, flattening JSON and executing Spark interactively. The following is a table of the matrix, arranged alphabetically from A-D: See here for the Matrix Table from E-J See here for the Matrix Tabke from K-Z For NiFi 1.5 NiFi 1.4 NiFi 1.3 NiFi 1.2 AttributeRollingWindow AttributeRollingWindow AttributeRollingWindow AttributeRollingWindow AttributesToJSON AttributesToJSON AttributesToJSON AttributesToJSON Base64EncodeContent Base64EncodeContent Base64EncodeContent Base64EncodeContent CaptureChangeMySQL CaptureChangeMySQL CaptureChangeMySQL CaptureChangeMySQL CompareFuzzyHash CompareFuzzyHash CompareFuzzyHash CompareFuzzyHash CompressContent CompressContent CompressContent CompressContent ConnectWebSocket ConnectWebSocket ConnectWebSocket ConnectWebSocket ConsumeAMQP ConsumeAMQP ConsumeAMQP ConsumeAMQP ConsumeAzureEventHub ConsumeEWS ConsumeEWS ConsumeEWS ConsumeEWS ConsumeIMAP ConsumeIMAP ConsumeIMAP ConsumeIMAP ConsumeJMS ConsumeJMS ConsumeJMS ConsumeJMS ConsumeKafka ConsumeKafka ConsumeKafka ConsumeKafka ConsumeKafka_0_10 ConsumeKafka_0_10 ConsumeKafka_0_10 ConsumeKafka_0_10 ConsumeKafka_0_11 ConsumeKafka_0_11 ConsumeKafkaRecord_0_10 ConsumeKafkaRecord_0_10 ConsumeKafkaRecord_0_10 ConsumeKafkaRecord_0_10 ConsumeKafkaRecord_0_11 ConsumeKafkaRecord_0_11 ConsumeKafka_1_0 ConsumeKafkaRecord_1_0 ConsumeMQTT ConsumeMQTT ConsumeMQTT ConsumeMQTT ConsumePOP3 ConsumePOP3 ConsumePOP3 ConsumePOP3 ConsumeWindowsEventLog ConsumeWindowsEventLog ConsumeWindowsEventLog ConsumeWindowsEventLog ControlRate ControlRate ControlRate ControlRate ConvertAvroSchema ConvertAvroSchema ConvertAvroSchema ConvertAvroSchema ConvertAvroToJSON ConvertAvroToJSON ConvertAvroToJSON ConvertAvroToJSON ConvertAvroToORC ConvertAvroToORC ConvertAvroToORC ConvertAvroToORC ConvertCharacterSet ConvertCharacterSet ConvertCharacterSet ConvertCharacterSet ConvertCSVToAvro ConvertCSVToAvro ConvertCSVToAvro ConvertCSVToAvro ConvertExcelToCSVProcessor ConvertExcelToCSVProcessor ConvertExcelToCSVProcessor ConvertExcelToCSVProcessor ConvertJSONToAvro ConvertJSONToAvro ConvertJSONToAvro ConvertJSONToAvro ConvertJSONToSQL ConvertJSONToSQL ConvertJSONToSQL ConvertJSONToSQL ConvertRecord ConvertRecord ConvertRecord ConvertRecord CreateHadoopSequenceFile CreateHadoopSequenceFile CreateHadoopSequenceFile CreateHadoopSequenceFile CountText DebugFlow DebugFlow DebugFlow DebugFlow DeleteDynamoDB DeleteDynamoDB DeleteDynamoDB DeleteDynamoDB DeleteGCSObject DeleteGCSObject DeleteGCSObject DeleteGCSObject DeleteHDFS DeleteHDFS DeleteHDFS DeleteHDFS DeleteElasticsearch5 DeleteElasticsearch5 DeleteRethinkDB DeleteRethinkDB DeleteS3Object DeleteS3Object DeleteS3Object DeleteS3Object DeleteMongo DeleteSQS DeleteSQS DeleteSQS DeleteSQS DetectDuplicate DetectDuplicate DetectDuplicate DetectDuplicate DistributeLoad DistributeLoad DistributeLoad DistributeLoad DuplicateFlowFile DuplicateFlowFile DuplicateFlowFile DuplicateFlowFile

cstanca · ‎10-12-2017

@Robert Krimper Could you edit the question and post the query formatted. There are a lot of missing spaces and it is hard to figure out the actual query.

Online	Offline
Last Visited	‎03-22-2019 03:12 AM

Member Since	‎03-16-2016 04:06 PM
Last Visited	‎03-22-2019 03:12 AM
Posts	707
Kudos received	1728

Cloudera Community

Re: 5th attempt at getting an answer to this quest...

Re: Trying to reinstall Apache NiFi 1.5 on HDF 3.1

Re: Is it mandatory that we should have exact moun...

Re: Alternate to smartsense

Re: Tracking of Hive tables metadata changes in re...

Re: Ambari Files View - Service hdfs check failed

Re: How to Change Host IP in Ambari View

Re: Forcefully Exit NameNode Safe Mode

Re: Nifi QueryDatabaseTable processor returns 0 if...

Re: HDF 3.1.1 Service Distribution

Re: create SQL table from AVRO Schema

Re: Problems upgrading HDF from 2.6.4.0 -> 3.1.1.0

Re: HDP Spark Connector for Google Cloud

Apache NiFi Journey from HDF 3.0 to 3.1 - Part 1

Re: HDP 2.6.2: Hive Throws ConcurrentModificationE...