Member since 
    
	
		
		
		05-07-2020
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                32
            
            
                Posts
            
        
                2
            
            
                Kudos Received
            
        
                1
            
            
                Solution
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 5964 | 03-15-2022 08:41 AM | 
			
    
	
		
		
		09-13-2022
	
		
		12:34 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks for the reply @nramanaiah. I seem to be unable to find an option 'metastore.msck.repair.enable.partition.retention' does it need to be added as a custom option and if so under which drop down? Thanks 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		09-08-2022
	
		
		01:44 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I am trying to set partition retention times on existing Hive manged tables using the following:      ALTER TABLE <table name> SET TBLPROPERTIES ('discover.partitions'='true');  ALTER TABLE <table name> SET TBLPROPERTIES ('partition.retention.period'='1d');     as stated on this page below, however I am still able to search partitions older than a day so it appears to not be working? It does mention on the page that this is for 'external' tables, can anyone let me know if this an 'age off' retention period is possible on managed tables? am I missing any commands etc?   https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive-set-partition-retention.html     Thanks in advance 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
 - 
						
							
		
			Apache Hive
 - 
						
							
		
			Apache NiFi
 
			
    
	
		
		
		09-02-2022
	
		
		03:04 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 @araujo thank you! I had gotten around the issue by using a replaceText loop to search for capitals within' the keys 1 at a time and prepend underscores, followed by a jolt to lowercase all keys.      You're script returns the same result, very impressed. Thanks again.    
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-22-2022
	
		
		08:16 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I have stolen to spec below from here: https://stackoverflow.com/questions/54696540/jolt-transformation-lowercase-all-keys which works in lowercasing all keys, however I ideally need the hyphens between the words.     [    {      // unwrap the keys and values into literal      // "key" : "A", "value" : "b"      "operation": "shift",      "spec": {        "*": {          "$": "&1.key",          "@": "&1.value"        }      }    },    {      "operation": "modify-overwrite-beta",      "spec": {        "*": {          // Now that the origional key          //  is on the "right hand side"          //  lowercase it          "key": "=toLower"        }      }    },    {      // pivot back, the now lowercased keys      "operation": "shift",      "spec": {        "*": {          "value": "@(1,key)"        }      }    }  ] 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-22-2022
	
		
		07:36 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this?     Example input:  {"aRandomFieldName":"Random Value"}     Required Output:  {"a_random_field_name":"Random Value"}     Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great.     Thanks in advance  Andy. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
 - 
						
							
		
			Apache NiFi
 
			
    
	
		
		
		03-15-2022
	
		
		08:41 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hello @Azhar_Shaikh      Thanks for the reply, as it turns out it wasn't a service account problem. We found that the ListS3's output included a 'key' field, and this is what was required in The FetchS3Object processor for 'Object Key'.      So the fix I applied was to split the json into individual records (SplitJson), then pull the keys out as attributes (EvaluateJsonPath) then input ${key} into the FetchS3 processor.. worked a treat.        
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		03-15-2022
	
		
		01:34 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi @VidyaSargur      I do not see an option for 'Accept as Solution' below the post, I assume because I didn't ask the original question.     Regards     
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		03-11-2022
	
		
		01:31 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hello     I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output:         I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong?      FetchS3Object Config:         Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error.     Cheers  Andy 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
 - 
						
							
		
			Apache NiFi
 
			
    
	
		
		
		03-10-2022
	
		
		08:46 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @araujo Thanks very useful indeed! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		03-10-2022
	
		
		01:34 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @araujo      Thanks André     What if I wanted to turn the key value pairs separated by '=' in json content as in the original question? for instance the following syslog:         converted into:  [ {  "sig" : "0",  "arch" : "c000003e",  "syscall" : "87"  }]     I'm aware this can be done using regex to create attributes and then attributeToJson but some of my logs have hundreds of key value pairs so that's not an option, there must be a way to convert it using record processing? i.e convertRecord 
						
					
					... View more