Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Stuck on "PutElasticSearchHttpRecord" - "Record Reader" and schemas

Highlighted

Stuck on "PutElasticSearchHttpRecord" - "Record Reader" and schemas

New Contributor

103397-nifi-diagram.jpg

Hello, I'm brand new to NiFi and am exploring it for a few things (cloudtrail logs in S3 -> Elasticsearch, SFTP to external parties, S3 data synch etc). While it was insanely easy to set it up initially, I am now horribly stuck...

I can pull a file with a bunch of AWS Cloudtrail logs in JSON format (a sample shown below) off an S3 bucket, I can for fun write it to a local file, split the CloudTrail file into individual JSON records, parse out the "eventID" and rename the filename so that I can write the JSON records into separate files with the eventID as the name. I can SFTP the file no problem. Don't ask me for the reasons for my data flows, at the moment I'm trying to find out what it can do for us ...

But then I hit the wall...

No matter what I do, I cannot get a piece of JSON through "PutElasticsearchHttpProcessor" and I'm getting stuck on the "Record Reader". I have tried JsonPathReader vith different "Schema Access Strategy" settings (noth "Schema Name" and "Schema Text"), setting "schema.name" manually etc., but I don't really understand why I have to specify a schema to be able to push an already well formed JSON record through . Does anyone have any idea where I might be going wrong?

{"eventVersion":"1.05","userIdentity":{"type":"AssumedRole","principalId":"xxxx:GlueJobRunnerSession","arn":"arn:aws:sts::xxxxx:assumed-role/AWSGlueServiceRole-lamdba_aws_athena/GlueJobRunnerSession","accountId":"xxxxx","sessionContext":{"attributes":{"mfaAuthenticated":"false","creationDate":"2019-01-09T00:38:21Z"},"sessionIssuer":{"type":"Role","principalId":"xxxxx","arn":"arn:aws:iam::xxxxx:role/service-role/AWSGlueServiceRole-lamdba_aws_athena","accountId":"xxxxx","userName":"AWSGlueServiceRole-lamdba_aws_athena"}},"invokedBy":"glue.amazonaws.com"},"eventTime":"2019-01-09T00:38:21Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeNetworkInterfaces","awsRegion":"eu-west-2","sourceIPAddress":"glue.amazonaws.com","userAgent":"glue.amazonaws.com","requestParameters":{"networkInterfaceIdSet":{"items":[{"networkInterfaceId":"eni-0119663ebaa3c066c"}]},"filterSet":{}},"responseElements":null,"requestID":"94f24bc7-b1c1-48ce-952e-5f7a6f00f97b","eventID":"ea3af835-ab6c-4305-a5a7-d239b680c377","eventType":"AwsApiCall","recipientAccountId":"xxxxx"}