Posts: 14
Registered: ‎10-21-2018

How do i stream data from a file to hdfs using spring XD

I am trying to create a pipeline using spring XD. Taking file(from local) as source and sinking the data to hdfs-dataset in avro format.

for this I have create a stream as below.

xd:>stream create --name mydataset --definition "file --dir=/home/hdfs/spring-xd-1.3.2.RELEASE/tmp/xd/input/filetest --outputType=test/plain | hdfs-dataset --fsUri=hdfs://ip-10-0-243-166.ec2.internal:8020" --deploy

I am getting the .avro files but when I genereate .avsc(schema) I got to know that .avro files ar not generated properly.

I am not sure whether above command is correct or not.

my source file has below data with comma, "", and pipe symbols.





can anyone suggest what is wrong here...