Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do i stream data from a file to hdfs using spring XD

How do i stream data from a file to hdfs using spring XD

Explorer

I am trying to create a pipeline using spring XD. Taking file(from local) as source and sinking the data to hdfs-dataset in avro format.

for this I have create a stream as below.

xd:>stream create --name mydataset --definition "file --dir=/home/hdfs/spring-xd-1.3.2.RELEASE/tmp/xd/input/filetest --outputType=test/plain | hdfs-dataset --fsUri=hdfs://ip-10-0-243-166.ec2.internal:8020" --deploy

I am getting the .avro files but when I genereate .avsc(schema) I got to know that .avro files ar not generated properly.

I am not sure whether above command is correct or not.

my source file has below data with comma, "", and pipe symbols.

"EMP-IND","EMP-CO","EMP-ID"

|Z|,|Z9|,|CFN45|

|Z|,|Z9|,|CFN54|

|Z|,|Z9|,|CFN57|

can anyone suggest what is wrong here...