Load data from remote windows server to HDFS

answered 2019-05-29 09:52:12 -0600

metadaddy

5464 ●26 ●41 ●82 https://about.me/patpa...

updated 2019-05-31 19:42:03 -0600

You have a few choices:

Mount the Windows machine's drive via SMB/NFS etc and access it via the Directory origin
Run an SFTP/FTP server on the Windows machine and use the SFTP/FTP Client origin.
Deploy StreamSets Data Collector Edge on the Windows machine and use its Directory or File Tail origin. Set the data format in the origin appropriately to parse the data there, then send the data to a receiving pipeline on Data Collector via a supported destination. Note - you could use Kinesis or Kafka as the transport from Edge to Data Collector as well as those listed in the docs.

edit flag offensive delete link

Comments

Currently we are able to read file using SFTP / FTP Client with NFS mount to load to HDFS. Would like to know- 1. Can we read a windows file directly (Streamsets Data Collector Edge uses HTTP Client as destination and no whole file option) 2. Any other alternative to SFTP

KeerthiS ( 2019-05-29 10:49:05 -0600 )edit

What is the data format?

metadaddy ( 2019-05-29 11:04:32 -0600 )edit

the file formats could any of the following - gz, zip, bz2 and some other file formats

KeerthiS ( 2019-05-29 14:21:56 -0600 )edit

we are looking we could read a Windows path (\\ServerName\Path) without using NFS or SMB. If there is no possibility, which one is best recommended and what are other options which you would suggest similar to NFS / SMB

KeerthiS ( 2019-05-29 14:23:13 -0600 )edit

For SFTP , what needs to be installed and SFTP client options and best recommended ?

KeerthiS ( 2019-05-29 14:51:47 -0600 )edit

see more comments

Load data from remote windows server to HDFS

1 Answer

Comments

Question Tools

Stats

Related questions

Load data from remote windows server to HDFS edit

1 Answer

Comments

Question Tools

Stats

Related questions

Load data from remote windows server to HDFS