- September 10, 2021
- By Greg Bean
- 3 min read
AWS DataSync For Your Hybrid Data Transfer Needs
AWS DataSync simplifies hybrid cloud data transfers. Move data between NFS, SFTP, and AWS without managing protocols or writing custom code.
HYBRID INTEGRATION PROBLEMS
Introduction
In short
- Legacy file transfer protocols like NFS and SFTP are complex, unmaintained, and create security risks when connecting corporate systems to AWS.
- AWS DataSync replaces these protocols with secure, fast, and simple file transfers between S3, NFS, EFS, SMB, and FSx with scheduling and incremental sync support.
- DataSync replaced flaky NFS libraries and eliminated firewall holes, giving reliable, fast hybrid data workflows without custom code to maintain.
As engineers working with enterprise hybrid cloud architectures, we occasionally must work with legacy corporate IT systems. Oftentimes these integrations are not elegant or ideal - dealing with NFS or SFTP in application code is few people’s ideas of fun or productive work. The protocols used in legacy systems can be quite large and obtuse and the open-source implementations (where they exist) are typically single authors, single-purpose, unmaintained, and far from robust. The error messages can be cryptic and troubleshooting failed connections or transfers can lead to many sleepless nights and premature grey hairs.
Sometimes we even write bespoke code for something that should ideally be an infrastructure concern, increasing our tech debt and maintenance burdens. When working with something like NFS overall network security takes a hit. To connect to an internal corporate NFS share from an AWS network requires both sides to poke many holes in their firewalls across TCP and UDP. Most corporate security teams will not be incredibly pleased with doing this.
The problem is compounded for those of us who enjoy using more modern systems architectures like event-driven lambdas. Where would you mount an NFS share while running a lambda? How do you handle large file listings or exceptionally large file transfers without your Lambda hitting the execution timeout? What if you need to transfer many thousands of files quickly?
The complexity can quickly spiral out of control.
AWS DATASYNC
DataSync
Luckily, Amazon has come to our rescue! DataSync is a new offering that provides secure, robust, and simple file transfer solutions for integrating with legacy and modern systems alike. It provides the ability to transfer entire directory trees and their metadata between S3 buckets, NFS shares, EFS volumes, SMB shares, and Amazon FSx. It supports incremental file transfers, deleted file handling, and scheduling with cron-like syntax. It is also amazingly fast; the agents are multithreaded applications that are likely to be far more performant and robust than a quick-and-dirty solution worked out under a tight deadline.
Our team has recently implemented DataSync for a hybrid data processing workflow that previously was handled by flaky NFS libraries and the results are amazingly robust and performant. Our workflow consisted of lambdas that would run on a CloudWatch schedule, connect to an NFS server and copy files to an S3 bucket to be further processed by other Lambdas. It was written in python and used open-source implementation of NFS.
Sounds simple enough, right? Unfortunately, it required us to support a fork of this mostly unmaintained library with some changes that were needed for our use case, open many obscure ports in our on-premise corporate firewall, and was plagued with performance issues. Troubleshooting connection and transfer errors was a herculean effort, requiring the efforts of many teams and wire-level network traces.
RESULTS
Conclusion
With DataSync now on the scene, we were able to install a single agent in our on-premise network and replace our crusty old NFS library and attendant lambda code with a single, simple job that is far more robust and performant than we could’ve reasonably achieved. DataSync simply runs on a schedule and incrementally syncs an on-premise directory structure into an S3 bucket where our processing lambda picks it up. No obscure legacy protocols, no hard to diagnose issues, less code to support - what is not to love?
If you have a hybrid cloud architecture, check it out and see if it fits your needs. We have found that it works very well for our enterprise archival and data processing workflow needs.
KEEP READING
Related Resources
Article
Getting Started with AWS CDK
Start building infrastructure as code with AWS CDK best practices
Read the article
Article
A Brief Survey and Overview of AWS ETL Tools - Part 1
AWS has an incredible array of data-centric services. Some of these services have functionality that overlaps with one another and some services may take two…
Read the article
Case Study
Automating 30,000 Daily Records for a Regional Bank
An AWS Lambda and Python pipeline now runs the transfer daily with no manual work, delivered in just under four months.
Read the regional bank case study
