Introducing FTP and FTPS support for AWS Transfer

April 24, 2020

Introducing FTP and FTPS support for AWS Transfer

Amazon has extended the feature set of AWS Transfer for SFTP to also support FTP and FTPS. In this post we will explore why this is important, how to use it, but also some of the complexities introduced.

Let’s start at the basics: what is AWS Transfer? Introduced at re:Invent 2018, it’s a service that allows you to use the SSH File Transfer Protocol (SFTP) to transfer files to and from S3. Today, support for the FTPS and FTP protocols have been added and the service has been renamed to the AWS Transfer Family.

The main reasons for AWS Transfer’s existence are to simplify file interactions with S3 and to enable older applications and systems to interact with S3.

An example: let’s say you have an application built in the 2000’s running your core business. For business reasons you can’t migrate away, but you do want to secure the application with a modern and cost efficient backup solution. However, the application only supports backups over SFTP. With AWS Transfer for SFTP (before support for FTP and FTPS was added), you get a secure SFTP endpoint to which you can write your files. The actual file storage is done on S3, with benefits like low cost, no maintenance, high availability, high durability and different storage classes, including Glacier.

A brief overview of SFTP, FTP and FTPS

The SFTP protocol has been around since the late nineties. As the name (SSH File Transfer Protocol) implies, it’s a file protocol on top of SSH. That means it’s very secure and compatible with the many systems that support SSH. It also means that authentication and authorization is based on existing SSH technologies. It is important to note that SFTP is not based on FTP. Instead it’s its own protocol.

FTP, on the other hand, is a lot older. It actually predates SSH by more than 20 years. Because it’s been around so long, almost every computing system supports FTP. Many applications written in the last century include some interaction with or dependency on FTP. The base FTP protocol does not support encryption.

FTPS (also known as FTP-SSL and FTP Secure) is a direct derivative of FTP and supports the same commands and interactions as FTP, as opposed to SFTP. There are two ways FTPS can apply security: implicit and explicit. Implicit FTPS initiates a secured connection from the start and is not compatible with non-FTPS servers. Explicit FTPS initiates a classic FTP connection and upgrades the security if both sides support it.

Introducing FTP and FTPS support for AWS Transfer

When Amazon introduced AWS Transfer for SFTP a new and easy way to interact with S3 became available. With today’s release, even more (and older) systems and applications can integrate with Simple Storage Service using the FTP and FTPS protocols.

This benefits many use cases, like:

  • Backups and archival
  • Data migration
  • Application integration
  • Data analytics

Some examples:

  1. If you’re running an older application and would like to migrate to a more modern solution, you can use AWS Transfer to upload the files to S3, automatically process them with AWS Glue and load them into an RDS database.
  2. If other applications need to access the data of your classic application, store its files in S3 and let the other applications access the data through modern channels like API Gateway or Athena.
  3. If you need to store data for longer periods of time for compliance reasons, upload them to S3 and automatically transition them to Glacier or Glacier Deep Archive.
  4. If you need analytics on a classic application, upload the data to S3 and use Glue to load it into Redshift. From there, use Quicksight, Tableau or QlikSense to visualize and analyze your data.

What are the differences in AWS Transfer for SFTP, FTP and FTPS?

Let’s immediately jump to the most significant difference. AWS Transfer for FTP does not support public endpoints. In other words, SFTP and FTPS servers can be reached over the internet, but FTP requires a Site-to-Site VPN, Direct Connect or Client VPN connection for external clients. Local clients like EC2 instances can connect to the FTP server directly.

Interesting to note is that the original Transfer for SFTP would provide public IP addresses pointing directly to the service, but this is not supported for FTP and FTPS. Instead, elastic network interfaces will be deployed in your VPC. For FTPS these ENIs can be assigned Elastic IP addresses to make the service publicly available, but this is not supported for FTP. Most likely, Amazon has chosen not to make FTP publicly available because FTP can’t be secured in any meaningful way.

The new VPC hosted option is now also available for AWS Transfer for SFTP.

Another significant difference is the authentication mechanism. AWS Transfer for SFTP supports ‘service managed’ users. With service managed users, you can define and manage the users in the AWS Transfer console.

users

Service managed users are not supported for AWS Transfer for FTP and FTPS. A ‘custom identity provider type’ is required instead. We will go into the details of the custom identity provider type below.

The third major difference is that FTPS requires you to specify a server certificate. Much like HTTPS, this ACM certificate identifies the server and verifies its hostname.

All differences are summarized in the table below:

Protocol Data encrypted in transit Can be public User management
SFTP Yes Yes, both directly and through VPC Service managed or custom
FTP No No Custom only
FTPS Yes Yes, but only through VPC Custom only

Custom identity providers with API Gateway

As described above, authorization for the FTP and FTPS protocols uses API Gateway. This is a bit of an odd implementation - it feels like a jerry-rigged solution to avoid maintaining their own authorization mechanism.

When setting up a new Transfer Server, you will be presented this screen:
api-gateway

As you can see, you need to configure an API Gateway endpoint. From the documentation you’ll find that this Gateway needs to be configured with AWS_IAM, which enforces that only AWS Transfer can call the API.

The API itself needs to be configured with a path /servers/{serverId}/users/{username}/config. When you execute a query on FTP or FTPS, AWS Transfer will execute a GET request on that path.

apig-config-method-fields

You then configure a Lambda function to review the credentials and allow or deny the connection. When the connection is allowed, the Lambda returns the IAM role that will be assumed by AWS Transfer to read / write from the S3 bucket.

Although this solution allows for very deep customization, it is rather complex to build and maintain. And because AWS does not support any other authentication methods for FTP and FTPS, you will have to implement it if you want to enable these protocols. Additionally, rolling your own security mechanism in Lambda increases the chance of errors and mistakes, possibly leading to denial of service to valid users or worse - unwanted access for unauthorized users.

Pricing

Pricing has remained unchanged at $0.04 per GB transferred (in or out). Keep in mind that you pay $0.30 per hour per enabled protocol.

Compatibility

When desiging or deploying AWS Transfer for FTP(S), you should be aware of the following details:

  • Only Passive mode is supported. Our service does not make outbound connections.
  • Only Explicit mode for FTPS is supported. Our service does not support implicit mode.
  • Renaming file names is supported, but renaming directory (S3 BucketName) is not supported.
  • Append operations are not supported.

Conclusion

AWS Transfer for FTP and FTPS is a great addition to enable older systems to connect to and interact with the AWS ecosystem. This allows for easier migration, cheaper backups and improved analytical insight into your legacy applications.

AWS Transfer for FTP(S) comes with a few caveats; FTP can’t be made public (which is for the best, really), and user management for FTP and FTPS is a lot more involved than for SFTP.

Regardless, if you have a requirement for FTP(S) in your organisation, the serverless AWS Transfer for FTP and FTPS service beats running and maintaining your own servers hands down.

Luc van Donkersgoed

Luc van Donkersgoed