Sunday, July 3, 2016

Accessing External IP Addresses from Lambda in a VPC

AWS Lambda is to compute what S3 is to storage. It is a simplified model where programs become stateless tasks named Lambda functions in Java, Python or Nodejs. Lambda functions can be triggered by HTTP call to the API Gateway, Cloudwatch time-based schedules or other AWS services. This is "just" stateless functions relying on the library you want. In return, don't bother with the infrastructure at all. Get a compute-model that scales out and is reliable. And best of all, pay for what you really use and not what you provision. I love Lambda as much as I love S3!

However, the devil remains in the details and there are a few of them: the security model; how Lambda connects to networks; How it scales; How to manage code; or how to deal with programming/debugging. Obviously, you can guess it takes some work to be able to kick off Lambda in milliseconds on distributed systems and it does not come for free. Even if you develop stateless functions. This article presents Lambda VPC setup or, more specifically, how to allow Lambda to access external networks and external endpoints from within a VPC.

But lets start with some good news:

  • If Lambda only accesses Internet addresses, you don't have to attach it to your VPC and it should just work fine
  • If you want Lambda to only access VPC internal IPs, including a S3 VPC endpoint, add it to the VPC and make sure it has access to the right network services/IPs by setting security groups appropriately. It does not require any specific settings either.
You cannot set a specific IP address for Lambda, even thought you could create a very narrow subnet; that is because, when needed, the service should be able to scale out and use several IP simultaneously. In addition, you should also attach Lambda functions to subnets located in different AZ so that if one is not available, they can execute in another AZ. 
It is not unusual to have Lambda requiring access to both VPC and external IPs. There are 2 ways to deal with it:
  • You can develop programs running on internal resources that would manage external accesses by themselves. For instance, say you want to pull data from an EC2 instance and push them into DynamoDB. You might prefer to write a code uploading data in DynamoDB from the EC2 instance rather that a lambda function handling the pull/push task. One of the reasons for such a choice is because you probably already have some extra power that you overpay if you run EC2 instances ;-)
  • You can configure the VPC so that Lambda accesses both internal and external resources like below: 

Lambda can not directly access the router to the external network. Instead, it has to access a NAT Gateway. As a result, and because you cannot guess Lambda IP addresses, you must proceed like this:

  • Create at least 2 subnets, in yellow above, each one in a separate availability zone, so that, when you kick'off a Lambda function, it can do it even if one AZ is down. Those subnets should not be able to access the external network. That's why you will change route tables in the 3rd step of this configuration.
  • Create one NAT gateway per availability zone with the "lambda subnet" you've just created. The NAT gateway should be able to access the external network. Note that if you create a NAT Gateway, it will use an Elastic IP address, even if, it is actually not useful for this particular scenario ; it will also cost you a few. To avoid using a EIP and reduce the cost, you might want to create a NAT instance instead of a NAT Gateway. It provides the same feature but will require you manage an EC2 instance by yourself.
  • Create one route table per Lambda subnet so that, TCP/IP connections to external addresses get redirected to the NAT Gateway/Instance. Route tables are specific to every Lambda subnet because they reference the NAT Gateway/Instance from the local AZ. Attach tables to subnets.
  • Test connections from Lambda subnets and make sure they can access both internal and external addresses. If you don't manage to make it work, test it with an EC2 instance running from inside subnets to figure out if the problem comes from your route tables or from Lambda code/security groups.
This is it. You can continue to play with Lambda.

No comments:

Post a Comment