terraform-aws-privatelink-consumer

Consumer side of AWS PrivateLink for the Tamr API. Run this in your own AWS account to reach the Tamr API privately, without traffic crossing the public internet.

It creates an interface VPC endpoint against Tamr's endpoint service, a security group scoping which of your subnets may use it, and a Route 53 private hosted zone that shadows your Tamr tenant domain inside your VPC only.

Usage

module "tamr_api_privatelink" {
  source = "git::https://github.com/Datatamer/terraform-aws-privatelink-consumer.git?ref=v1.0.0"

  vpc_id                = "vpc-0123456789abcdef0"
  subnet_ids            = ["subnet-aaa", "subnet-bbb", "subnet-ccc"]
  endpoint_service_name = "com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0"
  allowed_cidrs         = ["10.0.0.0/16"]
  api_domain            = "tamr.cloud"
}

Use the endpoint service name and tenant domain Tamr gives you for your region, for example tamr.cloud, or eu.tamr.cloud, uk.tamr.cloud or ap.tamr.cloud outside the US. Your AWS region must match the region of that endpoint service.

Onboarding sequence

1. Send Tamr the ARN of the IAM role that will apply this module, or your AWS account ID if you can't name a role. Tamr sends back the endpoint service name and the AZ IDs the service is available in.
2. Apply this module. The endpoint starts in pendingAcceptance.
3. Wait for Tamr to accept the connection. Someone at Tamr does this by hand. After that the endpoint is available, which the module's vpc_endpoint_state output shows.
4. Test with enable_test_client = true (see below), or from an instance in allowed_cidrs:

   dig +short your-tenant.tamr.cloud     # expect endpoint ENI private IPs
   curl -v https://your-tenant.tamr.cloud/

What this does to DNS

Inside this VPC, the private hosted zone takes over all of api_domain: every name under it, at any depth, resolves to the endpoint. Resolution outside the VPC is untouched.

An endpoint reaches a single Tamr region. If your tenants are in more than one region, give each region its own VPC and endpoint.

If you already manage a hosted zone for that domain in this VPC, set create_private_hosted_zone = false and add an alias record yourself, pointing at the endpoint_dns_name output.

Testing without an instance

enable_test_client = true creates a Lambda inside your VPC that makes the request for you, named <name>-test-client (tamr-api-test-client with the default name):

module "tamr_api_privatelink" {
  ...
  enable_test_client = true
  test_client_host   = "your-tenant.tamr.cloud"   # must be one label under api_domain
  test_client_path   = "/api/v1alpha/record/list"
}

To control where it logs, for example to set a retention period, pass a log group you own in test_client_log_group_name. Otherwise Lambda creates its default /aws/lambda/<name>-test-client group the first time it runs.

aws lambda invoke --function-name tamr-api-test-client \
  --payload '{"path": "/api/v1alpha/record/list"}' \
  --cli-binary-format raw-in-base64-out /dev/stdout

The response lists each request's status, headers and body, so the result doesn't depend on logs.

Result in requests	Meaning
"status": 403	Working end to end. TLS terminated with the right cert, Tamr routed the request, and the API rejected an unauthenticated request
"status": 404	Reached Tamr but no route matched — check test_client_host is your real tenant hostname
"error": "TimeoutError"	Connection not accepted yet, or no inbound rule on the endpoint security group
"error": "SSLCertVerificationError"	The hostname requested is not what the private hosted zone resolves, or create_private_hosted_zone is false

Several requests on one connection

count sends the request that many times over one connection, interval_seconds apart. Each entry in requests says whether it reused the connection. For example, to check that a connection survives five minutes idle:

aws lambda invoke --function-name tamr-api-test-client \
  --payload '{"count": 2, "interval_seconds": 300}' \
  --cli-binary-format raw-in-base64-out --cli-read-timeout 0 /dev/stdout

The whole run must fit in test_client_timeout, 900 seconds by default. The AWS CLI stops waiting after 60 seconds unless you pass --cli-read-timeout 0.

The same script runs on your own machine, so you can compare with the public route:

python3 lambda/api_test_client/index.py https://your-tenant.tamr.cloud/ --count 2 --interval 300

Notes

• AZ IDs, not names. AZ names are per-account aliases, so compare your subnets' AZ IDs with the ones Tamr sends. A subnet in an AZ the service does not serve fails at apply with "endpoint service does not support the Availability Zone (AZ) for the subnet"; drop that subnet from subnet_ids.
• Your source IP is not visible to Tamr. Traffic is translated at the endpoint and again at Tamr's load balancer. Authorization is entirely at the application layer — your API credentials, not your network address.
• Idle connections. AWS drops a TCP connection through an endpoint after 350 seconds with no traffic. Enable TCP keepalives on long-lived clients.
• Cost. You pay AWS for endpoint hours and data processed, per the standard PrivateLink pricing. Tamr does not charge for the connection.

Development

The test client's Lambda ships prebuilt as lambda/api_test_client.zip, so applying the module needs no build step. After changing lambda/api_test_client/index.py, rebuild and commit the zip:

make lambda/zip

make all also formats the code and regenerates the inputs and outputs below. It needs terraform, python3 and terraform-docs v0.24.0. CI runs make all on every pull request and fails if it changes anything.

Requirements

Name	Version
terraform	>= 1.2
aws	>= 6.0, < 7.0

Providers

Name	Version
aws	>= 6.0, < 7.0

Modules

No modules.

Resources

Name	Type
aws_iam_role.test_client	resource
aws_iam_role_policy_attachment.test_client_vpc_access	resource
aws_lambda_function.test_client	resource
aws_route53_record.wildcard	resource
aws_route53_zone.this	resource
aws_security_group.endpoint	resource
aws_security_group.test_client	resource
aws_vpc_endpoint.this	resource
aws_vpc_security_group_egress_rule.test_client_to_endpoint	resource
aws_vpc_security_group_ingress_rule.endpoint_from_test_client	resource
aws_vpc_security_group_ingress_rule.from_cidr	resource
aws_vpc_security_group_ingress_rule.from_sg	resource
aws_cloudwatch_log_group.test_client	data source
aws_iam_policy_document.test_client_trust	data source

Inputs

Name	Description	Type	Default	Required
allowed_cidrs	CIDRs within the consumer VPC permitted to reach the endpoint on 443	list(string)	[]	no
allowed_security_group_ids	Security group IDs permitted to reach the endpoint on 443	list(string)	[]	no
api_domain	Tamr tenant domain to shadow inside this VPC, e.g. tamr.cloud or eu.tamr.cloud	string	n/a	yes
create_private_hosted_zone	Create a Route 53 private hosted zone for api_domain. Set false if you already manage that zone in this VPC and will add the record yourself.	bool	true	no
enable_test_client	Create a Lambda in the VPC that requests the API through the endpoint, for testing the connection	bool	false	no
endpoint_service_name	Tamr-provided endpoint service name, e.g. com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0	string	n/a	yes
name	Name prefix for every resource this module creates	string	"tamr-api"	no
subnet_ids	Subnet IDs for the endpoint ENIs, at most one per AZ	list(string)	n/a	yes
tags	Tags applied to every resource that supports them	map(string)	{}	no
test_client_host	Tenant hostname the test client requests, one label under api_domain, e.g. your-tenant.tamr.cloud. Required when enable_test_client is true.	string	null	no
test_client_log_group_name	Existing CloudWatch log group the test client writes to. When null, Lambda creates its default /aws/lambda/-test-client group on first invocation, outside Terraform.	string	null	no
test_client_method	HTTP method the test client uses	string	"GET"	no
test_client_path	URL path the test client requests	string	"/"	no
test_client_security_group_ids	Additional security groups to attach to the test client, for example one that allows reaching CloudWatch Logs	list(string)	[]	no
test_client_timeout	Seconds the test client may run	number	900	no
test_client_verify_tls	Whether the test client verifies the API certificate	bool	true	no
vpc_id	Consumer VPC ID to place the endpoint in	string	n/a	yes

Outputs

Name	Description
endpoint_dns_entries	All DNS entries (regional plus per-AZ), for debugging resolution
endpoint_dns_name	Regional DNS name of the endpoint
hosted_zone_id	ID of the private hosted zone, if this module created it
security_group_id	Security group on the endpoint ENIs. Reference this from a client's egress rules.
test_client_function_name	Name of the test client Lambda, for aws lambda invoke
test_client_security_group_id	Security group on the test client, already permitted inbound on the endpoint
test_client_url	URL the test client requests by default
vpc_endpoint_id	ID of the interface endpoint, for Tamr to accept
vpc_endpoint_state	pendingAcceptance until Tamr accepts the connection, then available