Mailing list system providing address validation and unsubscribe URIs.
Source: https://github.com/mbland/elistman
(Try force reloading the page to get the latest badges if this is a return visit. The browser cache may hide the latest results.)
Only serves one list at a time as defined by deployment parameters.
Implemented in Go using the following Amazon Web Services:
- API Gateway
- Lambda
- DynamoDB
- Simple Email Service
- Simple Notification Service
- Web Application Firewall
Uses CloudFormation and the AWS Serverless Application Model (SAM) for deploying the Lambda function, binding to the API Gateway, managing permissions, and other configuration parameters.
Originally implemented to support https://mike-bland.com/subscribe/.
The very earliest stages of the implementation were based on hints from victoriadrake/simple-subscribe, but all the code is original.
This software is made available as Open Source software under the Mozilla Public License 2.0. For the text of the license, see the LICENSE.txt file.
Run the bin/check-tools.sh
script to check that the required tools are
installed.
-
This script will try to install some missing tools itself. If any are missing, the script will provide a link to installation instructions.
-
Note: The script does not check for the presence of
make
, as it comes in many flavors and aliases and already ships with many operating systems. The notable exception is Microsoft Windows.However, if the
winget
command is available, you can install the GnuWin32.Make package:winget install -e --id GnuWin32.Make
Then:
- Press Win-R and enter
systempropertiesadvanced
to open the System Properties > Advanced pane. - Click the Environment Variables... button.
- Select Path in either the User variables or System variables pane, then click the corresponding Edit... button.
- Click the New button, then click the Browse... button.
- Navigate to This PC > Local Disk (C:) > Program Files (x86) > GnuWin32 > bin.
- Click the OK button, then keep clicking the OK button until all of the System Properties panes are closed.
Make should then be available as either
make
ormake.exe
. - Press Win-R and enter
Configure your credentials for a region from the Email Receiving Endpoints section of Amazon Simple Email Service endpoints and quotas.
Follow the guidance on the AWS Command Line Interface: Quick Setup page if necessary.
Set up SES in the region selected in the above step. Make sure to enable DKIM and create a verified domain identity per Verifying your domain for Amazon SES email receiving.
Create a Receipt Rule Set and set it as Active. EListMan will add a Receipt Rule for an unsubscribe email address to this Receipt Rule Set.
Create a Receipt Rule to receive Email notifications for the postmaster
and abuse
accounts, along with any other accounts that you'd like.
- You can add these recipient conditions manually, which would require creating a Simple Notification Service (SNS) topic manually as well.
- Alternatively, consider using mbland/ses-forwarder to automate configuring the Receipt Rule Set with an appropriate Receipt Rule.
When you're ready for the system to go live, publish an MX record for Amazon SES email receiving.
It's also advisable to configure your account-level suppression list to automatically add addresses resulting in bounces and complaints.
Assuming you have your AWS CLI environment set up correctly, this should confirm that SES is properly configured (with your own identity listed, of course):
$ aws sesv2 list-email-identities
{
"EmailIdentities": [
{
"IdentityType": "DOMAIN",
"IdentityName": "mike-bland.com",
"SendingEnabled": true,
"VerificationStatus": "SUCCESS"
}
]
}
You can also view other account attributes, such as account suppression list status, send quotas, and send rates, via:
$ aws sesv2 get-account
{
...
"ProductionAccessEnabled": true,
"SendQuota": {
"Max24HourSend": ...,
"MaxSendRate": ...,
"SentLast24Hours": ...
},
"SendingEnabled": true,
"SuppressionAttributes": {
"SuppressedReasons": [
"BOUNCE",
"COMPLAINT"
]
},
"Details": {
"MailType": "MARKETING",
"WebsiteURL": "https://mike-bland.com/",
"ContactLanguage": "EN",
"UseCaseDescription": "This is for publishing blog posts to email subscribers.",
"AdditionalContactEmailAddresses": [
"mbland@acm.org"
],
...
}
}
Set up a custom domain name in API Gateway in the region selected in the above step. Create a SSL certificate in Certificate Manager for it as well.
If done correctly, the following command should produce output resembling the example:
$ aws apigatewayv2 get-domain-names
{
"Items": [
{
"ApiMappingSelectionExpression": "$request.basepath",
"DomainName": "api.mike-bland.com",
"DomainNameConfigurations": [
{
"ApiGatewayDomainName": "<...>",
"CertificateArn": "<...>",
"DomainNameStatus": "AVAILABLE",
"EndpointType": "REGIONAL",
"HostedZoneId": "<...>",
"SecurityPolicy": "TLS_1_2"
}
]
}
]
}
Next, set up an IAM role to allow the API to write CloudWatch logs. You only
need to execute the steps in the Create an IAM role for logging to
CloudWatch section. One possible name for the new IAM role would be
ApiGatewayCloudWatchLogging
.
The last step from the above instructions is to
$ ARN="arn:aws:iam::...:role/ApiGatewayCloudWatchLogging"
$ aws apigateway update-account --patch-operations \
op='replace',path='/cloudwatchRoleArn',value='$ARN'
If successful, the output should resemble the following, where <ARN>
is the value of $ARN
from above:
{
"cloudwatchRoleArn": "<ARN>",
"throttleSettings": {
"burstLimit": ...,
"rateLimit": ...
},
"features": []
}
Note: Per the documentation for the AWS::ApiGateway::Account CloudFormation
entity, "you should only have one AWS::ApiGateway::Account
resource per
region per account." This is why it's not included in template.yml
in favor of
the one-time-per-account instructions above.
However, if you want to try using SAM/CloudFormation to manage it, see:
To make sure the local environment is in good shape, and your AWS services are
properly configured, run the main test suite via make test
. (Note that the
example output below is slightly edited for clarity.)
$ make test
go vet -tags=all_tests ./...
go run honnef.co/go/tools/cmd/staticcheck -tags=all_tests ./...
go build -tags=all_tests ./...
go test -tags=small_tests ./...
ok github.com/mbland/elistman/agent 0.110s
ok github.com/mbland/elistman/db 0.392s
ok github.com/mbland/elistman/email 0.187s
ok github.com/mbland/elistman/handler 0.260s
ok github.com/mbland/elistman/ops 0.461s
ok github.com/mbland/elistman/types 0.523s
go test -tags=medium_tests -count=1 ./...
ok github.com/mbland/elistman/db 2.970s
ok github.com/mbland/elistman/email 1.150s
go test -tags=contract_tests -count=1 ./db -args -awsdb
ok github.com/mbland/elistman/db 44.264s
If you're using Visual Studio Code, you can run all but the last test via
the Test: Run All Tests command (testing.runAll
). The default keyboard shortcut is ⌘; A.
- The project VS Code configuration is in .vscode/settings.json.
- For other helpful testing-related keyboard shortcuts, press ⌘K ⌘S, then
search for
testing
.
The tests are divided into suites of varying test sizes, described below, using Go build constraints (a.k.a. "build tags"). These constraints are specified on the first line of every test file:
$ head -n1 */*_test.go
==> agent/agent_test.go <==
//go:build small_tests || all_tests
# ...snip...
# See "Test coverage" section below for an explanation of the
# dynamodb_contract_test build constraints.
==> db/dynamodb_contract_test.go <==
//go:build ((medium_tests || contract_tests) && !no_coverage_tests) || coverage_tests || all_tests
# ...snip...
==> email/mailer_contract_test.go <==
//go:build medium_tests || contract_tests || all_tests
# ...etc...
The small_tests
all run locally, with no external dependencies. These tests
cover all fine details and error conditions.
Each of the medium_tests
exercises integration with specific dependencies.
Most of these dependencies are actual, live AWS services that require a network
connection.
These tests are designed to set up required state and clean up any side effects. Other than ensuring the network is available, and the required resources are running and accessible, no external intervention is necessary.
medium_tests
validate high level use cases and fundamental assumptions, not
exhaustive details and error conditions. That's what the small_tests
are for,
resulting in fewer, less complicated, faster, and more stable medium_tests
.
Each of the contract_tests
are also medium_tests
. In fact, it's arguable
that these tags are redundant, but I want the reader to contemplate both
concepts and their equivalence.
The medium/contract tests in db/dynamodb_contract_test.go
run against:
- a local Docker container running the amazon/dynamodb-local image when
run without the
-awsdb
flag- e.g. When run via
go test -tags=medium_tests -count=1 ./...
, in VS Code via ⌘; A, or in CI via-tags=coverage_tests
, described below.
- e.g. When run via
- the actual DynamoDB for your AWS account when run with the
-awsdb
flag- e.g. When run via
go test -tags=contract_tests -count=1 ./db -args -awsdb
- e.g. When run via
Note: -count=1
is the Go idiom to ensure tests are run with caching
disabled, per go help testflag
.
There are no end-to-end large_tests
yet, outside of bin/smoke-tests.sh
. The
smoke tests are described below, as are the plans for adding end-to-end tests
one day.
To check code coverage, you can run:
$ make coverage
go test -covermode=count -coverprofile=coverage.out \
-tags=small_tests,coverage_tests ./...
ok github.com/mbland/elistman/agent 0.351s coverage: 100.0% of statements
ok github.com/mbland/elistman/db 3.214s coverage: 100.0% of statements
ok github.com/mbland/elistman/email 0.539s coverage: 100.0% of statements
ok github.com/mbland/elistman/handler 0.613s coverage: 100.0% of statements
ok github.com/mbland/elistman/ops 0.457s coverage: 100.0% of statements
ok github.com/mbland/elistman/types 0.675s coverage: 100.0% of statements
go tool cover -html=coverage.out
[ ...opens default browser with HTML coverage results... ]
You can also check coverage in VS Code by searching for the Go: Toggle Test Coverage in Current Package command via Show All Commands (⇧⌘P).
Note that db/dynamodb_contract_test.go
is the one and only medium_test
that
we need for test coverage purposes. It contains the coverage_tests
build
constraint, enabling the CI pipeline to collect its coverage data without
running other medium_tests
.
Build the elistman
command line interface program in the root directory via:
go build
Run the command and check the output to see if it was successful:
$ ./elistman -h
Mailing list system providing address validation and unsubscribe URIs
See the https://github.com/mbland/elistman README for details.
To create a table:
elistman create-subscribers-table TABLE_NAME
To see an example of the message input JSON structure:
elistman preview --help
To preview a raw message before sending, where `generate-email` is any
program that creates message input JSON:
generate-email | elistman preview
To send an email to the list, given the STACK_NAME of the EListMan instance:
generate-email | elistman send -s STACK_NAME
Usage:
elistman [command]
Available Commands:
[...commands snipped...]
Flags:
-h, --help help for elistman
-v, --version version for elistman
Use "elistman [command] --help" for more information about a command.
Run elistman create-subscribers-table <TABLE_NAME>
to create the DynamoDB
table, replacing <TABLE_NAME>
with a table name of your choice. Then run aws dynamodb list-tables
to confirm that the new table is present.
Create the deploy.env
configuration file in the root directory containing the
following environment variables (replacing each value with your own as
appropriate):
# This will be the name of the CloudFormation stack. The `--stack-name` flag of
# `elistman` CLI commands will require this value.
STACK_NAME="mike-blands-blog-example"
# This is the domain name configured in the "Configure AWS API Gateway" step.
API_DOMAIN_NAME="api.mike-bland.com"
# This will be the first component of the EListMan API endpoints after the
# hostname, e.g., api.mike-bland.com/email/subscribe.
API_MAPPING_KEY="email"
# The domain from which emails will be sent. This should likely match the
# website on which the subscription form appears.
EMAIL_DOMAIN_NAME="mike-bland.com"
# The proper name of the website from which emails will appear to be sent. It
# need not match to the site's <title> exactly, but should clearly describe what
# subscribers expect.
EMAIL_SITE_TITLE="Mike Bland's blog"
# The proper name of the email sender. It need not match EMAIL_SITE_TITLE, but
# again, should not surprise subscribers.
SENDER_NAME="Mike Bland's blog"
# The username of the email sender. The full address will be of the form:
# SENDER_USER_NAME@EMAIL_DOMAIN_NAME, e.g., posts@mike-bland.com.
SENDER_USER_NAME="posts"
# The username of the unsubscribe email recipient. The full address will be of
# the form: UNSUBSCRIBE_USER_NAME@EMAIL_DOMAIN_NAME, e.g.,
# unsubscribe@mike-bland.com.
UNSUBSCRIBE_USER_NAME="unsubscribe"
# The path to the unsubscribe form relative to EMAIL_DOMAIN_NAME. See the
# "Understand the {{UnsubscribeUrl}} template" and "Publish your HTML
# unsubscribe form" sections below.
UNSUBSCRIBE_FORM_PATH="/unsubscribe"
# The name of the Receipt Rule Set created in the "Configure AWS Simple Email
# Service (SES)" step.
RECEIPT_RULE_SET_NAME="mike-bland.com"
# The name of the DynamoDB table created via `elistman create-subscribers-table`
# in the "Create the DynamoDB table" step.
SUBSCRIBERS_TABLE_NAME="<TABLE_NAME>"
# Percentage of daily quota to consume before self-limiting bulk sends via
# `elistman send -s STACK_NAME`. See the "Send rate throttling and send quota
# capacity limiting" step for a detailed description. (Does not apply when
# running `elistman send` with specific subscriber addresses specified on the
# command line.)
MAX_BULK_SEND_CAPACITY="0.8"
# EListMan will redirect API requests to the following URLs according to the
# "Algorithms" described below.
INVALID_REQUEST_PATH="/subscribe/malformed.html"
ALREADY_SUBSCRIBED_PATH="/subscribe/already-subscribed.html"
VERIFY_LINK_SENT_PATH="/subscribe/confirm.html"
SUBSCRIBED_PATH="/subscribe/hello.html"
NOT_SUBSCRIBED_PATH="/unsubscribe/not-subscribed.html"
UNSUBSCRIBED_PATH="/unsubscribe/goodbye.html"
bin/smoke-test.sh
invokes curl
to send HTTP requests to the running Lambda,
all of which expect an error response without any side effects (save for
logging).
To check that your configuration works locally, you'll need two separate
terminal windows to run bin/smoke-test.sh
. In the first, run:
$ make run-local
[ ...validates template.yml, builds lambda, etc... ]
bin/sam-with-env.sh deploy.env local start-api --port 8080
[ ...more output... ]
You can now browse to the above endpoints to invoke your functions....
2023-05-29 16:08:04 WARNING: This is a development server....
* Running on http://127.0.0.1:8080
2023-05-29 16:08:04 Press CTRL+C to quit
In the next terminal, run:
$ ./bin/smoke-test ./deploy.env --local
INFO: SUITE: Not found (403 locally, 404 in prod)
INFO: TEST: 1 — invalid endpoint not found
Expect 403 from: POST http://127.0.0.1:8080/foobar/mbland%40acm.org
curl -isS -X POST http://127.0.0.1:8080/foobar/mbland%40acm.org
HTTP/1.1 403 FORBIDDEN
Server: Werkzeug/2.3.4 Python/3.8.16
Date: Mon, 29 May 2023 20:19:57 GMT
Content-Type: application/json
Content-Length: 43
Connection: close
{"message":"Missing Authentication Token"}
PASSED: 1 — invalid endpoint not found:
status: 403
INFO: TEST: 2 — /subscribe with trailing component not found
Expect 403 from: POST http://127.0.0.1:8080/subscribe/foobar
[ ...more test output/results... ]
PASSED: 6 — invalid UID for /unsubscribe:
status: 400
PASSED: All 6 smoke tests passed!
Then enter CTRL-C in the first window to stop the local SAM Lambda server.
Before deploying to production, we need to talk about spam.
The EListMan system tries to validate email addresses through its own up front analysis and by sending validation links to subscribers. However, opportunistic spam bots can still—and will—submit many valid email addresses without either the knowledge or consent of the actual owner.
Fortunately, the validation link mechanism prevents most bogus subscriptions, and DynamoDB's Time To Live feature cleans them from the database automatically. A bounce or complaint also notifies the EListMan Lambda to remove the address and add it to the account-level suppression list. The suppression list ensures the system won't send to that address again, even if someone attempts to resubmit it.
This means most bogus subscriptions will not pollute the verified subscriber list, and such recipients will not receive further emails. However, generating these bogus subscriptions still consumes resources, and their verification emails can yield bounces and complaints that will harm your SES reputation metrics.
Having learned this the hard (naïve) way, I recommend using a CAPTCHA to prevent spam bot abuse:
- When I first published my EListMan subscription form, my instance received dozens of bogus subscription requests a day—before I'd even announced it on my blog. (The form had been available before, but used a different subscription system.)
- After deploying a CAPTCHA, the number of bogus subscriptions dropped to zero. (I hope I hadn't inadvertently been allowing subscription verification spam all those years before....)
EListMan's CloudFormation/SAM template configures an AWS Web Application Firewall (WAF) CAPTCHA, creating one Web ACL and one Rule associated with it. If you choose to use it, note that it does incur additional charges. See AWS WAF Pricing for details.
If you choose not to use it, comment out or delete the WebAcl
and
WebAclAssociation
resources in template.yml.
To use EListMan's Web ACL configuration, you'll need to generate an API key for the CAPTCHA API. Include whichever domain will serve the submission form in the list of domains used to generate the API key.
The default EListMan configuration expects this domain to be the same as
EMAIL_DOMAIN_NAME
, described above. If you use a different domain, set
WebAcl > Properties > TokenDomains
in template.yml
appropriately.
If the smoke tests pass, deploy the EListMan system via:
make deploy
Once the deployment is running, run the smoke tests without the --local
flag
to ensure your instance is reachable:
./bin/smoke-tests.sh ./deploy.env
You'll need to publish a subscription <form> similar to the following,
substituting API_DOMAIN_NAME
with the custom domain name from the Configure
AWS API Gateway step:
<!-- subscribe.html -->
<form method="post" action="https://API_DOMAIN_NAME/email/subscribe">
<input name="email" type="email"
placeholder="Please enter your email address."/>
<button type="submit">Subscribe</button>
</form>
However, as mentioned above, spam bots are a thing, even for the humblest of sites publicly sporting a <form> element.
You may gain extra protection from spam bots by generating the subscription form using JavaScript instead of embedding a <form> element directly in your HTML.
In other words, instead of embedding the <form> directly in your subscription page as shown above, use something like this:
<!-- subscribe.html -->
<div class="subscribe-form">
<button>Show subscribe form</button>
</div>
// subscribe.js
"use strict";
document.addEventListener("DOMContentLoaded", () => {
var container = document.querySelector(".subscribe-form")
var showForm = () => {
var f = document.createElement("form")
// The following should generate the value for API_DOMAIN_NAME.
var api_domain_name = ["my", "api", "com"].join(".")
f.action = ["https:", "", api_domain_name, "email", "subscribe"].join("/")
f.method = "post"
var i = document.createElement("input")
i.name = "email"
i.type = "email"
i.placeholder = "Please enter your email address."
f.appendChild(i)
var s = document.createElement("button")
s.type = "submit"
s.appendChild(document.createTextNode("Subscribe"))
f.appendChild(s)
container.parentNode.replaceChild(f, container)
}
container.querySelector("button").addEventListener('click', showForm)
})
Of course, the ultimate protection would be to use an AWS WAF CAPTCHA to protect
the /subscribe
API endpoint.
Using the same HTML from above, the code below will render the AWS WAF CAPTCHA puzzle when the subscriber clicks the button. When they solve the puzzle, it will then reveal the submission form.
Remember to substitute YOUR_AWS_WAF_CAPTCHA_API_KEY
with your own API key:
// subscribe.js
"use strict";
document.addEventListener("DOMContentLoaded", () => {
var container = document.querySelector(".subscribe-form")
var showForm = () => {
// Same implementation as above
}
container.querySelector("button").addEventListener('click', () => {
AwsWafCaptcha.renderCaptcha(container, {
apiKey: YOUR_AWS_WAF_CAPTCHA_API_KEY,
onSuccess: showForm,
dynamicWidth: true,
skipTitle: true
});
})
})
The {{UnsubscribeUrl}}
generated for each recipient will be of the format:
https://${EMAIL_DOMAIN_NAME}/${UNSUBSCRIBE_FORM_PATH}?email=<email>&uid=<uid>
where:
<email>
is the recipient's query encoded email address<uid>
is the recipient's query encoded user ID generated by the system
For example:
https://mike-bland.com/unsubscribe?email=foo%40bar.com&uid=00000000-1111-2222-3333-444444444444
For more background on URI encoding:
You'll need to publish a page with an unsubscribe <form> at the
location https://${EMAIL_DOMAIN_NAME}/${UNSUBSCRIBE_FORM_PATH}
. This form will
allow the user to confirm they really intend to unsubscribe.
Use JavaScript to fill out the <form> using the email
and uid
URL
query parameters provided when the user clicks on their unique
{{UnsubscribeUrl}}
.
For example, following the same pattern as the Generate your email submission form programmatically (optional) section above:
<!-- unsubscribe.html -->
<h2>Unsubscribe</h2>
<div class="unsubscribe"><p>If you would like to stop receiving email
updates, please click the "Unsubscribe" link at the bottom of one of the
emails.</p></div>
// unsubscribe.js
"use strict";
document.addEventListener("DOMContentLoaded", () => {
var params = new URLSearchParams(window.location.search)
if (!params.has("email") || !params.has("uid")) {
return
}
var instructions = document.querySelector(".unsubscribe p")
instructions.innerHTML = instructions.innerHTML.replace(/[\n ]+/g, " ")
.replace("link at the bottom of one of the emails", "button below")
var f = document.createElement("form")
// The following should generate the value for API_DOMAIN_NAME.
var api_domain_name = ["my", "api", "com"].join(".")
f.action = [
"https:", "", api_domain_name, "email", "unsubscribe",
encodeURI(params.get("email")), encodeURI(params.get("uid")),
].join("/")
f.method = "post"
var s = document.createElement("button")
s.type = "submit"
s.appendChild(document.createTextNode("Unsubscribe"))
f.appendChild(s)
instructions.parentNode.appendChild(f)
})
After deploying EListMan and publishing your subscription form, use the form to
subscribe to the list. Then you can run the following command to send a test
email to yourself (replacing STACK_NAME
and MY_EMAIL_ADDRESS
as
appropriate):
$ ./bin/generate-test-message.sh ./deploy.env |
./elistman send -s STACK_NAME MY_EMAIL_ADDRESS
Run ./elistman send -h
to see an example email:
$ ./elistman send -h
Reads a JSON object from standard input describing a message:
{
"From": "Foo Bar <foobar@example.com>",
"Subject": "Test object",
"TextBody": "Hello, World!",
"TextFooter": "Unsubscribe: {{UnsubscribeUrl}}",
"HtmlBody": "<!DOCTYPE html><html><head></head><body>Hello, World!<br/>",
"HtmlFooter": "<a href='{{UnsubscribeUrl}}'>Unsubscribe</a></body></html>"
}
You will need to generate a similar JSON object to feed into the standard input
of ./elistman send
:
From
,Subject
,TextBody
, andTextFooter
are required.- If
HtmlBody
is present,HtmlFooter
must also be present. TextFooter
, andHtmlFooter
if present, must contain one and only one instance of the{{UnsubscribeUrl}}
template. The EListMan Lambda will replace this template with the unsubscribe URL unique to each subscriber.TextFooter
andHtmlFooter
will appear on a new line immediately afterTextBody
andHtmlBody
, respectively.
Provided you have a program to generate the JSON object above called
generate-email
, you can then send an email to the list via:
generate-email | ./elistman send -s STACK_NAME
The Makefile is very short and readable. Use it to run common tasks, or learn common commands from it to use as you please.
For guidance on writing Go developer documentation, see Go Doc Comments.
There are two ways to view the developer documentation in a web browser.
godoc is reportedly deprecated, but still works well. See:
- golang/go: x/tools/cmd/godoc: document as deprecated #49212
- 349051: cmd/godoc: deprecate and point to cmd/pkgsite
# Install the godoc tool.
$ go install -v golang.org/x/tools/cmd/godoc@latest
# Serve documentation from the local directory at http://localhost:6060.
$ godoc -http=:6060
You can then view the EListMan docs locally at:
One of the nice features of godoc
is that you can view documentation for
unexported symbols by adding ?m=all
to the URL. For example:
pkgsite is the newer development documentation publishing system.
# Install the pkgsite tool.
$ go install golang.org/x/pkgsite/cmd/pkgsite@latest
# Serve documentation from the local directory at http://localhost:8080.
$ pkgsite
You can then view the EListMan docs locally at:
Note that, unlike godoc
, pkgsite
doesn't provide an option to serve documentation for unexported symbols.
https://<api_hostname>/<route_key>/<operation>
mailto:<unsubscribe_user_name>@<email_domain_name>?subject=<email>%20<uid>
Where:
<api_hostname>
: Hostname for the API Gateway instance<route_key>
: Route key for the API Gateway<operation>
: Endpoint for the list management operation:/subscribe
/verify/<email>/<uid>
/unsubscribe/<email>/<uid>
<email>
: Subscriber's email address<uid>
: Identifier assigned to the subscriber by the system<unsubscribe_user_name>
: The username receiving unsubscribe emails, typicallyunsubscribe
, set viaUNSUBSCRIBE_USER_NAME
.<email_domain_name>
: Hostname serving as an SES verified identity for sending and receiving email, set viaEMAIL_DOMAIN_NAME
See also:
Unless otherwise noted, all responses will be HTTP 303 See Other, with the target page specified in the Location HTTP header.
- The one exception will be unsubscribe requests from mail clients using the
List-Unsubscribe
andList-Unsubscribe-Post
email headers.
- An HTTP request from the API Gateway comes in, containing the email address of a potential subscriber.
- Validate the email address.
- Parse the name as closely as possible to RFC 5322 Section 3.2.3 via net/mail.ParseAddress.
- Reject any common aliases, like "no-reply" or "postmaster."
- Check the MX records of the host by:
- Doing a reverse lookup on each mail host's IP addresses.
- Looking up the IP addresses of the hosts returned by the reverse lookup.
- Confirming at least one reverse lookup host IP address matches a mail host IP address.
- If it fails validation, return the
INVALID_REQUEST_PATH
.
- Look for an existing DynamoDB record for the email address.
- If it exists, return the
VERIFY_LINK_SENT_PATH
forPending
subscribers andALREADY_SUBSCRIBED_PATH
forVerified
subscribers.
- If it exists, return the
- Generate a UID.
- Write a DynamoDB record containing the email address, the UID, a timestamp,
and with
SubscriberStatus
set toPending
. - Generate a verification link using the email address and UID.
- Send the verification link to the email address.
- If the mail bounces or fails to send, return the
INVALID_REQUEST_PATH
.
- If the mail bounces or fails to send, return the
- Return the
VERIFY_LINK_SENT_PATH
.
- An HTTP request from the API Gateway comes in, containing a subscriber's email address and UID.
- Check whether there is a record for the email address in DynamoDB.
- If not, return the
NOT_SUBSCRIBED_PATH
.
- If not, return the
- Check whether the UID matches that from the DynamoDB record.
- If not, return the
NOT_SUBSCRIBED_PATH
.
- If not, return the
- If the subscriber's status is
Verified
, return theALREADY_SUBSCRIBED_PATH
. - Set the
SubscriberStatus
of the record toVerified
. - Return the
SUBSCRIBED_PATH
.
- Either an HTTP Request from the API Gateway or a mailto: event from SES comes in, containing a subscriber's email address and UID.
- Check whether there is a record for the email address in DynamoDB.
- If not, return the
NOT_SUBSCRIBED_PATH
.
- If not, return the
- Check whether the UID matches that from the DynamoDB record.
- If not, return the
NOT_SUBSCRIBED_PATH
.
- If not, return the
- Delete the DynamoDB record for the email address.
- If the request was an HTTP Request:
- If it uses the
POST
method, and the data containsList-Unsubscribe=One-Click
, return HTTP 204 No Content. - Otherwise return the
UNSUBSCRIBED_PATH
page.
- If it uses the
DynamoDB's Time To Live feature will eventually remove expired pending subscriber records after 24 hours.
EListMan calls the SES v2 getAccount
API method once a minute to monitor
sending quotas and to adjust the send rate. Every individual message sent,
including both subscription verification messages and messages sent to the list,
will honor the current send rate.
The MAX_BULK_SEND_CAPACITY
parameter specifies what percentage of the 24 hour
send quota may be used for sending emails to the list. This helps avoid
exceeding the daily quota before a message has been sent to all subscribers.
elistman send
will fail, before sending an email, if the percentage of the
daily send quota specified by MAX_BULK_SEND_CAPACITY
has already been
consumed.
The default is to use 80% of the available daily send quota for list messages,
expressed as MAX_BULK_SEND_CAPACITY="0.8"
. The remaining 20% acts as a buffer.
For example, for a quota of 50,000 messages, up to 40,000 (50,000 * 0.8) are
available to elistman send
within a 24 hour period.
Note that this mechanism tries to prevent the operator from accidentally
exceeding the 24 hour quota, but it's not foolproof. The operator is ultimately
responsible for ensuring that elistman send
won't exceed the quota if
MAX_BULK_SEND_CAPACITY
hasn't yet been reached, or for tuning it accordingly.
Building on the previous example, if there are 17,000 subscribers:
- The first
elistman send
consumes 17,000 of the quota. - The second
elistman send
consumes the next 17,000 of the quota, for a total of 34,000. - The third
elistman send
will proceed, since 34,000 is less than the 40,000 calculated byMAX_BULK_SEND_CAPACITY="0.8"
. However, it will consume 17,000 more of the quota, for 51,000 total, exceeding the 50,000 quota.
Subscription verification messages are not affected by the
MAX_BULK_SEND_CAPACITY
constraint. The buffer defined by
MAX_BULK_SEND_CAPACITY
can ensure that there is always daily send quota
available for such messages.
- Managing your Amazon SES sending limits
- Errors related to the sending quotas for your Amazon SES account
- How to handle a "Throttling – Maximum sending rate exceeded" error
- How to Automatically Prevent Email Throttling when Reaching Concurrency Limit
This is something I really want to pull off, but without blocking the first release.
Here is what I anticipate the implementation will involve (beyond some of the existing cases in bin/smoke-test.sh):
- Create a new random test username.
- Create a S3 bucket for the emails received by the random test user.
- Create a receipt rule for the active rule set on the domain to send emails to
the new random test username to the S3 bucket.
- Add the random test username to the recipient conditions.
- Add an action to write to the S3 bucket
- Bring up a CloudFormation/Serverless Application Model stack defining these resources.
For each permutation described below:
- Send a request to subscribe the valid random username.
- Note: The
/subscribe
endpoint is CAPTCHA-protected on the dev and prod instances. We may need to bring up an alternate API Gateway instance for the test without CAPTCHA protection, or callProdAgent.Subscribe()
through another method.
- Note: The
- Read the S3 bucket to get the validation URL.
- Request the validation URL.
- Form an unsubscribe request (either URL or mailto) from the validation URL.
- Subscribe via urlencoded params, unsubscribe via urlencoded params
- Subscribe via form-data, unsubscribe via form-data params
- Subscribe via urlencoded params, unsubscribe via email
- Try to resubscribe to expect an
ALREADY_SUBSCRIBED_PATH
response. - Modify the UID in the verification URL to expect an
INVALID_REQUEST_PATH
response.
- Tear down the stack, which will:
- Tear down the receipt rules
- Tear down the test bucket
- Building Lambda functions with Go
- Using AWS Lambda with other services
- Using AWS Lambda with Amazon API Gateway
- aws/aws-sdk-go
- aws/aws-lambda-go
- Blank AWS Lambda function in Go
- Installing or updating the latest version of the AWS CLI
- The Complete AWS Sam Workshop
- AWS Serverless Application Model (AWS SAM) specification
- AWS Serverless Application Model (SAM) Version 2016-10-31
- AWS Serverless Application Model (AWS SAM) Documentation
- AWS CloudFormation Parameters
- AWS CloudFormation Template Reference
- AWS::Serverless::HttpApi
- Setting up custom domain names for REST APIs
- AWS::Serverless::Connector
- How can I set up a custom domain name for my API Gateway API?
- Tutorial: Build a CRUD API with Lambda and DynamoDB
- AWS SAM policy templates
- Serverless Land: Lambda to SES
- How to use AWS secret manager and SES with AWS SAM
- AWS SDK for Go V2
- AWS Lambda function handler in Go
- aws-lambda-go APIGatewayV2 event structures
- aws-lambda-go APIGatewayV2 event example
- Working with AWS Lambda proxy integrations for HTTP APIs
- Using templates to send personalized email with the Amazon SES API
- AWS SES Sample incoming email event
- AWS CloudFormation AWS::SES::ReceiptRuleSet
- AWS CloudFormation AWS::SES::ReceiptRule
- AWS SES Invoke Lambda function action
- Using AWS Lambda with Amazon SES
- aws/aws-lambda-go/events/README_SES.md
- Regions and Amazon SES
- DMARC GUIDE | DMARC: What is DMARC?
- Packing multiple binaries in a Golang package
- One-Click List-Unsubscribe Header – RFC 8058
- The Use of URLs as Meta-Syntax for Core Mail List Commands and their Transport through Message Header Fields - RFC 2369
- Signaling One-Click Functionality for List Email Headers - RFC 8058
- List-Unsubscribe header critical for sustained email delivery
- The Email Marketers Guide to Using List-Unsubscribe
- List Unsubscribe Header in Email
- Prevent mail to Gmail users from being blocked or sent to spam
- Stack Overflow: Post parameter in path or in body
- How to Verify Email Address Without Sending an Email
- 25, 2525, 465, 587, and Other Numbers: All About SMTP Ports
- How to Choose the Right SMTP Port (Port 25, 587, 465, or 2525)
- Which SMTP port should I use? Understanding ports 25, 465 & 587
- Using curl to send email
- Email Sender Reputation Made Simple
- Why Go: Command-line Interfaces (CLIs)
- spf13/cobra
- AWS Lambda function logging in Go
- Working with stages for HTTP APIs
- AWS Lambda function errors in Go
- RFC 9110: HTTP Semantics
- Golang Auto Build Versioning
- jasonmf/go-embed-version
- Using ldflags to Set Version Information for Go Applications
- go tool link (also
go tool link -help
) - A better way than “ldflags” to add a build version to your Go binaries
- AWS Lambda function versions
- Sending test emails in Amazon SES with the simulator
- Setting up event notification for Amazon SES
- Receiving Amazon SES notifications using Amazon SNS
- Contents of event data that Amazon SES publishes to Amazon SNS
- How email sending works in Amazon SES
- Specifying a configuration set when you send email
- RFC 2782: A DNS RR for specifying the location of services (DNS SRV)
- RFC 4409: Message Submission for Mail
- RFC 6186: Use of SRV Records for Locating Email Submission/Access Services
- Multipurpose Internet Mail Extensions (MIME)
- Does the presence of a Content-ID header in an email MIME mean that the attachment must be embedded?
- RFC 2183: Communicating Presentation Information in Internet Messages: The Content-Disposition Header Field
- RFC 2392: Content-ID and Message-ID Uniform Resource Locators
- The precise format of Content-Id header
- RFC 7103: Advice for Safe Handling of Malformed Messages