Sending and Receiving Emails From SES Using Gmail

Published on Oct 25, 2024

Recently, I got interested in email services. You know those emails with a custom domain like “john@github.com”, yeah I want to get one of those for my domain (rickyxyz.dev). I know there are services like Google Workspace or Zoho Mail that allows you to use a custom domain for your email address, but I am a techie, of course I can’t just the simple solution and I need to engineer one myself. So, I tried to connect AWS SES with my Gmail client.

Expected Results

By the end of this process, I hoped I could (and you too):

Send an email from my custom domain through Gmail client
Receive an email sent to my custom domain’s email
Read email sent to my custom domain using Gmail client

Requirements

What you need to get started:

A domain (and access to change its DNS records) (my domain is registered with Cloudflare in this case)
A AWS account/user with access to AWS SES, S3, and Lambda
A Gmail account (for the Gmail client)

Before you use/create any service in your AWS account, make sure your AWS is set to the right region.

Connect Your Domain with AWS SES

AWS SES can only send an email from domain that is registered and verified in the SES identities list.

Add your domain to AWS SES Identities

In the AWS SES dashboard, go to the ‘identities’ tab and create a new identity.

Input your domain name, and check the custom MAIL FROM domain and input a your subdomain.

If your domain is registered with AWS Route53, you should probably check the publish DNS records to Route53 to let AWS update the domain record automatically. I skipped the advanced DKIM settings and the tags information. Afterward you should be taken to this screen.
Updating your DNS record From the previous screen, scroll down and you will find the DNS record you need to put to your domain’s DNS configuration. My domain is registered with Cloudflare, but as long as you have access to change the DNS record, any domain registrars should work. You will need to add 3 groups of DNS record to your DNS record: DKIM records, MAIL FROM record, and DMARC record. I am going to assume you won’t have any problem with adding the DNS record to your domain, but here is the AWS Guide if you need it.

The only thing you might need to watch out for is probably the MX record value. You see the ‘10’ in the MX record value, that is not part of the value but the MX record priority;

Make sure you do not insert the ‘10’ into the record value, but insert it into the MX record priority like so.
Check back to AWS SES After you updated your DNS records, give it a while and check back to AWS SES, if everything went alright you should see identity status verified.

Setup Email Receiving

Your domain is now registered with AWS SES, but you cannot receive or send email yet. To receive emails on AWS SES, you need to once again update your DNS record. AWS Documentation Page

Adding an MX record for email receiving to your domain

Add another MX record to your DNS record. Insert the root domain name as the MX record’s name and insert the value.

inbound-smtp.region.amazonaws.com

You can find the region value from the region selector on the top bar in AWS.

replace the region with your AWS SES region, and you’re supposed to set the priority to 10, but I set it to 20 to avoid possible conflict with the other MX record. So in my case it looks like this in Cloudflare.
Receive and store email to S3 bucket

Open the Email Receiving dashboard on SES and create a new rule set. Name it something you will understand when you read it later.

In the rule set create a new rule, and name it clearly. The TLS and virus scan options are up to you. Then fill the recipient conditions with the address you want SES to handle for you. For example if you want to send every email addressed to “admin@yourdomain.com” to an S3 bucket, you would add “admin@yourdomain.com” to the recipient conditions. In the next step add a new action to deliver to S3 bucket, and pick an S3 bucket or create a new one. For now disable the message encryption.

Finish the rule creation and make sure the ruleset is enabled.
Test the receiving setup

Now you should be able to receive and view email sent to AWS SES. Try sending an email from your Gmail to one of your SES recipient value. If everything is setup correctly, you should see a new entry in the S3 bucket you selected in the previous step.

Setup AWS Lambda to Forward Email to Your Gmail

If your SES is still in sandbox mode, you may need to add your email to AWS SES Identities so AWS SES can send to your email.

Now that AWS SES can store your email to S3 bucket, the next step is to forward that email to your Gmail address, so you can read emails sent to you in your Gmail.

Go to AWS Lambda and create a new function using node 20.x or node 18.x To make the code works with Node 20.x, change the Lambda file extension from ‘index.mjs’ to ‘index.js’.

AWS Lambda rename index file

For the function body, I used a slightly modified code from AWS Lambda SES Forwarder by Joe Turgeon.

The modified code I used can be found in this GitHub Gist link

Show AWS Lambda Code Here

// Original code by user https://github.com/arithmetric
// Original code https://github.com/arithmetric/aws-lambda-ses-forwarder/blob/master/index.js

"use strict";

// Import the required AWS SDK v3 clients and commands.
const { SESClient, SendRawEmailCommand } = require("@aws-sdk/client-ses");
const {
  S3Client,
  CopyObjectCommand,
  GetObjectCommand,
} = require("@aws-sdk/client-s3");

console.log("AWS Lambda SES Forwarder // @arithmetric // Version 5.1.0");

// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
//   trailing slash.
//
// - allowPlusSign: Enables support for plus sign suffixes on email addresses.
//   If set to `true`, the username/mailbox part of an email address is parsed
//   to remove anything after a plus sign. For example, an email sent to
//   `example+test@example.com` would be treated as if it was sent to
//   `example@example.com`.
//
// - forwardMapping: Object where the key is the lowercase email address from
//   which to forward and the value is an array of email addresses to which to
//   send the message.
//
//   To match all email addresses on a domain, use a key without the name part
//   of an email address before the "at" symbol (i.e. `@example.com`).
//
//   To match a mailbox name on all domains, use a key without the "at" symbol
//   and domain part of an email address (i.e. `info`).
//
//   To match all email addresses matching no other mapping, use "@" as a key.
var defaultConfig = {
  fromEmail: "noreply@example.com",
  subjectPrefix: "",
  emailBucket: "s3-bucket-name",
  emailKeyPrefix: "emailsPrefix/",
  allowPlusSign: true,
  forwardMapping: {
    "info@example.com": [
      "example.john@example.com",
      "example.jen@example.com",
    ],
    "abuse@example.com": ["example.jim@example.com"],
    "@example.com": ["example.john@example.com"],
    info: ["info@example.com"],
  },
};

/**
 * Parses the SES event record provided for the `mail` and `recipients` data.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.parseEvent = async function (data) {
  if (
    !data.event ||
    !data.event.Records ||
    data.event.Records.length !== 1 ||
    !data.event.Records[0].eventSource ||
    data.event.Records[0].eventSource !== "aws:ses" ||
    data.event.Records[0].eventVersion !== "1.0"
  ) {
    data.log({
      message: "parseEvent() received invalid SES message:",
      level: "error",
      event: JSON.stringify(data.event),
    });
    throw new Error("Error: Received invalid SES message.");
  }

  data.email = data.event.Records[0].ses.mail;
  data.recipients = data.event.Records[0].ses.receipt.recipients;
  return data;
};

/**
 * Transforms the original recipients to the desired forwarded destinations.
 */
exports.transformRecipients = async function (data) {
  let newRecipients = [];
  data.originalRecipients = data.recipients;

  data.recipients.forEach(function (origEmail) {
    let origEmailKey = origEmail.toLowerCase();
    if (data.config.allowPlusSign) {
      origEmailKey = origEmailKey.replace(/\+.*?@/, "@");
    }

    if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
      newRecipients = newRecipients.concat(
        data.config.forwardMapping[origEmailKey]
      );
      data.originalRecipient = origEmail;
    } else {
      let origEmailDomain, origEmailUser;
      let pos = origEmailKey.lastIndexOf("@");
      if (pos === -1) {
        origEmailUser = origEmailKey;
      } else {
        origEmailDomain = origEmailKey.slice(pos);
        origEmailUser = origEmailKey.slice(0, pos);
      }

      if (
        origEmailDomain &&
        data.config.forwardMapping.hasOwnProperty(origEmailDomain)
      ) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailDomain]
        );
        data.originalRecipient = origEmail;
      } else if (
        origEmailUser &&
        data.config.forwardMapping.hasOwnProperty(origEmailUser)
      ) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailUser]
        );
        data.originalRecipient = origEmail;
      } else if (data.config.forwardMapping.hasOwnProperty("@")) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping["@"]
        );
        data.originalRecipient = origEmail;
      }
    }
  });

  if (!newRecipients.length) {
    data.log({
      message:
        "Finishing process. No new recipients found for " +
        "original destinations: " +
        data.originalRecipients.join(", "),
      level: "info",
    });
    return data.callback();
  }

  data.recipients = newRecipients;
  return data;
};

/**
 * Fetches the message data from S3.
 */
exports.fetchMessage = async function (data) {
  data.log({
    level: "info",
    message:
      "Fetching email at s3://" +
      data.config.emailBucket +
      "/" +
      data.config.emailKeyPrefix +
      data.email.messageId,
  });

  const s3Client = new S3Client();

  try {
    await s3Client.send(
      new CopyObjectCommand({
        Bucket: data.config.emailBucket,
        CopySource:
          data.config.emailBucket +
          "/" +
          data.config.emailKeyPrefix +
          data.email.messageId,
        Key: data.config.emailKeyPrefix + data.email.messageId,
        ACL: "private",
        ContentType: "text/plain",
        StorageClass: "STANDARD",
      })
    );

    const result = await s3Client.send(
      new GetObjectCommand({
        Bucket: data.config.emailBucket,
        Key: data.config.emailKeyPrefix + data.email.messageId,
      })
    );

    const streamToString = (stream) => {
      return new Promise((resolve, reject) => {
        const chunks = [];
        stream.on("data", (chunk) => chunks.push(chunk));
        stream.on("error", reject);
        stream.on("end", () =>
          resolve(Buffer.concat(chunks).toString("utf-8"))
        );
      });
    };

    data.emailData = await streamToString(result.Body);
  } catch (err) {
    data.log({
      level: "error",
      message: "getObject() returned error:",
      error: err,
      stack: err.stack,
    });
    throw new Error("Error: Failed to load message body from S3.");
  }

  return data;
};

/**
 * Processes the message data, making updates to recipients and other headers before forwarding.
 */
exports.processMessage = async function (data) {
  const match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
  let header = match && match[1] ? match[1] : data.emailData;
  const body = match && match[2] ? match[2] : "";

  // Add "Reply-To:" with the "From" address if it doesn't already exist
  if (!/^reply-to:[\t ]?/im.test(header)) {
    const fromMatch = header.match(/^from:[\t ]?(.*(?:\r?\n\s+.*)*\r?\n)/im);
    const from = fromMatch && fromMatch[1] ? fromMatch[1] : "";
    if (from) {
      header = header + "Reply-To: " + from;
      data.log({
        level: "info",
        message: "Added Reply-To address of: " + from,
      });
    }
  }

  // Replace "From:" header with verified domain or original recipient
  header = header.replace(
    /^from:[\t ]?(.*(?:\r?\n\s+.*)*)/gim,
    function (match, from) {
      const fromText = data.config.fromEmail
        ? `From: ${from.replace(/<(.*)>/, "").trim()} <${data.config.fromEmail}>`
        : `From: ${from.replace("<", "at ").replace(">", "")} <${data.originalRecipient}>`;
      return fromText;
    }
  );

  // Add subject prefix
  if (data.config.subjectPrefix) {
    header = header.replace(/^subject:[\t ]?(.*)/gim, (match, subject) => {
      return `Subject: ${data.config.subjectPrefix}${subject}`;
    });
  }

  // Replace "To" header with manually defined one
  if (data.config.toEmail) {
    header = header.replace(
      /^to:[\t ]?(.*)/gim,
      `To: ${data.config.toEmail}`
    );
  }

  // Remove certain headers
  const headersToRemove = [
    "return-path:",
    "sender:",
    "message-id:",
    "dkim-signature:",
  ];
  headersToRemove.forEach((headerToRemove) => {
    const regex = new RegExp(
      `^${headerToRemove}[\\t ]?.*\\r?\\n(\\s+.*\\r?\\n)*`,
      "mgi"
    );
    header = header.replace(regex, "");
  });

  data.emailData = header + body;
  return data;
};

/**
 * Send email using the SES sendRawEmail command.
 */
exports.sendMessage = async function (data) {
  const sesClient = new SESClient();

  const params = {
    Destinations: data.recipients,
    Source: data.originalRecipient,
    RawMessage: {
      Data: data.emailData,
    },
  };

  data.log({
    level: "info",
    message:
      "sendMessage: Sending email via SES. Original recipients: " +
      data.originalRecipients.join(", ") +
      ". Transformed recipients: " +
      data.recipients.join(", ") +
      ".",
  });

  try {
    const result = await sesClient.send(new SendRawEmailCommand(params));
    data.log({
      level: "info",
      message: "sendRawEmail() successful.",
      result: result,
    });
  } catch (err) {
    data.log({
      level: "error",
      message: "sendRawEmail() returned error.",
      error: err,
      stack: err.stack,
    });
    throw new Error("Error: Email sending failed.");
  }

  return data;
};

/**
 * Handler function to be invoked by AWS Lambda with an inbound SES email as
 * the event.
 */
exports.handler = async function (event, context, callback) {
  const steps = [
    exports.parseEvent,
    exports.transformRecipients,
    exports.fetchMessage,
    exports.processMessage,
    exports.sendMessage,
  ];

  let data = {
    event: event,
    context: context,
    callback: callback,
    config: defaultConfig,
    log: function (entry) {
      entry.level = entry.level || "info";
      entry.message = entry.message || "";
      if (entry.level !== "debug" || data.config.debug) {
        console.log("LOG " + entry.level + ": " + entry.message);
      }
    },
  };

  for (let step of steps) {
    try {
      await step(data);
    } catch (error) {
      data.log({
        level: "error",
        message: "Step failed: " + step.name,
        error: error,
      });
      return callback(error);
    }
  }
};

Also, don’t forget to change the defaultConfig values to match yours. Please read the comment on top of the variable for information about each field value.

For the fromEmail, you may need to set it to your FROM EMAIL subdomain address.

var defaultConfig = {
  fromEmail: "admin@mail.loopzoop.win",
  subjectPrefix: "",
  emailBucket: "loopzoop-emails",
  emailKeyPrefix: "incoming-emails/",
  allowPlusSign: true,
  forwardMapping: {
    "admin@loopzoop.win": ["your_email@gmail.com"],
  },
};

Set up Lambda permission to use S3 and SES

After setting up the Lambda function, you will need to allow that Lambda instance to access S3 to read the email data and access to SES to send raw email. Go to the configuration tab and go to the permission section. Click on the role name to get redirected to IAM.

In the IAM create a new inline policy.

There are two ways to create the policy, using a visual editor or a JSON editor. I am using the visual editor for this. First add S3 to the services and add persmission for ‘GetObject’ and ‘PutObject’.

Then for the resources, limit the access to only the specific S3 bucket that stores the incoming SES email, make sure there is a wildcard (’*’) to the end of the ARN string.

Then add SES to the services list and add the ‘SendRawEmail’ permission,

for the resources limit the access to your specific AWS SES identity.

You can find the ARN string for your SES identity in the Identities dashboard in SES

Review and save your changes.
Add Lambda invocation to the SES receiving rule action After creating the Lambda function and adding the necessary permission, go back to SES receiving rule and edit the existing rule to add an action to it. Add the Lambda function you just created, then save it.

At this point AWS may ask you to grant persmission for SES to access the Lambda resource
Test the email forwarding Now if I didn’t miss any step while writing this, any email you send to your AWS SES address will be forwarded to the email of your choice.

If your SES is still in sandbox mode, you will need to add your personal email to SES Identity. In sandbx mode SES will not send email to address that is not in the identity list

Setup SMTP Receive email from Gmail

Now (hopefully) your SES instance could receive and forward emails to your Gmail. The next step is to setup SMTP between SES and your Gmail client, so you can send email using your custom domain from Gmail. This part should be the easiest.

Create a new SMTP credentials

Go to SMTP settings in your SES dashboard and create a new SMTP credentials. Again, name it something clear that you will understand later.

Make sure to download the SMTP credential .csv file.
Input your SMTP details to your Gmail client

In Gmail web, go to settings and open up all settings and open the ‘Accounts and Import’ section.

A new popup window will show up, in the popup window fill all fields needed. The email address, is the address where the email will be sent from.

Then fill up the rest of the fields with details of your SMTP credentials.
Verify your email

After filling up all the details, now Google should send a verification email to your SES. If the previous steps are working, the verification email should be forwarded to your Gmail inbox (please check your spam folder, it might end up there) or you can just directly check from your S3 bucket content.
Try sending out an email

Now try sending out an email using your custom domain address. You should probably try sending an email to an account you own first to make sure everything is working.

If your SES is still in sandbox mode, you need to add the email you want to send to to the SES Identities list.

Summary

Hopefully now you shold be able to send and receive email using your custom domain through Gmail using SES.

If there any steps I missed or something is wrong with these steps, please leave a comment, and I will try to amend this post.

~ Thank you for reading