Laravel and Amazon SQS Queues

In this lesson, we will explore how to create and configure queues using Amazon Simple Queue Service (SQS). By the end of this lesson, you will understand how Amazon SQS works and how it can be used with Laravel workers.

The benefits of Amazon SQS are high scalability, thus it can handle a virtually unlimited number of messages per second. This makes it ideal for scenarios where the workload is expected to grow rapidly. Also, it is cost-effective as pricing is based on usage, which means that you only pay for the messages you send and receive.

We can look up pricing in this table:

Few things to note:

• Every Amazon SQS action counts as a request, this means that send/receive/delete/change visibility actions are also counted as a request.
• Each 64 KB chunk of a message payload is billed as 1 request (for example, an API action with a 256 KB payload is billed as 4 requests). This is especially important if you have jobs that require a lot of data and it gets more complicated.

Along pricing Amazon also provides the Calculator so you can measure your monthly estimated costs.

1. Requirements

To consume AWS services such as Amazon SQS first we need to install the AWS SDK package using composer:

composer require aws/aws-sdk-php:~3.0

2. Setting up Amazon SQS

2.1 Create IAM User

1. To get credentials for our Laravel app first we need to create IAM (Identity and Access Management) User. Navigate to the Security Credentials page. The menu can be found in the top right corner by pressing your account name.

2. Then press Add Users button.

3. Enter the user name and press Next

4. We do not need to set any permissions. Access permissions will be configured while creating a queue. Keep defaults and press Next

5. Click Create User to finish the process.

6. Now in the users list press on the user name you just created.

7. In the user's summary window copy ARN (Amazon Resource Number) for this user. It has the following form arn:aws:iam::****:user/**** and will be required when setting send/receive permissions for the queue messages.

8. Now below locate the Security credentials tab and press on it.

9. In the Access Keys section press Create access key. This will create credentials for our Laravel app to have access to AWS services.

10. Select Application running outside AWS option and press Next

11. Optionally you can set a description tag for this key. Then press the Create access key button.

12. Now save both Access key and Secret access key and press Done. Optionally you can download credentials by pressing the Download .csv file button.

Secret access key is displayed only once in this view, so if you fail to save that key you need to create new keys.

2.2 Create Queue

1. Now navigate to Simple Queue Service management. This can be found by typing sqs in the top search bar.

2. Before proceeding with creating a queue, it is important to pick your region where SQS will be hosted. It is recommended to pick the closest location to your server. Region name (on the right side of the list) will be required for later configuration, so please note that for example eu-central-1.

3. After selecting a region you will be presented with a placeholder on the page to get started. Press the Create queue button.

4. Fill in Name for your queue and note it down. For this example, we chose the name mailing to send user verification emails.

A queue name is case-sensitive and can have up to 80 characters. You can use alphanumeric characters, hyphens (-), and underscores ( _ ).

Now let's briefly discuss queue Types.

• Standard queue doesn't offer message de-duplication and it will behave like any other Laravel queue, and if you send the same job more than once, it will be processed multiple times as you would expect. It is important to note that there's no guarantee that jobs will be processed in the same order as they were dispatched. Amazon SQS will attempt to process them in the order they are received however it is not always possible because standard queues are designed to be massively scalable using a highly distributed architecture.

• FIFO (first-in-first-out) queues preserve the exact order in which messages are sent and received. Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues avoid sending duplicates to a queue.

5. Let's move to the configuration section:

Visibility Timeout

Amazon SQS is the only queue connection that does not contain the retry_after (timeout) value. On Amazon SQS it is called Visibility timeout

Visibility timeout value sets the length of time that a message received from a queue will not be visible to other workers. This timeout starts when Amazon SQS returns a message. If the job fails to process and delete the message before the visibility timeout expires, the message becomes visible to other consumers (workers) and this is applied to all messages.

Typically you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue.

For example, if the visibility timeout value is set to 30 seconds and it takes more than 30 seconds to process a job, another queue worker will start processing it again, so when configuring the supervisor stopwaitsecs value must be not higher than the visibility timeout.

Other values can be left at defaults.

6. On the Access policy section pick Basic criteria to define access. For both send and receive policies select Only the specified AWS accounts, IAM users, and roles, and in the text area fields fill in the ARN of the IAM User we created in part 2.1. It has the following format arn:aws:iam::****:user/****.

7. Now scroll to the bottom and press the Create queue button.

8. You will be greeted with the success alert and details page. Save the URL of the queue in the second column for later configuration. It has the following format https://sqs.<REGION>.amazonaws.com/<IAM-USER-ID>/<QUEUE-NAME>

2.3 Laravel configuration

It is time to update the environment file with the credentials and SQS values for a queue we just created.

SQS_PREFIX should have a Queue URL without the last segment, which means we don't include the last part of the URL with the queue name /mailing. SQS_QUEUE is where we put the queue name.

.env

QUEUE_CONNECTION=sqs
 
//...
 
AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY-ID>
AWS_SECRET_ACCESS_KEY=<YOUR-SECRET-ACCESS-KEY>
AWS_DEFAULT_REGION=<YOUR-REGION>
SQS_PREFIX=<YOUR-QUEUE-URL-WITHOUT-LAST-SEGMENT>
SQS_QUEUE=<YOUR-QUEUE-NAME>

2.4 Supervisor workers and SQS

Finally, we can write supervisor configuration to run SQS queues.

For Ubuntu users - /etc/supervisor/conf.d/amazon-sqs.conf For Fedora users - /etc/supervisord.d/amazon-sqs.ini

[program:amazon-sqs]
directory=/home/web/laravel-queues
command=php artisan queue:work
 
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=web
numprocs=10
redirect_stderr=true
stdout_logfile=/home/web/.supervisor/amazon-sqs.log
stopwaitsecs=30

Save the file and then update the supervisor configuration with this command:

sudo supervisorctl update

Try registering a new account, and a verification email should be delivered.

Notes

It is possible to specify queue connection while invoking the queue:work command like this instead of the .env file.

command=php artisan queue:work sqs

This also doesn't limit you to only one SQS queue, you can have multiple configurations of worker groups on the supervisor for each queue with different commands like:

command=php artisan queue:work sqs --queue=mailing

command=php artisan queue:work sqs --queue=another-queue-name