Handling Failed Jobs

In this lesson, we will discuss what to do and how to handle such cases if the job fails. There can be uncovered a case in logic, the mail server might not respond, etc.

This practical example will continue with the demo project setup from Chapter 1 using the database queue driver.

Now let's update our SendEmailVerificationNotification listener to make it fail 33% of the time to have some failed jobs.

app/Listeners/SendEmailVerificationNotification.php

namespace App\Listeners;
 
use Illuminate\Auth\Events\Registered;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Auth\MustVerifyEmail;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\Jobs\Job;
use Illuminate\Queue\SerializesModels;
 
class SendEmailVerificationNotification implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    public function handle(Registered $event)
    {
        if ($event->user instanceof MustVerifyEmail && ! $event->user->hasVerifiedEmail()) {
            if (rand(0, 2) === 0) {
                throw new \Exception('Error Processing Job', 1);
            }
 
            $event->user->sendEmailVerificationNotification();
        }
    }
}

Then run queue workers without any flags:

php artisan queue:work

We tried to register 5 users and it was a pretty unlucky run.

In our inbox were only 2 emails delivered.

And in the failed_jobs table we can see there are 3 failed jobs.

Manually retrying jobs

The first logical thought would be "Can we just re-run these jobs". Yes, and there is a command for that:

php artisan queue:retry {job_uuid}

php artisan queue:retry 4fc699eb-a235-48e3-8c05-f6fbe36b820f
php artisan queue:retry 9a96c8bb-9dda-4093-a14d-4c97fe7a4015
php artisan queue:retry 8f58cd9e-3bf6-4bd8-9f75-d858f85ca7d8

After issuing that command to three failed jobs we still had one of them failing.

Manually retrying jobs might be very useful in some scenarios if it is some sort of edge case and you need a hotfix.

Let's clean up the failed_jobs table using the queue:flush command and see how we can mitigate this without developer's intervention.

php artisan queue:flush

Automatically Retry Failed Job

We can set the --tries flag on the queue worker to set a number of retries if the job fails and try registering 5 users again.

php artisan queue:work --tries=3

In the queue worker output, we can see that in total we had jobs failing 5 times, but despite that, all users got their emails.

 INFO  Processing jobs from the [default] queue.
 
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 28.92ms FAIL
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 1,994.38ms DONE
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 8.14ms FAIL
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 584.56ms DONE
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 7.26ms FAIL
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 576.43ms DONE
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 8.21ms FAIL
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 574.61ms DONE
App\Listeners\SendEmailVerificationNotification .... RUNNING
App\Listeners\SendEmailVerificationNotification .... 575.11ms DONE

Retrying jobs might help in many cases but also can be not an optimal solution if the remote server has temporary downtime since jobs will be retried in rapid succession.

To avoid the job being retried immediately after a failure we can set the --backoff flag:

php artisan queue:work --tries=3 --backoff=60

Now if the job fails worker will wait for 60 seconds to retry it again.

Forfeit Failed Jobs

Wait, give up on failed jobs? This seems completely opposite of what we tried to do in the previous paragraph. Let me explain.

• We can't retry the same job an infinite amount of times because the queue will be clogged up, maybe the error is more serious and can be resolved only by updating the code.
• Maybe there are network connectivity issues and it is not going to recover in a window we are retrying jobs.
• By the time job will be retried data provided is no longer relevant and we need an updated dataset.

Let's say sending email verification failed even after retrying several times, this issue still can be resolved without intervention just by simply allowing the user to send it again.

Update EmailVerificationNotificationController with the following content to dispatch a job to send an email instead of sending it immediately:

app/Http/Controllers/Auth/EmailVerificationNotificationController.php

namespace App\Http\Controllers\Auth;
 
use App\Http\Controllers\Controller;
use App\Jobs\SendEmailVerification;
use App\Providers\RouteServiceProvider;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
 
class EmailVerificationNotificationController extends Controller
{
    public function store(Request $request): RedirectResponse
    {
        SendEmailVerification::dispatch($request->user());
 
        return back()->with('status', 'verification-link-sent');
    }
}

Monitoring Failed Jobs

It is especially important to monitor failed jobs because when issues occur it is not immediately apparent if something went wrong at all and if not fixed, it can and will negatively affect your product and can cause more problems down the line.

In this section, we will run through some tools to monitor your application if you're not already doing that.

Native: Event Listeners

There might be cases where you want to implement your own logic when a job fails. Laravel provides you with a JobFailed event where you can register your own listeners to do something more complex.

The new listener can be created using the make:listener Artisan command:

php artisan make:listener FailedJobListener

Then register it in EventServiceProvider

app/Providers/EventServiceProvider.php

use Illuminate\Queue\Events\JobFailed;
use App\Listeners\FailedJobListener;
 
// ...
 
protected $listen = [
    // ...
 
    JobFailed::class => [
        FailedJobListener::class
    ]
];

All events of Illuminate Queue can be found on Laravel's API documentation https://laravel.com/api/10.x/Illuminate/Queue/Events.html.

First-Party: Laravel Horizon

Laravel Horizon as covered in Chapter 3 provides reports on failed jobs. We won't cover installation again.

Let's change our queue connection to redis:

.env

QUEUE_CONNECTION=redis

And run Horizon:

php artisan horizon

Horizon provides UI for failed jobs on Failed Jobs tab:

Jobs can be restarted in a convenient way by pressing the retry button on the right side of the table and it will be dispatched again. When the job completes it won't be removed from that list and the retried badge will be added next to its name. Failed jobs expire and will be removed from this list after 7 days.

Details provide how many times the job was attempted, and failed, and a stack trace.

Natively Horizon doesn't provide notifications on failed jobs via email. This can be resolved by adding an event listener to Horizon events like binding any other actions using EventServiceProvider.

app/Providers/EventServiceProvider.php

use Laravel\Horizon\Events\JobFailed;
use App\Jobs\JobFailedListener;
 
// ...
 
protected $listen = [
    // ...
 
    JobFailed::class => [
        JobFailedListener::class
    ]
];

Horizon provides the following events JobDeleted, JobFailed, JobPushed, JobReleased, JobReserved, JobsMigrated, LongWaitDetected, MasterSupervisorDeployed, MasterSupervisorLooped, MasterSupervisorReviving, RedisEvent, SupervisorLooped, SupervisorProcessRestarting, UnableToLaunchProcess, WorkerProcessRestarting.

Free from the community: Spatie

Laravel Failed Job Monitor package is specifically designed for this single task.

You need to set up a mailer for Laravel to send emails

1. It can be installed via the composer command:

composer require spatie/laravel-failed-job-monitor

2. Then you need to publish the configuration.

php artisan vendor:publish --tag=failed-job-monitor-config

3. And update your email address in configuration to get notified by email.

config/failed-job-monitor.php*

// ...
 
'mail' => [
    'to' => ['email@example.com'],
],
 
// ...

The email will provide an exception message, a class that failed, the payload of the job, and a stack trace.

Documentation can be found on the GitHub page https://github.com/spatie/laravel-failed-job-monitor.

Full-blown error logger: BugSnag

BugSnag is a more advanced tool to track your application's health and errors. Although it is not free and provides a 14-day trial plan for free. After registering an account wizard will lead you to set up your project. We quickly run through how essential steps.

Bugsnag will log all errors in your application, not only failed jobs.

1. Install bugsnag/bugsnag-laravel via composer:

composer require "bugsnag/bugsnag-laravel:^2.0"

2. Add your BUGSNAG_API_KEY to your environment file:

.env

BUGSNAG_API_KEY=<YOUR-BUGSNAG-API-KEY>

3. Add Bugsnag\BugsnagLaravel\BugsnagServiceProvider::class before App\Providers\AppServiceProvider::class in your config/app.php file:

config/app.php

/*
 * Package Service Providers...
 */
Bugsnag\BugsnagLaravel\BugsnagServiceProvider::class,
 
/*
 * Application Service Providers...
 */
App\Providers\AppServiceProvider::class,

4. Set up a bugsnag logging channel in config/logging.php, and add it to your logging stack:

config/logging.php

'stack' => [
    'driver'            => 'stack',
    'channels'          => ['single', 'bugsnag'],
    'ignore_exceptions' => false,
],
 
'bugsnag' => [
    'driver' => 'bugsnag',
],

5. If you did everything correctly when a job fails you will get detailed summaries of what and where happened including not only stack trace but also database queries, request variables, etc.

We won't cover there all the possibilities and leave it to explore for yourself.

6. By default BugSnag won't send you any emails, but this can be adjusted to your needs by clicking on the gear icon in the top right corner My email notifications.

Emails will be sent directly from BugSnag servers, so you don't need to set up a mailer in Laravel.

Documentation on possible Laravel integrations can be found on the Official BugSnag Docs page.

There are also many alternatives to BugSnag out in the market like Flare, LaraBug, Sentry, RollBar. They offer different prices and features for different needs.