
=encoding utf8

=head1 NAME

Minion::Guide - An introduction to Minion

=head1 OVERVIEW

This document contains an introduction to L<Minion> and explains the most important features it has to offer.

=head1 INTRODUCTION

Essentials every L<Minion> developer should know.

=head2 Job queue

Job queues allow you to process time and/or computationally intensive tasks in background processes, outside of the
request/response lifecycle of web applications. Among those tasks you'll commonly find image resizing, spam filtering,
HTTP downloads, building tarballs, warming caches and basically everything else you can imagine that's not super fast.

  Mojo::Server::Prefork                              +--------------+                     Minion::Worker
  |- Mojo::Server::Daemon [1]       enqueue job ->   |              |   -> dequeue job    |- Minion::Job [1]
  |- Mojo::Server::Daemon [2]                        |  PostgreSQL  |                     |- Minion::Job [2]
  |- Mojo::Server::Daemon [3]   retrieve result <-   |              |   <- store result   |- Minion::Job [3]
  +- Mojo::Server::Daemon [4]                        +--------------+                     |- Minion::Job [4]
                                                                                          +- Minion::Job [5]

They are not to be confused with time based job schedulers, such as cron or systemd timers, but L<Minion> has built in
support for cron style L</"Recurring jobs"> when you want a task to run on a wall clock schedule.

=head2 Mojolicious

You can use L<Minion> as a standalone job queue or integrate it into L<Mojolicious> applications with the plugin
L<Mojolicious::Plugin::Minion>.

  use Mojolicious::Lite -signatures;

  plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};

  # Slow task
  app->minion->add_task(poke_mojo => sub ($job, @args) {
    $job->app->ua->get('mojolicious.org');
    $job->app->log->debug('We have poked mojolicious.org for a visitor');
  });

  # Perform job in a background worker process
  get '/' => sub ($c) {
    $c->minion->enqueue('poke_mojo');
    $c->render(text => 'We will poke mojolicious.org for you soon.');
  };

  app->start;

Background worker processes are usually started with the command L<Minion::Command::minion::worker>, which becomes
automatically available when an application loads L<Mojolicious::Plugin::Minion>.

  $ ./myapp.pl minion worker

The worker process will fork a new process for every job that is being processed. This allows for resources such as
memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the
overhead.

  Minion::Worker
  |- Minion::Job [1]
  |- Minion::Job [2]
  +- ...

By default up to four jobs will be processed in parallel, but that can be changed with configuration options or on
demand with signals.

  $ ./myapp.pl minion worker -j 12

Jobs can be managed right from the command line with L<Minion::Command::minion::job>, and recurring schedules with
L<Minion::Command::minion::schedule>.

  $ ./myapp.pl minion job
  $ ./myapp.pl minion schedule

You can also add an admin ui to your application by loading the plugin L<Mojolicious::Plugin::Minion::Admin>. Just make
sure to secure access before making your application publicly accessible.

  # Make admin ui available under "/minion"
  plugin 'Minion::Admin';

=head2 Deployment

To manage background worker processes with systemd, you can use a unit configuration file like this.

  [Unit]
  Description=My Mojolicious application workers
  After=postgresql.service

  [Service]
  Type=simple
  ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
  KillMode=process

  [Install]
  WantedBy=multi-user.target

=head2 Consistency

Every new job starts out as C<inactive>, then progresses to C<active> when it is dequeued by a worker, and finally ends
up as C<finished> or C<failed>, depending on its result. Every C<failed> job can then be retried to progress back to the
C<inactive> state and start all over again.

                                                      +----------+
                                                      |          |
                                             +----->  | finished |
   +----------+            +--------+        |        |          |
   |          |            |        |        |        +----------+
   | inactive |  ------->  | active |  ------+
   |          |            |        |        |        +----------+
   +----------+            +--------+        |        |          |
                                             +----->  |  failed  |  -----+
        ^                                             |          |       |
        |                                             +----------+       |
        |                                                                |
        +----------------------------------------------------------------+

The system is eventually consistent and will preserve job results for as long as you like, depending on
L<Minion/"remove_after">. But be aware that C<failed> results are preserved indefinitely, and need to be manually
removed by an administrator if they are out of automatic retries.

While individual workers can fail in the middle of processing a job, the system will detect this and ensure that no job
is left in an uncertain state, depending on L<Minion/"missing_after">. Jobs that do not get processed after a certain
amount of time, depending on L<Minion/"stuck_after">, will be considered stuck and fail automatically. So an admin can
take a look and resolve the issue.

=head1 FEATURES

L<Minion> has many great features. This section is still very incomplete, but will be expanded over time.

=head2 Priorities

Every job enqueued with L<"enqueue" in Minion|Minion/"enqueue1"> has a priority. Jobs with a higher priority get
performed first, the default priority is C<0>. Priorities can be positive or negative, but should be in the range
between C<100> and C<-100>.

  # Default priority
  $minion->enqueue('check_links', ['https://mojolicious.org']);

  # High priority
  $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => 30});

  # Low priority
  $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => -30});

You can use L<Minion::Job/"retry"> to raise or lower the priority of a job.

  $job->retry({priority => 50});

=head2 Job results

The result of a job has two parts. First there is its state, which can be C<finished> for a successfully processed job,
and C<failed> for the opposite. And second there's a C<result> data structure, that may be C<undef>, a scalar, a hash
reference, or an array reference. You can check both at any time in the life cycle of a job with L<Minion/"job">, all
you need is the job id.

  # Check job state
  my $state = $minion->job($job_id)->info->{state};

  # Get job result
  my $result = $minion->job($job_id)->info->{result};

While the C<state> will be assigned automatically by L<Minion>, the C<result> for C<finished> jobs is usually assigned
manually with L<"finish" in Minion::Job|Minion::Job/"finish1">.

  $minion->add_task(job_with_result => sub ($job) {
    sleep 5;
    $job->finish({message => 'This job should have taken about 5 seconds'});
  });

For jobs that C<failed> due to an exception, that exception will be assigned as C<result>.

  $minion->add_task(job_that_fails => sub ($job) {
    sleep 5;
    die 'This job should always fail after 5 seconds';
  });

But jobs can also fail manually with L<Minion::Job/"fail">.

  $minion->add_task(job_that_fails_with_result => sub ($job) {
    sleep 5;
    $job->fail({errors => ['This job should fail after 5 seconds']});
  });

Retrieving job results is of course completely optional, and it is very common to have jobs where the result is
unimportant.

=head2 Named queues

Each job can be enqueued with L<"enqueue" in Minion|Minion/"enqueue1"> into arbitrarily named queues, independent of all
their other properties. This is commonly used to have separate classes of workers, for example to ensure that free
customers of your web service do not negatively affect your service level agreements with paying customers. The default
named queue is C<default>, but aside from that it has no special properties.

  # Use "default" queue
  $minion->enqueue('check_links', ['https://mojolicious.org']);

  # Use custom "important" queue
  $minion->enqueue('check_links', ['https://mojolicious.org'], {queue => 'important'});

For every named queue you can start as many workers as you like with the command L<Minion::Command::minion::worker>. And
each worker can process jobs from multiple named queues. So your workers can have overlapping responsibilities.

  $ ./myapp.pl minion worker -q default -q important

There is one special named queue called C<minion_foreground> that you should avoid using directly. It is reserved for
debugging jobs with L<Minion/"foreground">.

=head2 Job progress

Progress information and other job metadata can be stored in notes at any time during the life cycle of a job with
L<Minion::Job/"note">. The metadata can be arbitrary data structures constructed with scalars, hash references and array
references.

  $minion->add_task(job_with_progress => sub ($job) {
    sleep 1;
    $job->note(progress => '25%');
    sleep 1;
    $job->note(progress => '50%');
    sleep 1;
    $job->note(progress => '75%');
    sleep 1;
    $job->note(progress => '100%');
  });

Notes, similar to job results, can be retrieved with L<Minion/"job">, all you need is the job id.

  # Get job metadata
  my $progress = $minion->job($job_id)->info->{notes}{progress};

You can also use notes to store arbitrary metadata with new jobs when you create them with
L<"enqueue" in Minion|Minion/"enqueue1">.

  # Create job with metadata
  $minion->enqueue('job_with_progress', [], {notes => {progress => 0, something_else => [1, 2, 3]}});

The admin ui provided by L<Mojolicious::Plugin::Minion::Admin> allows searching for jobs containing a certain note, so
you can also use them to tag jobs.

=head2 Delayed jobs

The C<delay> option of L<"enqueue" in Minion|Minion/"enqueue1"> can be used to delay the processing of a job by a
certain amount of seconds (from now).

  # Job will not be processed for 60 seconds
  $minion->enqueue('check_links', ['https://mojolicious.org'], {delay => 20});

You can use L<Minion::Job/"retry"> to change the delay.

  $job->retry({delay => 10});

=head2 Recurring jobs

For jobs that should run on a wall clock schedule, use L<Minion::Command::minion::schedule> with a cron expression.
Schedules are stored in the database as the source of truth, so they survive restarts and only fire once per interval
no matter how many workers are running.

  # Daily cleanup at 4am
  $ ./myapp.pl minion schedule -e daily_cleanup -c '0 4 * * *' -t cleanup

A cron expression is five fields separated by spaces, one for each part of the wall clock. All times are interpreted
in B<UTC>, not local time.

  * * * * *
  | | | | |
  | | | | +---- day of week  (0-6, 0=Sun; or SUN-SAT; 7 also means Sun)
  | | | +------ month        (1-12; or JAN-DEC)
  | | +-------- day of month (1-31)
  | +---------- hour         (0-23)
  +------------ minute       (0-59)

A C<*> means "every value", so C<* * * * *> reads as "every minute of every hour, every day". Pin a field to a
specific number to fix that part of the clock.

  0 4 * * *       # 04:00 every day
  30 9 * * 1      # 09:30 every Monday
  0 0 1 1 *       # midnight on January 1st (once a year)

Use C</N> for "every Nth value", C<a-b> for ranges, and commas for lists.

  */15 * * * *    # every 15 minutes
  0 9-17 * * *    # at the top of every hour from 09:00 to 17:00
  0 9 * * 1-5     # 09:00 every weekday
  0 0,12 * * *    # midnight and noon

Common patterns also have nicknames that expand to one of the forms above.

  @hourly         # 0 * * * *
  @daily          # 0 0 * * *
  @midnight       # 0 0 * * *
  @weekly         # 0 0 * * 0
  @monthly        # 0 0 1 * *
  @yearly         # 0 0 1 1 *

When both C<day-of-month> and C<day-of-week> are restricted, the job fires when B<either> matches, following the
standard Vixie L<cron(8)> behavior. So C<0 0 1 * 0> means "midnight on the 1st of the month B<and> every Sunday at
midnight". Times are interpreted in UTC.

Arguments, priorities, queues and retries can be set per schedule with the same options as
L<Minion::Command::minion::job>.

  # Every five minutes, with arguments and high priority
  $ ./myapp.pl minion schedule -e refresh -c '*/5 * * * *' -t refresh_cache -a '["users"]' -p 5

  # Weekdays at 9, with retries
  $ ./myapp.pl minion schedule -e weekday_report -c '0 9 * * 1-5' -t report -A 3

Adding a schedule with an existing name updates it; reusing the same cron expression preserves the next firing time,
changing it recomputes it. Invalid expressions are rejected immediately.

  # Pause and resume schedules without removing them
  $ ./myapp.pl minion schedule -P daily_cleanup
  $ ./myapp.pl minion schedule -r daily_cleanup

  # Show or list schedules
  $ ./myapp.pl minion schedule
  $ ./myapp.pl minion schedule daily_cleanup

  # Remove a schedule
  $ ./myapp.pl minion schedule -R daily_cleanup

Schedules can also be declared in code with L<"schedule" in Minion|Minion/"schedule"> and managed from the admin UI.

  $minion->schedule(daily_cleanup => '0 4 * * *' => 'cleanup');

Running workers tick L<"dispatch_schedules" in Minion|Minion/"dispatch_schedules"> every C<dispatch_interval> seconds
(default C<30>) to enqueue any due jobs, coordinated through a Postgres advisory lock so a single dispatch cycle never
fires the same schedule twice. If no worker has been running, the dispatcher will not catch up missed runs; the next
firing time is always the next match strictly after the current time.

=head2 Expiring jobs

The C<expire> option of L<"enqueue" in Minion|Minion/"enqueue1"> can be used to limit for how many seconds (from now) a
job should be valid before it expires and gets deleted from the queue.

  # Job will vanish if it is not dequeued within 60 seconds
  $minion->enqueue('check_links', ['https://mojolicious.org'], {expire => 60});

You can use L<Minion::Job/"retry"> to reset the expiration time.

  $job->retry({expire => 30});

=head2 Job dependencies

The C<parents> option of L<"enqueue" in Minion|Minion/"enqueue1"> can be used to make a new job depend on
one or more existing jobs, which all need to have reached the state C<finished> before it can be processed.

  # Second job will not be processed until first job has finished
  my $first = $minion->enqueue('step_one');
  $minion->enqueue('step_two', [], {parents => [$first]});

This is commonly used to build workflows where tasks need to run in a specific order.

  my $a = $minion->enqueue('prepare');
  my $b = $minion->enqueue('process', [], {parents => [$a]});
  my $c = $minion->enqueue('cleanup', [], {parents => [$b]});

If a parent job transitions to the C<failed> state, all dependent jobs will remain C<inactive> and wait until
it has been retried successfully or removed. You can use the C<lax> option to allow a job to be processed
even if its parents have C<failed>.

  # Job will be processed even if parent jobs have failed
  $minion->enqueue('cleanup', [], {parents => [$id], lax => 1});

=head2 Automatic retries

The C<attempts> option of L<"enqueue" in Minion|Minion/"enqueue1"> can be used to allow a job to be retried
automatically if it fails, up to the specified number of attempts.

  # Job will be attempted up to five times
  $minion->enqueue('check_links', ['https://mojolicious.org'], {attempts => 5});

The delay between retries is determined by L<Minion/"backoff">, which defaults to C<(retries ** 4) + 15>
(15, 16, 31, 96, 271, 640...) seconds, allowing roughly 25 attempts to be made over 21 days. You can change
the formula by providing your own callback.

  # Add some randomness to spread retry load
  $minion->backoff(sub ($retries) { ($retries ** 4) + 15 + int(rand 30) });

=head2 Named locks

L<Minion/"lock"> and L<Minion/"unlock"> can be used to limit the concurrency of certain tasks, or to
implement rate limiting for access to external services.

  # Only one job should run at a time
  $minion->add_task(do_unique_stuff => sub ($job, @args) {
    return $job->finish('Previous job is still active')
      unless $minion->lock('fragile_backend_service', 7200);
    ...
    $minion->unlock('fragile_backend_service');
  });

The second argument is the number of seconds after which the lock will expire automatically, in case the job
fails to release it. An optional C<limit> argument can be used to allow more than one lock with the same name
to be active at the same time.

  # Only five jobs should run at a time and we try again later if necessary
  $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
    return $job->retry({delay => 30})
      unless $minion->lock('some_web_service', 60, {limit => 5});
    ...
    $minion->unlock('some_web_service');
  });

For convenience you can also use L<Minion/"guard"> to release the lock automatically as soon as the job is
finished, even if it failed.

  $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
    return $job->retry({delay => 30})
      unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
    ...
  });

=head2 Result promises

L<Minion/"result_p"> can be used to wait for the result of a job asynchronously, without blocking the event
loop. It returns a L<Mojo::Promise> object that will be fulfilled when the job reaches the state C<finished>,
or rejected when it reaches the state C<failed>.

  # Enqueue job and receive the result at some point in the future
  my $id = $minion->enqueue('foo');
  $minion->result_p($id)->then(sub ($info) {
    say "Finished: $info->{result}";
  })->catch(sub ($info) {
    say "Failed: $info->{result}";
  })->wait;

=head2 Foreground debugging

L<Minion/"foreground"> can be used to retry a job in the C<minion_foreground> queue and perform it right
away in the current process, making it very easy to debug a failed job with the full output of your
application available. This is usually done from the command line with L<Minion::Command::minion::job>.

  $ ./myapp.pl minion job -f 10023

=head2 Worker signals

Worker processes can be controlled at runtime with signals. C<INT> and C<TERM> will stop the worker
gracefully after finishing its current jobs, while C<QUIT> will stop it immediately.

  # Stop worker gracefully
  $ kill -INT $worker_pid

Job processes spawned by the worker can also receive signals. C<INT> and C<TERM> start out with the
operating system default, allowing jobs to install custom signal handlers to stop gracefully. C<USR1> and
C<USR2> start out being ignored, and are available for jobs to use as they see fit.

  $minion->add_task(myapp_task => sub ($job, @args) {
    local $SIG{INT} = sub { ...; exit };
    ...
  });

=head2 Remote control

L<Minion/"broadcast"> can be used to send remote control commands to one or more workers. Several commands
are built in, such as C<stop> to kill a job, C<kill> to send a signal to a job process, and C<jobs> to
change how many jobs a worker runs concurrently. This is usually done from the command line with
L<Minion::Command::minion::job>.

  # Kill job 10025
  $ ./myapp.pl minion job -b stop -a '[10025]'

  # Send USR1 signal to job 10026
  $ ./myapp.pl minion job -b kill -a '["USR1", 10026]'

  # Reduce concurrency of worker 23 to two jobs
  $ ./myapp.pl minion job -b jobs -a '[2]' 23

You can also add your own commands to workers with L<Minion::Worker/"add_command">.

  $worker->add_command(my_command => sub ($worker, @args) {
    ...
  });

=head2 Testing

L<Minion/"perform_jobs"> can be used to perform all jobs in the queue with a temporary worker, making it
very easy to test background tasks without having to start a real worker process.

  use Test::More;
  use Mojolicious::Lite -signatures;

  plugin Minion => {Pg => 'postgresql://postgres@/test'};

  app->minion->add_task(add => sub ($job, $a, $b) {
    $job->finish($a + $b);
  });

  my $id = app->minion->enqueue(add => [1, 1]);
  app->minion->perform_jobs;
  is app->minion->job($id)->info->{result}, 2, 'right result';

  done_testing;

=head2 Job iteration

L<Minion/"jobs"> can be used to safely iterate through job information, with a variety of filters to narrow
down the results.

  # Iterate through all failed jobs for a task
  my $jobs = $minion->jobs({states => ['failed'], tasks => ['check_links']});
  while (my $info = $jobs->next) {
    say "$info->{id}: $info->{result}";
  }

This is commonly used to build custom monitoring or to perform bulk operations on jobs.

  # Retry all failed jobs in a queue
  my $jobs = $minion->jobs({states => ['failed'], queues => ['important']});
  while (my $info = $jobs->next) {
    $minion->job($info->{id})->retry;
  }

=head2 Rate limiting

Named locks can also be used to limit how often a job accesses an external service, by letting them expire
naturally instead of releasing them manually with L<Minion/"unlock">.

  # Only a hundred jobs should run per hour
  $minion->add_task(do_rate_limited_stuff => sub ($job, @args) {
    return $job->retry({delay => 3600})
      unless $minion->lock('some_web_service', 3600, {limit => 100});
    ...
  });

Each lock acquired this way will count against the limit for the duration of its expiration time, effectively
capping throughput to C<limit> jobs per expiration period.

=head2 Custom workers

In cases where you don't want to use L<Minion> together with L<Mojolicious>, you can just skip the plugins and write
your own worker scripts.

  #!/usr/bin/perl
  use strict;
  use warnings;

  use Minion;

  # Connect to backend
  my $minion = Minion->new(Pg => 'postgresql://postgres@/test');

  # Add tasks
  $minion->add_task(something_slow => sub ($job, @args) {
    sleep 5;
    say 'This is a background worker process.';
  });

  # Start a worker to perform up to 12 jobs concurrently
  my $worker = $minion->worker;
  $worker->status->{jobs} = 12;
  $worker->run;

The method L<Minion::Worker/"run"> contains all features you would expect from a L<Minion> worker and can be easily
configured with L<Minion::Worker/"status">. For even more customization options L<Minion::Worker> also has a very rich
low level API you could for example use to build workers that do not fork at all.

=head2 Task plugins

As your L<Mojolicious> application grows, you can move tasks into application specific plugins.

  package MyApp::Task::PokeMojo;
  use Mojo::Base 'Mojolicious::Plugin', -signatures;

  sub register ($self, $app, $config) {
    $app->minion->add_task(poke_mojo => sub ($job, @args) {
      $job->app->ua->get('mojolicious.org');
      $job->app->log->debug('We have poked mojolicious.org for a visitor');
    });
  }

  1;

Which are loaded like any other plugin from your application.

  # Mojolicious
  $app->plugin('MyApp::Task::PokeMojo');

  # Mojolicious::Lite
  plugin 'MyApp::Task::PokeMojo';

=head2 Task classes

For more flexibility, or if you are using L<Minion> as a standalone job queue, you can also move tasks into dedicated
classes. Allowing the use of Perl features such as inheritance and roles. But be aware that support for task classes is
still B<EXPERIMENTAL> and might change without warning!

  package MyApp::Task::PokeMojo;
  use Mojo::Base 'Minion::Job', -signatures;

  sub run ($self, @args) {
    $self->app->ua->get('mojolicious.org');
    $self->app->log->debug('We have poked mojolicious.org for a visitor');
  }

  1;

Task classes are registered just like any other task with L<Minion/"add_task"> and you can even register the same class
with multiple names.

  $minion->add_task(poke_mojo => 'MyApp::Task::PokeMojo');

=head1 MORE

You can continue with L<Mojolicious::Guides> now or take a look at the L<Mojolicious
wiki|https://github.com/mojolicious/mojo/wiki>, which contains a lot more documentation and examples by many different
authors.

=head1 SUPPORT

If you have any questions the documentation might not yet answer, don't hesitate to ask in the
L<Forum|https://forum.mojolicious.org> or the official IRC channel C<#mojo> on C<irc.libera.chat>
(L<chat now!|https://web.libera.chat/#mojo>).

=cut
