PHP SimpleQueue

Overview

PHP SimpleQueue is a PHP/MySql based generic queuing mechanism. If you have ever needed to be able to manage tasks by queuing them up and processing them all at once, or if you have ever had a use for a throttling mechanism instead of processing everything at request time, this queue may be just what you're looking for.

By keeping the queue very generic and lightweight, it can be used to house as many different types of events as needed, each in its own context. Then, with an appropriately indexed MySql table, specific contexts can be queried for outstanding queued items very quickly.

Additionally, PHP SimpleQueue has a built in retry mechanism that will allow you to define both the number of retries for processing a specific item as well as how long to delay between retries.

PHP SimpleQueue has a self installer, so you simply need to FTP the package to your server, launch the install.php script from your browser, follow the prompts for configuration, and you will be able to use the queue from any of your scripts.

Note

PHP SimpleQueue is packaged with a logging class on which it depends to function. By default, it will write to the current directory which the script is running. If you define a LOG_PATH variable, this path will be used instead.

PHP SimpleQueue also checks to see if you have defined a DEBUG variable. If so, and it is set to true, verbose logging messages will be posted to the log file for your perusal.

For testing purposes, you could simply turn debugging on and write to your /tmp directory by adding the following two lines to your script:

<?php
define
('DEBUG'TRUE);
define('LOG_PATH''/tmp');
?>

This would result in verbose logging being dumped to /tmp/SimpleQueue.log.

Example

Let's create a basic scenario: you have a webpage that allows users to contact a 3rd party API (let's say Twitter). As traffic increases to your site, you realize you don't want every request to be processed immediately and tie up resources, but you'd rather throttle the requests a few at a time.

The first step is to simply queue the requests that come in and give the user a successful page load. This gives them a great user experience, and it also helps you spread your server load more evenly. Your queuing code could be as simple as this:

<?php
// Optional context is "twitter" for this example
// $request can be any data you need to process
require('lib/PHPSimpleQueue.php');
$q = new SimpleQueue();
$q->queue(json_encode($request), 'twitter');
?>

Now that you have things queued up, we need to define a callback function to process the request and tell PHP SimpleQueue whether or not each item was successfully processed. If not, it will be requeued according to the configs, otherwise, it will be permanently deleted from the queue.

We need a script that will run on a CRON job (let's say every 5 minutes). We know we want to process 25 twitter queue items every 5 minutes, so our script could be as simple as this:

<?php
// Return a boolean from your callback
function processTwitter($data) {
    
$request json_decode($data);
    
// Replace "process_request" with your process logic for the data stored
    
if (process_request($request)) {
        return 
TRUE// Tell queue to delete item
    
} else {
        return 
FALSE// Tell queue to retry item
    
}
}

// Set up queue rules
require('lib/PHPSimpleQueue.php');
$q = new SimpleQueue();
$q->setExecLimit(25); // 25 per run
$q->setRetryLimit(3); // After 3 failures, remove from the queue
$q->setRetryDelay(15); // Wait 15 minutes between retries
$q->setCallbackAction('processTwitter''twitter'); // Set callback for "twitter" context
$q->dequeue('twitter'); // Tell the queue to process this context based on configs
?>

Download

PHP SimpleQueue is hosted on GitHub for download.

You can clone the read-only version of the project from git://github.com/guahan-web/PHP-SimpleQueue.git.

This project is currently under heavy development, so please feel free to contact me via the GitHub repository page for suggestions or bug reports.

Install

Once you have downloaded the source, you can install PHP SimpleQueue by following these steps:

  1. Upload the files to a web accessible directory for installation. Once you have installed ths script, you can move it into a lib directory or wherever else you like.
  2. Be sure that the SimpleQueueDB.php file is writable by apache on your server, as this file will be modified by the installation.
  3. View the install.php file via a web browser and follow the on screen prompts.
  4. Set the LOG_PATH variable in any scripts in which you use this queue to a directory where apache will be able to write log files.
  5. Once you verify everything is working as intended, feel free to move the queue to any lib directory as you see fit.