Semaphore or Automatic cleaner - built-in PHP

I built a system that migrate all sort of data from one legacy system to a new database. It has to be run every 15 minutes and must start with data from today and work it's way back to the oldest data. It takes some time to migrate that much data so it does one day per 15 minutes.

The problem is... what if one day takes more than 15 minutes? That is what happened to me when I weas testing my script on a staging machine that was slower than my own laptop. So slow that for a job that took 3 minutes on my laptop in a VM, it took 30 minutes on the staging VM (not on my laptop). So cron started a new migration and another, and another. We ended up with 6 simultanious migration running. Server was barely usable. We were able to recover by killing the scripts that were running.

Now the question came: How to we prevent that?

I decided to use a locking mecanism that works somewhat like a semaphore. IT is not full proof but it does not require to be. If you run more than one script at the same time, race condition occurs both will think it's the only one running and will run simulatinously. That's ok in my case.

My project already had a table for variables so I decided to use it to store the timestamp of when my scripts starts running.

At the begining of my script I check to see if a row exists in one of my "variable" table. If the row exists, check if that timestamp is more than 3 (or whatever) hours old. If it has been that long, I assume that the previous script is stuck or died and delete the variable and continue. If it was less than that limit, then I exist right away, thinking that the previous script is still running. If it does not exists, I assume I am the only one running and I continue.

If the script is ready to continue, it create the variable with the current timestmap. Then run the migration. When finished, it deletes the variable. The cycle starts over.

But what if the script takes more than 30 seconds to run (max_execution_time)? The script will die but the variable will not be deleted. The script will have to wait 3 hours before being able to start over. The solution?

register_shutdown_function() is the answer.

Simply create a function that will delete your variable and call register_shutdown_function() and pass the name of your function. When php will shutdown, it will call your function. The only reason your function would not be called is: PHP fatal error, or process is killed by an admin. Maybe there are other reasons but those two I know about.

I invite you to read the documentation on this function. It is a tricky function that have a few rules and limitations. But it is usefull to increase the chances that your stuff gets cleaned up. It could be used to delete temporary files for examples.

What do you use this function for?

how would you include on a feedapi or feeds modules

how would you include on a feedapi or feeds module?

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote>
  • You may post PHP code. You should include <?php ?> tags.
  • Lines and paragraphs break automatically.
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>. Beside the tag style "<foo>" it is also possible to use "[foo]". PHP source code can also be enclosed in <?php ... ?> or <% ... %>.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options