Helios

distributed job processing system for Perl

Welcome to Helios!

Helios is a system for building asynchronous distributed job processing applications. Applications that need to process hundreds, thousands, even millions of small units of work in parallel can use the Helios system to distribute their workload across the multiple processes and servers that form a Helios collective. Helios may also be used to improve the user experience on websites. By utilizing the framework's APIs, potential timeout issues can be eliminated and response times decreased for larger tasks invoked in response to user input. The web server application can "fire and forget" to the background, immediately returning control to the user. Using Helios, simple Perl applications can be written to distribute massive workloads throughout the Helios collective while still retaining centralized management.

The Helios system consists of 2 parts:

  • Helios - the core system framework installed with your application on the hosts that will actually perform work, and
  • Helios::Panoptes - the web-based administration console that acts as the control center for a Helios collective of hosts.
Most users wanting to deploy a Helios collective will need both pieces: Helios is deployed with the application classes on hosts that actually perform work (the "worker hosts"), while Helios::Panoptes is deployed on a web server to monitor the work being performed.

You can download Helios and Helios::Panoptes distributions from CPAN or GitHub using the links to the right. Installation instructions for Debian/Ubuntu, Red Hat/CentOS, and other Unix-type operating systems are included in the main Helios distribution. You can also check the Helios GitHub wiki for more information.

Helios News

Helios 2.82 Released!

Helios 2.82 has been released to CPAN and GitHub! This release is primarily a bugfix and documentation release, with some modifications to improve Kwalitee scores as well. Additionally, the examples in the eg/ directory have been updated. Thank you to everyone who contributed in this development cycle!

Download the Helios 2.82 tarball here or check out our project on GitHub.

Helios 2.82 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the GitHub project linked above or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 18 Oct 2014

HeliosX::Job::JSON 1.00 Released!

HeliosX::Job::JSON 1.00 has been released! As previously mentioned here, HeliosX::Job::JSON allows Helios services to use job arguments written in JSON format instead of Helios's default XML format. Download HeliosX::Job::JSON from CPAN or check out the HeliosX::Job::JSON GitHub project for more details.

Posted by lajandy on Fri, 10 Oct 2014

Helios 2.811_4150 Development Release

Helios 2.811_4150 has been released to CPAN and pushed to the Helios GitHub project. The main change in this release adds Helios::Error::JobTypeError and Helios::Error::ObjectDriverError to the list of exception classes loaded when a program uses the Helios::Error module. There are also some improvements to improve Helios's Kwalitee scores. Additionally, the examples in the eg/ directory have been updated to use the newer Helios 2.8 API and have been refactored to prevent top-level namespace pollution on CPAN. Download the Helios 2.811_4150 tarball here.

This release will most likely be the last one in the current development cycle. Thanks to everyone who contributed. Helios 2.82 is coming soon!

Posted by lajandy on Fri, 10 Oct 2014

Helios 2.811_3850 Development Release

Helios 2.811_3850 has been released to CPAN and committed to GitHub. This release contains a change in helios_job_submit.pl to fix bug [RT98845] and a change to Helios::Config to fix bug [RT98284]. Download the release tarball or check out the release on GitHub.

Posted by lajandy on Fri, 19 Sep 2014

HeliosX::Job::JSON 0.02_3670 Development Release

HeliosX::Job::JSON 0.02_3670 is a new development release of HeliosX::Job::JSON aimed at making job submission easier and cleaning up various things in the distribution. Now the config, jobtype, and argument string can be supplied as parameters to new(), and if the config hash is not specified, HeliosX::Job::JSON will use Helios::Config to get the global config hash itself. This should make HeliosX::Job::JSON much easier to work with before. Check it out on MetaCPAN (at the link above) or on GitHub.

Posted by lajandy on Sun, 07 Sep 2014

New HeliosX Module: HeliosX::Job::JSON

HeliosX::Job::JSON allows you to specify job arguments for a service in JSON format instead of Helios's default XML(ish) syntax. If you work with JSON frequently, or especially if you want to specify complex data structures (i.e. arrays, arrays of hashes) as job arguments, HeliosX::Job::JSON may make your work easier. Take a look at the MetaCPAN link above or HeliosX::Job::JSON's GitHub page

Posted by lajandy on Sat, 23 Aug 2014

Helios 2.811_3460 Development Release

Helios 2.811_3460 has been released to CPAN. It contains some documentation fixes to helios_job_submit.pl (thanks @jscook!) and helios_jobtype_info. Download the tarball or check out the release on GitHub.

Posted by lajandy on Sat, 23 Aug 2014

Helios 2.81 Released!

Helios 2.81 is now available on CPAN and GitHub! This is a maintenance and minor bugfix release. It contains a new command, helios_jobtype_info, a -H shortcut for --hostname on all of the helios_config_* commands, and a few other minor fixes.

Helios 2.81 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 16 Aug 2014

Helios 2.801_3270 Development Release

Helios 2.801_3270 has been released to CPAN. It contains some minor fixes to the new helios_* commands and a new helios_jobtype_info command. It also contains some further clarifications of jobtypes vs. service names. Download it from your local CPAN mirror or check out the release on GitHub.

Posted by lajandy on Sun, 10 Aug 2014

Helios 2.80 Released!

Helios 2.80 is now available on CPAN and GitHub! This release brings a lot of new features to improve responsiveness, configurability, scriptability, and flexibility. There are some bugfixes as well. The full change log is available here.

Helios 2.80 is now the current stable release of the Helios framework. This release has been extensively tested, and existing Helios applications should experience no problems running with Helios 2.80. If you encounter bugs or problems, please report them using the Helios RT.

Helios 2.80 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 15 Mar 2014

Helios 2.72_0950 (2.80 Release Candidate 1) Released

Helios 2.72_0950 has been released. This release contains some minor fixes, changes, and documentation updates compared to the previous release. If there are no problems, we will promote this version to the official 2.80 release soon.

Download the release from the link above or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Fri, 28 Feb 2014

Helios 2.71_4770 Development Release

Helios 2.71_4770 has been released. This release brings some minor fixes and improvements to Helios::Config and the helios.pl daemon. Additionally, the Helios::TheSchwartz module has been removed, as the Helios::TS module has completely replaced it. There have been a few other code cleanups in various places, and the new shell commands have had their .pl extensions removed (the previously existing commands are unchanged for backwards compatibility).

The Helios 2.7x code has stabilized fairly well, our we hope the next release will be the first 2.80 release candidate. Check the Changes file for more info, or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Mon, 25 Nov 2013

Helios 2.71_4460 Development Release

Helios 2.71_4460 has been released. This release brings some minor fixes and adjustments to the new helios_* shell commands. See the Changes file for more information, or check out the unstable-2.7x Helios branch on GitHub.

This development series of Helios is nearly feature-complete. We hope to start building release candidates soon!

Posted by lajandy on Sat, 02 Nov 2013

Helios 2.71_4350 Development Release

Helios 2.71_4350 has been released. This release brings several new shell commands to help administer Helios services and jobs from the command line:

  • helios_service_status.pl - retrieve status of running Helios services.
  • helios_job_status.pl - retrieve the exitstatus of a Helios job.
  • helios_job_info.pl - retrieve job metadata, args, and log messages.
While Helios::Panoptes is the primary Helios administration interface, these new commands make Helios information more accessible from the command line and increases the Helios framework's scriptability. Combined with the new helios_config_* commands, you should only need to resort to the web interface for administration tasks if you *want* to.

Check out the Changes file for this release for more information, or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Sat, 26 Oct 2013

Helios 2.71_4250 Development Release

Helios 2.71_4250 has been released. This release brings significant additions to the Helios configuration subsystem and a major addition to the job retry API.

The new Helios::Config module adds several new methods to set, unset, and get config parameter values straight from the database. This allows you to programmatically access config parameters directly without having to read all the values in hash form or resorting to SQL to change or delete a parameter.

The Helios::Config changes also enable 4 new Helios shell commands:

  • helios_config_get.pl - retrieve a config parameter from the collective database.
  • helios_config_set.pl - set a config parameter in the collective database.
  • helios_config_unset.pl - delete a config parameter in the collective database.
  • helios_config_import.pl - import config parameters from an INI-style config file into the collective database.
These new commands allow easy access to Helios config parameters from the shell, so you can more easily script operations with your applications or the Helios system itself. You can also use the Helios config subsystem as a source of persistent config options for your shell scripts. The helios_config_import.pl allows you to ease setup and deployment of Helios applications by putting configuration parameters in a file ahead of time and importing the configuration into the collective database with one command.

For a long time, the Helios job retry API (the MaxRetries() and JobRetryInterval() values in your service classes) had a significant weakness: if you wanted to retry a job more frequently than an hour, you had to reach into TheSchwartz job queuing layer and override the grab_for() method, which was practically undocumented. The new JobLockInterval() takes over grab_for() duties, so you can set all the job handling values in your service class in an obvious way.

In addition to these changes, there are other, smaller changes to make the framework faster, more reliable, and generally better. Check out the changes in the unstable-2.7x Helios branch on GitHub or download the new build from your CPAN mirror.

Posted by lajandy on Fri, 18 Oct 2013

Helios 2.71_4051 Development Release

Helios 2.71_4051 has been released. This release includes code to implement "virtual jobtypes." Jobtypes in Helios are more or less like queues in the job queuing system. Previously, Helios services only worked on one type of job at a time, and that jobtype's name had to match the name of the Helios service. If you wanted a service to work on more than one jobtype, or distribute job load in a special way using multiple jobtypes, you would have to create one or more subclasses of your Helios::Service subclass with the same name as your jobtype(s), and potentially run multiple helios.pl service daemons (assuming you could even do what you wanted at all).

With the new jobtype handling code, you can define whatever jobtypes you want using the helios_jobtype_add.pl command, and set which jobtypes your service should work on at runtime with the new --jobtypes option of helios.pl. No more weird empty service subclasses. No more having to manage multiple service class configurations for what essentially are the same service. No more having to keep your old service names if you upgrade because you still have code using an old jobtype name.

In order to accomplish the new jobtype system, there are two new Helios classes in this release:

  • Helios::JobType - handles jobtype creation, lookup, removal.
  • Helios::TS - a TheSchwartz subclass that adds features and bugfixes TheSchwartz does not have. In effect, Helios::TS will become the basis of the Helios internal job queuing system going forward.
If testing with the 2.7x series goes well, we hope these two classes will be the basis for a completely revamped and more complete Helios API in Helios 3.0.

The Helios::Service class and helios.pl daemon in this release have been modified to support the new jobtypes feature and use the new Helios::JobType and Helios::TS classes. Helios::Service has a bunch of new or updated methods to handle multiple jobtypes. The helios.pl program's command line option parsing has been overhauled, so it will be easier to add more command line options in the future.

Work on the 2.7x development series continues in the unstable-2.7x Helios branch on GitHub. Follow the progress or join in there!

Posted by lajandy on Sat, 05 Oct 2013

Helios 2.71_3860 Development Release

Helios 2.71_3860 is the first release in the 2.7x development series. This release includes two new features: job prioritization and worker launch patterns. Job prioritization allows the user to add priorities to jobs. Worker launch patterns allow the user to adjust the speed at which helios.pl launches workers to match better the application and environment.

Work on the 2.7x development series will be happening in the unstable-2.7x Helios branch on GitHub. The list of goals we want to achieve in the 2.7x series (and the preliminary 2.9x feature list) is available on the Development Roadmap wiki page. Follow the progress or, better yet, join in!

Posted by lajandy on Sat, 21 Sep 2013

Helios 2.61 Released

Helios 2.61 is now available on CPAN and on GitHub. This is a bugfix release, containing fixes for 2 outstanding Helios bugs and a patch for another that appears to be a result of LOB binding problems in the DBD layer. Consult the Changes file and the Helios RT for more information. If you are experiencing problems due to these bugs, you are encouraged to upgrade.

Posted by lajandy on Sat, 14 Sep 2013

Helios 2.601_3670 Development Release

Helios 2.601_3670 has been released to CPAN. This development build contains code to enhance the handling of bug RT79690. Download the Helios 2.601_3670 tarball here.

Posted by lajandy on Sun, 08 Sep 2013

Helios 2.601_3610 Development Release

Helios 2.601_3610 has been released to CPAN. This development build contains fixes for several of the bugs in the Helios RT and some internal documentation additions, including extra metadata for sites like MetaCPAN and search.cpan.org. Download the tarball here.

Posted by lajandy on Sun, 01 Sep 2013

Forthcoming Bugfixes and More

Helios now has a couple of bugfixes committed to the bug/* branches on the Helios GitHub project. Soon we will get those together and do a Helios bugfix release.

The planned 2.7x development series has had to be put on hold for awhile because of other commitments, but we would like to start an active Helios development phase soon. Stay tuned for details!

Posted by lajandy on Sun, 04 Aug 2013

Helios 2.60 Released!

Helios 2.60 is now available on CPAN and GitHub! This release brings performance enhancements and new configuration options.

Major features include:

  • Major changes to database connection handling. Helios now aggressively caches database connections at the DBI layer and more properly handles database connect/disconnect between daemon and worker processes. These improvements provide significant performance increases with a reduced number of open database connections. These increases are especially noticeable with Oracle-based collectives, but collectives using MySQL should experience significant improvements as well.
  • A new modular configuration API. The Helios::Config module provides a modular, customizable API to provide configuration information to Helios applications. In addition, the new API's modularity allows scripts, utilities, and other application pieces external to the user's Helios service easier access to configuration information.
There are several other features and improvements as well. The full change log is available here.

Helios 2.60 is now the current stable release of the Helios framework. Despite the significant changes under the hood, existing Helios applications should experience no problems running with Helios 2.60. All current Helios users are encouraged to upgrade to this version due to the major performance increases. If you encounter bugs or problems, please feel free to report them via RT.

Helios 2.60 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Thu, 15 Nov 2012

Helios 2.60 Release Candidate 3 is out!

Helios 2.52_4310 is the third release candidate for Helios 2.60. Changes since the last release candidate include a minor change to Helios::Config to prevent a subtle config parameter parsing bug. Read about the changes in this release and the entire 2.5x development cycle in the Change Log or download the distribution from CPAN.

Posted by lajandy on Fri, 26 Oct 2012

New Website!

Welcome one and all to the new website for the Helios job processing system! Though Helios has been under development for many years, this is the first time it has had a proper website outside of its CPAN and github pages. Hopefully we can use this site as a single point of contact for all the Helios resources in development on the web: the CPAN page, the github project, the wiki, etc. Come back frequently, because even bigger things are coming soon!

Posted by lajandy on Fri, 26 Oct 2012