Helios

distributed job processing system for Perl

Welcome to Helios!

Helios is a system for building asynchronous distributed job processing applications. Applications that need to process hundreds, thousands, even millions of small units of work in parallel can use the Helios system to distribute their workload across the multiple processes and servers that form a Helios collective. Helios may also be used to improve the user experience on websites. By utilizing the framework's APIs, potential timeout issues can be eliminated and response times decreased for larger tasks invoked in response to user input. The web server application can "fire and forget" to the background, immediately returning control to the user. Using Helios, simple Perl applications can be written to distribute massive workloads throughout the Helios collective while still retaining centralized management.

The Helios system consists of 2 parts:

  • Helios - the core system framework installed with your application on the hosts that will actually perform work, and
  • Helios::Panoptes - the web-based administration console that acts as the control center for a Helios collective of hosts.
Most users wanting to deploy a Helios collective will need both pieces: Helios is deployed with the application classes on hosts that actually perform work (the "worker hosts"), while Helios::Panoptes is deployed on a web server to monitor the work being performed.

You can download Helios and Helios::Panoptes distributions from CPAN or GitHub using the links to the right. Installation instructions for Debian/Ubuntu, Red Hat/CentOS, and other Unix-type operating systems are included in the main Helios distribution. You can also check the Helios GitHub wiki for more information.

Helios News

New Helios Commands in x/feature-2.821-commands

We have pushed a set of new command line utility prototypes to the x/feature-2.821-commands branch of the Helios GitHub project.

There are several new commands:

  • The helios_job_* commands allow Helios users some control over queued jobs in the system. Jobs can now be locked and unlocked outside of services, and jobs can be cancelled without being processed (the jobs will be recorded as failed in Job History).

    • helios_job_lock

    • helios_job_unlock

    • helios_job_cancel

  • The helios_jobtype_list command lists all of the jobtypes defined in the Helios collective database. If you have forgotten which jobtypes exist in your Helios setup, you can use this command instead of using SQL or searching for each jobtype one-by-one with helios_jobtype_info.

  • The helios_config_* commands are not new, but have new shortcuts for specifying options that allow a user to leave off the command line switch names. So instead of using:

    helios_config_get -s MyService -p my_parameter

    On the command line, you can now use:

    helios_config_set MyService my_parameter

If the testing on these prototypes goes well, we will polish them up and get them ready for the next Helios release.

Posted by lajandy on Fri, 04 Sep 2015

Helios 2.83 and HeliosX::Logger::HiRes 1.00 Released - Happy CPAN Day!

Today, the 20th anniversary of the Comprehensive Perl Archive Network (CPAN), we have released Helios 2.83 and HeliosX::Logger::HiRes 1.00.

Helios 2.83 is a minor release containing official SQLite support, better schema DDL for Oracle databases, and some cleanup of some files with mixed-format line endings.

Helios::Logger::HiRes 1.00 is the first stable release of a plugin module providing enhanced logging features to the Helios framework, including subsecond timestamp precision (provided by Perl's Time::HiRes) and a command line log searching tool.

Helios 2.83 and HeliosX::Logger::HiRes are open source software packages. Helios 2.83 is available under a dual Artistic/GNU GPL dual license (the same as Perl itself). HeliosX::Logger::HiRes is available under the terms of the Artistic License 2.0. If you have questions or want to get involved with the Helios project, check out the Logical Helion GitHub page or contact us via email at helios [at] logicalhelion dot org or via our Twitter page.

Posted by lajandy on Sun, 16 Aug 2015

New Helios 2.821-unstable

We have merged the x/feature-2.821-schemas Helios branch on GitHub into the unstable-2.821x branch and fixed some long-standing line ending problems in some other files in the Helios distribution. Check out the changes in the Helios unstable-2.821x branch on GitHub.

We have had good feedback on the new SQLite support and simpler Oracle schema DDL, so if there are no reports of problems, we will target a new Helios release with these changes for the upcoming CPAN Day on August 16th. If you test the new unstable branch and find anything wrong, please do not hesitate to let us know by sending an email to helios at logicalhelion dot org.

Posted by lajandy on Fri, 14 Aug 2015

HeliosX::Logger::HiRes 0.10_3270 Released

HeliosX::Logger::HiRes 0.10_3270 has been released to CPAN. This development release adds some extra tests if your environment already has a Helios setup (HELIOS_INI is set) and adds some enhancements to the heliosx_logger_hires_search command. You can now specify multiple log priorities for your search, and your results can now be returned in JSON and some delimited formats. Download it from CPAN or grab it from GitHub!

Posted by lajandy on Tue, 11 Aug 2015

HeliosX::Logger::HiRes 0.10_2751 Released

HeliosX::Logger::HiRes 0.10_2571 has been released to CPAN. This is the first release of this module to CPAN. The main differences between this development release and the last commit to GitHub are some cleanup of the heliosx_logger_hires_search command and the addition of a lot of missing documentation. Test it by downloading it from CPAN or grab the source from GitHub.

Posted by lajandy on Sun, 05 Jul 2015

HeliosX::Logger::HiRes: New Update on GitHub

HeliosX::Logger::HiRes has a new update on GitHub. This update contains corrected schema DDL and a new version of heliosx_logger_hires_search.

The new heliosx_logger_hires_search command now has several new capabilities. The most notable are the --tail and --follow switches, which allow you to get a running display of incoming log messages similar to the behavior of the Unix tail command with the -f option. You can now also search for log messages with specific priorities.

If no major problems are reported, this update will probably be built into a development CPAN release. So do test it out and give us feedback at helios at logicalhelion dot org if there are any problems!

Posted by lajandy on Mon, 01 Jun 2015

New Project: HeliosX::Logger::HiRes

A new Helios-related project has been added to GitHub: HeliosX::Logger::HiRes. HeliosX::Logger::HiRes provides a high-resolution logging subsystem for Helios applications, using timestamps provided by Time::HiRes. The logging data is stored in a table with a better structure than the Helios default, and there is also a command line utility, heliosx_logger_hires_search to view and search log information from the command line. The project is currently alpha-quality, but the logging class itself is solid, and we should be able to start releasing test builds to CPAN soon.

Though there was a Helios::Logger module with this name previously on CPAN, this module is a completely new module forked from the Helios 2.9x prototype code. Since Helios::Logger modules are relatively self-contained, we decided to take the opportunity to get this new piece of code in the hands of users while the Helios 2.9x codebase continues to develop.

Posted by lajandy on Mon, 18 May 2015

New Experiemental Helios Database Schemas

Now pushed to the Helios GitHub project is a new x/feature-2.821-schemas feature branch with an initial version of the SQL DDL to define a Helios collective database with SQLite. This branch also includes an updated version of the Oracle schema DDL. The Oracle schema DDL that has shipped with Helios since version 2.40 was the result of an MySQL-to-Oracle conversion tool; the new version is more concise, straightforward, and cuts out unnecessary things like public synonyms. If you are using Helios on Oracle already, there is no need to change your current schema. If you are installing a new instance of Helios on Oracle, this new schema DDL may be easier to adapt to your needs.

Official support for SQLite is completely new. Though using a SQLite-based collective database will limit you to one host, it can be an ideal development and testing tool. You can also use Helios with SQLite in smaller application setups where even MySQL/MariaDB would be overkill.

These schemas should be considered experimental until we actually build a development release for CPAN. But if you want ot experiment with using Helios with SQLite or want a simpler schema to use with Oracle, see how well these schemas work in your environment.

Posted by lajandy on Mon, 18 May 2015

Helios Update - 18 May 2015

Despite the lack of forward motion in public, work on Helios and its related projects has been continuing slowly over the past few months. Over the next days and weeks, we will start pushing new development versions of Helios and Helios extensions to GitHub and CPAN. Check here and our GitHub project regularly for updates!

Posted by lajandy on Mon, 18 May 2015

Helios::Panoptes 1.7x System Log Lands on GitHub

An alpha-quality version of the new System Log application has landed in the Helios::Panoptes unstable-1.7x branch on GitHub. System Log replaces the old Error Log, which had issues so numerous it was better to replace it rather than try to fix them. The new System Log is functional, but lacks result pagination and some UI elements in the default template.

In addition to the new System Log, Makefile.PL has been changed to correct some issues and update the PREREQ_PM list. The header.html has also been updated to add a System Log link and remove links to Error Log and Submit Job, whose functionality will be removed for the next production release.

As System Log is still considered alpha-quality, we will not be releasing a build of this code to CPAN yet. When more of the planned Helios::Panoptes 1.7x modules have been added and stabilized somewhat, we will start building development releases for CPAN. Until then, Helios::Panoptes users are encouraged to grab the new code from GitHub, test it, and file any bug reports on GitHub or the Helios::Panoptes CPAN RT.

Posted by lajandy on Sat, 06 Dec 2014

Helios 2.90 Prototype 0

Helios 2.90 Prototype 0 has just been committed to the unstable-2.9x branch on GitHub. This commit merges the HeFC Prototype 0 code committed in November with the recent Helios::TS 2.90 Prototype 0 code committed yesterday. In order to make the merged code functional, a few more modifications were required:

  • Changes to Helios::Service to use the new Helios::Logger::Default instead of the old Helios::Logger::Internal.

  • Modified helios_job_info and helios_job_status commands to work with the new HeFC Helios::Job API.

  • A brand new helios_job_submit command based on the submit command in HeliosX::Job::JSON. In addition to working with the new HeFC Helios::Job class, this new submit command also provides a cleaner command line syntax and supports job arguments in JSON as well as XML syntax.

  • Removed the old Makefile.PL and replaced it with HeFC_Makefile.PL from the HeFC branch so the new modules and commands can be built properly.

While this code has been proven to process jobs under certain circumstances, it is pre-alpha quality code and should not be used in any production environment. Additionally, the HeFC APIs are still very preliminary and subject to change. Given this early stage of development, we will not be releasing a build of this code to CPAN. Once development is farther along and the new code and APIs are more stable, we we start to release alpha-quality builds to CPAN. For now, those wanting to experiment with the new APIs or see how Helios 2.9x will work with their existing applications can use the above link to get the code from GitHub.

Posted by lajandy on Thu, 04 Dec 2014

Helios 2.9x - Helios::TS 2.90 Prototype 0

The first prototype of Helios::TS 2.90 has been pushed to the api-2.9x-helios-ts-worker branch of the Helios GitHub project. This commit contains work to separate Helios::TS and Helios::Service from TheSchwartz::Worker and to more properly modularize the relationship between TheSchwartz, Helios::TS, and the rest of the Helios system. This new modularization should allow for new and replaceable Helios worker and job queuing modules in the future.

The modifications include changes to Helios::TS and Helios::Service and a new class for worker code, Helios::TS::Worker:

  • Helios::TS - The work_once() method has been modified to call Helios::TS::Worker instead of a TheSchwartz::Worker subclass. This allows Helios::Service (and its subclasses) to be completely independent of TheSchwartz.

  • Helios::TS::Worker - This new module separates "worker" code from Helios "service" code. The new work_safely() method defines new calling signature so Helios::TS can define a service class as well as the job to be run. The work_safely() method also calls Helios::Service->work() using its new calling signature.

  • Helios::Service - Helios::Service is now its own base class instead of a subclass of TheSchwartz::Worker. Several methods have been added or modified:

    • work() - defines new new calling signature to used by Helios:TS::Worker and other future potential "worker" code. Modified to work with new HeFC Helios::Job API.
    • keep_exit_status_for() - method added to keep Helios::TS::Worker happy.
    • getJobArgs() - modified to work with new HeFC Helios::Job API.

    Additionally, several deprecated methods and their POD were removed: getConfigFromIni(), getConfigFromDb(), getFuncidFromDb(), get/setFuncid().

The next step in the Helios 2.9 development cycle will be to merge this Helios::TS prototype with HeFC Prototype 0 to attempt to create a functioning Helios 2.9 system prototype.

Posted by lajandy on Wed, 03 Dec 2014

Helios 2.9x - Helios Foundation Classes (HeFC) Prototype 0

One of the goals of the Helios 2.9x development cycle is to develop a new set of modular classes to provide a foundation for the various pieces of Helios. Some parts of the system already have good foundations, but other parts of the system do not, while other foundational pieces are too dependent on prerequisites and do not allow Helios to be as modular as it could be.

We have decided to base this new set of Helios Foundation Classes (HeFC) on the minimalist Class::Tiny object framework. Though there are more featureful object frameworks for Perl 5, using Class::Tiny will allow Helios to keep its dependency list as short as possible while still allowing interoperability with the Moo(se) frameworks. Using Class::Tiny will also allow Helios to continue support for Perl 5.8, which we would like to keep for at least a little longer.

HeFC Prototype 0 consists of 4 fundamental classes for the Helios system:

  • Helios::JobType - Helios::JobType made its debut in Helios 2.80. For Helios 2.9, Helios::JobType has been rebased on Class::Tiny, but is otherwise unchanged.

  • Helios::JobHistory REPLACED - The Helios::JobHistory class in Helios 2.8 and earlier is only a Data::ObjectDriver class mapping job history objects to the job history table in the Helios collective database. The new Helios::JobHistory class will define a more proper job history API and completely abstract job history operations from the actual job history system implementation details.

  • Helios::Log - Helios::Log is a new class to define an API for logging features. It will define a log querying API as well as a log writing API. For now, the Helios::Logger interface will be available and will be the system used for logging by Helios services because the interface is minimal and lightweight. Helios::Log will be most useful to applications that need to query logging systems for information outside of Helios services.

  • Helios::Job REPLACED - The Helios::Job class in Helios 2.8 and earlier was originally designed to offer only a thin layer of abstraction over TheSchwartz::Job and the rest of the TheSchwartz job queuing system. As a result, the default Helios::Job does not define a complete API for working with jobs in Helios. This lack of a complete API has made it very difficult to subclass Helios::Job to provide new features (as we recently learned during the development of HeliosX::Job::JSON). Thus, the old Helios::Job class has been thrown out completely.

    The new Helios::Job class will be able to use the new Helios::JobType, Helios::JobHistory, and Helios::Log classes to provide enhanced features. Additionally, it will be easy to subclass Helios::Log, Helios::JobHistory, and even Helios::Job itself, so swapping out the default logging or job history systems will be easy. Since the new Helios::Job class will define a more complete API abstracted from any specific underlying job implementation, the new class will make it easy to subclass methods and provide new or enhanced features to better fit your Helios application's needs. In particular, the new Helios::Job class's job argument (de-)serialization methods will be publicly exposed in a clean way (unlike the current class), and writing a custom Helios::Job subclass to accept job arguments in new formats will be trivial.

Though all of the classes mentioned above will have attributes and methods specific to their respective functions, all of the HeFC classes will define a set of common class methods:

  • new() - a constructor that accepts a hash of attribute values to initialize an object. Actually provided by Class::Tiny::Object.
  • BUILD() - an initializer called if new() is given parameters. Called by the new() provided by Class::Tiny::Object.
  • lookup() - a class method to retrieve an object from the system.

In addtion to the class methods, each class will define a common set of low-level object methods:

  • create() - manages the creation of a new object in the system.
  • remove() - manages the deletion of an existing object in the system. Given the nature of HeFCs as foundational system components, should almost always be used with caution.
  • inflate() - if new() is called with an obj parameter, BUILD() will assume the parameter value is an elemental object to the system and create a new HeFC object from the basic system object's values. In the default Helios 2.9 HeFC classes, the 'elemental' objects are all Data::ObjectDriver objects, but the modularity the HeFCs provide allows such objects in future to be based on DBIx::Class or even a non-SQL-based database system.

In addition to these low-level methods, each HeFC will provide its own higher-level methods appropriate to its function (i.e. submit() for Helios::Job, log_msg() for Helios::Log).

This initial group of HeFCs is now available in the HeFC API branch on GitHub. Each of the new classes has its own group of tests in the t/ directory. Since some HeFC features require new database structures, the SQL to create the new database tables are supplied in the sql/ directory. Because the new Helios::Job class uses a different calling signature than the old, Helios::Service will have to be reworked to be compatible with the new classes. A reworked Helios::Service should land on GitHub shortly, but that should not stop experimenters from taking a look at the new classes now.

Posted by lajandy on Sun, 23 Nov 2014

Data::ObjectDriver Fork on GitHub

Data::ObjectDriver is the simple-to-use but powerful object-relational mapping framework that supports certain parts of the Helios system. Originally developed by SixApart, Inc., it has been maintained by others in the Perl community in recent years.

Recently, we have noticed Data::ObjectDriver's tests failing when installed with either Perl 5.20 or recent versions of DBD::SQLite. The test failures with Perl 5.20 appear to be due to an old version of Test::Builder in the Data::ObjectDriver distribution's inc/ directory. The test failures with recent DBD::SQLite versions appear to be due to a change in the text of an DBD::SQLite error message in the event of a unique constraint violation.

On the Logical Helion GitHub we have forked Data::ObjectDriver from its canonical repo and incorporated changes to fix the problems above. If you are having trouble installing Helios on Perl 5.20 or with a recent version of DBD::SQLite because of these issues, try using our fork by cloning our forked Data::ObjectDriver repo or by downloading the Data::ObjectDriver fork in ZIP format from this site.

Posted by lajandy on Fri, 14 Nov 2014

Helios 2014 Q4 Project Goals

Helios project release schedules have generally been "how fast can we add these features" or "how urgent is this bug?" For the upcoming development phases, however, we are going to try shifting to a quarterly release schedule, with a new version of Helios or Helios-related software released at the end of the 3 month cycle. This should help the project sustain a more constant forward motion than the more sporadic releases of the previous schedules.

Goals for Q4 of 2014 will be split between Helios and Helios::Panoptes:

  • Helios::Panoptes is ancient and has severe issues that need to be addressed. The public version still cannot properly use Oracle-based collective databases, it is slow, has no way to run via FastCGI or PSGI, and the default templates, though simple and functional, are also ugly and clumsy in places. For starters. The overall goal for Helios::Panoptes for Q4 is to modernize it while maintaining (as much as possible) its role as a lightweight, simple web admin console application.

  • Additionally, we'll start development releases of the first phase of Helios 2.9x. While these are still in the prototype phase and will be for awhile, we hope to reach alpha-quality code for the first phase by the end of December. There are many changes we want to work into the Helios 2.9x development cycle; we'll post more about the new features and other changes soon.

Want to help us achieve our Q4 goals? Send us an email at helios at logicalhelion dot org if you want to get involved in the development process. Also, be sure to check out the unstable-* and x/* branches of our Helios and Helios::Panoptes GitHub projects!

Posted by lajandy on Fri, 31 Oct 2014

Helios 2.82 Released!

Helios 2.82 has been released to CPAN and GitHub! This release is primarily a bugfix and documentation release, with some modifications to improve Kwalitee scores as well. Additionally, the examples in the eg/ directory have been updated. Thank you to everyone who contributed in this development cycle!

Download the Helios 2.82 tarball here or check out our project on GitHub.

Helios 2.82 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the GitHub project linked above or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 18 Oct 2014

HeliosX::Job::JSON 1.00 Released!

HeliosX::Job::JSON 1.00 has been released! As previously mentioned here, HeliosX::Job::JSON allows Helios services to use job arguments written in JSON format instead of Helios's default XML format. Download HeliosX::Job::JSON from CPAN or check out the HeliosX::Job::JSON GitHub project for more details.

Posted by lajandy on Fri, 10 Oct 2014

Helios 2.811_4150 Development Release

Helios 2.811_4150 has been released to CPAN and pushed to the Helios GitHub project. The main change in this release adds Helios::Error::JobTypeError and Helios::Error::ObjectDriverError to the list of exception classes loaded when a program uses the Helios::Error module. There are also some improvements to improve Helios's Kwalitee scores. Additionally, the examples in the eg/ directory have been updated to use the newer Helios 2.8 API and have been refactored to prevent top-level namespace pollution on CPAN. Download the Helios 2.811_4150 tarball here.

This release will most likely be the last one in the current development cycle. Thanks to everyone who contributed. Helios 2.82 is coming soon!

Posted by lajandy on Thu, 09 Oct 2014

Helios 2.811_3850 Development Release

Helios 2.811_3850 has been released to CPAN and committed to GitHub. This release contains a change in helios_job_submit.pl to fix bug [RT98845] and a change to Helios::Config to fix bug [RT98284]. Download the release tarball or check out the release on GitHub.

Posted by lajandy on Thu, 18 Sep 2014

HeliosX::Job::JSON 0.02_3670 Development Release

HeliosX::Job::JSON 0.02_3670 is a new development release of HeliosX::Job::JSON aimed at making job submission easier and cleaning up various things in the distribution. Now the config, jobtype, and argument string can be supplied as parameters to new(), and if the config hash is not specified, HeliosX::Job::JSON will use Helios::Config to get the global config hash itself. This should make HeliosX::Job::JSON much easier to work with before. Check it out on MetaCPAN (at the link above) or on GitHub.

Posted by lajandy on Sun, 07 Sep 2014

New HeliosX Module: HeliosX::Job::JSON

HeliosX::Job::JSON allows you to specify job arguments for a service in JSON format instead of Helios's default XML(ish) syntax. If you work with JSON frequently, or especially if you want to specify complex data structures (i.e. arrays, arrays of hashes) as job arguments, HeliosX::Job::JSON may make your work easier. Take a look at the MetaCPAN link above or HeliosX::Job::JSON's GitHub page

Posted by lajandy on Sat, 23 Aug 2014

Helios 2.811_3460 Development Release

Helios 2.811_3460 has been released to CPAN. It contains some documentation fixes to helios_job_submit.pl (thanks @jscook!) and helios_jobtype_info. Download the tarball or check out the release on GitHub.

Posted by lajandy on Fri, 22 Aug 2014

Helios 2.81 Released!

Helios 2.81 is now available on CPAN and GitHub! This is a maintenance and minor bugfix release. It contains a new command, helios_jobtype_info, a -H shortcut for --hostname on all of the helios_config_* commands, and a few other minor fixes.

Helios 2.81 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 16 Aug 2014

Helios 2.801_3270 Development Release

Helios 2.801_3270 has been released to CPAN. It contains some minor fixes to the new helios_* commands and a new helios_jobtype_info command. It also contains some further clarifications of jobtypes vs. service names. Download it from your local CPAN mirror or check out the release on GitHub.

Posted by lajandy on Sat, 09 Aug 2014

Helios 2.80 Released!

Helios 2.80 is now available on CPAN and GitHub! This release brings a lot of new features to improve responsiveness, configurability, scriptability, and flexibility. There are some bugfixes as well. The full change log is available here.

Helios 2.80 is now the current stable release of the Helios framework. This release has been extensively tested, and existing Helios applications should experience no problems running with Helios 2.80. If you encounter bugs or problems, please report them using the Helios RT.

Helios 2.80 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Sat, 15 Mar 2014

Helios 2.72_0950 (2.80 Release Candidate 1) Released

Helios 2.72_0950 has been released. This release contains some minor fixes, changes, and documentation updates compared to the previous release. If there are no problems, we will promote this version to the official 2.80 release soon.

Download the release from the link above or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Fri, 28 Feb 2014

Helios 2.71_4770 Development Release

Helios 2.71_4770 has been released. This release brings some minor fixes and improvements to Helios::Config and the helios.pl daemon. Additionally, the Helios::TheSchwartz module has been removed, as the Helios::TS module has completely replaced it. There have been a few other code cleanups in various places, and the new shell commands have had their .pl extensions removed (the previously existing commands are unchanged for backwards compatibility).

The Helios 2.7x code has stabilized fairly well, our we hope the next release will be the first 2.80 release candidate. Check the Changes file for more info, or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Mon, 25 Nov 2013

Helios 2.71_4460 Development Release

Helios 2.71_4460 has been released. This release brings some minor fixes and adjustments to the new helios_* shell commands. See the Changes file for more information, or check out the unstable-2.7x Helios branch on GitHub.

This development series of Helios is nearly feature-complete. We hope to start building release candidates soon!

Posted by lajandy on Sat, 02 Nov 2013

Helios 2.71_4350 Development Release

Helios 2.71_4350 has been released. This release brings several new shell commands to help administer Helios services and jobs from the command line:

  • helios_service_status.pl - retrieve status of running Helios services.
  • helios_job_status.pl - retrieve the exitstatus of a Helios job.
  • helios_job_info.pl - retrieve job metadata, args, and log messages.
While Helios::Panoptes is the primary Helios administration interface, these new commands make Helios information more accessible from the command line and increases the Helios framework's scriptability. Combined with the new helios_config_* commands, you should only need to resort to the web interface for administration tasks if you *want* to.

Check out the Changes file for this release for more information, or check out the unstable-2.7x Helios branch on GitHub.

Posted by lajandy on Sat, 26 Oct 2013

Helios 2.71_4250 Development Release

Helios 2.71_4250 has been released. This release brings significant additions to the Helios configuration subsystem and a major addition to the job retry API.

The new Helios::Config module adds several new methods to set, unset, and get config parameter values straight from the database. This allows you to programmatically access config parameters directly without having to read all the values in hash form or resorting to SQL to change or delete a parameter.

The Helios::Config changes also enable 4 new Helios shell commands:

  • helios_config_get.pl - retrieve a config parameter from the collective database.
  • helios_config_set.pl - set a config parameter in the collective database.
  • helios_config_unset.pl - delete a config parameter in the collective database.
  • helios_config_import.pl - import config parameters from an INI-style config file into the collective database.
These new commands allow easy access to Helios config parameters from the shell, so you can more easily script operations with your applications or the Helios system itself. You can also use the Helios config subsystem as a source of persistent config options for your shell scripts. The helios_config_import.pl allows you to ease setup and deployment of Helios applications by putting configuration parameters in a file ahead of time and importing the configuration into the collective database with one command.

For a long time, the Helios job retry API (the MaxRetries() and JobRetryInterval() values in your service classes) had a significant weakness: if you wanted to retry a job more frequently than an hour, you had to reach into TheSchwartz job queuing layer and override the grab_for() method, which was practically undocumented. The new JobLockInterval() takes over grab_for() duties, so you can set all the job handling values in your service class in an obvious way.

In addition to these changes, there are other, smaller changes to make the framework faster, more reliable, and generally better. Check out the changes in the unstable-2.7x Helios branch on GitHub or download the new build from your CPAN mirror.

Posted by lajandy on Fri, 18 Oct 2013

Helios 2.71_4051 Development Release

Helios 2.71_4051 has been released. This release includes code to implement "virtual jobtypes." Jobtypes in Helios are more or less like queues in the job queuing system. Previously, Helios services only worked on one type of job at a time, and that jobtype's name had to match the name of the Helios service. If you wanted a service to work on more than one jobtype, or distribute job load in a special way using multiple jobtypes, you would have to create one or more subclasses of your Helios::Service subclass with the same name as your jobtype(s), and potentially run multiple helios.pl service daemons (assuming you could even do what you wanted at all).

With the new jobtype handling code, you can define whatever jobtypes you want using the helios_jobtype_add.pl command, and set which jobtypes your service should work on at runtime with the new --jobtypes option of helios.pl. No more weird empty service subclasses. No more having to manage multiple service class configurations for what essentially are the same service. No more having to keep your old service names if you upgrade because you still have code using an old jobtype name.

In order to accomplish the new jobtype system, there are two new Helios classes in this release:

  • Helios::JobType - handles jobtype creation, lookup, removal.
  • Helios::TS - a TheSchwartz subclass that adds features and bugfixes TheSchwartz does not have. In effect, Helios::TS will become the basis of the Helios internal job queuing system going forward.
If testing with the 2.7x series goes well, we hope these two classes will be the basis for a completely revamped and more complete Helios API in Helios 3.0.

The Helios::Service class and helios.pl daemon in this release have been modified to support the new jobtypes feature and use the new Helios::JobType and Helios::TS classes. Helios::Service has a bunch of new or updated methods to handle multiple jobtypes. The helios.pl program's command line option parsing has been overhauled, so it will be easier to add more command line options in the future.

Work on the 2.7x development series continues in the unstable-2.7x Helios branch on GitHub. Follow the progress or join in there!

Posted by lajandy on Sat, 05 Oct 2013

Helios 2.71_3860 Development Release

Helios 2.71_3860 is the first release in the 2.7x development series. This release includes two new features: job prioritization and worker launch patterns. Job prioritization allows the user to add priorities to jobs. Worker launch patterns allow the user to adjust the speed at which helios.pl launches workers to match better the application and environment.

Work on the 2.7x development series will be happening in the unstable-2.7x Helios branch on GitHub. The list of goals we want to achieve in the 2.7x series (and the preliminary 2.9x feature list) is available on the Development Roadmap wiki page. Follow the progress or, better yet, join in!

Posted by lajandy on Sat, 21 Sep 2013

Helios 2.61 Released

Helios 2.61 is now available on CPAN and on GitHub. This is a bugfix release, containing fixes for 2 outstanding Helios bugs and a patch for another that appears to be a result of LOB binding problems in the DBD layer. Consult the Changes file and the Helios RT for more information. If you are experiencing problems due to these bugs, you are encouraged to upgrade.

Posted by lajandy on Fri, 13 Sep 2013

Helios 2.601_3670 Development Release

Helios 2.601_3670 has been released to CPAN. This development build contains code to enhance the handling of bug RT79690. Download the Helios 2.601_3670 tarball here.

Posted by lajandy on Sat, 07 Sep 2013

Helios 2.601_3610 Development Release

Helios 2.601_3610 has been released to CPAN. This development build contains fixes for several of the bugs in the Helios RT and some internal documentation additions, including extra metadata for sites like MetaCPAN and search.cpan.org. Download the tarball here.

Posted by lajandy on Sun, 01 Sep 2013

Forthcoming Bugfixes and More

Helios now has a couple of bugfixes committed to the bug/* branches on the Helios GitHub project. Soon we will get those together and do a Helios bugfix release.

The planned 2.7x development series has had to be put on hold for awhile because of other commitments, but we would like to start an active Helios development phase soon. Stay tuned for details!

Posted by lajandy on Sun, 04 Aug 2013

Helios 2.60 Released!

Helios 2.60 is now available on CPAN and GitHub!
This release brings performance enhancements and new configuration options.

Major features include:

  • Major changes to database connection handling. Helios now aggressively caches database connections at the DBI layer and more properly handles database connect/disconnect between daemon and worker processes. These improvements provide significant performance increases with a reduced number of open database connections. These increases are especially noticeable with Oracle-based collectives, but collectives using MySQL should experience significant improvements as well.
  • A new modular configuration API. The Helios::Config module provides a modular, customizable API to provide configuration information to Helios applications. In addition, the new API's modularity allows scripts, utilities, and other application pieces external to the user's Helios service easier access to configuration information.
There are several other features and improvements as well. The full change log is available here.

Helios 2.60 is now the current stable release of the Helios framework. Despite the significant changes under the hood, existing Helios applications should experience no problems running with Helios 2.60. All current Helios users are encouraged to upgrade to this version due to the major performance increases. If you encounter bugs or problems, please feel free to report them via RT.

Helios 2.60 is open source software made available under a dual Artistic/GNU GPL dual license (the same as Perl itself). If you have questions or want to get involved with the Helios project, check out the Helios project on GitHub or contact us via email at helios [at] logicalhelion.com or via Twitter.

Posted by lajandy on Thu, 15 Nov 2012

Helios 2.60 Release Candidate 3 is out!

Helios 2.52_4310 is the third release candidate for Helios 2.60. Changes since the last release candidate include a minor change to Helios::Config to prevent a subtle config parameter parsing bug. Read about the changes in this release and the entire 2.5x development cycle in the Change Log or download the distribution from CPAN.

Posted by lajandy on Fri, 26 Oct 2012

New Website!

Welcome one and all to the new website for the Helios job processing system! Though Helios has been under development for many years, this is the first time it has had a proper website outside of its CPAN and github pages. Hopefully we can use this site as a single point of contact for all the Helios resources in development on the web: the CPAN page, the github project, the wiki, etc. Come back frequently, because even bigger things are coming soon!

Posted by lajandy on Thu, 25 Oct 2012