New Helios Commands in x/feature-2.821-commands
We have pushed a set of new command line utility prototypes to the x/feature-2.821-commands
branch of the Helios GitHub project.
There are several new commands:
The helios_job_* commands allow Helios users some control over queued jobs
in the system. Jobs can now be locked and unlocked outside of services, and
jobs can be cancelled without being processed (the jobs will be recorded as
failed in Job History).
helios_job_lock
helios_job_unlock
helios_job_cancel
The helios_jobtype_list command lists all of the jobtypes defined in the
Helios collective database. If you have forgotten which jobtypes exist in
your Helios setup, you can use this command instead of using SQL or searching
for each jobtype one-by-one with helios_jobtype_info.
The helios_config_* commands are not new, but have new shortcuts for
specifying options that allow a user to leave off the command line switch
names. So instead of using:
helios_config_get -s MyService -p my_parameter
On the command line, you can now use:
helios_config_set MyService my_parameter
If the testing on these prototypes goes well, we will polish them up and
get them ready for the next Helios release.
Posted by lajandy on Fri, 04 Sep 2015
Helios 2.83 and HeliosX::Logger::HiRes 1.00 Released - Happy CPAN Day!
Today, the 20th anniversary of the Comprehensive Perl Archive Network
(CPAN), we have released Helios 2.83 and HeliosX::Logger::HiRes 1.00.
Helios 2.83 is a minor release containing official SQLite support,
better schema DDL for Oracle databases, and some cleanup of some files with
mixed-format line endings.
Helios::Logger::HiRes 1.00 is the first stable release of a plugin module
providing enhanced logging features to the Helios framework, including
subsecond timestamp precision (provided by Perl's Time::HiRes) and a
command line log searching tool.
Helios 2.83 and HeliosX::Logger::HiRes are open source software packages.
Helios 2.83 is available under a dual Artistic/GNU GPL dual license (the same
as Perl itself). HeliosX::Logger::HiRes is available under the terms of the
Artistic License 2.0. If you have questions or want to get involved with the
Helios project, check out the Logical Helion GitHub page or contact us via
email at helios [at] logicalhelion dot org or via our Twitter page.
Posted by lajandy on Sun, 16 Aug 2015
New Helios 2.821-unstable
We have merged the x/feature-2.821-schemas Helios branch on GitHub into the
unstable-2.821x branch and fixed some long-standing line ending problems in
some other files in the Helios distribution. Check out the changes in the
Helios unstable-2.821x branch on GitHub.
We have had good feedback on the new SQLite support and simpler Oracle schema
DDL, so if there are no reports of problems, we will target a new Helios
release with these changes for the upcoming CPAN Day on August 16th. If you
test the new unstable branch and find anything wrong, please do not hesitate
to let us know by sending an email to helios at logicalhelion dot org.
Posted by lajandy on Fri, 14 Aug 2015
HeliosX::Logger::HiRes 0.10_3270 Released
HeliosX::Logger::HiRes 0.10_3270
has been released to CPAN. This development release adds some extra tests if
your environment already has a Helios setup (HELIOS_INI is set) and adds some
enhancements to the heliosx_logger_hires_search command. You can now specify
multiple log priorities for your search, and your results can now be returned
in JSON and some delimited formats. Download it from CPAN or grab it from
GitHub!
Posted by lajandy on Tue, 11 Aug 2015
HeliosX::Logger::HiRes 0.10_2751 Released
HeliosX::Logger::HiRes 0.10_2571 has been released to CPAN. This is the
first release of this module to CPAN. The main differences between this
development release and the last commit to GitHub are some cleanup of the
heliosx_logger_hires_search command and the addition of a lot of missing
documentation. Test it by downloading it from CPAN or grab the source from
GitHub.
Posted by lajandy on Sun, 05 Jul 2015
HeliosX::Logger::HiRes: New Update on GitHub
HeliosX::Logger::HiRes has a new update on GitHub. This update contains
corrected schema DDL and a new version of heliosx_logger_hires_search.
The new heliosx_logger_hires_search command now has several new capabilities.
The most notable are the --tail and --follow switches, which allow you to
get a running display of incoming log messages similar to the behavior of the
Unix tail command with the -f option. You can now also search for log
messages with specific priorities.
If no major problems are reported, this update will probably be built into a
development CPAN release. So do test it out and give us feedback at
helios at logicalhelion dot org if there are any problems!
Posted by lajandy on Mon, 01 Jun 2015
New Project: HeliosX::Logger::HiRes
A new Helios-related project has been added to GitHub: HeliosX::Logger::HiRes.
HeliosX::Logger::HiRes provides a high-resolution logging subsystem for
Helios applications, using timestamps provided by Time::HiRes. The logging
data is stored in a table with a better structure than the Helios default,
and there is also a command line utility, heliosx_logger_hires_search
to view and search log information from the command line. The project is
currently alpha-quality, but the logging class itself is solid, and we should
be able to start releasing test builds to CPAN soon.
Though there was a Helios::Logger module with this name previously on CPAN,
this module is a completely new module forked from the Helios 2.9x prototype
code. Since Helios::Logger modules are relatively self-contained, we decided
to take the opportunity to get this new piece of code in the hands of users
while the Helios 2.9x codebase continues to develop.
Posted by lajandy on Mon, 18 May 2015
New Experiemental Helios Database Schemas
Now pushed to the Helios GitHub project
is a new x/feature-2.821-schemas feature branch
with an initial version of the SQL DDL to define a Helios collective database
with SQLite. This branch also includes an updated version of
the Oracle schema DDL. The Oracle schema DDL that has shipped with Helios
since version 2.40 was the result of an MySQL-to-Oracle conversion tool; the
new version is more concise, straightforward, and cuts out unnecessary things
like public synonyms. If you are using Helios on Oracle already, there is no
need to change your current schema. If you are installing a new instance
of Helios on Oracle, this new schema DDL may be easier to adapt to your
needs.
Official support for SQLite is completely new. Though using a SQLite-based
collective database will limit you to one host, it can be an ideal
development and testing tool. You can also use Helios with SQLite in smaller
application setups where even MySQL/MariaDB would be overkill.
These schemas should be considered experimental until we actually build a
development release for CPAN. But if you want ot experiment with using Helios
with SQLite or want a simpler schema to use with Oracle, see how well these
schemas work in your environment.
Posted by lajandy on Mon, 18 May 2015
Helios Update - 18 May 2015
Despite the lack of forward motion in public, work on Helios and its related
projects has been continuing slowly over the past few months. Over the next
days and weeks, we will start pushing new development versions of Helios and
Helios extensions to GitHub and CPAN. Check here and our GitHub project
regularly for updates!
Posted by lajandy on Mon, 18 May 2015
Helios::Panoptes 1.7x System Log Lands on GitHub
An alpha-quality version of the new System Log application has landed in the
Helios::Panoptes unstable-1.7x branch
on GitHub. System Log replaces the old Error Log, which had issues so numerous
it was better to replace it rather than try to fix them. The new System Log is
functional, but lacks result pagination and some UI elements in the default
template.
In addition to the new System Log, Makefile.PL has been changed to correct
some issues and update the PREREQ_PM list. The header.html has also been
updated to add a System Log link and remove links to Error Log and Submit Job,
whose functionality will be removed for the next production release.
As System Log is still considered alpha-quality, we will not be releasing a
build of this code to CPAN yet. When more of the planned Helios::Panoptes 1.7x
modules have been added and stabilized somewhat, we will start building development
releases for CPAN. Until then, Helios::Panoptes users are encouraged to
grab the new code from GitHub, test it, and file any bug reports on GitHub or
the Helios::Panoptes CPAN RT.
Posted by lajandy on Sat, 06 Dec 2014
Helios 2.90 Prototype 0
Helios 2.90 Prototype 0 has just been committed to the
unstable-2.9x branch
on GitHub. This commit merges the HeFC Prototype 0 code committed in November
with the recent Helios::TS 2.90 Prototype 0 code committed yesterday. In order
to make the merged code functional, a few more modifications were required:
Changes to Helios::Service to use the new Helios::Logger::Default instead
of the old Helios::Logger::Internal.
Modified helios_job_info and helios_job_status commands to work with the
new HeFC Helios::Job API.
A brand new helios_job_submit command based on the submit command in
HeliosX::Job::JSON. In addition to working with the new HeFC Helios::Job
class, this new submit command also provides a cleaner command line syntax and
supports job arguments in JSON as well as XML syntax.
Removed the old Makefile.PL and replaced it with HeFC_Makefile.PL from
the HeFC branch so the new modules and commands can be built properly.
While this code has been proven to process jobs under certain circumstances,
it is pre-alpha quality code and should not be used in any production
environment. Additionally, the HeFC APIs are still very preliminary and
subject to change. Given this early stage of development, we will not be
releasing a build of this code to CPAN. Once development is farther along and
the new code and APIs are more stable, we we start to release alpha-quality
builds to CPAN. For now, those wanting to experiment with the new APIs or see
how Helios 2.9x will work with their existing applications can use the above
link to get the code from GitHub.
Posted by lajandy on Thu, 04 Dec 2014
Helios 2.9x - Helios::TS 2.90 Prototype 0
The first prototype of Helios::TS 2.90 has been pushed to the
api-2.9x-helios-ts-worker branch
of the Helios GitHub project. This
commit contains work to separate Helios::TS and Helios::Service from
TheSchwartz::Worker and to more properly modularize the relationship between
TheSchwartz, Helios::TS, and the rest of the Helios system. This new
modularization should allow for new and replaceable Helios worker and job
queuing modules in the future.
The modifications include changes to Helios::TS and Helios::Service and a new
class for worker code, Helios::TS::Worker:
Helios::TS - The work_once() method has been modified to call
Helios::TS::Worker instead of a TheSchwartz::Worker subclass. This allows
Helios::Service (and its subclasses) to be completely independent of
TheSchwartz.
Helios::TS::Worker - This new module separates "worker" code from Helios
"service" code. The new work_safely() method defines new calling signature
so Helios::TS can define a service class as well as the job to be run. The
work_safely() method also calls Helios::Service->work() using its new calling
signature.
Helios::Service - Helios::Service is now its own base class instead of a
subclass of TheSchwartz::Worker. Several methods have been added or modified:
work() - defines new new calling signature to used by Helios:TS::Worker and
other future potential "worker" code. Modified to work with new HeFC
Helios::Job API.keep_exit_status_for() - method added to keep Helios::TS::Worker happy.getJobArgs() - modified to work with new HeFC Helios::Job API.
Additionally, several deprecated methods and their POD were removed:
getConfigFromIni(), getConfigFromDb(), getFuncidFromDb(),
get/setFuncid().
The next step in the Helios 2.9 development cycle will be to merge this
Helios::TS prototype with HeFC Prototype 0 to attempt to create a functioning
Helios 2.9 system prototype.
Posted by lajandy on Wed, 03 Dec 2014
Helios 2.9x - Helios Foundation Classes (HeFC) Prototype 0
One of the goals of the Helios 2.9x development cycle is to develop a new set
of modular classes to provide a foundation for the various pieces of
Helios. Some parts of the system already have good foundations, but other
parts of the system do not, while other foundational pieces are too dependent
on prerequisites and do not allow Helios to be as modular as it could be.
We have decided to base this new set of Helios Foundation Classes (HeFC) on
the minimalist Class::Tiny object
framework. Though there are more featureful object frameworks for Perl 5, using
Class::Tiny will allow Helios to keep its dependency list as short as possible
while still allowing interoperability with the Moo(se) frameworks. Using
Class::Tiny will also allow Helios to continue support for Perl 5.8, which we
would like to keep for at least a little longer.
HeFC Prototype 0 consists of 4 fundamental classes for the Helios system:
Helios::JobType - Helios::JobType made its debut in Helios 2.80. For
Helios 2.9, Helios::JobType has been rebased on Class::Tiny, but is otherwise
unchanged.
Helios::JobHistory REPLACED - The Helios::JobHistory class in Helios
2.8 and earlier is only a Data::ObjectDriver class mapping job history objects
to the job history table in the Helios collective database. The new
Helios::JobHistory class will define a more proper job history API and
completely abstract job history operations from the actual job history system
implementation details.
Helios::Log - Helios::Log is a new class to define an API for logging
features. It will define a log querying API as well as a log writing API. For
now, the Helios::Logger interface will be available and will be the system used
for logging by Helios services because the interface is minimal and lightweight.
Helios::Log will be most useful to applications that need to query logging
systems for information outside of Helios services.
Helios::Job REPLACED - The Helios::Job class in Helios 2.8 and earlier
was originally designed to offer only a thin layer of abstraction over
TheSchwartz::Job and the rest of the TheSchwartz job queuing system. As a
result, the default Helios::Job does not define a complete API for working with
jobs in Helios. This lack of a complete API has made it very difficult to
subclass Helios::Job to provide new features (as we recently learned during the
development of HeliosX::Job::JSON).
Thus, the old Helios::Job class has been thrown out completely.
The new Helios::Job class will be able to use the new Helios::JobType,
Helios::JobHistory, and Helios::Log classes to provide enhanced features.
Additionally, it will be easy to subclass Helios::Log, Helios::JobHistory,
and even Helios::Job itself, so swapping out the default logging or job
history systems will be easy. Since the new Helios::Job class will define a
more complete API abstracted from any specific underlying job implementation,
the new class will make it easy to subclass methods and provide new or
enhanced features to better fit your Helios application's needs. In
particular, the new Helios::Job class's job argument (de-)serialization
methods will be publicly exposed in a clean way (unlike the current class),
and writing a custom Helios::Job subclass to accept job arguments in new
formats will be trivial.
Though all of the classes mentioned above will have attributes and methods
specific to their respective functions, all of the HeFC classes will define a
set of common class methods:
new() - a constructor that accepts a hash of attribute values to initialize
an object. Actually provided by Class::Tiny::Object.BUILD() - an initializer called if new() is given parameters. Called by
the new() provided by Class::Tiny::Object.lookup() - a class method to retrieve an object from the system.
In addtion to the class methods, each class will define a common set of
low-level object methods:
create() - manages the creation of a new object in the system.remove() - manages the deletion of an existing object in the system. Given
the nature of HeFCs as foundational system components, should almost always be
used with caution.inflate() - if new() is called with an obj parameter, BUILD() will assume
the parameter value is an elemental object to the system and create a new HeFC
object from the basic system object's values. In the default Helios 2.9 HeFC
classes, the 'elemental' objects are all Data::ObjectDriver objects, but the
modularity the HeFCs provide allows such objects in future to be based on
DBIx::Class or even a non-SQL-based database system.
In addition to these low-level methods, each HeFC will provide its own
higher-level methods appropriate to its function (i.e. submit() for
Helios::Job, log_msg() for Helios::Log).
This initial group of HeFCs is now available in the
HeFC API branch on GitHub.
Each of the new classes has its own group of tests in the t/ directory. Since
some HeFC features require new database structures, the SQL to create the new
database tables are supplied in the sql/ directory. Because the new
Helios::Job class uses a different calling signature than the old,
Helios::Service will have to be reworked to be compatible with the new classes.
A reworked Helios::Service should land on GitHub shortly, but that should not
stop experimenters from taking a look at the new classes now.
Posted by lajandy on Sun, 23 Nov 2014
Data::ObjectDriver Fork on GitHub
Data::ObjectDriver is the
simple-to-use but powerful object-relational mapping framework that supports
certain parts of the Helios system. Originally developed by SixApart, Inc.,
it has been maintained by others in the Perl community in recent years.
Recently, we have noticed Data::ObjectDriver's tests failing when installed
with either Perl 5.20 or recent versions of DBD::SQLite. The test failures
with Perl 5.20 appear to be due to an old version of Test::Builder in the
Data::ObjectDriver distribution's inc/ directory. The test failures with
recent DBD::SQLite versions appear to be due to a change in the text of an
DBD::SQLite error message in the event of a unique constraint violation.
On the Logical Helion GitHub we have
forked Data::ObjectDriver from its canonical repo and incorporated changes to
fix the problems above. If you are having trouble installing Helios on Perl
5.20 or with a recent version of DBD::SQLite because of these issues, try using
our fork by cloning our forked Data::ObjectDriver repo
or by downloading the Data::ObjectDriver fork in ZIP format
from this site.
Posted by lajandy on Fri, 14 Nov 2014
Helios 2014 Q4 Project Goals
Helios project release schedules have generally been "how fast can we add these
features" or "how urgent is this bug?" For the upcoming development phases,
however, we are going to try shifting to a quarterly release schedule, with a
new version of Helios or Helios-related software released at the end of the 3
month cycle. This should help the project sustain a more constant forward
motion than the more sporadic releases of the previous schedules.
Goals for Q4 of 2014 will be split between Helios and Helios::Panoptes:
Helios::Panoptes is ancient and has severe issues that need to be addressed.
The public version still cannot properly use Oracle-based collective
databases, it is slow, has no way to run via FastCGI or PSGI, and the default
templates, though simple and functional, are also ugly and clumsy in places.
For starters. The overall goal for Helios::Panoptes for Q4 is to modernize it
while maintaining (as much as possible) its role as a lightweight, simple web
admin console application.
Additionally, we'll start development releases of the first phase of
Helios 2.9x. While these are still in the prototype phase and will be for
awhile, we hope to reach alpha-quality code for the first phase by the end of
December. There are many changes we want to work into the Helios 2.9x
development cycle; we'll post more about the new features and other changes
soon.
Want to help us achieve our Q4 goals? Send us an email at helios at
logicalhelion dot org if you want to get involved in the development process.
Also, be sure to check out the unstable-* and x/* branches of our
Helios and
Helios::Panoptes
GitHub projects!
Posted by lajandy on Fri, 31 Oct 2014
Helios 2.82 Released!
Helios 2.82 has been released to CPAN and GitHub! This
release is primarily a bugfix and documentation release, with some modifications
to improve Kwalitee scores as well. Additionally, the examples in the eg/
directory have been updated. Thank you to everyone who contributed in this
development cycle!
Download the Helios 2.82 tarball here or check out our project on GitHub.
Helios 2.82 is open source software made available under a dual Artistic/GNU
GPL dual license (the same as Perl itself). If you have questions or want to
get involved with the Helios project, check out the GitHub project linked above
or contact us via email at helios [at] logicalhelion.com or via Twitter.
Posted by lajandy on Sat, 18 Oct 2014
HeliosX::Job::JSON 1.00 Released!
HeliosX::Job::JSON 1.00 has
been released! As previously mentioned here, HeliosX::Job::JSON allows Helios
services to use job arguments written in JSON format instead of Helios's default XML
format. Download HeliosX::Job::JSON from CPAN or check out the
HeliosX::Job::JSON GitHub project for more details.
Posted by lajandy on Fri, 10 Oct 2014
Helios 2.811_4150 Development Release
Helios 2.811_4150
has been released to CPAN and pushed to the Helios GitHub project. The
main change in this release adds Helios::Error::JobTypeError and
Helios::Error::ObjectDriverError to the list of exception classes loaded when a
program uses the Helios::Error module. There are also some improvements to
improve Helios's Kwalitee scores. Additionally, the examples in the eg/
directory have been updated to use the newer Helios 2.8 API and have been
refactored to prevent top-level namespace pollution on CPAN. Download the Helios 2.811_4150 tarball here.
This release will most likely be the last one in the current development cycle. Thanks
to everyone who contributed. Helios 2.82 is coming soon!
Posted by lajandy on Thu, 09 Oct 2014
Helios 2.811_3850 Development Release
Helios 2.811_3850
has been released to CPAN and committed to GitHub. This release contains a
change in helios_job_submit.pl to fix bug [RT98845] and a change to
Helios::Config to fix bug [RT98284]. Download the release tarball
or check out the release on GitHub.
Posted by lajandy on Thu, 18 Sep 2014
HeliosX::Job::JSON 0.02_3670 Development Release
HeliosX::Job::JSON 0.02_3670 is
a new development release of HeliosX::Job::JSON aimed at making job submission
easier and cleaning up various things in the distribution. Now the config,
jobtype, and argument string can be supplied as parameters to new(), and if the
config hash is not specified, HeliosX::Job::JSON will use Helios::Config to get
the global config hash itself. This should make HeliosX::Job::JSON much easier
to work with before. Check it out on MetaCPAN (at the link above) or
on GitHub.
Posted by lajandy on Sun, 07 Sep 2014
New HeliosX Module: HeliosX::Job::JSON
HeliosX::Job::JSON allows you to specify job arguments for a
service in JSON format instead of Helios's default XML(ish) syntax. If you
work with JSON frequently, or especially if you want to specify complex data
structures (i.e. arrays, arrays of hashes) as job arguments, HeliosX::Job::JSON
may make your work easier. Take a look at the MetaCPAN link above or HeliosX::Job::JSON's GitHub page
Posted by lajandy on Sat, 23 Aug 2014
Helios 2.811_3460 Development Release
Helios 2.811_3460
has been released to CPAN. It contains some documentation fixes to helios_job_submit.pl
(thanks @jscook!) and helios_jobtype_info. Download the tarball
or check out the release on GitHub.
Posted by lajandy on Fri, 22 Aug 2014
Helios 2.81 Released!
Helios 2.81 is now available on CPAN and GitHub! This
is a maintenance and minor bugfix release. It contains a new command,
helios_jobtype_info, a -H shortcut for --hostname on all of the helios_config_*
commands, and a few other minor fixes.
Helios 2.81 is open source software made available under a dual Artistic/GNU
GPL dual license (the same as Perl itself). If you have questions or want to
get involved with the Helios project, check out the Helios project on GitHub
or contact us via email at helios [at] logicalhelion.com or via Twitter.
Posted by lajandy on Sat, 16 Aug 2014
Helios 2.801_3270 Development Release
Helios 2.801_3270 has been released to CPAN. It contains some
minor fixes to the new helios_* commands and a new helios_jobtype_info
command. It also contains some further clarifications of jobtypes vs. service
names. Download it from your local CPAN mirror or
check out the release on GitHub.
Posted by lajandy on Sat, 09 Aug 2014
Helios 2.80 Released!
Helios 2.80 is now available on CPAN and GitHub! This
release brings a lot of new features to improve responsiveness, configurability,
scriptability, and flexibility. There are some bugfixes as well. The full change log is available here.
Helios 2.80 is now the current stable release of the Helios framework. This
release has been extensively tested, and existing Helios applications should
experience no problems running with Helios 2.80. If you encounter bugs or
problems, please report them using the Helios RT.
Helios 2.80 is open source software made available under a dual Artistic/GNU
GPL dual license (the same as Perl itself). If you have questions or want to
get involved with the Helios project, check out the Helios project on GitHub
or contact us via email at helios [at] logicalhelion.com or via Twitter.
Posted by lajandy on Sat, 15 Mar 2014
Helios 2.72_0950 (2.80 Release Candidate 1) Released
Helios 2.72_0950
has been released. This release contains some minor fixes, changes, and
documentation updates compared to the previous release. If there are no
problems, we will promote this version to the official 2.80 release soon.
Download the release from the link above or check out the
unstable-2.7x Helios branch on GitHub.
Posted by lajandy on Fri, 28 Feb 2014
Helios 2.71_4770 Development Release
Helios 2.71_4770
has been released. This release brings some minor fixes and improvements to
Helios::Config and the helios.pl daemon. Additionally, the Helios::TheSchwartz
module has been removed, as the Helios::TS module has completely replaced it.
There have been a few other code cleanups in various places, and the new shell
commands have had their .pl extensions removed (the previously existing
commands are unchanged for backwards compatibility).
The Helios 2.7x code has stabilized fairly well, our we hope the next
release will be the first 2.80 release candidate. Check the Changes
file for more info, or check out the
unstable-2.7x Helios branch on GitHub.
Posted by lajandy on Mon, 25 Nov 2013
Helios 2.71_4460 Development Release
Helios 2.71_4460
has been released. This release brings some minor fixes and adjustments to the
new helios_* shell commands. See the Changes
file for more information, or check out the
unstable-2.7x Helios branch on GitHub.
This development series of Helios is nearly feature-complete. We hope to start
building release candidates soon!
Posted by lajandy on Sat, 02 Nov 2013
Helios 2.71_4350 Development Release
Helios 2.71_4350
has been released. This release brings several new shell commands to help
administer Helios services and jobs from the command line:
- helios_service_status.pl - retrieve status of running Helios services.
- helios_job_status.pl - retrieve the exitstatus of a Helios job.
- helios_job_info.pl - retrieve job metadata, args, and log messages.
While Helios::Panoptes is the primary Helios administration interface, these
new commands make Helios information more accessible from the command line and
increases the Helios framework's scriptability. Combined with the new
helios_config_* commands, you should only need to resort to the web interface
for administration tasks if you *want* to.
Check out the Changes
file for this release for more information, or check out the
unstable-2.7x Helios branch on GitHub.
Posted by lajandy on Sat, 26 Oct 2013
Helios 2.71_4250 Development Release
Helios 2.71_4250
has been released. This release brings significant additions to the Helios
configuration subsystem and a major addition to the job retry API.
The new Helios::Config module adds several new methods to set, unset, and
get config parameter values straight from the database. This allows you to
programmatically access config parameters directly without having to read
all the values in hash form or resorting to SQL to change or delete a
parameter.
The Helios::Config changes also enable 4 new Helios shell commands:
- helios_config_get.pl - retrieve a config parameter from the collective
database.
- helios_config_set.pl - set a config parameter in the collective
database.
- helios_config_unset.pl - delete a config parameter in the collective
database.
- helios_config_import.pl - import config parameters from an INI-style
config file into the collective database.
These new commands allow easy access to Helios config parameters from the
shell, so you can more easily script operations with your applications or the
Helios system itself. You can also use the Helios config subsystem as a source
of persistent config options for your shell scripts. The
helios_config_import.pl allows you to ease setup and deployment of Helios
applications by putting configuration parameters in a file ahead of time and
importing the configuration into the collective database with one command.
For a long time, the Helios job retry API (the MaxRetries() and
JobRetryInterval() values in your service classes) had a significant weakness:
if you wanted to retry a job more frequently than an hour, you had to reach
into TheSchwartz job queuing layer and override the grab_for() method, which was
practically undocumented. The new JobLockInterval() takes over grab_for()
duties, so you can set all the job handling values in your service class in an
obvious way.
In addition to these changes, there are other, smaller changes to make the
framework faster, more reliable, and generally better. Check out the changes
in the unstable-2.7x Helios branch on GitHub
or download the new build from your CPAN mirror.
Posted by lajandy on Fri, 18 Oct 2013
Helios 2.71_4051 Development Release
Helios 2.71_4051
has been released. This release includes code to implement "virtual jobtypes."
Jobtypes in Helios are more or less like queues in the job queuing system.
Previously, Helios services only worked on one type of job at a time, and that
jobtype's name had to match the name of the Helios service. If you wanted a
service to work on more than one jobtype, or distribute job load in a special
way using multiple jobtypes, you would have to create one or more subclasses of
your Helios::Service subclass with the same name as your jobtype(s), and
potentially run multiple helios.pl service daemons (assuming you could even do
what you wanted at all).
With the new jobtype handling code, you can define whatever jobtypes you
want using the helios_jobtype_add.pl command, and set which jobtypes your
service should work on at runtime with the new --jobtypes option of helios.pl.
No more weird empty service subclasses. No more having to manage multiple
service class configurations for what essentially are the same service. No
more having to keep your old service names if you upgrade because you still
have code using an old jobtype name.
In order to accomplish the new jobtype system, there are two new Helios
classes in this release:
- Helios::JobType - handles jobtype creation, lookup, removal.
- Helios::TS - a TheSchwartz subclass that adds features and
bugfixes TheSchwartz does not have. In effect, Helios::TS will become the
basis of the Helios internal job queuing system going forward.
If testing with the 2.7x series goes well, we hope these two classes will be
the basis for a completely revamped and more complete Helios API in Helios 3.0.
The Helios::Service class and helios.pl daemon in this release have been
modified to support the new jobtypes feature and use the new Helios::JobType
and Helios::TS classes. Helios::Service has a bunch of new or updated methods
to handle multiple jobtypes. The helios.pl program's command line option
parsing has been overhauled, so it will be easier to add more command line
options in the future.
Work on the 2.7x development series continues in the
unstable-2.7x Helios branch on GitHub.
Follow the progress or join in there!
Posted by lajandy on Sat, 05 Oct 2013
Helios 2.71_3860 Development Release
Helios 2.71_3860
is the first release in the 2.7x development series. This release includes two
new features: job prioritization and worker launch patterns. Job
prioritization allows the user to add priorities to jobs. Worker launch
patterns allow the user to adjust the speed at which helios.pl launches workers
to match better the application and environment.
Work on the 2.7x development series will be happening in the
unstable-2.7x Helios branch on GitHub.
The list of goals we want to achieve in the 2.7x series (and the preliminary
2.9x feature list) is available on the
Development Roadmap wiki page.
Follow the progress or, better yet, join in!
Posted by lajandy on Sat, 21 Sep 2013
Helios 2.61 Released
Helios 2.61 is now available on CPAN
and on GitHub. This
is a bugfix release, containing fixes for 2 outstanding Helios bugs and a
patch for another that appears to be a result of LOB binding problems in the
DBD layer. Consult the Changes file
and the Helios RT
for more information. If you are experiencing problems due to these bugs, you
are encouraged to upgrade.
Posted by lajandy on Fri, 13 Sep 2013
Helios 2.601_3670 Development Release
Helios 2.601_3670
has been released to CPAN. This development build contains code to enhance the
handling of bug RT79690. Download the Helios 2.601_3670 tarball here.
Posted by lajandy on Sat, 07 Sep 2013
Helios 2.601_3610 Development Release
Helios 2.601_3610 has been released to CPAN. This
development build contains fixes for several of the bugs in the Helios RT and some internal documentation
additions, including extra metadata for sites like MetaCPAN and
search.cpan.org. Download the tarball here.
Posted by lajandy on Sun, 01 Sep 2013
Forthcoming Bugfixes and More
Helios now has a couple of bugfixes committed to the bug/* branches on
the Helios GitHub project.
Soon we will get those together and do a Helios bugfix release.
The planned 2.7x development series has had to be put on hold for awhile
because of other commitments, but we would like to start an active Helios
development phase soon. Stay tuned for details!
Posted by lajandy on Sun, 04 Aug 2013
Helios 2.60 Released!
Helios 2.60 is now available on CPAN and GitHub!
This release brings performance enhancements and new configuration options.
Major features include:
- Major changes to database connection handling. Helios now aggressively
caches database connections at the DBI layer and more properly handles
database connect/disconnect between daemon and worker processes. These
improvements provide significant performance increases with a reduced
number of open database connections. These increases are especially noticeable
with Oracle-based collectives, but collectives using MySQL should experience
significant improvements as well.
- A new modular configuration API. The Helios::Config module provides a
modular, customizable API to provide configuration information to Helios
applications. In addition, the new API's modularity allows scripts, utilities,
and other application pieces external to the user's Helios service easier
access to configuration information.
There are several other features and improvements as well. The full change log
is available
here.
Helios 2.60 is now the current stable release of the Helios framework. Despite
the significant changes under the hood, existing Helios applications should
experience no problems running with Helios 2.60. All current Helios users are
encouraged to upgrade to this version due to the major performance increases.
If you encounter bugs or problems, please feel free to report them via RT.
Helios 2.60 is open source software made available under a dual Artistic/GNU
GPL dual license (the same as Perl itself). If you have questions or want to
get involved with the Helios project, check out the Helios project on GitHub or
contact us via email at helios [at] logicalhelion.com or via Twitter.
Posted by lajandy on Thu, 15 Nov 2012
Helios 2.60 Release Candidate 3 is out!
Helios 2.52_4310 is the third release candidate for Helios 2.60. Changes since the last release candidate include a minor change to Helios::Config to prevent a subtle config parameter parsing bug. Read about the changes in this release and the entire 2.5x development cycle in the Change Log or download the distribution from CPAN.
Posted by lajandy on Fri, 26 Oct 2012
New Website!
Welcome one and all to the new website for the Helios job processing system! Though
Helios has been under development for many years, this is the first time
it has had a proper website outside of its CPAN and github pages. Hopefully
we can use this site as a single point of contact for all the Helios resources
in development on the web: the CPAN page, the github project, the wiki, etc. Come
back frequently, because even bigger things are coming soon!
Posted by lajandy on Thu, 25 Oct 2012
