kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 30th December, 2013 06:38 (UTC)

Writing about C++, Programming, Fost 4, the web, Thailand and anything else that catches my attention—with some photos thrown in

Fost 4 Docker images

Created 2nd September, 2014 02:15 (UTC), last edited 18th October, 2014 11:01 (UTC)

We're starting to put together a few docker images including base containers suitable for building other services on top of as well as some demo containers.

All images are based on the official Ubuntu image, and we are supporting Ubuntu Precise (12.04) and Ubuntu Trusty (14.04). We will include new releases as they happen. The releases are tagged, e.g.:

  • kayess/ubuntu-updated:precise
  • kayess/ubuntu-updated:trusty
  • kayess/ubuntu-updated:latest (currently Trusty)

Basic images

kayess/ubuntu-updated

This is a totally bare bones image that includes the latest security updates and is able to execute apt commands properly.

kayess/minit

Based on kayess/ubuntu-updated this image provides a simple minimal init process that reaps zombies. This should help to improve the long term stability of services that are to run inside the container.

kayess/minit-restart

Very similar to kayess/minit, the main difference being that when the init system finds that the server process has died for any reason it will restart it.

kayess/fost-runtime

Includes packages needed to execute Fost code, but assumes that you will be building the entirety of the Fost dependencies to include in your images.

kayess/fost-builder

Configured to allow building of Fost libraries. It provides a command fost-build that is passed the build command to run. It sets the output location to /src/dist-docker and installs the correct version of the Boost libraries.

sudo docker run -v $(pwd):/src -u $(id -u):$(id -g) -it kayess/fost-builder fost-build ./compile toolset=gcc release

The -v makes your source folder available to the container, and the -u option ensures that the files created during the build are owned by your account on the host.

Beanbag

Beanbag is a lightweight JSON based no-SQL database that supports transactions. It is fully ACID.

kayess/beanbag

This is a small development/runtime base. It provides a beanbag web server pre-configured to handle a number of beanbags together with the ability to serve static files. The beanbag seed project shows how to use this with AngularJS.

kayess/beanbag-gtf

This is a stand-alone demo of the GPLed TLA FAQ. You can run it using:

sudo docker pull kayess/beanbag-gtf && sudo docker run -td -p 9000:2222 kayess/beanbag-gtf

And then connect to the demo application at http://localhost:9000/.


Categories:
Posted: 19th October, 2014 04:40 (UTC)
First announced on the front page.

Django Async

Created 25th September, 2013 12:06 (UTC), last edited 23rd September, 2014 05:12 (UTC)

Django Async is an asynchronous execution queue for Django with proper database transaction management

Building a database backed task queue is a fairly trivial thing, but getting the database transactions exactly right is no simple matter.

Installing Django Async

Get Django Async with pip from pypi:

pip install django-async

Installation is very simple, just add the async application to your Django applications in settings.py. You also really want to use the transaction middleware (see below) and a proper transactional database (like PostgreSQL).

Using Django Async from your code

To run a job asynchronously just use the schedule function:

from async import schedule
schedule('my.function', args=(1, 2, 3), kwargs=dict(key='value'))

Tasks can be run by executing the management command flush_queue:

python manage.py flush_queue

See Using Django Async from your code for more details.

Using Django Async from the command line

Command line interaction with Django Async is through Django's management commands.

python manage.py flush_queue

For more details on options see Using Django Async from the command line.

Transaction handling

Database transactions are hard to get right, and unfortunately Django doesn't make them much easier. Firstly, you really want to be using a proper transactional database system.

Django has two major flaws when it comes to transaction handling:

  1. The Django transaction functionality fails to create composable transactions.
  2. The Django documentation makes a very poor recommendation about where to put the django.middleware.transaction.TransactionMiddleware.

The first problem is not going to get fixed in Django, but the second can be handled by putting the middleware in the right place — that is, as early as possible. The only middleware that should run before the transaction middleware is any whose functionality relies on it being first.

Within the async task execution each task is executed decorated by django.db.transaction.commit_on_success. This means that you cannot execute a task directly from within a page request if you are using the transaction middleware (this is due to problem number one above).

Pages

  1. Using Django Async from your code
  2. Using Django Async from the command line

Categories:
Posted: 23rd September, 2014 11:56 (UTC)

django-async 0.6 has been released.

  • Added group model for grouping job, slumber operation for that model.
  • Brought in time zone support in Django 1.4. Thanks to Peter Brooks for initiating the effort.
  • Added jobs limit option into flush_queue command.
  • Add field cancelled to job
  • Add a function that can be placed into the async queue for clearing out old jobs.
  • Add the ability to have multiple workers so that throughput and latency can be improved.

Detailed changes

  • Fix admin error when a job has no group
  • Fixed a bug that stopped us from running jobs in top level modules.
  • Added database indexes to speed up selection of jobs, and the progress operation and re-arranged the bookkeeping of job execution to lower transaction costs. Also added in scripts to make testing of the queue overhead simpler.
  • Fixed the generation of the progress operation URL.
  • When giving a group name to the schedule API it now does the right thing and creates a new group instance if appropriate.
  • A final job can be set on a group (using on_completion) which will be executed after all of the other jobs have been done.
  • Include the time consumed in the progress report and switch to ISO formatted dates.
  • Fix pylint 1.2.1 issues.
  • Changed the progress estimated_group_duration to estimated_total_time and added in remaining_seconds to the Slumber progress operation.
  • Modified api to accept target group parameter.
  • Make remove old job function include cancelled jobs

Fost 4 release 4.14.09.44924 now out

Posted 21st September, 2014 07:22 (UTC), last edited 21st September, 2014 08:07 (UTC)

There are a couple of new libraries that we've tried to get included previously, but are now actually fully integrated into the meta-build system that we use to test and publish the libraries — this means we won't accidentally miss them out again.

This release has not been tested on Mac — my ancient iMac has finally died, and I have no other way to properly test this platform. For now I'm leaving the Mac instructions in here, but if I don't get a new Mac or a volunteer to run the test builds then I'm going to have to drop the Mac support.

We've also got some initial experimental Docker images available to make building and deploying Fost based systems easier. Later on we hope to be able to augment these with full development environments which should make using the libraries significantly simpler.

Linux & Mac

git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • The meta data set on the progress for a task is now made available to the meter readings.
  • There is a new fostlib::cli::monitor function that can be used to display a progress bar monitoring work towards a future.
  • Futures now return const&s instead of copies of the value when they're dereferenced.
  • Allow fostlib::future<> instances to be compared for equality.
  • The fostlib::accessors now use perfect forwarding if FOST_HAS_MOVE is defined.
  • Need to not pretty print the JSON values when showing the full settings database as we're outputting something approximating a CSV file.

fost-postgres

  • Change the test database configuration to assume ident/trust based authentication.

Categories:

Early test images

Created 12th September, 2014 04:01 (UTC), last edited 13th September, 2014 09:02 (UTC)

These are basic test images that just allow me to check that the ray tracer is outputting what it should.

These can all be built using some commands like:

git clone —recursive git@github.com:KayEss/AnimRay.git animray
cd animray
Boost/install $(lsb_release -sc)
animray/compile animray scenes release

You will find each example in the dist/bin folder and they can be run from there, e.g.:

dist/bin/affine-transformations affine.tga 1920 1080

Pictures & Photos


Categories:
Posted: 13th September, 2014 10:11 (UTC)
First put on the front page to coincide with the 0.1 release of the code.

Fost 4 release 4.14.06.44917 now out

Posted 19th July, 2014 05:36 (UTC), last edited 19th July, 2014 06:43 (UTC)

The new release was once again tagged a while ago, but I went on holiday before managing to debug the failure of the release process to push the tag to github.

Linux & Mac

git clone --branch=4.14.06.44917 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.06.44917 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-hello — Sample project — git@github.com:KayEss/fost-hello.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-orm

  • Made the internal API for adding a transformation function to the JSON database through fostlib::jsondb::local::transformation public.

Categories: