kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 30th December, 2013 06:38 (UTC)

Writing about C++, Programming, Fost 4, the web, Thailand and anything else that catches my attention—with some photos thrown in

Moving from Fost 4 to Fost 5

Created 28th December, 2014 06:09 (UTC), last edited 31st December, 2014 02:00 (UTC)

The new version of Fost is going to have a large number of breaking changes that will affect pretty much every aspect of the libraries' use. The reason for this is quite simple, C++11 is really a new language when compared to C++03. Many of these changes will be brought about due to simple changes brought about by the promotion of many Boost libraries into the standard library.

Targets and approach

The plan is to support C++14 with whatever compilers we can use to do this. The current latest version of Ubuntu at least has the features we need to make a start here. I hope that Microsoft Visual C++ 2013 also has enough support to at least get started. We still aren't in a position to support Mac OS X as I don't have access to the platform in a manner suitable for testing — it isn't clear to me that this is even possible without a huge investment due to inherent limitations of OS X virtualisation.

There's likely to be a problem with compilers on the older Ubuntu platforms that's likely to necessitate building both a compiler and Boost. Hopefully Docker can help make this somewhat simpler.

Initially we'll be wanting to support Ubuntu Precise (12.04) and later, and Windows with Microsoft Visual C++ 2013. We'll also support Android with NDK 10c and later.

For Python we intend to move to Python 3 and drop support for 2.

The change from Fost.3 to 4 was done by a complete rebuild of all of the files from the ground up. That isn't what will happen this time. Instead we'll start to build Fost 5 with the newer compile flags and then start to make the breaking changes that we want. These breaking changes will continue for the first releases of Fost 5 — this appears to be unavoidable given the resources we have to do this work with.

Change roadmap

There's a number of changes that are obviously needed even before starting work on anything: Firstly, there's a number of Boost libraries that we can now replace with standard libraries:

  • Smart pointers
  • Threads
  • Many uses of type traits, lambdas and functors.
  • Regex
  • Filesystem (subject to compiler support)

It's clear that Fost 5 will continue to require a large part of Boost.

There's also a list of changes we want to make to the libraries as well:

  • The argument parsing implementation dates from 1995 and isn't all that great. It'd make more sense to either drop it entirely in favour of a more modern implementation, or at least build something that is more POSIX like. Initialiser lists should make configuring the options in code far simpler.
  • There are many places were we'd want to use the JSON implementation, but have wanted to augment the types that can be stored in the structure. This can be done by having a base template implementation that can then be extended by additional user types. Variadic templates will make this far simpler.
  • Much of the thread implementation can be replaced with standard items, hopefully including the futures. There's still likely to be uses for the worker and work pools though. The message passing here will be substantially easier to use with lambdas. The current counter can be dropped in favour of atomics, but it's likely that we'll still want something that can aggregate counts across multiple threads far more efficiently.
  • Boost still uses the oldest version of Spirit for parsing, and has a number of number of wrappers to get around threading problems (which probably aren't needed). It would be best to upgrade this to the current version.

Question marks

Boost build is a pain, and hard for other people to work with. There's a lot of things I'd like to be able to get out of the build process, but customising Boost build is always a constant frustration. It isn't at all clear that any other build system would be a significant improvement. The only way to really know whether something else is going to be better is to try it out.

SCons looks good in that it's building Python, already a dependency in the build process. But for large projects it's slow, and we use this to build large projects (sometimes hundreds of thousands of build targets). CMake also looks interesting, but extending it will be a nightmare and it's hard to know if it has the features that Fost needs or not, let alone features for things we'd like to do.

It may also be time to look again at testing. C++11's lambda syntax gives a lot of extra flexibility there. It ought to be possible to produce something much closer to a Jasmine style of tests now rather than the current macro based system.

A bigger question hangs over Boost ASIO. Fost 4 doesn't take advantage of the asynchronous features, preferring instead to use threads as building blocks rather than callbacks. This means a certain amount of fighting the API so as to get time outs and blocking working well. What's worse though is the use of OpenSSL — but it isn't at clear what a better alternative would look like. Maybe LibreSSL will become usable across our target platforms in time.

Pages

  1. Using Fost 5
  2. Upgrading a Fost 4 project

Categories:
Posted: 31st December, 2014 02:10 (UTC)
First on the front page.

Fost 4 release 4.14.12.44938 now out

Posted 21st December, 2014 09:38 (UTC), last edited 26th December, 2014 11:46 (UTC)

First things first, there's still no official Mac support because I still don't have access to the necessary hardware.

The biggest change that's happened with this release is that the version of Boost build used to build anything other than Boost itself is now a fixed version that is a git submodule from the Boost folder. I'm afraid that I don't really know how this will play out with new check outs on Windows — but it's pretty likely that it's still broken. We are working on this and getting everything working and tested on Visual Studio 2013.

We've made a start at proper Android support. At the moment there is some support spread across fost-android and fost-android-ndk. The project structure right now is wrong and rather inflexible. This should be sorted out by the time the next release is happening. The Android code is all compiled as C++11, which brings me to another point.

One thing that has become clear through development of the Android code and AnimRay is that C++11 is a much better language than C++03. Fost has always used the sorts of features that are now in the language, but it's far more pleasurable to use things like the built in lambda syntax rather than the awkward Boost lambda bind constructs that are currently used.

Our thinking right now is to handle this initially through a long lived fost5 branch in the library repositories, and to start to make use of these for Android and AnimRay. Once we're happy with this we'll do a final release in the Fost 4 line, and then start on Fost 5. Given that Fost 4 already compiles properly under C++11 it's quite likely that this could happen pretty quickly. Expect Fost 4.15.03 to be the final Fost 4 release.

After this date we'll keep a Fost 4 branch live in the repositories and will continue to make small changes to it, but the future will be in C++11 and C++14 and onwards in the new Fost 5 releases.

Linux & Mac

git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • Added a mechanism for joining paths (`fostlib::join_paths`) in the same sort of way that URL paths are joined.
  • Improved FSL_CHECK so it doesn't need to declare a local variable.
  • Added a new test macro `FSL_CHECK_ERROR` that ensures that the error between two values (normally reals) is less than an error term.

fost-beanbag

  • Fix a test so it doesn't depend on being built within the beanbag project.

fost-internet

  • Allow MIME binary bodies to be made from start/end memory pointers.
  • Only set Date and Host headers if they're not already set.

fost-orm

  • Expose the mechanism for determining the JSON database file locations.
  • Make sure that the directory we're about to save a JSON DB blob into actually exists.
  • Allow a path prefix to be set for the JSON DB save/load location.
  • Improved the exception reported when constructing a JSON DB fails.

fost-web

  • Allow web server views to have their own specific setting values.

Categories:

Fost 4 Docker images

Created 2nd September, 2014 02:15 (UTC), last edited 18th October, 2014 11:01 (UTC)

We're starting to put together a few docker images including base containers suitable for building other services on top of as well as some demo containers.

All images are based on the official Ubuntu image, and we are supporting Ubuntu Precise (12.04) and Ubuntu Trusty (14.04). We will include new releases as they happen. The releases are tagged, e.g.:

  • kayess/ubuntu-updated:precise
  • kayess/ubuntu-updated:trusty
  • kayess/ubuntu-updated:latest (currently Trusty)

Basic images

kayess/ubuntu-updated

This is a totally bare bones image that includes the latest security updates and is able to execute apt commands properly.

kayess/minit

Based on kayess/ubuntu-updated this image provides a simple minimal init process that reaps zombies. This should help to improve the long term stability of services that are to run inside the container.

kayess/minit-restart

Very similar to kayess/minit, the main difference being that when the init system finds that the server process has died for any reason it will restart it.

kayess/fost-runtime

Includes packages needed to execute Fost code, but assumes that you will be building the entirety of the Fost dependencies to include in your images.

kayess/fost-builder

Configured to allow building of Fost libraries. It provides a command fost-build that is passed the build command to run. It sets the output location to /src/dist-docker and installs the correct version of the Boost libraries.

sudo docker run -v $(pwd):/src -u $(id -u):$(id -g) -it kayess/fost-builder fost-build ./compile toolset=gcc release

The -v makes your source folder available to the container, and the -u option ensures that the files created during the build are owned by your account on the host.

Beanbag

Beanbag is a lightweight JSON based no-SQL database that supports transactions. It is fully ACID.

kayess/beanbag

This is a small development/runtime base. It provides a beanbag web server pre-configured to handle a number of beanbags together with the ability to serve static files. The beanbag seed project shows how to use this with AngularJS.

kayess/beanbag-gtf

This is a stand-alone demo of the GPLed TLA FAQ. You can run it using:

sudo docker pull kayess/beanbag-gtf && sudo docker run -td -p 9000:2222 kayess/beanbag-gtf

And then connect to the demo application at http://localhost:9000/.


Categories:
Posted: 19th October, 2014 04:40 (UTC)
First announced on the front page.

Django Async

Created 25th September, 2013 12:06 (UTC), last edited 23rd September, 2014 05:12 (UTC)

Django Async is an asynchronous execution queue for Django with proper database transaction management

Building a database backed task queue is a fairly trivial thing, but getting the database transactions exactly right is no simple matter.

Installing Django Async

Get Django Async with pip from pypi:

pip install django-async

Installation is very simple, just add the async application to your Django applications in settings.py. You also really want to use the transaction middleware (see below) and a proper transactional database (like PostgreSQL).

Using Django Async from your code

To run a job asynchronously just use the schedule function:

from async import schedule
schedule('my.function', args=(1, 2, 3), kwargs=dict(key='value'))

Tasks can be run by executing the management command flush_queue:

python manage.py flush_queue

See Using Django Async from your code for more details.

Using Django Async from the command line

Command line interaction with Django Async is through Django's management commands.

python manage.py flush_queue

For more details on options see Using Django Async from the command line.

Transaction handling

Database transactions are hard to get right, and unfortunately Django doesn't make them much easier. Firstly, you really want to be using a proper transactional database system.

Django has two major flaws when it comes to transaction handling:

  1. The Django transaction functionality fails to create composable transactions.
  2. The Django documentation makes a very poor recommendation about where to put the django.middleware.transaction.TransactionMiddleware.

The first problem is not going to get fixed in Django, but the second can be handled by putting the middleware in the right place — that is, as early as possible. The only middleware that should run before the transaction middleware is any whose functionality relies on it being first.

Within the async task execution each task is executed decorated by django.db.transaction.commit_on_success. This means that you cannot execute a task directly from within a page request if you are using the transaction middleware (this is due to problem number one above).

Pages

  1. Using Django Async from your code
  2. Using Django Async from the command line

Categories:
Posted: 23rd September, 2014 11:56 (UTC)

django-async 0.6 has been released.

  • Added group model for grouping job, slumber operation for that model.
  • Brought in time zone support in Django 1.4. Thanks to Peter Brooks for initiating the effort.
  • Added jobs limit option into flush_queue command.
  • Add field cancelled to job
  • Add a function that can be placed into the async queue for clearing out old jobs.
  • Add the ability to have multiple workers so that throughput and latency can be improved.

Detailed changes

  • Fix admin error when a job has no group
  • Fixed a bug that stopped us from running jobs in top level modules.
  • Added database indexes to speed up selection of jobs, and the progress operation and re-arranged the bookkeeping of job execution to lower transaction costs. Also added in scripts to make testing of the queue overhead simpler.
  • Fixed the generation of the progress operation URL.
  • When giving a group name to the schedule API it now does the right thing and creates a new group instance if appropriate.
  • A final job can be set on a group (using on_completion) which will be executed after all of the other jobs have been done.
  • Include the time consumed in the progress report and switch to ISO formatted dates.
  • Fix pylint 1.2.1 issues.
  • Changed the progress estimated_group_duration to estimated_total_time and added in remaining_seconds to the Slumber progress operation.
  • Modified api to accept target group parameter.
  • Make remove old job function include cancelled jobs

Fost 4 release 4.14.09.44924 now out

Posted 21st September, 2014 07:22 (UTC), last edited 21st September, 2014 08:07 (UTC)

There are a couple of new libraries that we've tried to get included previously, but are now actually fully integrated into the meta-build system that we use to test and publish the libraries — this means we won't accidentally miss them out again.

This release has not been tested on Mac — my ancient iMac has finally died, and I have no other way to properly test this platform. For now I'm leaving the Mac instructions in here, but if I don't get a new Mac or a volunteer to run the test builds then I'm going to have to drop the Mac support.

We've also got some initial experimental Docker images available to make building and deploying Fost based systems easier. Later on we hope to be able to augment these with full development environments which should make using the libraries significantly simpler.

Linux & Mac

git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • The meta data set on the progress for a task is now made available to the meter readings.
  • There is a new fostlib::cli::monitor function that can be used to display a progress bar monitoring work towards a future.
  • Futures now return const&s instead of copies of the value when they're dereferenced.
  • Allow fostlib::future<> instances to be compared for equality.
  • The fostlib::accessors now use perfect forwarding if FOST_HAS_MOVE is defined.
  • Need to not pretty print the JSON values when showing the full settings database as we're outputting something approximating a CSV file.

fost-postgres

  • Change the test database configuration to assume ident/trust based authentication.

Categories: