kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 30th December, 2013 06:38 (UTC)

Writing about C++, Programming, Fost 4, the web, Thailand and anything else that catches my attention—with some photos thrown in

Django 1.8, Django Slumber and Django Async

Posted 10th April, 2015 03:55 (UTC), last edited 10th April, 2015 04:12 (UTC)

I've been doing some work to expand the test runners for Django Slumber and Django Async to include Django 1.7 and Django 1.8 with mixed results.

The biggest change in 1.7 was the alteration of Django's transaction handling. This was a change that I never thought that the Django project would make. I don't yet have enough experience with these later versions of Django to say whether or not the handling really makes sense, but from reading the documentation I'm hopeful — however it seems to me that the defaults for new projects are still broken and unsafe, but it's still too early for me to have any specific recommendations in this regard.

This unsafe-by-default behaviour is compounded by the difficulty of testing transactions in Django. Django's test runner tries to be clever with transactions in order to speed up test execution, something that works against getting transactions correct in the code. Then there's the removal of the transaction middleware. It used to be quite simple to ensure that requests had safe enough defaults within the context of a request by simply using this middleware. Now each view must be correctly marked up in order to get safe behaviour — something nearly impossible to test for, and difficult to know and remember. Are you sure that every single view in your latest Django project correctly uses atomic?

I think only people who have a deep understanding of database transactions and why they're needed have any hope of getting Django to do the right thing now. My advice used to be to include the transaction middleware in your application and then the behaviour you get would be what you'd probably expect — when a request errors then no changes are made to the database. This is very simple to reason about for expert and beginner alike.

So, although Django Async's tests all pass with both 1.7 and 1.8 I have little faith that the underlying behaviour is actually safe in the context of a request, although I believe that the queue itself is still safe.

With Slumber it's slightly more complex. There have been extensive changes in the meta layer (that describes applications and models) and although I think I have most of that worked through in a more or less acceptable manner there are test failures that defy explanation — tests not being able to see the data that they've just written to the database. There is probably some stupid error on my part here, but with Django nose still broken for Django 1.8 it's also very hard to get any insight through the logs as to what is happening. A large number of incompatibilities have been fixed though.

I've also added the use of Django's atomic decorator to the views (where this is available, at least where it can be imported). I believe that this ought to make all Slumber operations safe under 1.7 and 1.8. This change is already in the develop branch, and will be published as 0.8.1 once I get a chance (certainly in the next day or two).


Categories:

Big disks and LVM

Posted 6th April, 2015 05:55 (UTC), last edited 6th April, 2015 06:31 (UTC)

I have many very large disks spread across several machines at my home office — currently 14TB on my desktop and 13TB on my main file server. I have also have a few others knocking about for backups which I put into an external hot-drive slot on my desktop case. The environment that I'm in is extremely hostile to disks — I churn through a lot of data running compilations, tests, VMs and doing animations. Not to mention other data processing tasks associated with logs across large numbers of servers. Added to the heat in Thailand disks don't last very long. Recently I've been getting better at replacing disks before they totally fail.

Back in the days when I was using only Windows I used to stripe up smaller disks into single large logical volumes. This make using the disks far easier (and in many cases increases performance), but it comes at a great cost — if any disk fails then the entire logical volume dies with it. I've also tried out RAID using mdadm. This works well until anything at all goes wrong. I don't think I've ever come across any tool that was quite so user hostile as mdadm. Every time I've needed to use it it's taken me half a day to re-learn how to do even the simplest of tasks. LVM also has some RAID abilities, but I've never tried them.

I first started to use LVM because it has great tools for resizing disk partitions, but these days I don't use any of those. The main reason I use it now is that it is the only way I know of to correctly handle very large disks.

LVM

There are three layers that need to be initialised:

  1. Physical disks — see what is there with pvscan and pvdisplay.
  2. Volume groups — vgscan and vgdisplay
  3. Logical volumes — lvscan and lvdisplay

The configuration I'm always after is rather simple: a single volume group per disk and then a single logical volume that takes up the whole disk as well.

First create the physical volume directly onto the disk. The standard disk partitioning programs don't seem to work properly with disks larger than a couple of TB (and you may not notice until you're writing data above the 2TB mark).

sudo pvcreate /dev/sdf

We want a volume group for a single disk. It's tempting to do other things here and to try to be clever — pretty much always a bad idea (see above).

sudo vgcreate miro2 /dev/sdf

Create the logical volume:

sudo lvcreate --name files6 --extents 100%VG miro2

Finally make a file system:

sudo mkfs.ext4 /dev/mapper/miro2-files6

I use a naming convention for the volume groups that includes the machine name. This makes it easier to move a disk between machines without suffering a name clash. I also just use an increasing number for the logical volume names for simplicity's sake. These disks get mounted through fstab into a folder each under /mnt:

/dev/mapper/miro1-files5 /mnt/files5 ext4 noatime,errors=remount-ro  0  1
/dev/mapper/miro2-files6 /mnt/files6 ext4 noatime,errors=remount-ro  0  1

I always use noatime simply because I think it does save some writes and I never have reason to care about access times. I use bind mounts to put individual folders on the disks where I want them:

/mnt/files4/projects /home/kirit/Projects none bind 0  0

My next task is to try to work out how to use smartctl better and add that to my internal monitoring in the hopes of getting more advanced notice that disks are failing.


Categories:

Fost 4 release 4.15.03.44949 now out

Posted 23rd March, 2015 05:55 (UTC), last edited 24th March, 2015 01:48 (UTC)

This is going to be the last Fost 4 release. At some point soon the fost 5 branches will be removed and new long term 4.15.03 branches will be set up for long term maintenance of the C++03 compatible versions of the libraries.

Linux & Mac

git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • Removed a very unsafe string coercion that is no longer needed now we have tagged strings.

fost-beanbag

  • Refactored DELETE hook, some types and header layout.
  • Add a hook where a DELETE on a beanbag can be dropped, or other actions taken first.
  • Correct HTTP DELETE return value and implement deletion of beanbags.

fost-orm

  • Fix a bug where beanbags in the current directory would trigger an assert.
  • Work out the filename for the database once on construction rather than each time we want to use it.

fost-web

  • Added a generic 403 handler.

Categories:

Moving from Fost 4 to Fost 5

Created 28th December, 2014 06:09 (UTC), last edited 31st December, 2014 02:00 (UTC)

The new version of Fost is going to have a large number of breaking changes that will affect pretty much every aspect of the libraries' use. The reason for this is quite simple, C++11 is really a new language when compared to C++03. Many of these changes will be brought about due to simple changes brought about by the promotion of many Boost libraries into the standard library.

Targets and approach

The plan is to support C++14 with whatever compilers we can use to do this. The current latest version of Ubuntu at least has the features we need to make a start here. I hope that Microsoft Visual C++ 2013 also has enough support to at least get started. We still aren't in a position to support Mac OS X as I don't have access to the platform in a manner suitable for testing — it isn't clear to me that this is even possible without a huge investment due to inherent limitations of OS X virtualisation.

There's likely to be a problem with compilers on the older Ubuntu platforms that's likely to necessitate building both a compiler and Boost. Hopefully Docker can help make this somewhat simpler.

Initially we'll be wanting to support Ubuntu Precise (12.04) and later, and Windows with Microsoft Visual C++ 2013. We'll also support Android with NDK 10c and later.

For Python we intend to move to Python 3 and drop support for 2.

The change from Fost.3 to 4 was done by a complete rebuild of all of the files from the ground up. That isn't what will happen this time. Instead we'll start to build Fost 5 with the newer compile flags and then start to make the breaking changes that we want. These breaking changes will continue for the first releases of Fost 5 — this appears to be unavoidable given the resources we have to do this work with.

Change roadmap

There's a number of changes that are obviously needed even before starting work on anything: Firstly, there's a number of Boost libraries that we can now replace with standard libraries:

  • Smart pointers
  • Threads
  • Many uses of type traits, lambdas and functors.
  • Regex
  • Filesystem (subject to compiler support)

It's clear that Fost 5 will continue to require a large part of Boost.

There's also a list of changes we want to make to the libraries as well:

  • The argument parsing implementation dates from 1995 and isn't all that great. It'd make more sense to either drop it entirely in favour of a more modern implementation, or at least build something that is more POSIX like. Initialiser lists should make configuring the options in code far simpler.
  • There are many places were we'd want to use the JSON implementation, but have wanted to augment the types that can be stored in the structure. This can be done by having a base template implementation that can then be extended by additional user types. Variadic templates will make this far simpler.
  • Much of the thread implementation can be replaced with standard items, hopefully including the futures. There's still likely to be uses for the worker and work pools though. The message passing here will be substantially easier to use with lambdas. The current counter can be dropped in favour of atomics, but it's likely that we'll still want something that can aggregate counts across multiple threads far more efficiently.
  • Boost still uses the oldest version of Spirit for parsing, and has a number of number of wrappers to get around threading problems (which probably aren't needed). It would be best to upgrade this to the current version.

Question marks

Boost build is a pain, and hard for other people to work with. There's a lot of things I'd like to be able to get out of the build process, but customising Boost build is always a constant frustration. It isn't at all clear that any other build system would be a significant improvement. The only way to really know whether something else is going to be better is to try it out.

SCons looks good in that it's building Python, already a dependency in the build process. But for large projects it's slow, and we use this to build large projects (sometimes hundreds of thousands of build targets). CMake also looks interesting, but extending it will be a nightmare and it's hard to know if it has the features that Fost needs or not, let alone features for things we'd like to do.

It may also be time to look again at testing. C++11's lambda syntax gives a lot of extra flexibility there. It ought to be possible to produce something much closer to a Jasmine style of tests now rather than the current macro based system.

A bigger question hangs over Boost ASIO. Fost 4 doesn't take advantage of the asynchronous features, preferring instead to use threads as building blocks rather than callbacks. This means a certain amount of fighting the API so as to get time outs and blocking working well. What's worse though is the use of OpenSSL — but it isn't at clear what a better alternative would look like. Maybe LibreSSL will become usable across our target platforms in time.

Pages

  1. Using Fost 5
  2. Upgrading a Fost 4 project

Categories:
Posted: 31st December, 2014 02:10 (UTC)
First on the front page.

Fost 4 release 4.14.12.44938 now out

Posted 21st December, 2014 09:38 (UTC), last edited 26th December, 2014 11:46 (UTC)

First things first, there's still no official Mac support because I still don't have access to the necessary hardware.

The biggest change that's happened with this release is that the version of Boost build used to build anything other than Boost itself is now a fixed version that is a git submodule from the Boost folder. I'm afraid that I don't really know how this will play out with new check outs on Windows — but it's pretty likely that it's still broken. We are working on this and getting everything working and tested on Visual Studio 2013.

We've made a start at proper Android support. At the moment there is some support spread across fost-android and fost-android-ndk. The project structure right now is wrong and rather inflexible. This should be sorted out by the time the next release is happening. The Android code is all compiled as C++11, which brings me to another point.

One thing that has become clear through development of the Android code and AnimRay is that C++11 is a much better language than C++03. Fost has always used the sorts of features that are now in the language, but it's far more pleasurable to use things like the built in lambda syntax rather than the awkward Boost lambda bind constructs that are currently used.

Our thinking right now is to handle this initially through a long lived fost5 branch in the library repositories, and to start to make use of these for Android and AnimRay. Once we're happy with this we'll do a final release in the Fost 4 line, and then start on Fost 5. Given that Fost 4 already compiles properly under C++11 it's quite likely that this could happen pretty quickly. Expect Fost 4.15.03 to be the final Fost 4 release.

After this date we'll keep a Fost 4 branch live in the repositories and will continue to make small changes to it, but the future will be in C++11 and C++14 and onwards in the new Fost 5 releases.

Linux & Mac

git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.12.44938 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • Added a mechanism for joining paths (`fostlib::join_paths`) in the same sort of way that URL paths are joined.
  • Improved FSL_CHECK so it doesn't need to declare a local variable.
  • Added a new test macro `FSL_CHECK_ERROR` that ensures that the error between two values (normally reals) is less than an error term.

fost-beanbag

  • Fix a test so it doesn't depend on being built within the beanbag project.

fost-internet

  • Allow MIME binary bodies to be made from start/end memory pointers.
  • Only set Date and Host headers if they're not already set.

fost-orm

  • Expose the mechanism for determining the JSON database file locations.
  • Make sure that the directory we're about to save a JSON DB blob into actually exists.
  • Allow a path prefix to be set for the JSON DB save/load location.
  • Improved the exception reported when constructing a JSON DB fails.

fost-web

  • Allow web server views to have their own specific setting values.

Categories: