kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 22nd December, 2016 03:24 (UTC)

Writing about C++, Programming, Fost 5, the web, Thailand and anything else that catches my attention—with some photos thrown in

Fost 5 release 5.17.06.45032 now out

Posted 23rd June, 2017 04:09 (UTC), last edited 23rd June, 2017 04:24 (UTC)

Nothing new has landed over the past months since the last release, but that doesn't mean that nothing has happened.

A big change has been made to the low level JSON implementation which should be noticeably faster. Because this is such a big low level change we're letting it bake in testing for longer than normal. So far it looks very good, with no problems reported, and I think it'll turn out we only just missed merging it into master in time for this release.

With that change landed we'll be pushing that implementation further out to make it even faster, primarily by reducing memory allocations.

The previous version made use of two nested variants. One to hold the “atomic” part (null, booleans, integers, doubles and strings) of the JSON and the other to add in the object and array parts. Now this has been flattened down to a single variant that can hold any of them. The previous change also embedded the array and the object inside the variant. Now we hold a shared pointer to them in the variant. This means that in all cases copying a JSON object is now at most an atomic increment, where previously it might need to copy a map.

We also want to now add in other types—things like the f5::lstring so that string literals will never need an allocation at all, and empty types for array and object so that they don't need to allocate until data is actually put in them. We might also experiment with alternative data structures for small objects. I'd also like to see a similar change made to the object keys so that string literals used as object keys don't cause allocations—this is likely common enough that it'll be a noticeable win as well.

With this and a switch towards using the u8_view it also makes sense to start to look at a rope like structure that can be shared and is based around the same structure. This should allow parsed JSON to not need to allocate anything for the embedded data.

We also have some new threading primitives, implementing channels, which should make a lot of threaded code simpler. Right now they're implemented on top of eventfd, but we're actively looking at coroutines now that they've landed in clang 5.

Building on Linux

You will need a C++14 compiler. Recent versions of either gcc or clang are suitable.

git clone --branch=5.17.06.45032 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Categories:

Fost 4 Docker images

Created 2nd September, 2014 02:15 (UTC), last edited 28th April, 2017 07:22 (UTC)

We're starting to put together a few docker images including base containers suitable for building other services on top of as well as some demo containers.

All images are based on the official Ubuntu image. The releases are tagged, e.g.:

  • kayess/ubuntu-updated:trusty
  • kayess/ubuntu-updated:xenial
  • kayess/ubuntu-updated:yakkety
  • kayess/ubuntu-updated:zesty
  • kayess/ubuntu-updated:latest (currently Zesty)

Basic images

kayess/ubuntu-updated

This is a totally bare bones image that includes the latest security updates and is able to execute apt commands properly.

kayess/minit

Based on kayess/ubuntu-updated this image provides a simple minimal init process that reaps zombies. This should help to improve the long term stability of services that are to run inside the container.

kayess/minit-restart

Very similar to kayess/minit, the main difference being that when the init system finds that the server process has died for any reason it will restart it.

kayess/fost-runtime

Includes packages needed to execute Fost code, but assumes that you will be building the entirety of the Fost dependencies to include in your images.

kayess/fost-builder

Configured to allow building of Fost libraries. It provides a command fost-build that is passed the build command to run. It sets the output location to /src/dist-docker and installs the correct version of the Boost libraries.

sudo docker run -v $(pwd):/src -u $(id -u):$(id -g) -it kayess/fost-builder fost-build ./compile toolset=gcc release

The -v makes your source folder available to the container, and the -u option ensures that the files created during the build are owned by your account on the host.

Beanbag

Beanbag is a lightweight JSON based no-SQL database that supports transactions. It is fully ACID.

kayess/beanbag

This is a small development/runtime base. It provides a beanbag web server pre-configured to handle a number of beanbags together with the ability to serve static files. The beanbag seed project shows how to use this with AngularJS.

kayess/beanbag-gtf

This is a stand-alone demo of the GPLed TLA FAQ. You can run it using:

sudo docker pull kayess/beanbag-gtf && sudo docker run -td -p 9000:2222 kayess/beanbag-gtf

And then connect to the demo application at http://localhost:9000/.


Categories:
Posted: 28th April, 2017 07:31 (UTC)
Dropped support for obsolete versions of Ubuntu, and added newer versions.

Fost 5 release 5.17.03.45018 now out

Posted 28th March, 2017 03:47 (UTC), last edited 28th March, 2017 04:06 (UTC)

We've started to push through a few things that will, in the longer term, allow a number of performance improvements. Firstly is that all of the parsers have been changed from Boost Spirit Classic to the new version of Boost Spirit.

The new parsers are smaller and easier to understand than the old ones and this has also allowed us to drop the use of the parser lock which worked around a multi-threading problem in Boost Spirit Classic.

We have also change the internal structure of fostlib::json to alter where the shared pointer to the arrays and objects are handled. This will make copying of fostlib::json instances much cheaper, as they were always intended to be. The JSON parsers have also been changed so they can work from a number of array or string views. This removes a number of places where we used to have to allocate new strings to do conversions to UTF-16 which is what the parsers want to work with. This is now done on the fly inside the string view iterators.

We've also been working through and removing various old Boost functionality that is now in the C++ standard libraries along with general tidying up — adding move constructors and making things properly variadic where applicable.

Building on Linux

You will need a C++14 compiler. Recent versions of either gcc or clang are suitable.

git clone --branch=5.17.03.45018 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Detailed change log

cord

  • Add in iterators for converting between some combinations of UTF encodings.

fostgres

  • Added support for file uploads. JSON base 64 encoded only.
  • Update for new internal structure of fostlib::json.
  • Removed the old parser_lock which is no longer needed.
  • Fix the matcher so it properly decodes the file spec parts from their URL encoding.
  • Switch to the new version of Boost Spirit. Drop support for Boost version prior to !.59.

fost-base

  • Improve the way that the JSON atomic type can be constructed from integral types.
  • Allow us to move the underlying string out of a tagged string rvalue.
  • Make the object logging functions properly variadic.
  • Add a time profile for generating a histogram of timing results.
  • Make the performance counter movable.
  • Add better support for "dynamic" modules.
  • Log messages can be created directly from JSON.
  • Add support for stderr to the stdout logger.
  • Removed last uses of boost::bind.
  • Fix bug in the stdout logger when logging objects and arrays.
  • Remove Boost bind from CLI implementation.
  • Refactor the logging code to no longer use Boost function, lambda or bind.
  • Refactor atexit so it no longer uses Boost function.
  • Refactor the logging messages to do fewer allocations.
  • Refactor the JSON parser to make it more composable.
  • Go back to the C++14 compatible int64_t to string implementation.
  • Refactored the location of the shared_ptr in the json implementation.
  • Removed the old parser_lock which is no longer needed.
  • Allow access to the data buffer within the utf8_view.
  • Switch to the new version of Boost Spirit. Drop support for Boost version prior to !.59.
  • Construct the default logger sinks configuration rather than parse it from an embedded string.
  • Improve the types in the fostlib::string::const_iterator so they better model the InputIterator concept.

fost-internet

  • Removed the old parser_lock which is no longer needed.
  • Switch to the new version of Boost Spirit. Drop support for Boost version prior to !.59.

fost-postgres

  • Removed the old parser_lock which is no longer needed.
  • Switch to the new version of Boost Spirit. Drop support for Boost version prior to !.59.

fost-py

  • Update for new internal structure of fostlib::json.

fost-web

  • Log better information about failed HTTP requests to the server.

fost-wright

  • Capture proper timing data for job execution times.
  • Allow setting of the simulated work times.

Categories:

Fost 5 release 5.16.12.45005 now out

Posted 21st December, 2016 04:09 (UTC), last edited 28th March, 2017 03:04 (UTC)

The tags were pushed a few days ago now.

There is one potentially breaking change in the updates to the nullable type. Now that std::optional is in C++17 it seems sensible to move towards that. The new nullable implementation is a wrapper around std::experimental::optional, but retains the old nullable methods as well, but marks them as deprecated. It also adds a couple of methods that are in the upcoming std::optional but aren't in std::experimental::optional.

As part of the nullable changes we also stopped bringing in the std::rel_ops namespace. This could be a breaking change where you've been inadvertently relying on it for implementations of things like operator !=.

We've also improved the error handling in the fostlib::utf::save_file such that it should now throw if there is a problem saving the file. There's also a new file based log sink called panoptes which features built-in file rotation.

We didn't do the changes to the fost-orm library this time around, but expect them for the next release.

Building on Linux

git clone --branch=5.16.12.45004 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build 62 0
Boost/install 62 0
hello/compile
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • f5-cord — First version of a new string library —
  • f5-threading — Preview of the first Fost 5 library which includes help for threading.
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git

Detailed change log

fost-base

  • Extend exceptions::file_error to also take a boost::system::error_code.
  • Add error detection into save_file.
  • Refactor nullable to look much more like the upcoming std::optional.
  • Remove using namespace std::rel_ops.
  • Change the fostlib::nullable class to be implemented on top of std::experimental::optional.
  • Exceptions caught in FSL_MAIN are now printed to stderr rather than stdout.
  • Fix a problem for Boost 1.62.0
  • Add the Panoptes file logger as the first of a set of optional loggers.
  • Start to make use of the Unicode support in f5-cord.
  • Make hashing digester and hmac digester movable.
  • Add a set member to the jcursor that overwrites old values in JSON structures.
  • Add logging messages about why a JWT gets rejected.
  • We need to have < defined on JSON so do that instead of std::less.
  • Added std::less overload for JSON with new fost/json header.

fost-internet

  • Add a cookie header parser.
  • Replace older Boost code with std::function and lambdas.

fost-orm

  • Record jsondb saves and handle rename errors better.
  • Add a missing lock when adding database post-commit hooks.
  • Refactor the jsondb save process to make it more clear what the actual process is.

fost-postgres

  • Add support for passing a password for the PG DSN.

fost-web

  • Load shared objects before setting up the logger so that optional loggers can be loaded.
  • Add a generic 200 response.
  • Web view execution now gives a stack trace when an exception is thrown.
  • Improved error output where a view name is mis-configured.
  • Removed the old router function as it's been replaced by view::execute.
  • Add a middleware that logs the HTTP request details and some response details.

fostgres

  • Add default lookups for the Postgres connection string parameters.

cord

  • Start to add support for Unicode with UTF-8 decoding.

threading

  • Allow for fetching of the Boost.ASIO work pool thread count.
  • lower_bound on tsmap takes the key value by reference not by copy.
  • Added a missing header.
  • Removed a dependancy on algorithms header.

Categories:

Dependency injected exceptions or error handling

Posted 15th October, 2016 05:22 (UTC), last edited 27th June, 2017 11:41 (UTC)

Let's say we want to check that a Unicode code point is within the valid range before working out how many bytes it's UTF-8 representation is going to be:

void assert_valid(uint32_t cp) {
    if ( cp >= 0xd800 && cp <= 0xdbff ) {
        throw std::domain_error("UTF32 code point is in the leading UTF16 surrogate pair range");
    } else if ( cp >= 0xdc00 && cp <= 0xdfff ) {
        throw std::domain_error("UTF32 code point is in the trailing UTF16 surrogate pair range");
    } else if ( cp == 0xfffe || cp == 0xffff ) {
        throw std::domain_error("UTF32 code point is invalid");
    } else if ( cp > 0x10ffff ) {
        throw std::domain_error("UTF32 code point is beyond the allowable range");
    }
}

And we can now use it like this:

std::size_t u8length(uint32_t cp) {
    check_valid(cp);
    if ( cp < 0x00080 ) return 1u;
    else if ( cp < 0x00800 ) return 2u;
    else if ( cp < 0x10000 ) return 3u;
    else return 4u;
}

We now get an exception thrown if the Unicode is invalid. But what if we don't like the exception that is thrown? Maybe we want to throw our own exception class because it has some extra facilities?

We can refactor both functions to take the exception class to throw as a parameter:

template<typename E = std::domain_error> inline
void assert_valid(uint32_t cp) {
    if ( cp >= 0xd800 && cp <= 0xdbff ) {
        throw E("UTF32 code point is in the leading UTF16 surrogate pair range");
    } else if ( cp >= 0xdc00 && cp <= 0xdfff ) {
        throw E("UTF32 code point is in the trailing UTF16 surrogate pair range");
    } else if ( cp == 0xfffe || cp == 0xffff ) {
        throw E("UTF32 code point is invalid");
    } else if ( cp > 0x10ffff ) {
        throw E("UTF32 code point is beyond the allowable range");
    }
}

template<typename E = std::domain_error> inline
std::size_t u8length(uint32_t cp) {
    check_valid<E>(cp);
    if ( cp < 0x00080 ) return 1u;
    else if ( cp < 0x00800 ) return 2u;
    else if ( cp < 0x10000 ) return 3u;
    else return 4u;
}

Most of our users of u8length can still write u8length(cp) and get the default exception, but in my code I can now use it as u8length<fostlib::exceptions::unicode_error>(cp) and I get my exception class instead.

Error return

If we stopped there it's a little bit interesting, but hardly worthy of much note. With another couple of refactorings though we can have this same code returns errors for those who want that rather than exceptions. First of all let's factor out the throwing of the exception:

template<typename E> inline
void raise(const char *what) {
    throw E(what);
}

template<typename E = std::domain_error> inline
void assert_valid(uint32_t cp) {
    if ( cp >= 0xd800 && cp <= 0xdbff ) {
        raise<E>("UTF32 code point is in the leading UTF16 surrogate pair range");
    } else if ( cp >= 0xdc00 && cp <= 0xdfff ) {
        raise<E>("UTF32 code point is in the trailing UTF16 surrogate pair range");
    } else if ( cp == 0xfffe || cp == 0xffff ) {
        raise<E>("UTF32 code point is invalid");
    } else if ( cp > 0x10ffff ) {
        raise<E>("UTF32 code point is beyond the allowable range");
    }
}

The next thing to note is that the domains of our return values allow for errors to be returned. assert_valid can return a bool and u8length could return zero if the code point is invalid. This would then look like this:

template<typename E = std::domain_error> inline
bool assert_valid(uint32_t cp) {
    if ( cp >= 0xd800 && cp <= 0xdbff ) {
        raise<E>("UTF32 code point is in the leading UTF16 surrogate pair range");
    } else if ( cp >= 0xdc00 && cp <= 0xdfff ) {
        raise<E>("UTF32 code point is in the trailing UTF16 surrogate pair range");
    } else if ( cp == 0xfffe || cp == 0xffff ) {
        raise<E>("UTF32 code point is invalid");
    } else if ( cp > 0x10ffff ) {
        raise<E>("UTF32 code point is beyond the allowable range");
    } else {
        return true;
    }
    return false;
}

template<typename E = std::domain_error> inline
std::size_t u8length(uint32_t cp) {
    if ( !check_valid<E>(cp) ) return 0u;
    else if ( cp < 0x00080 ) return 1u;
    else if ( cp < 0x00800 ) return 2u;
    else if ( cp < 0x10000 ) return 3u;
    else return 4u;
}

Of course we'll still always get the exception because raise throws. What we can do though is to choose a type that means “don't throw” and then specialise raise on that type. void seems like it's probably a good choice:

template<> inline
void raise<void>(const char *) {}

Now we have two choices about how to use u8length:

auto bytes = u8length(cp);
dosomething(bytes, cp);

The use of exceptions by default is important from a security context. Unicode errors are a common attack vector, and throwing an exception if something is wrong simply makes the software a good deal safer as error handling code is just too easy to get wrong. But of course, sometimes we will want to do this sort of thing:

if ( auto bytes = u8length<void>(cp); bytes ) { // C++17
    dosomething(bytes, cp);
} else {
    // Handle error
}

Most API uses get the safety of the exception being thrown, but you can also opt out of that and just get an error return for the cases where that is preferable.

Real code for raise and the UTF encodings can be found in the f5-cord library* [*This code isn't in master yet. Check out the previously linked commits instead.]. Note that this version is also constexpr, which is also pretty cool.

The (approximate) final versions are reproduced below. And yes, raise is probably a stupid name now.

namespace f5 {
    
/// Raise an error of type E giving it the specified error text
template<typename E>
constexpr inline
void raise(f5::cord::lstring error) {
    throw E(error.c_str());
}

/// Specialisation for when we want an error return
template<>
constexpr inline
void raise<void>(f5::cord::lstring) {
}

/// A UTF-8 code point
typedef unsigned char utf8;
/// A UTF-32 code point
typedef uint32_t utf32;

/// Check that the UTF32 code point is valid. Throw an exception if not.
template<typename E = std::domain_error>
constexpr inline
bool check_valid(utf32 cp) {
    if ( cp >= 0xd800 && cp <= 0xdbff ) {
        raise<E>("UTF32 code point is in the leading UTF16 surrogate pair range");
    } else if ( cp >= 0xdc00 && cp <= 0xdfff ) {
        raise<E>("UTF32 code point is in the trailing UTF16 surrogate pair range");
    } else if ( cp == 0xfffe || cp == 0xffff ) {
        raise<E>("UTF32 code point is invalid");
    } else if ( cp > 0x10ffff ) {
        raise<E>("UTF32 code point is beyond the allowable range");
    } else {
        return true;
    }
    return false;
}

/// Return the number of UTF8 values that this code point will
/// occupy. If the code point falls in an invalid range then an
/// exception will be thrown.
template<typename E = std::domain_error>
constexpr inline
std::size_t u8length(utf32 cp) {
    if ( not check_valid<E>(cp) ) return 0u;
    else if ( cp < 0x00080 ) return 1u;
    else if ( cp < 0x00800 ) return 2u;
    else if ( cp < 0x10000 ) return 3u;
    else return 4u;
}

}

And some tests to show that it is also usable at compile time:

static_assert(f5::check_valid(' '), "Space is a valid UTF32 code point");
static_assert(not f5::check_valid<void>(0xff'ff'ff), "0xff'ff'ff is not a valid code point");

static_assert(f5::u8length(0) == 1, "Zero is still 1 byte long");
static_assert(f5::u8length(' ') == 1, "Space is 1 byte long");
static_assert(f5::u8length(0xa3) == 2, "Pounds are 2 bytes long");
static_assert(f5::u8length(0xe01) == 3, "Thai chickens are 3 bytes long");
static_assert(f5::u8length(0x1d11e) == 4, "The treble clef is 4 bytes long");

The most obvious improvement is for raise to perfectly forward any arguments on to the exception constructor, but we still can't partially specialise functions so the implementation will be a bit more complex as the specialisation will have to be done in a class.


Categories: