System framework improvements

The System's code has had a lot of iterations to it which i do wish i could have gotten to quicker, but it's definitely worth it. It is now a lot slimmer, removing two classes (four source files) without losing any features.

When i first wrote the code it became large because i wanted to make it very flexible. I have now removed redundant code and cleaned it up a lot. The System's code needs to be fast and very solid as its the base for a lot of the engine and its plug-ins. These latest changes should be the last major ones made on these classes now (at last!!).

As it stands we now have:

• TaskThread - Wraps a thread and provides processing of tasks
• SystemGroup - Manages a group of System's and enforces a specific start up and tear down sequence for each System registered to it, as well as periodical updates (ticks).
• System - Allows a class to be managed by a SystemGroup and receives ticks.

Each SystemGroup uses a single TaskThread to process its tasks. This provides the main part of the multi-threaded framework.

We had a problem where we would get 50fps in the samples even though they were set to be ticked at 60fps (as well as any movements in the fps sample would causes large frame rate decreases). This problem has disappeared now due to these changes, getting a very stable frame rate even whilst moving around. It is also responsive to small changes in tick intervals.

Ticking the graphics system at 16.6666667 gives a frame rate of 60fps and a tick interval of 16 gives a frame rate of 62fps. Obviously this is what it should do, but it does suggest (i still need to profile to get proof) that there is a very low latency between the primary thread waking up and all of the Systems being ticked.

My next aims are to provide built-in debugging code/features to help with development (especially to track down threading and release mode issues) and possibly improve memory management, as there are a few places where we are allocating and freeing memory a lot.

Engine initialisation and threading improvements

I haven't blogged in a while mainly because nothing major has really happened. I've mostly been learning to drive which is going very well. I have my theory test on Thursday which should be OK as I've been doing well on the practice CD i have.

I've just committed a fairly big update to the utilities and the core mainly aimed at threading as well as the engines initialisation sequence. One problem that existed was that for systems that are very slow to initialise (like OGRE) the engine would start running/ticking before it has completed.

There were many different possible solutions but the solution i went for is process every initial request (register/initialise/schedule) and only once every single one has completed, initialisation will finish. Ticking will not occur because the EngineTimingHandler has not been started yet, so game states etc cannot be ticked and cause the original problems.

A problem that existed before and probably emphasised with this solution is that all systems assumed the current time is 0 when they were created, and during initialisation they we're told it was 0. By the time everything has initialised, the time could be at least several seconds later, causing them to calculate very large time intervals which would result in incorrect calculations.

The solution now is that when a system is scheduled it is told the current time, which could be incorrect if the engine hasn't started running yet. The EngineTimingHandler solves this by telling every system that is scheduled the real current time before it starts running.

There are still many improvements to be made to OGE's threading but it has been fairly easy to make the necessary changes so i think our current approach is working well. The main improvement i want to make soon is to get OGE to work out how many threads to create at the start semi automatically. It will take various hints and requirements (from config) and retrieve details about the current CPU (such as number of packages, cores and logical processors) to work out how many threads to create.

On another note, i think its possible for us to get Intel's Threading Building Blocks working in OGE (the scheduler/algorithms part) because it allows you to specify the number of threads that the TBB scheduler can use/create. It would allow us to allocate all the threads we need and then let TBB create the remaining number of threads. On future CPU's with more cores than work that we can allocate from the various systems, we can put more workload onto the tbb scheduler.

Framework for Features

We have been writing framework code for quite a long time now for OGE, a large amount being utilities and a fair amount for the core. We are now starting to add new projects/modules, currently graphics, input and audio. We are really close to being able to start adding loads of features to OGE which i can't wait to be able to do. Finally the fun part! Sadly though we still have some more framework to add, such as the basic initialisation/shutdown of each of the plug-ins and adding more to the core, but I'm hoping this won't take long.

Once we have the first few plug-ins up and running, it will be a lot quicker to add the remaining ones as we won't have to discuss how to name various files and classes.

One of Each

I handed in my Advanced Processors coursework today, what a relief. I can now focus on the group project coursework which the presentation day is on the 6th of June, so not long away. I also have an exam on Monday, yay... So still lots to do but I'm getting there.

I had a bit of time to work on OGE. I've ported the Zip file code so now OGE can handle zip files through the file system. I haven't had much time to test it but it should work as I haven't changed much from the old code.

I've started writing some code to allow you to get a different thread to call any function with specified data. In the old code, we had Extensions but these felt a bit to hard to use and fiddly. Once I've finalised it and checked with Steven, i will clean it up and add it to the utilities. I'm not yet sure if Extensions will be needed at all after this, i will need to look at some use cases.

Configuration changes to OGE

I've had a bit of time since finishing my last bit of coursework and starting my next (ugh) so i worked on OGE's configuration/options code. There we're a few main features i wanted to add so i had to spend some time working out how to fit it in. I had tried previously but it came out a mess.

I still have quite a bit of work left on this but currently I have expanded oge::Option to allow:

• very basic tracking of what options have been added have been added
• tracking whether any values of an option have been added/changed/removed

These are very simple so don't allow complex tracking but will allow only the modified parts of an option hierarchy to be written to a config file. This can be used to have a read-only main config file and a user chosen or hardware dependant/specific settings config file. Only the settings that have changed compared to the read-only file have to be saved seperately.

I'm also working on the abstract option file parser and an XML implementation.
It seems to be going well at the moment and i'm hoping it won't be too long until i've finished and can commit but i've got a lot of coursework to do sadly.

I also realised i only have 24 days left of uni this year!

Lots of small changes

Since i finished some coursework that was due in today, I've managed to spend a bit of time working on OGE. Instead of making loads of changes and then committing the whole lot in one go, I'm now trying to commit more regularly with smaller additions. Various files have been moved around, data stream helper classes have been added, small changes and documentation to threading code have been made. Steven also ported the old serialisation code and is working on logging.

There is not much left to add to the utilities so very soon work will begin on the core which should be much more interesting. This will lead nicely into my summer holidays.

Looking at OGE now, it's so much cleaner and more organised than our old code. The utilities (at least in my opinion) is turning out very nicely and i at least will find it very useful when working on projects in the future, I hope others will also find it useful.

Threading in OGE

I recently committed the new threading code for OGE. It's still work in progress with changes needing to be made and more features to be added but the basis is there now.

In the old OGE code, a SystemThread allowed one or more System's to be initialised onto it and ticked/updated at regular intervals. In the new version instead of each System working out if it should update or not, each SystemThread goes through a SystemController. Using both of these, different ways of managing time can be implemented. Currently we have a "Fast" implementation which the SystemThreadFast class passes the current time (when it receives it) to all of the SystemController's. Each SystemController checks to see if it's System should be ticked.

Overall this keeps a lot of threading detail away from the user, so hopefully they won't need to know a huge amount about multi threading before using OGE.

Six Day Weekend

I havent blogged in a while because I have been fairly busy with uni and OGE, but hopefully from now I can be a bit more relaxed.

Firstly, I have a 6 day weekend now as most of my units in uni are coursework based, we managed to get our 2 lectures fit into a single day. This of course saves me a lot of time (an hour travel every day) and money, which i can now use to buy cheap DVDs...

Secondly, Leanne and I, came up with a schedule for me to use that will allow me to do all the work I need to but to have free time as well. Usually I end up thinking that I should be working nearly all the time, because there's no set schedule that I can stick to. So now I have OGE scheduled meaning I can definitely keep working on that as well as get all my work done.

For OGE, I just committed the first public preview of OGE utilities to SVN, there's obviously a lot to add to it still. Once it has been documented and enough features added to it then work can be started on the core, which will be a lot more fun.

The dependencies for the core is now POCO and Intel's Threading Building Blocks. The latter we added to be able to get as much performance as we can out of threading in OGE, as they include a lot of very useful utilities. I don't think we will end up using their parrallel code, as OGE will manage its own threads, but atomic primitives, fast mutexs and concurrent containers will be very useful.

Time flies again

Over the last year, my time spent programming often goes quite slow and is hard to get motivated enough. Yesterday and today i've been learning some fairly simple x86 assembly to begin making thread-safe lock/wait free containers (queue, list, stack etc) for OGE. I spent hours on it, and they flew by. I realise it feels exactly how it used to feel when i first started learning to program, which back then, i spent along time which flew by very fast.

Currently i have a set of atomic operations, such as add, exchange and add, swap, compare and swap etc. I also have an Atomic template class which uses these operations. This class can be used for thread-safe counters. This is leading up to create some containers, which should improve OGEs performance considerably, as there will be alot less locks being performed.

We didn't get accepted for Google Summer of Code this year, it seems they only accepted quite big/mature projects. We will hopefully have a better chance next year.

Virtual File System

Over the last few days i've started working on the virtual file system for OGE. The virtual file system allows access to files independant of where they come from, such as a compressed file or remotely on a server. The users code only ever interacts with Archive's, File's and DataStream's.

In the current version of OGE (in our CVS) the file system and resource management code were together. The naming was long winded and felt hard to maintain. This has changed in the new version. The two implementations that will come with OGE (normal files and compressed files (using zlib)) will be seperated as well to help keep the code cleaner and easier to maintain.

Once OGE has plugin management, it will be possible to register new types through plugins. This should be made easier with POCO too, as it has shared library classes and other utilities, such as a ClassLoader (inspired from Java).

Java habits

Work on OGE's core and utilities is progressing slowly but steadily. I'm currently working on a group of utility classes (Map, List and possibly Queue and Stack). For each of these, there will be a thread-safe and non thread-safe version. These probably won't be used directly in the core or the systems but are provided for faster prototyping or for users that aren't as familiar with multithreaded programming.

Over the past semester at university i've been using Java alot and didn't get alot of time to program in C++. I'm now finding various Java habits are affecting my programming. An example, i keep writing the visibility of member functions and variables with the declarations, like "public int a;" which is invalid C++ code. Another habit is writing boolean instead of bool, which is the opposite of the problem i had when starting to program in Java after using C++ alot. This is very fustrating...

A little update

Before i went to bed last night, i uninstalled everything that visual studio express installed, and installed just the C++ part (i couldnt be bothered to wait for the other languages). It seems MC.exe is in the SDK now, i have no idea why it didn't install the first time. So POCO now builds fine.

I'm writing simple scripts to automate the tedious tasks of copying all of the dependencies OGE needs into the dependencies folder. There will also be scripts to make SDKs much easier too. This should mean that we can keep the latest prebuilt dependencies up to date on sourceforge, as well as uploading new SDKs.

POCO and VC++ 2008

i've been working on the new structure for OGE over the last few days. I think it will be an improvement from the old one. I took some time to get POCO built and to see if it can be used to replace boost. So far i haven't been able to build it. I get an error that mc is not a valid command. mc.exe is a windows message compiler that comes with the SDK, but when i looked for it, it wasn't there. Installing the win32 tools from the microsoft website contained mc.exe, i have no idea why it hasn't been installed with 2008 express.

I'm going to copy the mc.exe over to the other SDK, hopefully it will work.

It's a shame problems like these arise, they waste a lot of time trying to solve them. Thanks to members of POCO project for their help.

Multi-processor builds

In a gap between revision i decided to test out the multi-processor build feature of visual studio 2008 (express). I tried it out on OGRE (CVS HEAD) as its quite a large project and i need it built for the future anyway. It took me a while to work out how to enable multi-processor builds, there doesn't seem to be anything in the project settings. Adding /MP to the command line options for the source files didn't work, which later i noticed that the build output said that because minimal rebuild was enabled, it was disabling /MP.

I built OgreMain.dll (in release mode) first without /MP which took about 12 minutes. With /MP it took 4 minutes 25 seconds, which is a great improvement. I built the rest of ogre, which from what i remember, takes about 40 minutes, this took 8 minutes (not including OgreMain as it's already built). This is on a Q6600 (2.4Ghz) and 2GB of RAM.

This will be very useful when developing OGE, as a lot of my time was wasted waiting for builds. Also, the new structure of OGE that I'll be working on will allow less full rebuilds, except when altering the core lib.

Uni Coursework/Exams

I've been very busy lately with uni coursework, namely Distributed Web Systems, where we are building a (fairly simple) distributed file sharing system, in small groups. The aim is to have a group of storage servers that store a consistent list of files, and any number of clients (web based front ends) can connect to any of the servers. The system should be able to handle server crashes.

I'm working on the server and the part of the client that interacts with them. I'm almost there, but like most projects i work on, every time i get a bit closer, the end gets a bit further away. It's a shame that the deadline doesn't follow suit. I have only a week left to finish off my part, integrate it with my team mates work, full test it and submit a report. My exams start the week after and go on for three weeks. I haven't really started revising yet because of coursework.

I'm looking forward to after the exams, i can begin work on OGE again. A big refactor is being planned, with the aim being to make it more modular and easy to develop with. I will go into more detail later...