Although i haven't blogged in a long time, I thought i'd happily note that i now have a Master's degree (MEng) in Computer Engineering with Merit (2.1).
It's been a long slog but well worth it. Now i just need to write/update my CV and start searching for jobs.
Monday, 6 July 2009
Monday, 20 October 2008
Processor Detection
I haven't blogged in a while because I've been busy with uni and trying to get as much programming done as i can.
I've been working on the processor detection code as it is very important to allow OGE to scale depending on the hardware its running on. It has two possible methods of detecting the information it needs but it depends on the compiler and the operating system that it runs on.
In Visual C++ 2008, it will compile with the newer windows API that provides package/core/logical information. This is only available on windows XP SP3, Vista and Windows Server 2008 (possibly some others that i am not aware of). OGE will check to see if this is available when it is running, if it not, i will revert to the secondary approach.
The secondary approach retrieves the number of logical processors an OS API call, it will then attempt to retrieve the rest of the information from the CPU itself using cpuid. Combining all of this information it can work out the number of cores, although currently it can't work out the number of packages from this.
If both approaches fail, it will default to 1 package, 1 core and 1 logical processor.
On Linux, the primary method is to read in /proc/cpuinfo and parse it to find out all of the processor information it needs. If for some reason it can't do this, it will revert to the secondary approach which is the same as on windows.
Mac support still needs to be done but that can wait till later.
With using these approaches and the features of each compiler, i realise that there is a very specific situation that will result in only the logical processors being detected. This will occur if OGE is compiled using visual studio 2005 as 64bit AND it is executed on a computer that has windows XP SP2 or earlier AND an Intel multi-core processor.
The reason is that visual studio 2005 does not allow use of inline assembly when compiling 64 bit applications, so __cpuid intrinsic must be used. The flaw with it is that it only accepts one input value in EAX, and part of detecting the number of cores requires that ECX be set to 0. This means we can't compile this in, and instead say that it could not be done. As this is the secondary approach, if the target computer is running XP SP3, vista etc it won't ever use this as it will use the primary approach instead. The problem only occurs on Intel multi-core processors because there only they can have hyper-threading.
This is unlikely to ever cause problems as it can be solved in many different ways, such as using visual studio 2008 to compile 64 bit applications (it has __cpuidex intrinsic allowing ECX to be specified).
It has been interesting implementing this, but has taken far too long as well. I will happy to get on with a new task.
I've been working on the processor detection code as it is very important to allow OGE to scale depending on the hardware its running on. It has two possible methods of detecting the information it needs but it depends on the compiler and the operating system that it runs on.
In Visual C++ 2008, it will compile with the newer windows API that provides package/core/logical information. This is only available on windows XP SP3, Vista and Windows Server 2008 (possibly some others that i am not aware of). OGE will check to see if this is available when it is running, if it not, i will revert to the secondary approach.
The secondary approach retrieves the number of logical processors an OS API call, it will then attempt to retrieve the rest of the information from the CPU itself using cpuid. Combining all of this information it can work out the number of cores, although currently it can't work out the number of packages from this.
If both approaches fail, it will default to 1 package, 1 core and 1 logical processor.
On Linux, the primary method is to read in /proc/cpuinfo and parse it to find out all of the processor information it needs. If for some reason it can't do this, it will revert to the secondary approach which is the same as on windows.
Mac support still needs to be done but that can wait till later.
With using these approaches and the features of each compiler, i realise that there is a very specific situation that will result in only the logical processors being detected. This will occur if OGE is compiled using visual studio 2005 as 64bit AND it is executed on a computer that has windows XP SP2 or earlier AND an Intel multi-core processor.
The reason is that visual studio 2005 does not allow use of inline assembly when compiling 64 bit applications, so __cpuid intrinsic must be used. The flaw with it is that it only accepts one input value in EAX, and part of detecting the number of cores requires that ECX be set to 0. This means we can't compile this in, and instead say that it could not be done. As this is the secondary approach, if the target computer is running XP SP3, vista etc it won't ever use this as it will use the primary approach instead. The problem only occurs on Intel multi-core processors because there only they can have hyper-threading.
This is unlikely to ever cause problems as it can be solved in many different ways, such as using visual studio 2008 to compile 64 bit applications (it has __cpuidex intrinsic allowing ECX to be specified).
It has been interesting implementing this, but has taken far too long as well. I will happy to get on with a new task.
Thursday, 25 September 2008
Results!
I've passed my driving exam! yay!!! It went very well. We did the maneuvers very early on so i could relax a bit and just concentrate on driving around. Overall i only got 2 minors, i am very happy!
That's one thing that i don't have to worry about now and i can concentrate on starting uni.
That's one thing that i don't have to worry about now and i can concentrate on starting uni.
Wednesday, 24 September 2008
Nerves
I've got my driving exam tomorrow, part of me is looking forward to it being quietly confident that It'll be OK but the other part of me is nervous. The main reason being that you can't control what other drivers will do, it could only take one dodgy driver to cause me to fail. I'm pretty confident that i can drive to the required standard and perform the maneuvers well enough. I will just have to see how it goes.
This week has been fairly busy where i was at a wedding over the weekend which was good, the last few driving lessons and my exam tomorrow. I also have enrollment day at uni on Monday and i need to buy various things for when i start back.
Doesn't seem long ago that i had a whole four months off, now I'm down to a week...
This week has been fairly busy where i was at a wedding over the weekend which was good, the last few driving lessons and my exam tomorrow. I also have enrollment day at uni on Monday and i need to buy various things for when i start back.
Doesn't seem long ago that i had a whole four months off, now I'm down to a week...
Tuesday, 9 September 2008
System framework improvements
The System's code has had a lot of iterations to it which i do wish i could have gotten to quicker, but it's definitely worth it. It is now a lot slimmer, removing two classes (four source files) without losing any features.
When i first wrote the code it became large because i wanted to make it very flexible. I have now removed redundant code and cleaned it up a lot. The System's code needs to be fast and very solid as its the base for a lot of the engine and its plug-ins. These latest changes should be the last major ones made on these classes now (at last!!).
As it stands we now have:
We had a problem where we would get 50fps in the samples even though they were set to be ticked at 60fps (as well as any movements in the fps sample would causes large frame rate decreases). This problem has disappeared now due to these changes, getting a very stable frame rate even whilst moving around. It is also responsive to small changes in tick intervals.
Ticking the graphics system at 16.6666667 gives a frame rate of 60fps and a tick interval of 16 gives a frame rate of 62fps. Obviously this is what it should do, but it does suggest (i still need to profile to get proof) that there is a very low latency between the primary thread waking up and all of the Systems being ticked.
My next aims are to provide built-in debugging code/features to help with development (especially to track down threading and release mode issues) and possibly improve memory management, as there are a few places where we are allocating and freeing memory a lot.
When i first wrote the code it became large because i wanted to make it very flexible. I have now removed redundant code and cleaned it up a lot. The System's code needs to be fast and very solid as its the base for a lot of the engine and its plug-ins. These latest changes should be the last major ones made on these classes now (at last!!).
As it stands we now have:
- TaskThread - Wraps a thread and provides processing of tasks
- SystemGroup - Manages a group of System's and enforces a specific start up and tear down sequence for each System registered to it, as well as periodical updates (ticks).
- System - Allows a class to be managed by a SystemGroup and receives ticks.
We had a problem where we would get 50fps in the samples even though they were set to be ticked at 60fps (as well as any movements in the fps sample would causes large frame rate decreases). This problem has disappeared now due to these changes, getting a very stable frame rate even whilst moving around. It is also responsive to small changes in tick intervals.
Ticking the graphics system at 16.6666667 gives a frame rate of 60fps and a tick interval of 16 gives a frame rate of 62fps. Obviously this is what it should do, but it does suggest (i still need to profile to get proof) that there is a very low latency between the primary thread waking up and all of the Systems being ticked.
My next aims are to provide built-in debugging code/features to help with development (especially to track down threading and release mode issues) and possibly improve memory management, as there are a few places where we are allocating and freeing memory a lot.
Monday, 25 August 2008
Busy Busy...
Last week i had my driving theory test which i passed! I can't be fully happy until I've passed the practical though which is in a months time. My family, girlfriend and i went to my cousins wedding on the weekend as well, which was good.
Next month is going to be busy as i have another wedding to go to, my driving practical test and enrolment for my final year of uni. My holiday has gone so fast and i don't really feel like I've got much done at all. I had better make this last month count.
Next month is going to be busy as i have another wedding to go to, my driving practical test and enrolment for my final year of uni. My holiday has gone so fast and i don't really feel like I've got much done at all. I had better make this last month count.
Saturday, 16 August 2008
Engine initialisation and threading improvements
I haven't blogged in a while mainly because nothing major has really happened. I've mostly been learning to drive which is going very well. I have my theory test on Thursday which should be OK as I've been doing well on the practice CD i have.
I've just committed a fairly big update to the utilities and the core mainly aimed at threading as well as the engines initialisation sequence. One problem that existed was that for systems that are very slow to initialise (like OGRE) the engine would start running/ticking before it has completed.
There were many different possible solutions but the solution i went for is process every initial request (register/initialise/schedule) and only once every single one has completed, initialisation will finish. Ticking will not occur because the EngineTimingHandler has not been started yet, so game states etc cannot be ticked and cause the original problems.
A problem that existed before and probably emphasised with this solution is that all systems assumed the current time is 0 when they were created, and during initialisation they we're told it was 0. By the time everything has initialised, the time could be at least several seconds later, causing them to calculate very large time intervals which would result in incorrect calculations.
The solution now is that when a system is scheduled it is told the current time, which could be incorrect if the engine hasn't started running yet. The EngineTimingHandler solves this by telling every system that is scheduled the real current time before it starts running.
There are still many improvements to be made to OGE's threading but it has been fairly easy to make the necessary changes so i think our current approach is working well. The main improvement i want to make soon is to get OGE to work out how many threads to create at the start semi automatically. It will take various hints and requirements (from config) and retrieve details about the current CPU (such as number of packages, cores and logical processors) to work out how many threads to create.
On another note, i think its possible for us to get Intel's Threading Building Blocks working in OGE (the scheduler/algorithms part) because it allows you to specify the number of threads that the TBB scheduler can use/create. It would allow us to allocate all the threads we need and then let TBB create the remaining number of threads. On future CPU's with more cores than work that we can allocate from the various systems, we can put more workload onto the tbb scheduler.
I've just committed a fairly big update to the utilities and the core mainly aimed at threading as well as the engines initialisation sequence. One problem that existed was that for systems that are very slow to initialise (like OGRE) the engine would start running/ticking before it has completed.
There were many different possible solutions but the solution i went for is process every initial request (register/initialise/schedule) and only once every single one has completed, initialisation will finish. Ticking will not occur because the EngineTimingHandler has not been started yet, so game states etc cannot be ticked and cause the original problems.
A problem that existed before and probably emphasised with this solution is that all systems assumed the current time is 0 when they were created, and during initialisation they we're told it was 0. By the time everything has initialised, the time could be at least several seconds later, causing them to calculate very large time intervals which would result in incorrect calculations.
The solution now is that when a system is scheduled it is told the current time, which could be incorrect if the engine hasn't started running yet. The EngineTimingHandler solves this by telling every system that is scheduled the real current time before it starts running.
There are still many improvements to be made to OGE's threading but it has been fairly easy to make the necessary changes so i think our current approach is working well. The main improvement i want to make soon is to get OGE to work out how many threads to create at the start semi automatically. It will take various hints and requirements (from config) and retrieve details about the current CPU (such as number of packages, cores and logical processors) to work out how many threads to create.
On another note, i think its possible for us to get Intel's Threading Building Blocks working in OGE (the scheduler/algorithms part) because it allows you to specify the number of threads that the TBB scheduler can use/create. It would allow us to allocate all the threads we need and then let TBB create the remaining number of threads. On future CPU's with more cores than work that we can allocate from the various systems, we can put more workload onto the tbb scheduler.
Friday, 18 July 2008
First driving lesson!
I just had my first driving lesson today which went really well. Learnt some of the basics and got to drive around a car park a bit. I am at least happy i haven't stalled the car...yet. Bit of a squeeze getting into the car as its smallish and I'm not. My instructor seems good as well, i think I'll be able to learn at a good rate. I'm hoping the rest of my lessons go as well as (or better than) today's.
Monday, 14 July 2008
Framework for Features
We have been writing framework code for quite a long time now for OGE, a large amount being utilities and a fair amount for the core. We are now starting to add new projects/modules, currently graphics, input and audio. We are really close to being able to start adding loads of features to OGE which i can't wait to be able to do. Finally the fun part! Sadly though we still have some more framework to add, such as the basic initialisation/shutdown of each of the plug-ins and adding more to the core, but I'm hoping this won't take long.
Once we have the first few plug-ins up and running, it will be a lot quicker to add the remaining ones as we won't have to discuss how to name various files and classes.
Once we have the first few plug-ins up and running, it will be a lot quicker to add the remaining ones as we won't have to discuss how to name various files and classes.
Tuesday, 24 June 2008
Final results!
I checked my results again on my uni's portal and they have all been finalised. I've got five 2.1's and one 2.2. Over the three years, i've got 15 1sts (>= 70%), 13 2.1's (60 - 60%), 5 2.2's (50 - 59%) and only 3 passes (40 - 40%). I'm really going to have work hard in the last year as i want to pull my average up a lot. Not all of the years count, in BEng the second and third years count but I'm not sure about MEng, do the last 2 or 3 years count?
I am trying to think of some software projects i can do for my final year project, I'm hoping it can be something related to games or even OGE. I really want to show off my programming abilities in this project to hopefully give me a better chance at getting a good job.
I am trying to think of some software projects i can do for my final year project, I'm hoping it can be something related to games or even OGE. I really want to show off my programming abilities in this project to hopefully give me a better chance at getting a good job.
Wednesday, 18 June 2008
Rough Core
I've been working on the core recently. I haven't been able to program as much as would like. To be able to get more done, i am focusing on getting a basic core up and running, where most configuration is done manually. This will allow the separate systems to be developed in parallel with the core. I will then continue to work on the core but being able to put all of the features that is needed/wanted in it.
These features are mainly thread handling and system creation and configuration. For the moment, you tell the core how many threads to create and it will just distribute the different systems by adding them in a cycle or round-robin way. The more advanced way, which is the goal, is to get OGE to detect the number of processors the computer has and to use various user defined settings to create the correct number and types of threads. It will also allow different ways of distributing the systems between the threads. The goal is to allow the engine to scale to the hardware its running on without altering the engine or application.
These features are mainly thread handling and system creation and configuration. For the moment, you tell the core how many threads to create and it will just distribute the different systems by adding them in a cycle or round-robin way. The more advanced way, which is the goal, is to get OGE to detect the number of processors the computer has and to use various user defined settings to create the correct number and types of threads. It will also allow different ways of distributing the systems between the threads. The goal is to allow the engine to scale to the hardware its running on without altering the engine or application.
Provisional Results!
I just checked my uni's portal for the latest results from the last semester. All of them are filled which 4 of 5 units are a 2.1 and the last is a 2.2. One of my units is also a double credit unit, so that's five 2.1's and one 2.2, so I'm happy! These of course, are provisional and can change when they are moderated, but I can finally relax.
Friday, 6 June 2008
I'm Free!
At last, this uni year has finished! I now have to wait for my exams. I can work on OGE a lot more now, which i have some personal goals for that i really need to complete over the holiday. So hopefully by the end OGE will be at a very basic, usable state that at least simple games can be made.
Wednesday, 28 May 2008
One of Each
I handed in my Advanced Processors coursework today, what a relief. I can now focus on the group project coursework which the presentation day is on the 6th of June, so not long away. I also have an exam on Monday, yay... So still lots to do but I'm getting there.
I had a bit of time to work on OGE. I've ported the Zip file code so now OGE can handle zip files through the file system. I haven't had much time to test it but it should work as I haven't changed much from the old code.
I've started writing some code to allow you to get a different thread to call any function with specified data. In the old code, we had Extensions but these felt a bit to hard to use and fiddly. Once I've finalised it and checked with Steven, i will clean it up and add it to the utilities. I'm not yet sure if Extensions will be needed at all after this, i will need to look at some use cases.
I had a bit of time to work on OGE. I've ported the Zip file code so now OGE can handle zip files through the file system. I haven't had much time to test it but it should work as I haven't changed much from the old code.
I've started writing some code to allow you to get a different thread to call any function with specified data. In the old code, we had Extensions but these felt a bit to hard to use and fiddly. Once I've finalised it and checked with Steven, i will clean it up and add it to the utilities. I'm not yet sure if Extensions will be needed at all after this, i will need to look at some use cases.
Thursday, 22 May 2008
Time dragging by
It feels like I've only had a few weeks left for months now. Each day seems to go by quite fast allowing me to get a lot less coursework done than i need to but the weeks are going so slowly.
My coursework for advanced processors, which is to write an IEEE style research paper is a nightmare at the moment. Mine is on multi-core processors. We have a ten page limit, no minimum although i would prefer to have 8 or 9 pages in the end.
I'm currently on 2 and a quarter pages, with 20 references, and its taken me a few days, which i have till Wednesday to hand it in. It keeps feeling like it will all be OK and ill finish with a lot of time to spare but then suddenly it will feel like I'm never going to get to 5 pages in that time, let alone 8 or 9.
I can't wait for all of this to be over and to start my holidays, i really need the break. I can only imagine what next year will be like...
My coursework for advanced processors, which is to write an IEEE style research paper is a nightmare at the moment. Mine is on multi-core processors. We have a ten page limit, no minimum although i would prefer to have 8 or 9 pages in the end.
I'm currently on 2 and a quarter pages, with 20 references, and its taken me a few days, which i have till Wednesday to hand it in. It keeps feeling like it will all be OK and ill finish with a lot of time to spare but then suddenly it will feel like I'm never going to get to 5 pages in that time, let alone 8 or 9.
I can't wait for all of this to be over and to start my holidays, i really need the break. I can only imagine what next year will be like...
Wednesday, 14 May 2008
Configuration changes to OGE
I've had a bit of time since finishing my last bit of coursework and starting my next (ugh) so i worked on OGE's configuration/options code. There we're a few main features i wanted to add so i had to spend some time working out how to fit it in. I had tried previously but it came out a mess.
I still have quite a bit of work left on this but currently I have expanded oge::Option to allow:
I'm also working on the abstract option file parser and an XML implementation.
It seems to be going well at the moment and i'm hoping it won't be too long until i've finished and can commit but i've got a lot of coursework to do sadly.
I also realised i only have 24 days left of uni this year!
I still have quite a bit of work left on this but currently I have expanded oge::Option to allow:
- very basic tracking of what options have been added have been added
- tracking whether any values of an option have been added/changed/removed
I'm also working on the abstract option file parser and an XML implementation.
It seems to be going well at the moment and i'm hoping it won't be too long until i've finished and can commit but i've got a lot of coursework to do sadly.
I also realised i only have 24 days left of uni this year!
Friday, 2 May 2008
Lots of small changes
Since i finished some coursework that was due in today, I've managed to spend a bit of time working on OGE. Instead of making loads of changes and then committing the whole lot in one go, I'm now trying to commit more regularly with smaller additions. Various files have been moved around, data stream helper classes have been added, small changes and documentation to threading code have been made. Steven also ported the old serialisation code and is working on logging.
There is not much left to add to the utilities so very soon work will begin on the core which should be much more interesting. This will lead nicely into my summer holidays.
Looking at OGE now, it's so much cleaner and more organised than our old code. The utilities (at least in my opinion) is turning out very nicely and i at least will find it very useful when working on projects in the future, I hope others will also find it useful.
There is not much left to add to the utilities so very soon work will begin on the core which should be much more interesting. This will lead nicely into my summer holidays.
Looking at OGE now, it's so much cleaner and more organised than our old code. The utilities (at least in my opinion) is turning out very nicely and i at least will find it very useful when working on projects in the future, I hope others will also find it useful.
Saturday, 26 April 2008
Threading in OGE
I recently committed the new threading code for OGE. It's still work in progress with changes needing to be made and more features to be added but the basis is there now.
In the old OGE code, a SystemThread allowed one or more System's to be initialised onto it and ticked/updated at regular intervals. In the new version instead of each System working out if it should update or not, each SystemThread goes through a SystemController. Using both of these, different ways of managing time can be implemented. Currently we have a "Fast" implementation which the SystemThreadFast class passes the current time (when it receives it) to all of the SystemController's. Each SystemController checks to see if it's System should be ticked.
Overall this keeps a lot of threading detail away from the user, so hopefully they won't need to know a huge amount about multi threading before using OGE.
In the old OGE code, a SystemThread allowed one or more System's to be initialised onto it and ticked/updated at regular intervals. In the new version instead of each System working out if it should update or not, each SystemThread goes through a SystemController. Using both of these, different ways of managing time can be implemented. Currently we have a "Fast" implementation which the SystemThreadFast class passes the current time (when it receives it) to all of the SystemController's. Each SystemController checks to see if it's System should be ticked.
Overall this keeps a lot of threading detail away from the user, so hopefully they won't need to know a huge amount about multi threading before using OGE.
Friday, 11 April 2008
Six Day Weekend
I havent blogged in a while because I have been fairly busy with uni and OGE, but hopefully from now I can be a bit more relaxed.
Firstly, I have a 6 day weekend now as most of my units in uni are coursework based, we managed to get our 2 lectures fit into a single day. This of course saves me a lot of time (an hour travel every day) and money, which i can now use to buy cheap DVDs...
Secondly, Leanne and I, came up with a schedule for me to use that will allow me to do all the work I need to but to have free time as well. Usually I end up thinking that I should be working nearly all the time, because there's no set schedule that I can stick to. So now I have OGE scheduled meaning I can definitely keep working on that as well as get all my work done.
For OGE, I just committed the first public preview of OGE utilities to SVN, there's obviously a lot to add to it still. Once it has been documented and enough features added to it then work can be started on the core, which will be a lot more fun.
The dependencies for the core is now POCO and Intel's Threading Building Blocks. The latter we added to be able to get as much performance as we can out of threading in OGE, as they include a lot of very useful utilities. I don't think we will end up using their parrallel code, as OGE will manage its own threads, but atomic primitives, fast mutexs and concurrent containers will be very useful.
Firstly, I have a 6 day weekend now as most of my units in uni are coursework based, we managed to get our 2 lectures fit into a single day. This of course saves me a lot of time (an hour travel every day) and money, which i can now use to buy cheap DVDs...
Secondly, Leanne and I, came up with a schedule for me to use that will allow me to do all the work I need to but to have free time as well. Usually I end up thinking that I should be working nearly all the time, because there's no set schedule that I can stick to. So now I have OGE scheduled meaning I can definitely keep working on that as well as get all my work done.
For OGE, I just committed the first public preview of OGE utilities to SVN, there's obviously a lot to add to it still. Once it has been documented and enough features added to it then work can be started on the core, which will be a lot more fun.
The dependencies for the core is now POCO and Intel's Threading Building Blocks. The latter we added to be able to get as much performance as we can out of threading in OGE, as they include a lot of very useful utilities. I don't think we will end up using their parrallel code, as OGE will manage its own threads, but atomic primitives, fast mutexs and concurrent containers will be very useful.
Tuesday, 18 March 2008
Time flies again
Over the last year, my time spent programming often goes quite slow and is hard to get motivated enough. Yesterday and today i've been learning some fairly simple x86 assembly to begin making thread-safe lock/wait free containers (queue, list, stack etc) for OGE. I spent hours on it, and they flew by. I realise it feels exactly how it used to feel when i first started learning to program, which back then, i spent along time which flew by very fast.
Currently i have a set of atomic operations, such as add, exchange and add, swap, compare and swap etc. I also have an Atomic template class which uses these operations. This class can be used for thread-safe counters. This is leading up to create some containers, which should improve OGEs performance considerably, as there will be alot less locks being performed.
We didn't get accepted for Google Summer of Code this year, it seems they only accepted quite big/mature projects. We will hopefully have a better chance next year.
Currently i have a set of atomic operations, such as add, exchange and add, swap, compare and swap etc. I also have an Atomic template class which uses these operations. This class can be used for thread-safe counters. This is leading up to create some containers, which should improve OGEs performance considerably, as there will be alot less locks being performed.
We didn't get accepted for Google Summer of Code this year, it seems they only accepted quite big/mature projects. We will hopefully have a better chance next year.
Subscribe to:
Posts (Atom)