Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Work in Progress / PerfAnDBPro (Performance Analyser for DBPro)

Author
Message
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 3rd Oct 2009 15:14 Edited at: 6th Dec 2009 19:10
Link
Hi to all,

I would like to present my utility, PerfAnDBPro (Performance Analyser for Dark Basic Professional).

Please read through the PerfAnDBPro system description in this post, and see the latest posts for an up-to-date view of this project's progress.


########################################################
General News:


On 1/12/09, PerfAnDBPro made it into the TGC Newsletter for December 2009. Thanks go to the newsletter editor BatVink for the entry.

On 15/11/09, I updated the image download in the second post to show a montage of all three of the programs: the precompiler assistant, the precompiler itself and the data viewer.

As of 5/11/09, a beta of the entire system is now available, and is being updated on a regular basis. Please see the links at the bottom of this post for the latest versions of the programs that make up the PerfAnDBPro system.


########################################################
Description of the system:


This is a system which is comprised of a set of three programs which work together to allow you to run a performance analysis on your project, and view the results generated. The programs are as follows:

Precompiler Assistant - When the user presses "Compile" in any editor, this program takes in the .dbpro project file reference that is transmitted by the editor, and runs the precompilers that have been loaded into the assistant in sequence (eg. the PerfAnDBPro Precompiler). If the user runs the Precompiler Assistant directly (by double-clicking the .exe), one is able to load and choose what precompilers to run and the order in which they will by executed.

PerfAnDBPro Precompiler - This program takes in source code via the command line and appends performance analysis code to it. This code records the length of time that is taken for each line of code to execute (with a few exceptions here and there), and stores these times in a .dat file. It also stores a copy of the unmodified source code for use by the data viewer.

PerfAnDBPro Data Viewer - This program allows you to view the timings as recorded in the .dat files produced by the PerfAnDBPro Precompiler.

It has five modes: standard, average line-by-line, average ordered, cumulative line-by-line and cumulative ordered.
The standard mode shows the times for code lines in execution order.
The average line-by-line mode shows the average times taken for each line of code with reference to the source code.
The average ordered mode shows average times taken in descending order.
Cumulative line-by-line mode is like its average equivalent, but instead, the total cumulative time the program spent on each line during the entire lifetime of the program is displayed.
Cumulative ordered mode displays the cumulative times in order with the largest at the top.

By pressing the "F" key in the data viewer, one can display the cumulative time taken for functions, loops and gosubs. They will be displayed as a modified bar where the first command of such structures is located.

In the images present in the download, the values in the coloumn marked "Residue" on the right show the correction factor for the timing measurements (it is the time taken by certain critical sections of the profiling code). The "Ln" coloumn is the source code line number, and the "Ex" column is the code line execution number.

----------------------------------------------------------------------------------------------------------
A couple of things to note:

The system does not modify directly the source code that is present in the .dba files (the files that are usually present along side the .dbpro file in the "Projects" folder). What it does is it modifies the _Temp.dba source produced by the editor (the source produced after all and any "include" files have been joined together, etc.).

The added code does cause the host project .exe to slow down by some amount, but this is accounted for in the code that is added.

Not all the code has to be profiled. The use of the commands "#PADBProBegin" and "#PADBProFinish" allows one to select what code to profile. Note that these commands are only recognisable by the precompiler, so they must be commented out if the precompiler is not it use.

You may be wondering why I wrote "...load and choose what precompilers to run...etc." for the Precompiler Assistant. This is because this program is able to run any precompiler .exe that has been designed to take in source code on its command line for modification. This could be handy for extendability purposes.

----------------------------------------------------------------------------------------------------------

Work on this system will be ongoing (of course), but may be a bit sporadic at times due to uni work.

Constructive comments/ideas welcome.


########################################
Current versions of the programs that make up the PerfAnDBPro system:

++++++++++++++++++++++++++++++++++++++++++++
The PerfAnDBPro Data Viewer: Updated 6/12/09

The Data Viewer download in this post has now been updated - we are now on version 0.2.3 beta.
Please remove any old files that came with the older version, as they may no longer be compatible with this new version.

++++++++++++++++++++++++++++++++++++++++++++
The Precompiler Assistant: Updated 5/11/09

A beta of the Precompiler Assistant is now available - we are currently on version 0.1.2beta

Please read the instructions in the .html help file provided carefully.

It is available here https://forumfiles.thegamecreators.com/download/1888666

++++++++++++++++++++++++++++++++++++++++++++
The PerfAnDBPro Precompiler: Updated 6/12/09

A beta of the PerfAnDBPro precompiler is now available - we are currently on version 0.2.3 beta.

Please read the instructions in the .txt help file provided carefully.

It is available here: https://forumfiles.thegamecreators.com/download/1890234

Attachments

Login to view attachments
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 3rd Oct 2009 15:15 Edited at: 15th Nov 2009 19:23
Link
Images Updated 15/11/09

The image in the download is a montage of all three programs:

The Precompiler Assistant
The PerfAnDBPro Precompiler
The PerfAnDBPro Data Viewer (x2 images)

The data viewer is showing images of a performance analysis run of a different program than the one which is visible in the IDE image behind the first two programs.

Attachments

Login to view attachments
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 8th Oct 2009 22:59 Edited at: 28th Nov 2009 23:18
Link
Update 8/10/09

Attached to this post is the current WIP version of the data viewer. There are a few known issues with it, but it should give folks a feel for what kind of data this system can display and how the data is handled in the data viewer.

File size is a little hefty but not too bad - 6.636 MB

-----------------------------------------------
Updates to the system as a whole

Amongst other things...

Added two "cumulative" modes to the data viewer - one can now view how long a line of code took to execute throughout the lifetime of the program.
Removed most hardcoded file references (all programs) - I think there maybe one or two left in the system.
Adjusted the internal arrangement of the data in the .dat files
Fixed a bug in the data viewer which chopped off the top/bottom lines of data (appeared in various forms)
Removed some uneccesary code that was being added by the precompiler to a host project's source code.
Bypassed a "double float based bug" which caused the data viewer to crash after many values were added together.

======================================
Edited 28/11/09

Alpha version of data viewer download removed - it is well out of date, and I should really have removed it earlier.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 9th Oct 2009 01:59 Edited at: 9th Oct 2009 02:06
Link
Just looking at the posts you made so far, I think this could be a handy tool. Do you have an estimate on how much it increases compile time?

Tagging this for future updates. Looks promising

Edit: Any estimates on when this will be ready for release?
Back to top
Profile PM Email Website
Plystire
21
Years of Service
User Offline
Joined: 18th Feb 2003
Location: Staring into the digital ether
Posted: 11th Oct 2009 01:14
Link
The screenshots look like it provides quite a detailed analysis on processing of the program! I like it, it coul dbe very useful, especially when polishing up a game for a release.


One suggestion that would make this even better. Do you think it would be possible to have it analyze execution time of subroutines, functions, and loops? By that I mean how long it took to run them as a whole, like the totaled up time for all the lines contained within each. Having it show the average execution time for each one would greatly help narrow down problematic areas of code in large projects.


I'll keep my eye on this one. *ticks MailBack box*


The one and only,


Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 12th Oct 2009 21:10
Link
GIDustin,

I have done a one-off test on a project of approximately 3500 lines of code, and the precompiling + compiling process takes about 3.4 times as long than if you were to compile the project "as-is" without adding any profiling code.

Having said this, bear in mind the following points:
1. In the test, the system added profiling code to all of the lines of code that were present (or any that allowed such code to be added). You don't have to add profiling code to every single line, and if you only want to profile a small part of your project, then the impact the system will have will not be nearly as great.
2. Most of the extra time is being taken up by the compiler having to process the extra profiling commands that have been added. As mentioned in the updates below, I have managed to reduce the time it takes for the precompiler to run from about 2-7 seconds down to roughly 1/3 of a second or so.
3. It may be possible to reduce the extra time taken by some measure if I were to "functionlise" the timing code. However, this would a). add a second residue factor into the timing of each line of code being measured, and b). it would make the host project .exe run slower. It does take quite a bit more time to execute tight loops with the added profiling code as it is (that is, if you are profiling on a line-by-line basis).

If, when this is released, people find that it takes too long to compile, I may add an option to "functionlise" the profiling code, but in doing so make the host program run slower.

-----------------------------------------------

Estimates for release are probably one of the most difficult to get right, and can result in a lot of frustration for waiting users if a program's development time takes longer than the developer announced. For me, I have a number of factors that are playing a part in this uncertainty:
1. I am currently in university on a full-time basis (final year).
2. I have never released a computer program before into the wider domain (don't have past experience on release times as guidance).
3. If lots of people want me to add in features that they want to see into the program, this will increase development time.
4. Unexpected bugs/slugs that takes ages to fix (probably one of the more common issues developers face).

I know everyone may say this, but as I consider this program to be rather essential to my DBPro development, I will release this in some shape or form.

So with that said, lets say an official release estimate of around the new year, plus/minus a month or so. Probably a bit conservative, but one never knows what may come up while coding.
The data viewer will be updated whenever I feel I've made sufficent progress to justify an update.

++++++++++++++++++++++++++++++++++++++++++++++++
Plystire,

Your suggestion for timing subs, functions and loops is something that I intended to add in anyway. I feel that subs and functions may be a bit more easier to tackle than loops, but I'll only know for certain when I actually start trying to code this.

You may be interested to know that this is not my first attempt at trying to make this kind of utility. My previous attempts had been scuppered due to a lack of coding experience, but more importantly, due to a host program exiting a loop/function unexpectedly. On this occasion, I have made some functions which can detect keywords in the host source code (before it is compiled and run) and I can thus work out the possible program flow that may occur. They need a bit of upgrading, as they were made for another project that I was working on, but this is one of the primary tasks at the moment.
In addition, the positioning of the profiling code needs to be readjusted, because the values being produced by it at the moment may seem rather odd to know who don't know how the profiling code is added to the host source. The values are fine for stand-alone lines of code like "Turn Object Left objID, fValue" and similar, but entering/exiting subs/functions at this time does come up with some odd results.

++++++++++++++++++++++++++++++++++++++++++++++++
Updates as of 12/10/09

Amongst other things...

The scroll bar has been fixed in the data viewer.
The time taken by the precompiler has be reduced substantially from 2-7 seconds to about 0.3 seconds.
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 21st Oct 2009 18:01 Edited at: 21st Oct 2009 18:02
Link
Update 21/10/09

Amongst other things...

The .dat data log files and the .dba source code files have now been combined into a single .dat file, so you can now name them anything you wish, as long as you keep the file extension the same.
Timer-based movement has been added to the scrolling actions.

========================================

The Data Viewer download in the first post has now been updated - we are now on version 0.2. Please remove any old files that came with the older version, as they may no longer be compatible with this new version.

========================================

The state of play at this time regarding development:

I am about to start work on adjusting the placement of the timing code that the precompiler adds to any host project before it is compiled. This should not affect timings of single lines of code, but will affect the timings of loop-like and function-like structures - the facility is being put into place to allow them to be timed as single blocks, as was suggested by Plystire. I will also have to take into account function and loop exits in the timing, along with structures such as recursive functions, which can effectivly act as loops in some cases.

After this is added, I will need to make a few adjustments to external support files, remove the one or two hard coded file references that remain, and add in some other odds and ends. At this point, the system will be ready for release.

----------------------------------------

Extensions to the system:

I am considering adding another type of graph to accompany the bar chart that is already present. This will be of a line graph style, and should allow for easier viewing of function-like and loop-like structures, and will be accesible as another data viewing "mode". As I may have mentioned in an earlier post, this is not the first time that this project has been attempted, and I actually have a prototype of this style of graph for folks to see if this would interest you.

It will be in the form of a screenshot or two, as the prototype that produces these graphs was never fully completed, and is not very stable.

A further extension to the system, is to allow folks to monitor CPU peformance and memory usage (both RAM and video). This will be included as an option that people may toggle on/off as desired.

----------------------------------------

As things stands regarding release estimates...
Without extensions, the release will be either in November or December.

I am happy to add the extensions that I have mentioned above if people would like them, but it will obviously be a bit longer before the system will be released - the extensions will probably push the release into December.

If people are more interested in getting to grips with the current system (after the needed changes to the timing, and the like have been made), then I will release the system earlier, and the extensions will come in versions 1.1, 1.2, and so on...

I know that some people don't like an ever-increasing roll of additional features after a program has been released, so I'm asking now what people would like.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 28th Oct 2009 06:52 Edited at: 28th Oct 2009 06:53
Link
Well Phjon, I just created a 300 line include file for DBPro to handle text. It works similarly to Cloggy's D3DFunc text commands:


It automatically wraps text within the text box, automatically converts "\n" to new lines, allows color changes mid-line, and allows text/image replacements. In the above example, [SPADES] is replaced by a spades image for a card game. Most notable uses for that are emoticons.

It works great, but somewhat slow. Something is slowing it down where each XP_BOX_TEXT() command takes 10ms. Might not be that bad, but your program could probably tell me where improvements could be made.

So, I eagerly await some type of beta to test out. In the meantime, I have the rest of the game to work on.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 28th Oct 2009 16:51 Edited at: 28th Oct 2009 18:25
Link
GIDustin,

Thanks for posting.

I assume that this text handler of yours is whatever that function in your code snippet refers to. I believe that my program could certainly pin down whatever is causing the slow-down problem in that function - that is when I have finished all the tasks that I describe below in my task list.

(There was an idea for the beta testing here, but I've removed it for the time being, as I need to think it through a bit more carefully).

====================================================================

I haven't managed to take a look at PerfAnDBPro over this past week due to other things taking up my time.

However, what I have done is broken down that large task which I mentioned in my previous post that I needed to do into small, bitesized chunks for me to work on. Here is that task list:

1. Add code to locate all labels in the host code.
2. Destack host code lines to allow for easier positioning of the profiling code.

so...

Sync : Wait Key : Cls

will become...

Sync
Wait Key
Cls

3. Destack "If...Then" statements
4. Reposition timing code for standard lines.
5. Adjust the timing code for "If...EndIf" branches.
6. Adjust the timing code for functions
7. Adjust the timing code for loops
8. Adjust the timing code for gosubs

This should take care of some of the strange timing results that I am getting for functions at the moment.

One of the main challenges for the development of this program is taking into account all of the possible combinations of a set of commands or statements that this program might encounter. I'm doing OK at the moment, but it does take a little while, and can be a bit draining if there are a lot of combinations.

Although the above list looks lengthy, each of the tasks are reasonably small. I will continue to post updates every week to ten days or so on this project. Things are being done - it's just at a slow plod at the moment due to time allocation difficulties.
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 2nd Nov 2009 21:47
Link
Update 2/11/09

Amongst other things...

Code has now been added to locate labels in the host source code.
Lines stacked with the colon are now destacked before the profiling code is added.
"If" statements are now destacked before the profiling code is added.
The timing code has be readjusted for ordinary lines of code, and has been adjusted for "IF" statements.
A class ID number is added by the precompiler to each line along with the timing code to help determine the type of code line that is present (standard, function-like, if, looping, etc...) and to assist the data viewer in working out how to display the profiling data.
The timing code has also been modified for loops and functions, but still needs further attention.

=======================================================

Good progress has been made on PerfAnDBPro these past few days.

I am currently working on how to implement Plystire's suggestion about displaying the cumulative times for loops, functions and similar structures. It is proving a little more difficult to implement than I had anticipated, but I am trying out a number of solutions, both in terms of gathering the data, and displaying it.

You may be wondering why I cannot simply record the time at the top of a loop and at the bottom, and subtract one from the other. This is because the timing code itself takes time to run, and makes the program slower. There is code in place to correct for this extra time, but only on a line-by-line basis. What I have to do is add up all the times taken for each line of code that is inside the loop and use that (also doing the same for the residues). In addition, I also have to take into account the effects of nested loops, and the time taken to execute the surrounding loop code, which makes the task a little more difficult than one may think. However, as I have mentioned, I have a number of ideas on how to solve this and it will be solved.

Following on from this, once I have ensured that the timing codes work and produce correct results for all types of code structures, I will be addressing issues with the precompiler commmands such as "PADBPRO Begin/Finish" and "PADBPRO BlockStart/BlockFinish". I need to ensure that no side effects are caused by placing these commands in odd places (e.g. straddling a loop or if statement, which may produce strange results in other timings at best and crash the system at worst). I don't think it will take that long to ensure that they work correctly.

Finally, I will be tidying up some loose odds and ends such as calls to supporting files, writing the help files, and such and such.

Depending on how things go coding-wise, and other external matters not getting too much in the way, it is very likely now that a beta release of the entire system should be out in three weeks time or less.
Back to top
Profile PM Email
tiresius
21
Years of Service
User Offline
Joined: 13th Nov 2002
Location: MA USA
Posted: 2nd Nov 2009 23:10
Link
Can you explain how the precompiler will be run from the IDE?

And instead of an actual command for start/end, can your profiler perhaps parse a comment with special characters? That way a profiled and non-profiled code can be generated easily by just not running it through the precompiler

Something like this:


I'm not a real programmer but I play one with DBPro!
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 3rd Nov 2009 16:34 Edited at: 5th Nov 2009 23:53
Link
tiresius,

The precompiler is run from the IDE via the Precompiler Assistant.

After you have set up the precompiler assistant, when you press "Compile" or "Compile and run", the precompiler assistant will be called up first.
The precompiler assistant will then call the precompiler to act upon the host project source code.
After the precompiler has finished and closed down, the precompiler assistant will hand control over to the DBPro compiler, which will then compile the newly-modified project code.

--------------------------------------------------

I have decided to release a beta of the precompiler assistant, not just to assist in helping me explain my system further, but because it is pretty much finished and fully operational as a program in itself. It is available in the "Download" button on this post.

Though I haven't had any problems with the precompiler assistant, you might want to make a couple of backups of your source code files just in case.

Have a look at this, and post if you still don't understand what's going on.

--------------------------------------------------

As an aside, I'm curious as to why no-one has thought of this idea before - it's reasonably simple in its design. Or maybe there's something that I need to know, but I haven't found out about it yet.

==================================================

Your second point was about the precompiler commands. I should be able to modify the commands to something like this without too much trouble, and I agree that it should also help to avoid messing up the compiler when PerfAnDBPro is not being used.

==================================================
Updated 5/11/09

Precompiler Assistant download updated to version 0.1.2 beta.
Caption in the program header was not correct in the previous version - this has now been fixed.

Attachments

Login to view attachments
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 4th Nov 2009 00:21
Link
Is the assistant designed to only work on DBPro's IDE or will any work? I use Codesurge, and I get an error when I compile. Your assistant pops up, but then an error appears "System cannot find the file specified" or something similar. Tried twice with same results, one with an empty precompiler list, and once with your Simple Precompiler in there.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 4th Nov 2009 15:18 Edited at: 4th Nov 2009 15:52
Link
GIDustin,

The assistant can work with any IDE that is used to make DBPro games/applications. It should definately work with Codesurge, as that is what I'm using to program the PerfAnDBPro precompiler and the data viewer.

A couple of things that might help me work out what's going wrong:

Did you take a look at the help file that came with the assistant, specifically the fact that you need to alter the file name given in the file "Settings.cfg" to wherever your DBPro compiler is now placed?

Could you post a picture of the error message and/or a copy of the text that it produces?
Could you confirm that you were able to get the program to work in "Settings Mode" (i.e. simply double-clicking on the program)?
If I'm still not sure what's going on, I'll take a look at the assistant's code, add in a few more error tracing abilities, and we'll see if that helps.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 5th Nov 2009 00:31
Link
If you copy a folder, and paste it in the same location, it renames it to "Copy of <Folder>". Well, not in Windows 7, which is why it didn't work. I had the path wrong in the Settings.

The precompiler works in settings mode and now works when compiling projects.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 5th Nov 2009 12:54 Edited at: 5th Nov 2009 12:56
Link
GIDustin,

Ah good, so it works OK now in both modes.

================================================
Update 5/11/09

Amongst other things:

The timing code alterations for loops, functions and gosubs has now been made, and is fully operational.
Plystire's suggestion is now in place - this is a mode that will allow you to view the combined times taken for loops, functions and gosubs. By this, I mean that the time displayed will be a sum of the times taken by everything in a loop (for example), plus the loop commands themselves.
tiresius's suggestion is now in place - the precompiler commands are now in the form of "REM" statements.
The data viewer "Settings.cfg" file name has been altered to "PADVSettings.cfg" to avoid any conflict with other "Settings.cfg" files that may be needed by other programs.

----------------------------------------------

Work is ongoing to ensure that the precompiler commands work under all circumstances. The commands "PADBProBlockStart" and "PADBProBlockFinish" are being added, and these will allow you to time a block of code as a whole, and not just time every single line of code within it. This is useful when you just want an overview of what time is being used where in a program without going into fine detail. It's also useful for deciding which part of a large project needs the most optimisation attention, as running PerfAnDBPro on a large host project in the standard line-by-line mode can have a tendancy to slow a project down by a fair amount, especially in tight inner loops.

After I have ensured that the precompiler commands work well, I will be writing the help files, and tying up any loose ends that remain. Then I will release a beta of the entire system.

I believe that such a release will now occur within a few days.
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 5th Nov 2009 23:50 Edited at: 6th Dec 2009 19:11
Link
Update 5/11/09

The beta version of PerfAnDBPro is now released!

It is available via the "Download" button at the bottom of this post.

Please read the instructions carefully and post if you have any problems.

I strongly recommend that you try this out first on small projects before larger ones, just in case you encounter any problems. I also recommend that you make a backup of your source code files before using the precompiler just in case.

For the moment, I have had to disable the BlockStart/BlockFinish commands, as they were producing results that, while I could understand them, it is very unlikely that other people may be able to. They will be put back in at a later date (this is only a beta at the moment, and not a full release).

There are a few code combinations that the system can't handle too well, such as...



...but it should be able to cope with most other things.

---------------------

Updates have also be made to the Precompiler Assistant and the data viewer.

The precompiler assistant is now at version 0.1.2beta - the only change that was made was to the caption of the program which wasn't set correctly for version 0.1.1beta

The data viewer is now on version 0.1beta - we have now moved away from the alpha phase as the entire PerfAnDBPro system is now in a beta state.

Changes include, amongst other things,

An option to view the combined times taken by code inside loops, functions and gosubs.
Modification of the loading code to take into account a new profiling arrangement that produces slightly less residue.

---------------------

On a personal note, my work load from uni will be increasing by some amount relatively soon, so updates may become a bit more sporadic again. I'm not too sure how things will pan out at the moment, but I just wanted to let you all know just in case I don't post a reply for a couple of days or so.

======================
Updated 6/12/09

The precompiler download is now at version 0.2.3 beta.

Please remove any older versions of this program and their supporting files, as they may not be compatible with this newer version.

Attachments

Login to view attachments
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 19:09 Edited at: 15th Nov 2009 19:28
Link
Update 15/11/09

If the precompiler finds that a command has been placed in an illegal location, the precompiler will exit when a user presses a key.
In addition, any modifcations to the host code will be removed if such an error occurs.

The new version of the precompiler (v0.1.1beta) is available via the links provided at the bottom of the first post.

==================================================

I would really appreciate it if people could grab a copy of this system and give it a go.

I'm conscious of the fact that this thread has been a little quieter than I had expected, given the fact that a beta has been released, and I'm not sure if it's just the fact that people are busy, or that there's something very wrong with this system, but people don't want to mention it.

==================================================

I have also updated the image montage to show a couple of images of the data viewer in action, in addition to the assistant and the precompiler itself.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 19:44
Link
Quote: "There are a few code combinations that the system can't handle too well, such as..."


This is what is stopping me from trying it atm. My code is full INCs and comments, and the setup routines are all gosubs.

If the newest version supports these then I can have a go at it on my "nearly finished, just waiting for the company's approval" game.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 20:35 Edited at: 15th Nov 2009 20:48
Link
GIDustin,

The present version of the system can handle "Inc's" and comments without trouble.

It can also handle "Gosubs", as long as their not in the format shown in that code snippet above in one of my previous posts. A "gosub" code structure like the one shown below...



...should pose no problems for it.

When I mentioned "a few code combinations", I was reffering to code structures that would be very rare/unusual for a programmer to write. On the whole, the system "should" cope with just about anything.

You can always run it anyway, and if it encounters something that it doesn't like, it'll let you know one way or another.

Just make sure that you make backups of your source code before you run it, just in case you do encounter something unusually bad (although I think that is very unlikely).

==================================

If you would like, I'll make a short program with "If's", "Gosub's" and comments in it to show you that the system can handle them OK.

----------------------------------

Sample is uploaded and has been attached to this post.
It contains the code and the profiled data which can be opened with the data viewer.

Attachments

Login to view attachments
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 21:17
Link
OK, setup the precompiler and added PerfAnDBPro v0_1_1beta to the list. Fired up my project and hit compile. The Precompiler list came up, then a typical DBPro window with the "PerfANDbpro" title. I see one line, something about keywords I think, and then I get an error:

File not found at line 521
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 21:22 Edited at: 15th Nov 2009 21:48
Link
GIDustin,

Right, I'll take a look at what might be causing that.

Standby...

(Edit 1)

It would seem that I failed to remove a hard-coded file reference from the precompiler. I thought I'd removed all of them!

Ah, well, I'd better get on and correct the fault. It shouldn't take me too long.

(Edit 2)

The bug has been fixed and a new version of the precompiler has been uploaded.

I've added a file called 'PADBProSettings.cfg'. In it, is the file directory where your DBPro keywords are stored. You'll need to change the directory given to the directory where the keywords are located on your PC.

Let me know if you have any more problems.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 21:47 Edited at: 15th Nov 2009 21:50
Link
Updated the precompile assistant? to 0.1.2, problem still exists. Looking at your initial post, it looks as if you uploaded a new precompile assistant, and not a new precompiler.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 21:51 Edited at: 15th Nov 2009 21:52
Link
GIDustin,

A case of post collision!

I forgot to mention the new changes in the new precompiler version in my previous post, my apologies for that.

I did upload a new precompiler, not a new assistant, I'm sure of that.

You need to update the precompiler, not the assistant.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 21:54 Edited at: 15th Nov 2009 21:54
Link
Checked all the download links in the thread. Maybe it's lack of sleep, but I dont see any download links for the precompiler anymore. The one that used to exist was replaced with another version of the precompiler assistant.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 22:00 Edited at: 15th Nov 2009 22:04
Link
GIDustin,

You're right, it is!
Something strange is going on.
Can you give me a few moments to fix things?

(EDIT)

OK, I've fixed things now.

It would seem that I did upload the assistant after all by mistake.

I think, because both have the same version number, I must of chosen the assistant to upload instead of the precompiler by accident.

All is well now...hopefully.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 22:09
Link
Happens to the best of us.

OK, got the newest percompiler and assitant, changed the precompiler.ini to point to the new one, changed the Padpro settings to point to my keywords.

It got further than before.

It loaded keywords, checked for keywords and labels (i think), and then died again:

Runtime Error 110: Only positive numbers are allowed at line 1231
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 22:16 Edited at: 15th Nov 2009 22:32
Link
GIDustin,

That sort of message is usually produced when one of the variables in IanM's string commands is given a zero instead of something greater than zero.

This one may take me a little longer to track down.

Could you try running PerfAnDBPro on the "Gosub Demo" code, and see if it produces the same error. It's attached to post number 20 (it's the most recent post with a download attachment).

It might also be helpful if you could tell me how far the program reached before it crashed. I know it managed to load and store the keywords, but did the program print any more lines afterwards, or did it just happen too quick?

-----------------------------

I guessing that it's not possible to send me a copy of the code your trying to profile (sounds like it's some proprietary company thing you're working on).
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 22:28 Edited at: 15th Nov 2009 22:31
Link
ok, ran the gosub test. It looks like it precompiled fine, and compiled fine. The exe does nothing (which, looking at the source code, is exactly what it is supposed to do), but I dont see it making any data files to load in the viewer.

Edit: I think i found the data files. I was expecting them to be in the project folder. I think they are in the compiler folder.
Edit2: The data viewer crashes when loading the dat files in the compiler folder that I found. Opening the data files proves that it is in fact the gosub demo since the source begins the data file.


As for the line 1231 error, I use Matrix stuff everywhere. Any specifics on how to track that down?
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 22:42
Link
GIDustin,

A couple more things,

Do you know what the specifics of that data viewer crash are?

There should be a .dat file wherever you placed the gosub demo called "Gosub Profiled Data.dat". That was made by me for folks to test the data viewer out, and should load fine. Could you get the data viewer to load that and see if that crashes?

------------------------------------

As for the line 1231 error, it's in a function that trims out unecessary keywords when the precompiler checks the host source code for keywords. However, the location of the error is strange, as I would have expected it to crash sooner that it did if the error is indeed inside that function.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 23:02 Edited at: 15th Nov 2009 23:06
Link
Phjon:

I didn't realize there was a newer version of the data viewer. The newest version opens the test one in the demo.zip, and then ones created when I compiled the gosub test.

So the only think not working now is my source code. I will try moving the BEGIN and END lines to see if I can get parts to work.

Edit: Moved the Begin and Finish lines to only encompass a single #constant declaration and the same problem exists. It appears as though I am unable to locate the problems lines alone.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 23:34
Link
GIDustin,

OK, that's good news the data viewer works.

--------------------------------------------

The precompiler commands will only act towards the end of precompiling process, when the profiling code is added on.

They have no effect when the precompiler checks for keywords; the program will check all lines of code for keywords regardless of whether they are to be precompiled or not.

The reason this is done is to ensure that the precompiler commands are not placed in illegal locations, such as straddling a "Do" command in a loop...if that happpens, strange results/crashes ususally occur, and if not in the precompiler, they will in the data viewer.

------------------------------------

I have confirmed that the error is in that function which I mentioned. I'll describe what the precompiler does at this point to give you an idea of how the bug may be coming about.

During the keyword recognition process, the precompiler checks if and where keywords are on each line of code and stores the keyword and its position in a comma-delimited string, which acts like an array of its own.

Sometimes, keywords are added which are superseeded by longer keywords, i.e. "Sync" is replaced by "Sync On".

What the function does is it takes in a keyword number and a line number, and removes the shorter keyword and its position from that comma-delimited list. Amongst other commands, I use IanM's "InStr" and "Get Split Word$" for this purpose.

--------------------------------------

What's happening with your profiling attempt on your own code, is that it seems as if the function is trying to remove a keyword from a list that has no entries.

==========================================

One thing you can try to help me pin this down in your code, but may take you a while:

Create a separate project and copy small sections of code into it, one section at a time, and try to profile those code sections.

When you encounter that bug again, try adding/removing code lines until you've narrowed it down to about 5-10 lines of code.

When you've done that, post up those 5-10 lines, or email them to me if you're conscious of someone stealing them and I'll see if I can duplicate that error on my PC.

==========================================

It's getting a bit late here in the UK where I am, but I'll be up for about 20 minutes more before I go to bed.

If I don't get back to you today, I will tomorrow.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 23:37
Link
The project is 14,271 lines long. I doubt I can narrow it down to 5-10 lines in 20 minutes

I'll give it a shot after this episode of Bleach. I do not use the instr command, but I make good use of get split word$. I can start there.

- GIDustin
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 23:43 Edited at: 17th Nov 2009 18:21
Link
GIDustin,

OK, just do your best. If it's too much, don't panic, I've got another idea that may help, but I need a little time to test this idea out.

Good luck!

===========================

Edit:

My idea works, and it'll save you doing all that copying/pasting work.

Attached to this post is a version of the precompiler that shows you which line number the precompiler is on when it's checking for the keywords.

From that, you should be able to work out what to post up (plus the few lines of code that surround it).

=============================================
Edit 2:

Download of the "debug version" of the precompiler has been removed - this version is now obsolete.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 15th Nov 2009 23:48 Edited at: 16th Nov 2009 00:11
Link


This block of code handles GUI themes. By itself, it gives the error. However, when I take it out of my code, I still get the error, meaning there is more than one location. While I look for the others, you can analyze the above.


Edit1:





These lines also errors out.

So, in the first 5000 lines, I get these 3 chunks that error out. I will wait to go through the last 9000 in case you find some easy fix
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 15th Nov 2009 23:59 Edited at: 16th Nov 2009 21:45
Link
GIDustin,

I confirm the line 1231 errors in both snippets.

I'll get to work on a solution, but you may have to wait until tomorrow for any updates, as I need some sleep.

===================================================
Update 16/11/09

I have managed to fix the bug that was arising (the line 1231 error) without too much trouble.

However, your first snippet uncovereed another bug in my program involving a gosub label detection function and related code structures. I am working to fix this, but it requires a re-write of the way I detect labels in host code, and ensuring that the colons at the ends of the labels don't get mistaken for "colon line-stack" characters, so an update may not appear for a couple of days or so, depending on how long I take to fix this.

I've also discovered that the timing code seems to be producing more residue than necessary, so I'll work on a fix for that as well.
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 17th Nov 2009 18:15
Link
Update 17/11/09

I have fixed both the gosub errors and the excessive residue issue in the precompiler. In addition, the precompiler will also profile "type" structures properly. It seems the program didn't profile these at all and just left them out - this has now been changed.

We are now at version 0.1.3 beta for the precompiler. The download is available at the usual location (the links in the 1st post).

In addition, a new version of the data viewer is up - we are now on version 0.2 beta. The single change made was to allow for the profiling data belonging to type structures.

=======================================
GIDustin,

Grab the new updates to the precompiler and the data viewer, and let me know if you run into any more problems.

A couple of helpful tips when working with large projects:

As your project is quite large, it might be useful to profile sections of it at a time, as the file sizes produced by the precompiler can be quite large, and the extra profiling code will also slow your program down by a fair amount (though this slow-down is accounted for via the residue timings made).

In addition, it does takes the data viewer a while to load particularly large files.

On the other hand, the file sizes generated may not be too bad - it does require a bit of playing around with different sized projects/code sections to see how the file sizes change.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 18th Nov 2009 00:54 Edited at: 18th Nov 2009 01:00
Link
Ran the newest version, here are the results:

I am disregarding your advice to do chunks of the game at once and continuing to try the whole thing.

Good news is that the precompiler got all the way done, no errors. BUT then the compile threw an error "could not understand command at line 1826". I opened the _temp.dbsource and found the problem. Your precompiler took:

and changed it to:


I am thinking that it saw the ":" in the comment and parsed it as if it wasn't. So, I took out that comment and ran again. Compiled just fine, and the exe ran just fine, no preformance hit or anything... Then I reloaded my project and tried again but this time I added REM #PerfAnDBProBegin to the beginning, and REM #PerfAnDBProFinish to the very end of the last source file (when listed in compiling order).

Only problem is that it sill compiled just fine, still ran just fine, and I don't think it created any data files other than the PerfAnDatXX file that contains the source code, or if it did I cannot find them.

My next thought is that they Begin and Finish were too far spread, so I placed them only around a block of code that loads user-defined values from an INI file (about 15 lines). Compiled fine, ran fine, still no clue where the created data files are if any were made at all.

Just for S&G, I used the data viewer to open a few of the "PerfAnDat??.dat" files in the compiler folder, and got the following:



So, did I miss something?

Attachments

Login to view attachments
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 18th Nov 2009 18:19 Edited at: 6th Dec 2009 19:13
Link
GIDustin,

Ok, what's happening in that picture is that the .dat file contains no data (your reasoning is correct).

The precompiler commands are actually:



I'm at fault for this, as the helpfile shows "REM #PerfAnDBProFinish" instead. They were originally the commands in the bold typeface in your post, but were changed to that shown above in the code snippet, as they were shorter. I must have failed to remember to change the help file docs to the new commands.

Try running it again with those commands in the snippet, and I'll fix the help in the meantime.

As for the bug you have reported, I'll take a look at that as well, but by the sounds of it, it's not a showstopper.

==========================================
EDIT:

I've put the new precompiler help file here to save you downloading everything else again.

##########################################
Edited 6/12/09

Help file download removed - the version is now obsolete.
Back to top
Profile PM Email
Gask
16
Years of Service
User Offline
Joined: 1st Sep 2007
Location:
Posted: 19th Nov 2009 03:06
Link
yeah got the same problem with

REM #PADBProBegin
REM #PerfAnDBProFinish

i test a lucky try by just changing begin by finish and it works ^^ so i realize that the help file was a little mess up ^^

Just two word for you Phjon : "Great work"

Really , i was sceptic at first when reading your topic , but as soon as i see your idea in pratical use i really love it.

I have already started to optimize my code with your program and see really no problem for the moment.

I hope that you will not let this die , it is really something.

thanks for your time on this.
Back to top
Profile PM
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 19th Nov 2009 04:58
Link
Not much time to devote to this today.

I did change the commands, and I did compile when the commands encompassed 3 lines of code. The data file loaded, and was quite useless as there is no way I can speed up a variable declaration. Then I wrapped the entire source and compiled again (20 minute compile when analyzing all 14,000 lines)

There were some problems though. My GetSubImage() function, which gets a portion of an image by first converting to a memblock, failed. Also, a simple "load bitmap" function half way through the code failed, even after several other "load bitmaps" succeeded. I didn't check, but I am guessing it was due to memory issues.

Running my game with 100% being analyzed for just 5 seconds created a 248MB data file. I attempted to load it in the viewer, but canceled it 2.5 hours later.

OK OK, I suppose I will analyze smaller chunks, but I am not going to optimize my code until the company its for OKs the initial draft.

The good news is, it works!!! It will definitely help in my future projects. I feel that having this tool at the beginning of a project will prove much more useful than acquiring it at the very end.

Kudos
- GIDustin
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 19th Nov 2009 20:19
Link
Gask,

Thanks for posting, and for the feedback. Glad to know that the program works for you!

I have no intention of letting this die, as I believe that this system is extreemly useful for assisting with programming my future projects, and for helping others with profiling theirs. There are a number of features and ideas that I would like to add in to the system, and I intend to start working on those soon after this is officially released. At the moment, there are still bugs for me to fix and one or two basic things to add in, so it may remain in beta mode for a little while yet, just to ensure that it is as stable as possible, and to ensure that the basic functionality works.

I would still like a couple more people to beta-test the system, but I'm not too worried if that doesn't occur - GIDustin's done a good job finding bugs in the system so far.


========================================================
GIDustin,

Regarding the faults with the functions, could you elaborate on the term "failed"?

Are you saying that the program put up a windows crash message - something like "This program has encountered a problem, and needs to close", or did a standard DBPro error message appear. Though I'm suspecting the former, if it was the latter, could you tell me what that message was?

In the second case, it does sound a bit like a memory issue. I might run a few tests of my own to see how much PerfAnDBPro uses profiling a largish project.

2.5 hours for a 248MB file does seem a very long time! I have had to open such large files with the data viewer in the past, but it hasn't taken that long for me. However, the loading function in the data viewer is doing somewhat more processing in recent versions, so again, running a few tests of my own may help here.

---------------------------------------------------------

I'm glad to hear that the system's beginning to work much better for you now, and I hope it helps you to optimise your large project much faster.


*********************************************************
A general update on where the project is heading,

As mentioned above, the system will be in beta mode for a little while to come. As well as the usual bug fixing and testing, I'll be running some checks on memory usage and determining how long large .dat files take to load with the recent data viewer. I will report back on my findings in due course.

I will also be making some effort to reactivate the "#PADBProBlockStart" and "#PADBProBlockFinish" commands. I disabled these commands earlier on because the results they were producing were very confusing. What follows is a description of these difficulties, and why these commands were disabled.

As folks may (or may not) know, PerfAnDBPro records the time taken by the timing commands, as well as the lines of host code themselves. These values are known as the residue, and for lines of code which take only a dozen or so nanoseconds to execute, like "x = 1" for example, the residue can be a significant proportion of this timing, and in some cases can exceed the time taken by the host code line. This is the reason why the residue is not automatically subtracted from the host line time, as negative values can result in all sorts for problems for the bar chart, the averager, and a number of other systems, as well as being potentially very confusing for users who don't have a great familiarity of how the timing system works.

Until recently, the residues themselves were unusually large, which meant that a timing of a block of code as a whole, rather than line-by-line, produced timings which were very small in comparision to times taken by adding up all the lines inside a block using the standard line-by-line timing method. On discovering that residues were the issue, but knowing that the block-timing system was not very critical for main operations, the commands were disabled when the beta was first released, so that people could get to grips with the more importian functionality, and see if the system actually worked on their own PC's.

With the overlarge residues having been fixed a few days ago, I can now re-investigate the of re-activating the block-timing commands, which should prove invaluable for people who have a lot of code to profile, but don't know where to start.

--------------------------------------------------

Eventually, when the beta testers seem happy with how the system is running, I will elevate this project to Program Announcements, and commence work on the line graph, which will be able to display significantly more data that the standard bar graph, and which subsequently, should make viewing loop structure timings a lot easier. The bar chart will still be kept, along with all its display modes as, amongst other reasons, it is an ideal format for displaying alongside host code lines.

--------------------------------------------------

As always, I'll keep you all up-to-date as the project progresses.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 20th Nov 2009 00:39 Edited at: 20th Nov 2009 01:10
Link
Quote: "Are you saying that the program put up a windows crash message - something like "This program has encountered a problem, and needs to close", or did a standard DBPro error message appear. Though I'm suspecting the former, if it was the latter, could you tell me what that message was?"


The first error I recieved was when running my program in 800x600 resolution. All my graphics are designed for 1024x768. When running 800x600, the first thing the game does is resize the graphics using sprites (make sprite, scale sprite, paste sprite, get image) The get image command failed: Cannot grab image due to an illegal area at line .... I set the game to 1024x768 and ran again. This time it loaded the main menu graphics fine, except it pasted them in weird spots... I hosted a LAN game and it got to the part where it loaded the game background image and it threw Could not load bitmap at line .... I know the media exists, it's even included in the EXE.

Come to think of it, it might not be a problem with your tool. All the problems spawn from the resolution settings (position of images depends on resolution, size to resize images to depends on resolution, folder in which bitmaps are contained depends on resolution), so let me check that. I am using the most current untested beta of DBP so it could be faulting there.

Some more thoughts: Timing on a line by line basis is neat, but in some cases not necessary. For instance, if I create some code and time it and realize it isn't as fast as I want, I will most likely recode the entire block. Then after retesting, I would have to add up all the lines within the blocks of both the old and new code to see if any improvement was gained. I suggest that you add the option to do only blocks that you define yourself. That way, we can get timings that mean much more and eliminate the timing of individual lines in areas where numerous lines are needed to perform one function. Timing on entire functions would also be very useful, whereas timing for each line within a function could be useful, but not as useful as the function as a whole.

Just my thoughts. I will be back in shortly once I look at my resolution handler.

- GIDustin

Edit: It seems like none of my resolution-based globals are being set right. I currently use:


I did a quick test and you can still define a variable as global and assign it a value on the same line. Perhaps profiling code does something? I am compiling again, but this time I am not profiling that specific source file. I'll see what happens in about 20 minutes when it's done compiling.

Edit2: After compiling again with profiling code removed from my global functions above, the game works fine. It actually ran much better than I thought it would with every single line being profiled.

The DAT file created is 297MB for being in the game about 15 seconds and navigating as much of the game as I could. I do not think I am going to try to load it.

On another note, I plan to format my computer tonight and reinstall Windows 7 64 bit. I stupidly installed the 32 bit version by mistake several weeks ago and just now noticed. I will still have my laptop on hand so as to continue trolling threads.
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 22nd Nov 2009 14:04 Edited at: 22nd Nov 2009 19:08
Link
GIDustin,

I have now fixed the bug that your code snippet with the comment in it produced.

For the code snippet involving the "Global" declarations, I couldn't find any problems with it, the code ran fine and the results in the data viewer looked OK as well. I did modify the code slightly so that it compiled properly without the precompiler in place - the adjusted code is shown below:


------------------------------------------------------------------

I have run some tests with the data viewer which involved loading large files. The results are shown below:

File size: 61.544 MB = 31.48 seconds
File size: 247.020 MB = 1 min 19.35 seconds

My PC specs

Intel Quad Core CPU
nVidia 8800GTS 320MB
4GB RAM
Windows XP

I'm just wondering - did you accurately measure the file that took 2.5 hours to load, or was this just an approximation?

------------------------------------------------------------------

I have also run a check on memory usage of a medium-sized project with profiling code added (3528 lines). I used Windows Task manager for the test, but I'm not sure if I trust what I'm reading. There seems to be a small increase in page file usage, but not enough to warrant a crash due to high memory usage. I may need to look for/make a RAM usage viewer.

==================================================================

The commands "PADBProBlockStart()" and "PADBProBlockFinish()" have now been fully re-activated and seem to work OK. There can be slight discrepancies in the times between timing line-by-line and by block, but I don't think it's anything to worry about and it could just be where and when the code is executed that may be the cause of the fluctuation. If anyone encounters large deviations, let me know.

Please note that there are no "#" signs before these commands, and they also need a set of brackets at the end. In addition the "PADBProBlockStart()" command need to have a literal string inside these brackets; this is the block label. An example of this is shown below:

REM PADBProBlockStart( "Code Block One" )

The help file provided explains a little more about how to use these commands.

Timing of entire function/loop/gosub blocks has been available for some time anyway - just press the "F" key to view this data. In standard mode, the "F"-key will only modify the top line of the code structure - the other lines will be left alone.

------------------------------------------------------------------

The new version of the precompiler is now available - we are now on version 0.2beta.

==================================================================
EDIT:

A new version of the data viewer is up - we are now on version 0.2.1 beta. This version has had one or two alterations made to it that will allow it to display the "Block" based data correctly.

EDIT 2:

It would seem that the results presented by the accumulated function/loop times (retrieved by pressing the "F"-key) seem to be producing odd results for nested code structures. I will investigate this.

Please note: This does not affect the "BlockStart/"BlockFinish" command timings - only the values which are displayed when the "F"-key is pressed.

I would still like people to take a look at the current version, as the "Block" commands will help make life easier timing large code blocks all at once, and you can use these commands to time functions and loops for the moment until I manage to get the function/loop accumulator fixed.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 22nd Nov 2009 21:42
Link
Quote: "I'm just wondering - did you accurately measure the file that took 2.5 hours to load, or was this just an approximation?"


2.5 hours is when I decided to CTRL+ALT+DELETE it. It was loading the entire time (HDD activity). I remember checking memory usage when I had the task manager thing up and when I killed the viewer it was not taking that much memory, maybe like 60mb?

Quote: "I have now fixed the bug that your code snippet with the comment in it produced."


See, I looked at the _Temp.dbsource, and how your analyzer altered the code for timings, and it should have worked. However, removing that block of 5 functions from the profiled code fixed all my problems... Perhaps we can throw this one to the "unknown" heap.

On a separate note, I formatted my computer a few days ago (stupidly installed windows 7 32-bit on a 64 bit computer, had to reinstall the 64 bit version). I will have to get my project out of my backup image and give the new versions a try.

And for those wondering how this affects large projects, my 14,000 line project became a 20,000 line project after profiling code was added, and the overall FPS of the game dropped from 60 to about 30-40, still well within the playable range, even with every single line being timed. However, the data file was increasing at a rate of 20MB per second, so watch out how long you let it run.

- GIDustin
Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 22nd Nov 2009 21:54
Link
Indeed - with every line being analysed, it will use up quite a bit of space, especially where you have loops which undergo a lot of iterations.

For very large projects, using the "Block" based commands PADBProBlockStart() and PADBProBlockFinish() in a number of places to monitor large sections of code can help you pin down the general area where problems might be located, without the resulting .dat file engulfing huge amounts of disk space.

Then you can use the standard PADBProBegin/PADBProFinish commands to drill down to the line-by-line level, in order to precisely locate what lines are causing the most trouble.
Back to top
Profile PM Email
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 28th Nov 2009 23:13
Link
Update 28/11/09

I have managed to fix a number of bugs with the data viewer and the precompiler. New versions of these are now available. For the data viewer, we are now on version 0.2.2 beta, and for the precompiler, we are on version 0.2.1 beta.

I will be making an attempt to optimise the data viewer so that it loads in large files faster than it does already. As always, if you find any more bugs, I will take a look at them and fix them.

==================================================================

The fixes made in these updates include, amongst other things:

The display of the combined times for loops, functions and gosubs.
Profiling globals should now work better. This bug was discovered when I tried to profile the data viewer code and got some stange results. The problem centred around stacking other commands around global variable declarations.
Fixed a bug involving stacked commands that have comments next to them. The precompiler was not splitting such stacked commands correctly.
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 28th Nov 2009 23:36
Link
Quote: "Profiling globals should now work better. This bug was discovered when I tried to profile the data viewer code and got some stange results. The problem centred around stacking other commands around global variable declarations."


Could be the cause of my troubles earlier. The whole set of resolution functions did nothing BUT declare and set globals.

I had another suggestion for ya, and maybe your newest version of the precompiler assistant already addresses this.

Every now and then I hit the compile button, then while it start compiling I read the code still on screen behind the compiler and see an error I had made. I hit the cancel, fix the error, and recompile. The precompile assistant of yours seems to catch the compiler cancel button, but doesn't pass that to the DBP Compiler. Even after hitting cancel, the compiler assistant closes, but the actual DBP Compiler continues to run in the background. After it finishes, I get my usual "Compiler error, duplication TRIM$ command in Styx and Matrix" which I have always ignored.

The problem is when I do this with large projects. I hit compile, OOPS, hit cancel, fix the problem, but now I have to wait until the DBPCompiler process finishes in the background before I can start compiling again, and on large projects it is a hefty waiting period. If I try to compile again before that, it causes a whole slew of problems. I could CTRL+ALT+DELETE it but I am always weary of forcefully killing any process, espacially those with my latest project in hand, and usually just check my email or something while I wait it out.

- GIDustin

Back to top
Profile PM Email Website
Phjon
18
Years of Service
User Offline
Joined: 28th Nov 2005
Location:
Posted: 29th Nov 2009 00:30
Link
GIDustin,

OK, I'll take a look at that.

I haven't actually touched the assistant for quite a while, and there haven't been any new versions of it ever since I forgot to change the label displaying what version it was to the correct number (which is version 0.1.2 beta).
Back to top
Profile PM Email
GIDustin
15
Years of Service
User Offline
Joined: 30th May 2008
Location:
Posted: 29th Nov 2009 05:25 Edited at: 29th Nov 2009 20:49
Link
Just finished writing my first precompiler and I am using your assistant to run it.

Basically, it keeps track of build numbers. The precompiler opens a "Revision.dat" in the project folder, gets the last build number, increases it by one, saves it, and adds that as a global variable in the project. Now I don't have to manually increase build numbers in my projects.

I am still trying to think of other uses for precompilers... Maybe adding line concatenation support to codesurge...?

Edit: In case anyone wants them:

My Version Tracking Precompiler:

Just add global REVISION to your source and the precompiler will add the value on the same line. I tried #constant, but it wouldn't compile.


Codesurge Line Concatenation Support (...)


Just be careful not to end any comment lines with ..., i.e.:



Back to top
Profile PM Email Website
Back to top

Login to post a reply

Server time is: 2024-04-16 21:40:50
Your offset time is: 2024-04-16 21:40:50