Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Code Snippets / [DBP] - [Matrix1] Precompiling functions for faster compile times!

Author
Message
SamKM
15
Years of Service
User Offline
Joined: 25th May 2009
Location:
Posted: 24th Sep 2015 04:59 Edited at: 25th Sep 2015 02:20
Link
Hey people, been playing around with this for a while, and it finally works
Should probably explain just for anyone who doesn't know already - the idea behind having precompiled code is that functions in your project which are completed and don't need changing can be saved in a compiled state. The benefit of doing this is that the compiler can then skip those functions each time the project is recompiled, instead of having to process them again and again for every minor change to your code. Using precompiled code has the most benefit when dealing with large function libraries, which can seriously slow down compile times when they have to be built every time.
Precompiling code is supported by languages like C++ - unfortunately, not so much in DBPro
Turns out it's possible though!

Here's the code for the system:


Here's a small example to show it working - replace the blank precomp_dlltemplate in the full code above with the one in this example when compiling it. Once you've run it once, the program should still work fine if you remove the functions for creating and updating the cube!


How to use it!
Basically, this system works by peeking the memory of compiled functions (accessed using the Matrix1Utils function ptr commands) and saving it to a header file. The code for the functions can then be removed from your project, and loaded from the header files at runtime. Apart from the fact you need to call them using 'call function ptr', functions loaded this way behave exactly the same as normal functions - they accept values, return values, run at the same speeds...
Depending on how much function code you remove from your project doing this, you should be able to massively speed up compile times, and, compared to how long the compiler normally takes to compile them, the time it takes to load functions from header files is pretty damn small - load times included, a spinning cube program using Evolved's Advanced Lighting library takes about 1 minute 30 seconds to compile on my machine. With the entire library saved to a header file, the program compiles/loads in about 6 seconds, with the whole library loading in less than half a second
Unfortunately, this is kind of hacky, and because of that, there's a trade off with using this system - having to structure your global variable and array declarations in weird ways
The reason for this is that function memory contains a number of pointers to different resources held internally by DBPro, such as DLLs, global variables, and arrays, which change each time your program is run - because precompiled functions are stored externally, any pointers they contain need to be updated when they're loaded. This means there needs to be a way for the system to identify the current pointers to these resources - hence the need for having some of your code in a specific structure!

Arrays:
Arrays are probably the worst part in terms of having to restructure your code. Apart from the one in the buffer template function, all arrays (including ones not used by functions you have/want to have precompiled) need to be DIMmed once before precomp_setup is called! Internally, arrays declared in your code are stored in an area of memory called (by me, probably not the official name ) the array buffer - having arrays in your code which are DIMmed after the setup function is called causes the setup to wrongly judge the size of the buffer, which is likely to screw things up. Here's an example:


Worth noting, it's totally fine to re-dim arrays with a different size or whatever anywhere else in your code, they just need to have been dimmed at least once before setup!
Another important thing - if you don't use precomp_declarelist (explained somewhere below), any functions you have precompiled will need to be recompiled if you add or remove any arrays in your code OR rename any existing arrays!

Global variables:
Global variables also get stored in a buffer, but having them declared before setup isn't necessary for judging it's size. It's fine to declare globals anywhere in your code, but it's important to move any global declarations out of functions you want to precompile, because you'll want to delete that function code afterwards - as with arrays, adding, removing or renaming any globals outside of the precomp_declarelist function will change the layout of the global buffer, which messes things up! Note that for any global variables using user-defined types, adding or removing fields to the type will also screw things up!

That's pretty much the annoying part - using the actual system is pretty straightforward - well, not so much the declare list maybe, but the rest of it

Functions:
precomp_setup(skip_stringbuffer as boolean, debug as boolean)
This needs to be called before compiling or loading any functions. When compiling functions, the string buffer needs to be scanned, but you can speed up load time by skipping it if you're loading functions!

precomp_compilefunctions(filename$, debug func_startname$, func_endname$)
Saves the compiled functions between and including func_startname$ and func_endname$ to the header file filename$. An example is if you had three functions in your code, func_1(), func_2(), and func_3(), each declared after the other - to compile all these to a single header file, you'd use the line 'precomp_compilefunctions("headerfile.dbh", "func_1", "func_3")'
Worth noting, func_endname$ can't be the last function in your code (because the system works out the size of a function's memory by getting the pointer to the next one) - an easy way around it is just adding a blank function to the end of your code.

precomp_loadfunctions(filename$, debugflag)
Loads previously saved functions to memory from the header file filename$. There's no limit to how many header files you can have loaded, so having functions saved into seperate header files works fine!

precomp_declarelist()
Probably the worst thing about this system is the fact that adding, removing, or renaming any arrays or globals, or adding/removing fields from a type used by a global, will most likely screw everything up (90% of the time ending in an 'Application has stopped working' error, but the other 10% giving you weird problems where loaded functions read/write to the wrong variables/arrays) - the declare list tries to make life easier! Whether it actually does is your call
The idea is that any arrays or globals you declare inside precomp_declarelist can be added, removed, or renamed without affecting loaded functions. Don't forget though, any arrays declared in the function still need to be DIMmed before the setup is called, so you should put a call to the declarelist function before the setup function.
Declaring arrays in the function is simple - just DIM them like normal. With globals, it's a little more complicated
Each global variable declared needs to be set to a value, so that the setup function knows how much space they occupy. Here's the list of sizes you need to assign:


Here's how the function should look declaring a mix of arrays and globals:



Worth noting it's totally fine to mix using the declare list with declaring arrays/globals outside the list - but any ones declared outside mean saved functions need to be recompiled
Apart from having to count up the size of UDTs, the most annoying thing about using function is you have to be really careful to make sure you've got it right - setting the wrong size value to variables won't give any kind of warning error, it'll just crash the program/make things go weird, so it's kind of a nightmare for debugging :/

precomp_dlltemplate()
The DLL template function is used by precomp_setup to get pointers to DBPro DLLs. Basically, the DLL template function should contain a list of commands, with each command representing a DLL plugin used in functions you want to precompile. Sounds pretty confusing, sorry, so here's an example - imagine you wanted to precompile this function:
function cube()
make object cube 1, 100
endfunction

There's only one command here, 'make object cube', which uses the DBPro Basic3D DLL. To precompile and load this function, the DLL template needs to contain any one command from the Basic3D DLL, so it can update pointers to that DLL - so in this example, the DLL template could look like this:
function precomp_dlltemplate()
rotate object 1, 0, 0, 0 `Any other command from the Basic3D DLL would work fine too!
endfunction

A DLL only needs to have one command representing it in the DLL template, so you wouldn't need to add anything more to the template if the cube function contained any more Basic3D commands!

Once you've precompiled functions, changing or removing lines in the template means you'll need to recompile the functions again, or it'll assign the wrong DLL pointers - instant meltdown D:

precomp_buffertemplate()
This template function can just be left alone - it's used by the setup function to get pointers to the array and string buffers. It should always look like this:
function precomp_buffertemplate()
string$ as string = "stringbuffer"
dim precompile_array(0) as boolean : precompile_array(0) = 1
endfunction

precomp_getfuncptr(funcname$)
Returns the pointer of the loaded function funcname$. Note that to call loaded functions, you need to use 'call function ptr' - using 'call function name' doesn't work

That's pretty much it! Here's the general pros and cons of using this system summed up:
Pros:
Massively increases compile times when used with large libraries such as Advanced Lighting
Allows for a truly unlimited number of functions - Credit to Chris Tate for discovering a weird bug which causes the compiler to crash when dealing with lots of functions!
Cons:
Pretty annoying and delicate to use
Loaded functions need to be called using 'call function ptr', which has a few limits itself (like a max of 10 arguments)

Sorry for the crazily long and probably really overdetailed post! To be honest my timing on finishing this is terrible, since it looks like DBPro will be getting an updated faster compiler after all, so it might not actually be worth anyone's time trying to get this working - hope it's useful to someone though
With more effort this whole system could probably be improved and made a lot easier to use, and I'm pretty sure there's potential in hacking DBPro functions by messing around with the memory - getting things like function overloading working probably wouldn''t be too hard! I'm probably done with coding for about a month now, getting this thing working has been a nightmare, but if anyone else wants to play around with the code, go for it
If you hit any problems with it, let me know!
Thanks

The code never bothered me anyway...
Back to top
Profile PM Email
Chris Tate
DBPro Master
16
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 24th Sep 2015 22:23
Link
Sounds like a charm of a project you have founded here; keep it up. Thanks for the credit, but I wish I did not have to find the problem.

I cannot test this out just yet, but will try it out in the near future. I have been struggling with the need for extreme code effeciency, but your work could lead to a more forgiving workflow.

I hope this gets attention.

Back to top
Profile PM Email Website
Booma
15
Years of Service
User Offline
Joined: 29th Mar 2009
Location:
Posted: 26th Sep 2015 11:37
Link
It works! (Windows 7, DEP disabled) I will test further. Excellent idea!
It seems to me that integration of this system in the editor has to be the following step.
Maybe it is necessary to talk to Balid? Or create own precompiler. The main idea that we mark function with precompile flag
and other work is done by the precompiler(if it is necessary, creates dbh files, update calls in code to call function ptr and other)
Thanks for the excellent project!
Back to top
Profile PM
CumQuaT
AGK Master
14
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 7th Oct 2015 17:20
Link
You mention at the end of this that it looks like DarkBasic Pro is getting a new compiler that is faster, etc... Where did you hear this? I've been hoping that would happen. I had actually thought the project was abandoned! (A terrible fear of mine given the scale of my project and the work left to do)
Back to top
Profile PM Email
CumQuaT
AGK Master
14
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 7th Oct 2015 17:23
Link
Actually, never mind, I just saw it hahaha that's what I get for leaving the forums alone for so long!

Boy, I hope this goes well... Given how amazing this community is, I'm sure it will!
Back to top
Profile PM Email
CumQuaT
AGK Master
14
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 8th Oct 2015 10:20
Link
Hello again! Just trying to use this here... How do I call a function which requires passed-in parameters? So for example if I had pre-compiled a function which had 3 arguments which could be passed into it, how would I call that function and pass in those arguments?
Back to top
Profile PM Email
SamKM
15
Years of Service
User Offline
Joined: 25th May 2009
Location:
Posted: 8th Oct 2015 12:36 Edited at: 8th Oct 2015 12:41
Link
Thanks so much for the interest people
@Balid - Good thinking, agreed, having a code precompiler would definitely make using this thing easier, I'll try and get back to this soon and work on that! Should be pretty easy to have it do things like writing the size of globals in the declare list for you, as well as the whole precompile flag thing and automatically recompiling any flagged functions if they're edited. I've got an idea for a way around having to use 'call function ptr' too, if I can find any way to write new function calls to memory, so I'll see if I can get that figured! The only real problem is how much impact it might have on compile times, but hopefully not much
@Cumquat - Not gonna lie, 'call function ptr' is really annoying to use in general :/
Call function ptr takes up to 10 arguments after it separated by commas - so if you wanted to pass 3 parameters it'd look like this:
call function ptr PTR, 1, 2, 3
If you're returning a value too, you've gotta put brackets around the ptr and any parameters like this:
call function ptr(PTR, 1, 2, 3)
Probably worth mentioning other 'call function ptr' oddities here too
Something it's easy to mess up with when using this is you need to make sure you don't pass any float arguments as integers, and vice versa - normally DBPro converts them for you, so doing this kind of thing is all good:
functionthing(5, 10)

function functionthing(int_param as integer, float_param# as float)
endfuntion
With call function ptr though, the second parameter passed needs to obviously be a float, otherwise the argument won't be read right:
call function ptr get ptr to function("functionthing"), 5, 10.0

The other problem with call function ptr is any values returned need to be passed straight to a variable for some reason - having them as an argument doesn't work
So this won't work:
if call function ptr(PTR) = 1
But this does:
value = call function ptr(PTR)
if value = 1
Thanks for trying it out, but to be honest, it's probably not worth messing around with at the moment - I'll try and improve it soon and see if there's a way to ditch call function ptr!
The code never bothered me anyway...
Back to top
Profile PM Email
CumQuaT
AGK Master
14
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 8th Oct 2015 13:00
Link
Truly you are some sort of wizard. I know you're saying it's not worth messing around with, but you have saved a 5 year project with this. I'm at a rather unique point where I can't afford to wait for any new official builds of the compiler due to Kickstarter requirements :\ so be safe in the knowledge that your work has massively helped at least one person!
Back to top
Profile PM Email
SamKM
15
Years of Service
User Offline
Joined: 25th May 2009
Location:
Posted: 9th Oct 2015 02:09 Edited at: 9th Oct 2015 02:21
Link
Thanks so much for the compliment (unless you're implying I'm an evil wizard ) - I'm honestly really honoured to have helped, I love the concept of Malevolence: SOA (and the game itself looks brilliant so far)! I've hit unexpected bugs with DBPro before too, I know the empty feeling from worrying you've probably wasted your time on the whole thing - those have mostly been projects I'd been working on for a few months though, so I guess I can't really relate to how hitting a major compiler bug 5 years and a Kickstarter into a game has gotta feel
Let me know if you hit any errors with the system and I'll try and fix it asap
The code never bothered me anyway...
Back to top
Profile PM Email
CumQuaT
AGK Master
14
Years of Service
User Offline
Joined: 28th Apr 2010
Location: Tasmania, Australia
Posted: 9th Oct 2015 05:27
Link
And let me know if you'd like a Steam key for Malevolence! Really, it's the least I can do! Shoot me an email at info@malevolencegame.com
Back to top
Profile PM Email
Brightside_
11
Years of Service
User Offline
Joined: 27th Oct 2013
Location:
Posted: 13th Nov 2015 06:52
Link
@SamKM
Lee is busy now on migrating Game Guru to C++ and DirectX 11 so I don't think that we'll see new compiler in near future.So the only way to speed up in creating big projects is to precompile them in your way.
I hope that it is possible to create precompiler or so so most of community will be able to use it. This is very important thing because DBPro is not updating any more so if you want something new you
have to create it from scratch all by yourself.
Just few questions
1.You have wrote that there is problem with dynamic deleting arrays.How you make AdvancedLighting compile? There is a lot of dynamic arrays there...
2.How many things you had to change in for example in AdvancedLighting? I get that all dim's must be declared before compiled functions and that return value from
function must be declared with particular value..anything else?
Back to top
Profile PM
Kuper
16
Years of Service
User Offline
Joined: 25th Feb 2008
Playing: Planescape:Torment
Posted: 30th Dec 2015 12:14 Edited at: 30th Dec 2015 12:15
Link
I hope that will give second breath for huge-size projects.
How to use precomp_declarelist() ?
I need to move all array and declared globals to it?
I ve made few tests and editing/array inserting works fine even if i just declared arrays and globals in the begining of main program file.
So I dont get why you added such function?

http://anyhowengine.wordpress.com
Back to top
Profile PM Email Website
Back to top

Login to post a reply

Server time is: 2024-11-09 00:32:08
Your offset time is: 2024-11-09 00:32:08