Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Code Snippets / [DBP] - fast string concatenation (functions attached]

Author
Message
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 9th Jun 2017 11:55 Edited at: 26th Feb 2019 05:11
Here is a piece of code that processes the concatenation of strings at high speed. I needed a way to export my DBO files to X format (including animated skinned meshes) in a fast way, so this code is utilized in that particular app. The code is undergoing optimisation, but the original code is here for now. Feel free to do with it what you want, improve and experiment with it:
[updated code 01Jul2018-now removed function CalcTotalStringBytes. ParseString function now returns string length and totsz is incremented with each call to ParseString, see updated code below]
Try changing the "iter" variable to a lot higher and compare time it takes to perform the concatenation using the traditional method to the faster alternative method.

Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 23rd Feb 2019 12:22 Edited at: 23rd Feb 2019 12:39
Will be updating this soon to be more efficient and optimized and making use of link array etc

Will add functions back in once updated and tested thoroughly
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 26th Feb 2019 05:10 Edited at: 26th Feb 2019 05:10
added original code bavk in on first post with small tweaks for now. still testing the link array code version.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 26th Feb 2019 06:26
actually there is a command "GET STRING PTR" which can be used to return the pointer to a specified string variable, so might make another version implementing this command see if it has better results.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 27th Feb 2019 05:11 Edited at: 27th Feb 2019 05:37
another method to do fast string concatenation using GET STRING PTR. Could also use data statements if you really wanted to to hold the strings depending on your scenario:



So this code now only uses the function FastConcatStrings and there are no memblocks or banks involved only MAKE MEMORY or you could use ALLOC. This example took 1.24 secs to concatenate 5 strings over 100000 iterations. 1 millions iterations takes about 12.6 secs on my laptop, will test on my desktop later. The traditional method will take minutes for 100000 iterations, a million iterations could take up to or even over an hour. Test it yourself if you want to. This faster concat approach is great for massive strings that you might want to export specific 3d model formats such as directX in ASCII format e,g, animation data strings in skinned meshes.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 28th Feb 2019 05:33 Edited at: 28th Feb 2019 05:42
here is another method using link array and adding a AddConcatString function which adds more flexibility, this way you can parse any array pointer that holds the list of strings to be concatenated:
of course you can change function names and use any method you want depending on the scenario for your program. Running this code using 1 million iterations took 3.658 seconds to process. I commented out the code that does it the traditional way as your talking about over an hour or more probably. There is a limit I found i.e. trying 2 million iterations on 5 joined strings just bombed out on my laptop which doesn't have a great deal of memory so might be ok on my pc which has around 12 GB of memory or more, but reducing the number of strings to join to say 2 allowed the program to run it's course without prematurely exiting. 2 million iterations on 2 joined strings took 3.167 seconds.

Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 4th Mar 2019 13:11
tested on updated code not yet attached) using 10 million iterations on my desktop PC (64 bit windows 10) and took 10 seconds to process the concatenation of 3 short strings and about 79 seconds to actually print to screen.
Will probably apply latest code to my DBO to directX app and see if there is a massive improvement.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 7th Mar 2019 13:23
still updating this and testing. I've removed "fill memory" and used alloc zeroed function instead which fills the memory with 0's. Perfect for strings.
Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others
Bored of the Rings
14
Years of Service
User Offline
Joined: 25th Feb 2005
Location: Middle Earth
Posted: 8th Mar 2019 08:17
of course, you could just do away with all functions and do straight code, see example code below:



Professional Programmer, languages: SAS, C++, SQL, PL-SQL, DBPro, Purebasic, JavaScript, others

Login to post a reply

Server time is: 2019-05-21 13:29:36
Your offset time is: 2019-05-21 13:29:36