Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Faster File I/O - Faster Read - Faster Write - Make File From Memblock - Make Memblock From Array - Cast DWORD - Cast Float - Batch Writes

Author
Message
Mage
17
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 30th Jul 2021 08:31 Edited at: 2nd Aug 2021 02:13
I have discovered how to read and write faster in DB Pro. Sorry the title is written to make this thread searchable later.
(DB is kind of limited we can't use things like file streams, etc)

The standard file commands are really inefficient and slow, and are only good for small files. This is because they only write a few bytes per I/O operation.
Normally people will use the standard file commands:
Open To Write
Write Long
Write String
Close File

If you are writing to a network drive, or have a huge number of writes to make, this is really slow. You will be bottlenecked by the I/O operations.
There is the concept of "Batch Writes" that can help here. Writing at several kb or much more at a time.

Make File From Memblock
This command can help speed up writing large files. It's about 50x faster. It will write bytes to a file in larger groups at a time optimizing the speed.
I found that I could throw all of my writing into an array of bytes. (bytes are 0-255)

Make Memblock From Array
This command can then convert the array into a memblock. This is needed because you can't resize a memblock which is an issue since you likely won't know the number of bytes being written to file ahead of time. You can create an array instead then convert it.

So you want to write a number or a string to a file, how do you get the bytes for each of these stored values?

Most of the popular data types like Integer, Long, Float, DWORD are 4bytes each. It's best to copy (cast) them into a DWORD, where there is no number conversion.
Integer and Long can simply be copied into a DWORD like any variable and then we can break that down.
Float was a problem. There seemed to be no way to cast FLOAT to DWORD. So I used the Matrix1Utils plugin command Cast Float To DWORD(float).
So the actual binary bits are copied into a DWORD unchanged.
You could also use a WORD data type but that's only 2 bytes and it will cut off numbers larger than 65535 if I recall.
The byte data type of course is the smallest with values of 0 to 255

So you've copied/casted your variable to a DWORD and the information is preserved.
You now need to break that into 4 bytes.

You can do that like this:

ByteA = (oDWORD << 24) >> 24
ByteB = (oDWORD << 16) >> 24
ByteC = (oDWORD << 8) >> 24
ByteD = oDWORD >> 24

Where ByteA is the smallest digits of a full 32bit number and ByteD is the largest of the digits.
So like a large number like 1,000,000,000 would be represented in the byte order DCBA.
What the operations above are actually using 'Bit Shifting' operations to trim sections of the binary data and then store that section as a number that can easily be written as a byte later.
So for a rough example 1,456,900. Imagine if 456 was the middle byte and we stored it as 456. In reality the above code does this with binary for each of the 4 bytes.

It's worth mentioning when you go to actually write these bytes to a file they are 'little endian'. So ByteA (with the least significant digits) is written first, and then the rest of the bytes in order.


What about Strings?
Strings are stored in proper order, with one byte for each character (the ASCII character code), then at the end of each string are the bytes 13 and 10 each separately written. These bytes terminate each string.


So you can basically put each byte in order into an array, then send it all to a file much faster than would normally be possible. About 50x faster from my testing.

Examples:
I created a series of wrapper functions that can simple be attached to any program using Include command.
These let me use several simple function calls to use this optimization:
This library is called VirtualFile as a naming convention to make things simpler.
vFile_OpenWrite() - This opens a 'virtual file', basically prepares a new byte array for writing. Returns an Index
vFile_WriteDword() - These functions take variables and put them in the byte array.
vFile_WriteWord()
vFile_WriteByte()
vFile_WriteFloat()
vFile_WriteString()
vFile_Write() - This writes the array to the file, using the optimizations.
vFile_Close() - This cleans up and closes things.
vFile_OpenRead() - This opens a 'virtual file', prepares a new byte array for reading, loads the file into that array. Returns an Index
vFile_ReadDword() - These functions extract bytes from the byte array into variables.
vFile_ReadWord()
vFile_ReadByte()
vFile_ReadFloat()
vFile_ReadString()
vFile_FileEnd() - This will determine if all of the bytes have been read from the loaded file.
See code here:


Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Mage
17
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 2nd Aug 2021 02:21
Updated original post to include reading from files, originally just writing. Tweaked some function names. Also slightly less caffeine.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!
Hannibal
17
Years of Service
User Offline
Joined: 5th Mar 2007
Location:
Posted: 11th Oct 2021 01:54
I got this memblock drawing code from this forum, doesn#t say who wrote it, but I just made a more friendly demo, showing it working, of course it would be better to update the screen with the changed areas than the entire screen,....

Mage
17
Years of Service
User Offline
Joined: 3rd Feb 2007
Location: Canada
Posted: 12th Oct 2021 00:03
Yeah I have found in recent years that writing a collection of wrapper functions that solve problems in simple, fast, and pretty ways really helps. They can just be included into an existing program and you save so much time being able to skip the issues they solve. I tend to post these efforts for other people to benefit and review.

Mage's Modular Code - Get awesome UI controls and powerful Bitmap Fonts!
Screen Device Recovery - Stop your apps from crashing when minimized/tabbed/screen locked.
CPU Friendly Frame Limiter - Make your app generate less computer heat + noise, and use less battery life!

Login to post a reply

Server time is: 2024-04-25 08:29:36
Your offset time is: 2024-04-25 08:29:36