Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Newcomers DBPro Corner / File Structure Method Question

Author
Message
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 20th Nov 2012 02:04
So, if you haven't seen my "Showing Off" thing, I'd recommend seeing it.

Within that folder, I had some files that were borderline sensitive. Should anyone choose to tamper with the structure, there might be consequences, especially later on in my program's development. Even if I'm storing my data within a pack, I don't know if it's particularly secure, as someone could just use DB to un-pack it.

So, here's the idea I had (note: not actually code)



Due to the complexity of my Assets tree, I was thinking about creating a file-builder that would let me make these pack files more easily, and then building a "tunneling" system that could wheedle out individual assets from places in the tree for the loader.

Now, is this approach sensible? Can anyone tell me where to start? Does anyone have code/recommendation to ease this process (and the ensuing headache)?

Bear in mind:
- I want to have user-modified or randomly-generated assets, some of which may be more complex than a default data set allows; these will therefore be stored within the save files.
- Characters within my game aren't a singular data type, as they will also need their perks and inventory as array lists. I'm also considering customizable models, in which case I would likely want to add model data into the save, probably also within the character's pack file.
Phaelax
DBPro Master
20
Years of Service
User Offline
Joined: 16th Apr 2003
Location: Metropia
Posted: 20th Nov 2012 06:35
You could write your data out to a memblock and store that. Then the structure of the files is proprietary to you and it'd be harder to extract individual files.

"You're not going crazy. You're going sane in a crazy world!" ~Tick
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 20th Nov 2012 20:10
Yeah, I thought that a memblock would be a lot more static than the "Russian Nesting Doll" method. And, knowing myself, I'd constantly have to resize/alter the storage space to account for expansions within the system. I use .ini files to so that I can grab checklists of contents without checklist code, so if I can get help developing this dynamic system all I have to do is work with those .inis.

nonZero, if you're reading this, I saw some of your code and would really appreciate your input specifically.
nonZero
12
Years of Service
User Offline
Joined: 10th Jul 2011
Location: Dark Empire HQ, Otherworld, Silent Hill
Posted: 21st Nov 2012 12:18
Quote: "nonZero, if you're reading this"



You called?

There was a thread some time ago where somebody actually exposed the weakness of DBP's encryption (It was something really pathetic like adding 3 to byte 1) so I'd agree that DBP's dirblocks are definitely out.
As Phaelax mentioned, memblocks/memory chunks are an option -- well actually a necessity -- as you'll never do any decryption/encryption/unpacking outside ram (unless you have a really fast hdd or time to kill).
A single file can be a double-edged sword. If you created a pack-file, you'd need to know where to look (the address) of the files within or you'd need to create a certain delim, say 5 bytes (101,102,103,104,105) or something as an indicator. The delim would obviously make anybody with a hex-editor and too much time able to figure out and extract each file. So that leaves "knowing" where they are and that's basically where a homebrew-header comes in handy. What you could do is have an ordinary single-folder pack file created by loading files to memblocks and then creating one huge chunk, saving the offset locations of each file in a statically-sized piece of memory (your header), say 400bytes, and then merge that with your file. You'll know that 400bytes must be added to each offset listed in the header but nobody else will. You could use an encryption scheme on the final chunk of memory before committing it to a file. I'd recommend something fairly un-complicated but sufficient to scramble:

Of course DBP is not the ideal language for byte-by-byte operations unless you're using pointers and direct memory access. Even then it would take quite some time to process large things.

A simpler approach is to leave files as they are and encrypt them individually. It would expose filenames (unless you simply rename them and refer to them with constants in your program) and smaller files would be less cpu load to crack. But there are ways to mitigate this using the previously-described information. One example of a custom-made system:

It'd be somewhat less taxing on resources than working with a single block of memory containing all files. It also means no worries about resizing the entire pack-file or worrying about the header being big enough and other allowances. Furthermore, you could use a redundancy system where all files were made the same size by padding them with n pointless bytes every n positions and add the remainder at the end. On its own it doesn't sound like much but if combined with an encrypted file, it makes it neigh-on impossible to detect the relevant bytes, especially if that bloating system is different per file-type.

Ultimately no system is fool-proof or tamper-proof. What you choose to do has to be based on the idea that "it's simply not worth the attacker's effort" or "it's simply infeasible without a massive cracking array". You also have to consider the trade between security and loading times. (EG: Not a good idea for each asset to take some five minutes to load).

If you want to learn more about encryption, you can have a look at the TrueCrypt documentation (TrueCrypt is freeware and you can download it at most places like Softpedia).

RP Functions Library v1.0

My signature has not been erased by a mod.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 21st Nov 2012 16:29
Quote: "Ultimately no system is fool-proof or tamper-proof. What you choose to do has to be based on the idea that "it's simply not worth the attacker's effort" or "it's simply infeasible without a massive cracking array". You also have to consider the trade between security and loading times. (EG: Not a good idea for each asset to take some five minutes to load)."


I've actually had a lot of ideas for encryption schema, the one test I ran had some major difficulty due to a lack of understanding of how to implement those ideas into actual code. Are there examples out there of memblock encryption within DB code itself? I think I might be able to pick something up if I had a working example.

I'm going to go pick up TrueCrypt documentation and take a hack at it, I guess. Thanks a bunch nonZero.
nonZero
12
Years of Service
User Offline
Joined: 10th Jul 2011
Location: Dark Empire HQ, Otherworld, Silent Hill
Posted: 24th Nov 2012 16:02
Quote: "Are there examples out there of memblock encryption within DB code itself? I think I might be able to pick something up if I had a working example."


Here's a couple of tutorials I cooked up:
Part 1 - Basics
Part 2 - Usable Implementation

Should help get you started. I might do one on homebrew file-containers too at some point too.

RP Functions Library v1.0

My signature has not been erased by a mod.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 24th Nov 2012 18:23
Thank you very much, nonZero.
I think I'll just leave the directory scheme open for viewing, but encrypt the individual files. If they really wanna tamper with them, I don't think I'll feel all that bad.
nonZero
12
Years of Service
User Offline
Joined: 10th Jul 2011
Location: Dark Empire HQ, Otherworld, Silent Hill
Posted: 24th Nov 2012 19:58
You're welcome

RP Functions Library v1.0

My signature has not been erased by a mod.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 30th Nov 2012 02:15
nonZero, I defer to your expertise once more.
So here I am with a bit-shifting cipher that I couldn't imagine someone solving for a long, long time that loads super-fast.

Thing is, I want all of my files to turn into my new custom file type, .enco. To avoid any confusion between file types, I want to have the start of the file read out as the unencrypted name of the file.

First 20 Bytes- ASCII file name. If the name is longer than that, I'll just take the first 20 characters.
Next 5 Bytes- (ASCII) a dot, followed by the file's extension.
Next 2 Bytes- Four randomly generated numbers between 0 and 7. I want to AND them onto each other (since they're only 4 bits a piece) and then separate before I decrypt. These will serve as something of a nonce for the decryption.

I also thought of making the next 6 or so become the size of the remaining file, so that if I just added the results of multiple encryptions together, I could know to read that many bytes and then start reading a new header. I'm not sure.

So I guess what I'm asking is, is your pack encryption tutorial going to account for writing things into files, byte for byte? I can't write my 64-bit encryption key to a custom file because I'm having trouble keeping track of pointers, as well. :\ Well it's 128 different numbers, but they're only 1-14. I want to store it in the 64 bits it actually requires, not the 1024 that DarkBASIC writes the array to.
Darkhog
11
Years of Service
User Offline
Joined: 25th Nov 2012
Location: Mushroom Kingdom
Posted: 30th Nov 2012 03:36
For additional level of security I'd suggest XOR-ing all your data memblocks with sequence of random numbers (very strong to break despite how it looks) using constant seed and then decoding it again with XOR-ing again by random number sequence from same seed.

The idea is that data won't be possible to decode without knowing seed it was coded with and even if said seed would be stored directly in EXE it can be positioned there just anywhere so without specific knowledge it'd be very hard to find.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 30th Nov 2012 20:02
That's actually how my encryption works, but it's based on something a little more complex than a seed. It's not addition based (like his tutorial) at all.
nonZero
12
Years of Service
User Offline
Joined: 10th Jul 2011
Location: Dark Empire HQ, Otherworld, Silent Hill
Posted: 1st Dec 2012 10:47 Edited at: 1st Dec 2012 10:49
Hope I'm understanding the question(s) correctly here...

Quote: "I can't write my 64-bit encryption key to a custom file because I'm having trouble keeping track of pointers,"


Here is a quick tutorial explaining pointers. May give you a different take:


Quote: "I want to store it in the 64 bits it actually requires, not the 1024 that DarkBASIC writes the array to. "


Now, you bit problem: The bit-size in arrays doesn't matter because you'll convert all the data to bytes instead.
1 byte = 8 bits on all architecture (that I know of), however ints, floats, doubles, etc. are a different kettle of fish which is why I always try to work with raw bytes where possible. So how DBP stores arrays won't really matter since you're not going to be using the arrays in their current form. In short yes, my tutorial specifically works with bytes only.

Quote: "Thing is, I want all of my files to turn into my new custom file type, .enco. To avoid any confusion between file types, I want to have the start of the file read out as the unencrypted name of the file."


About your header issue, you can encrypt the body separately. Here's my file-containers tut in rough, first draft:


^ Basically, while stored in a memblock, the "REAL" part of your file can be encrypted any way you like. Once committed to the final memory image (the memblock containing your file, the header and the bloating), that memory image is written byte-for-byte to a file. As in my tutorial, you could use the first 2 bytes to represent the header size. Then you're free to add whatever normal data you like upto 65000 bytes (65kb) worth (you can make it bigger but it'll cost more bytes). You can also use footers, as demonstrated, so you will be able to better secure your data by adding a few hundred bytes of "rubbish" or storing information about how to decrypt it.

Now about XORing: It is easy to crack under the right circumstances (ie the key is not as long as the data). So random or not, if the sequence repeats, it can be deciphered. This is not to say XOR cannot be used for encryption, merely that complex mechanisms are required (which is why I opt for mixing things up or simply using addition and subtraction + byte-swapping). Here is an example I cooked up (It's purely for illustrative purposes):



...I just think I beat the record for post-length in terms of chars. Hope there aren't any bugs or glitches in this mess.

RP Functions Library v1.0

My signature has not been erased by a mod.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 2nd Dec 2012 21:48
Thanks nonZero.
But, in all seriousness, I don't think you could crack my algorithm. It's not based in standard key logic. It's based on a crazy coordinate method I developed.

All that said, It's not all I plan to use. The simple idea of attaching the nonce to the encryption is a step in the right direction, but I've been experimenting with even more crazy methods.

Once more, thank you for being the genius you are. If you need anything, feel free to ask.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 3rd Dec 2012 02:07 Edited at: 3rd Dec 2012 02:21
Gah.



Ignoring the weird stuff in the header (and get_stream() and the XOR-ing part), is their anything I did wrong as far as writing that code? I can't tell because I hybridized it from the lesson 2 code and you seem to have completely rewritten your naming conventions, then I added even more strangeness in my own naming conventions XD So I can't keep track of what variable belonged to what part of your function.

EDIT:



Caught a couple of my own mistakes (notably, now it actually writes a file ) still, I don't want to go through a many-hours long headache only to realize I forgot something that my good friend nonZero would've caught right away.
nonZero
12
Years of Service
User Offline
Joined: 10th Jul 2011
Location: Dark Empire HQ, Otherworld, Silent Hill
Posted: 3rd Dec 2012 20:46
Apologies for the inconsistent naming. I can't use too much of my own naming-scheme for public source-code because it would be incomprehensible to others (many recurring names aren't even English) as I have my own "system" of common prefixes and descriptive suffixes. So when I make public stuff I try to use friendly naming schemes but I also try to throw in a descriptor here and there. Part of my naming convention which seeped into those examples was the Addr suffix:

Sometimes I inadvertently mangle my scheme, the typical scheme and the friendly scheme all together and produce a chimera. Sometimes I just type what I'm thinking by accident. That's why I usually like to look over my example after taking time to forget what's going on. Then I can say, "Can I follow this easily by using the names?" Unfortunately I didn't really have the time to do that with those.

Anyway, pointers the tutorial demonstrates different ways of accessing memory. I think you were mixing multiple different methods together. I picked up some errors here:

Error log:

But I don't know if that's all...
...Which is why it's always better to retype code than copy-paste and edit. That way, you can start with your own naming scheme from the ground up. One thing I will suggest is to avoid ambiguous names and give variables names that describe their purpose and type, eg:


What I would suggest is a rewrite if you take longer than an hour to get this right (sometimes it's quicker to trash than to to debug). Hopefully you won't need to rewrite though, I know what a pain it can be.

RP Functions Library v1.0

My signature has not been erased by a mod.
Mr909
11
Years of Service
User Offline
Joined: 2nd Jun 2012
Location:
Posted: 3rd Dec 2012 21:21
Well, due to an eye-opening test I performed, I learned that my code stream doesn't work even plausibly well with something like, say, a bitmap. Text/ini files, my main modus operandi, work fine. So I'm probably going to have to come up with a new generation algorithm and whatnot. I guess I will end up rewriting after all.
Thanks nonZero.

Login to post a reply

Server time is: 2024-03-29 13:59:21
Your offset time is: 2024-03-29 13:59:21