Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Geek Culture / Compression - anything better than 7-Zip? (floats)

Author
Message
Lampton Worm
22
Years of Service
User Offline
Joined: 4th Sep 2002
Location: United Kingdom
Posted: 20th Feb 2006 22:37
Link
Hi,

I've got a 60mb file, which is essentially a few million (mostly different) floats written out by DBP, and using 7-Zip to compress the file, I can get it down about 50%, just wondered if anyone knew if I could get it down further?

Actually, if you happen to know a better way to store that kind of data in a file (by better, I mean smaller, I don't mind lengthly compress/decompress times) I'd be interested.

Cheers
Back to top
Profile PM Website
Nicholas Thompson
20
Years of Service
User Offline
Joined: 6th Sep 2004
Location: Bognor Regis, UK
Posted: 20th Feb 2006 22:48
Link
what settings are you giving to 7zip? I'm surprised it can only manage 50% for what is essentially plain text...

Take a look here:
http://www.compression-links.info/

That looks a pretty usefull site.


http://forum.antiblog.com, http://thingy-ma-jig.co.uk/new/
Back to top
Profile PM Email Website
Lampton Worm
22
Years of Service
User Offline
Joined: 4th Sep 2002
Location: United Kingdom
Posted: 20th Feb 2006 23:03 Edited at: 20th Feb 2006 23:08
Link
Hi,

Currently -

Archive Format: 7z
Compression Level: Ultra
Compression Method: LZMA
Dictionary Size: 32MB
Word Size: 64

(File itself is 16,777,216 floats of varying sizes)

I played around with dictionary/word sizes but it didn't really make any odds. I'll have a look at that link also, ta.

EDIT: Hmm, I was using write float, but using write string while making a bigger un-compressed file, actually compresses better. Only problem is that is I need to then cast back to floats when reading the info back in later, I'll keep playing..

Cheers
Back to top
Profile PM Website
Nicholas Thompson
20
Years of Service
User Offline
Joined: 6th Sep 2004
Location: Bognor Regis, UK
Posted: 21st Feb 2006 00:42 Edited at: 21st Feb 2006 00:44
Link
might work... The help file is vague as ever!
Quote: "
VAL
This command will return an integer number of the string provided by converting the string to a numerical form.

SYNTAX
Return Float=VAL(String)
"


Basically - it either will or wont do what you want

Oh yeah - as for the string/float compression rate... My thoughts on this are that although a string will make a larger file, in your case each byte is only ever going to be 1 of 10 options (0-9). This means the algorithm is much more likely to see patterns it can losslessly compress down. Thats my logic anyway - but maybe thats why there is no ".NT" file extension which can compress data down into sub-bit level!


http://forum.antiblog.com, http://thingy-ma-jig.co.uk/new/
Back to top
Profile PM Email Website
JoelJ
21
Years of Service
User Offline
Joined: 8th Sep 2003
Location: UTAH
Posted: 21st Feb 2006 02:53
Link
is there a binary reader/writer for DBP?
aren't those better for something like this?


This just in: White lab coats cause cancer in mice. Details comming soon.
Back to top
Profile PM Website
Lampton Worm
22
Years of Service
User Offline
Joined: 4th Sep 2002
Location: United Kingdom
Posted: 21st Feb 2006 09:25
Link
Re: conversion.. yes, sorry that's the ticket I should have elaborated, conversion itself isn't a problem - I just meant I'd now have to cast all 16 million back, just another step to do that'll take another chunk of time. Using raw floats I wouldn't have to do that, tis all!
Back to top
Profile PM Website
Back to top

Login to post a reply

Server time is: 2024-11-16 15:58:04
Your offset time is: 2024-11-16 15:58:04