Sorry your browser is not supported!

i have a simple file with 5000 lines.
so i read it about 25 lines each sync loop, not more then that.
have tried different numbers for how many lines in between syncs, most of them are or too slow to reach the end of the file or drops down the fps to 1 2 3 5 ..etc
then i tried 25 lines per sync, so i thought to myself this should keep the fps OK but because the file is kinda big it against drops down to 5 fps.

when i am not reading from a file, the fps is on a steady reliable 30 fps so its not "too much stuff in the background"
i think that it is OK to read from a file on Android with agk every time laps rather then every frame but this may again cause the user to wait too long.

what i am doing is, i am searching for specific words inside that file, these files are not JSON related, i could turn them into JSON but i do not want to strict it to JSON only as i do not really believe in a single file format to rule them all this is not Mordor and we are not Hobbits

what do you offer to do? in my new ed apps i may encounter huge files of 100k+ lines.

I would recommend never reading a file during runtime unless it's absolutely necessary. Instead, read the values into an array, then handle the array as you are the file. Lookups in arrays, while not extremely fast when in the thousands of indexes, are still way faster than trying to read from files.

Reading 25 lines at a time per loop in an array should be plenty fast enough to keep up with runtime performance.

But i might have a file with 100k lines and even much more... Then what? ... I could not possibly insert all of them to an array...

you may have to,

Most OO databases work by the principle of loading it all into memory
Relational databases on the other hand can handle things much better
without having to until we get 3D storage and some huge improvements
in the algorythms

Yup, an array can easily handle 100K indexes of values. Memblocks also and can be leaner on memory. I use many times more than that in some of my work. Keys to performance are to perform operations in stages and handle data sets in as narrow of loops as possible.

I'm wondering what app could possibly require you to read 100k lines in one frame?

Not in one frame... But lets say this file posses 5789 lines
I want to search something..

This search value may be written in 100 lines and i need to show these lines. So there is no abstract way than to read the entire file. On windows it is flawless it takes about 7 seconds tk search everything ny keeping a steady fps reading 50 lines per frames.
On mobile reading 25 lines reduces the fps to 5 or 6
And by that making the app read 25 lines * 5 per second which makes everything worse with that said.if i make it to read 100 lines per frame it drops in a milisecond to 5 fps but then finish the search quick. But losing frame.rates is not wanted at all since animations are messed up.

I will check on relation db

And i am guessing i could read a sum of lines insert to an array and read more again and again to keep the fps high but still it will be slower then if ill bake the entire file.into an array and i do know that agks can not possible insert 100000 strings to an array it would simply crash since i have no idea what kind of phone will handle it and what kind will not..

Eventually i could also bake the searches.
Meaning. I could make a little outside app.
That will read the entire file.on pc.
Make a list of all the words add them to an array
Everytime i find that word, i will add a new getfilepos in that instance and then when the user search something i would actually search that file, that file that holds all the words will be baked into an array at startup of the mobile app where the new baked array will hold only the given word and a getfilepos to that line, this line will hold the word and all the getfilepos(s) that it ia found on the main file. Making the search immediately show the results by itterating the getfilepos(s) rather than reading the entire file.

^^^ You've solved your own problem!
If you can perform your search in one millisecond then that's the way to go. Just switch your animations so that they are frame rate independent (you should do this all the time anyway) and then there will be no glitch during the one millisecond slow-down.

ohh haha Scraggle i am sorry i did not make my self clear, the search does not takes one milisecond.

when i make it 100 lines to read per frame, it takes, one milisecond to get down to 5 fps for the rest of the search.
if i make it 25 lines per frame it takes a few seconds to drop to 5 fps and then continue.

sorry i was not clear , my english still needs some work

So i have searched the web an i have found Index Search thingy, so as i wrote before i was right on target ,
Indexing or Baking the search one time , creating a new file, and using that file to find my searches.
No memory needed at all, this way a 1GB file of txt could be searched in "Miliseconds"

i will present the solution later on, still got some stuff to figure out but it does work nicely so far.

Index searching is what Google / Apple / Microsoft is using in their Search engines and even in the File Browsers search. this is the fastest way to get on point without using any memory at all. since i am approaching Web and Mobile applications and pherhaps later on TV i have no idea what will be the memory capacity on all of these devices. This could actually work it self out into Assets Safety where you can with it protect your assets will all kind of cool strings/ bytes/ floats aka Unique Keys but we will see this later on.

I am having huge progress, please be patient.
The problem with Hebrew where it comes to indexing, is that there are more then one word for the exact word , and also to check all the words as if unique or not even with Array.sort
is a major pain for the CPU, i had to count the percentage if the progress in order to give it a break , so every 5 percent progress (using Mod(progress,5) = 0 ) i gave it a 20 seconds sleep .
and yeah, i turned on the AC all the way to the north pole.

so what i have now i extremely efficient, no memory at all is in use, it read every time just the designated line in the file and also display it in milliseconds. so now i can really move on to 1GB txt files.

Now, in English it should be even better why?

so this is a line in English :

I bought a car the other day.

in Hebrew the same sentence is:

קניתי מכונית ביום ההוא.

you can see one thing for sure, much less words, why? because in Hebrew, Present Past Future simple / Progressive are actually effecting each and every word rather then the sentence.

meaning:

in English : you might want to search for the word "Drive" right? it is somewhere inside the text. now Drive can be , Drive, Drove, Driving, and maybe Driven. the first and the last are about the same just with an "n" at the end.
the two in the middle are the past and future of the same word.

in Hebrew : if i am looking at the word "נסיעה" which is in English "Drive" it could be:
אסע // normal future
נסעתי // normal past
כשאסע // meaning when i will drive
נוסעים // we are driving
נסענו // we driven
נסע // we will drive
נוסעות // we will drive in female form (trust me you are better off then using both male and female words well actually for every word in Hebrew
נסעה // she drove
נסעו // they drove
וכשאסע // and when i will drive
כשנסעו // and when they drove

well i think you can grasp the idea a bit.
so at the current stage, i have a file which is separating all these words which are actually the same words, what i want to do now is to take all of these and when someone is searching something it will search eighter accurate word or about the same word, so just these have like 8 or 10 lines inside the text file which makes the search index file a bit long, i want it tighter , so actually if someone is looking for "סע" he or she should get all of these results this makes it a bit tougher but it can be done.

this is really exciting
for English it will be much easier. as if you are looking for Drive in a text you are not looking for Drove also when it comes to objects, Tree is עץ The Tree is העץ , no one will probably search for The but just Tree
i found about 20 chars in Hebrew that can change the word and if we are taking into account the old Hebrew then its about 10 more meaning 30 unique chars that can change all of the words in Hebrew.

stay tuned. this is the start of Assets Protection as i see it...

well I've never had a problem reading from a file I think your problem is not related to that. I think your problem is probably searching thousands of word strings it can get slow. you can load the file as an array or a MeM block. either will be much faster. I would also break up the file into types or catagories to speed it up. nouns proper nouns verbs adverbs possessive plural adjectives ect. so if drive is flagged as a verb ur program only has to check words that are verbs. and you could take it a step further and alphabetically organized them. if word to translate is drive. search array verb within verb array go
to the d's for drive for i = 1 to end string.verb.d[i] then check second letter see if it matches r if drive. no need to check d cause we are in ds array. loop this untill drive is found. drive:hebrewword if dive is found grab the right hand of the string

Yeah but then i will need to declare which is a verb and which is not.
This would just take too much time especially if i am going to use different translations.

The problem i see here is only that of huge txt files 100 500 mb and 1 Gb 5 Gb and more.
The easiest way is simply to insert them into memory if i use only pc. But i am aiming at mobiles which i simply do not know how.much free memory each device have. How much free memory etc...

So it is not really a problem but a miss conclusion on how to use large amounts of data inside a mobile device.

As far as i know if you use some apps on your phone they can each in background use more memory or less memory then the phone potential to run all of them in ease.

Yeah i figured out the use of arrays as for alpha betic search , if you search something that starts with "a" you do.not need the b and c and d ... So automatically you search much less.

I am building an Ed App, and planning to insert most of the educational books Israel has to offer inside it. It might be devided into several apps and it might not.

The Corona Holiday, made me think that if people cannot really go.to school in a similar fashion to the "before Corona time" we as a society are losing to a war of our young minds. This is not acceptable at least by me.
And obviously because we are talking mostly about kids, low frame rates at this era is not acceptable by them.

So it just made me think, what if i could get all of the educational stuff in one place and let it act like a "game"
Effects, Special keyboards, 2D, 3D shaders ,shadows, and ofcourse Searching stuff.

My app already devides stuff into chapters and signs.
I am on target just like you said.

But i wanted to keep all file sizes small.

So first file was to create a simple index
Then a search index
And then a super small search index.

So if i am searching now for Drive
It will check only 2 lines in the search index.
1 is for the accurate "Drive" found in the main txt file.
2 is the not accurate or similar to no matter if verb or something else.
I might also add a button for how much accuracy you want then it will still search these 2 lines only.

So first line is a milisecond search and present.
The second depends on accuracy.


About game assets protection. If ill take all of assets and connect them into one big asset file for images for example i could read using a smart search index in real time create them in the render itself rather then to load images at start up, because then i will be able for example to enter amounts of "rubbish" bytes before and after, this will automaticly change the "getfilepos" and "setthefilepos" of each assets as they are connected together with a chain, if you brake that chain, like hijacking my app and change its images then rerelease it under a different name or something it will fail because it is still searching for the right Index and file pos, by making 2 apps one for the indexing and one for release to the public it will be much harder to handle reverse engineering.

It is.not something that cannot be hacked but it will give pain to the ones who tries.

I can also see a lot of different usability for this.

And obviously if i am planning also on TV devices then memory will be a huge problem.

And i have a super rule in my apps.
No following people
Not using ads i did not approved
And not using the internet for content only updates.

This is because, if i am planning on kids as my main consumers, i will without a doubt not let them use the internet and not let other "bad people" find a crocked way to connect to these open web ports.

So everything has to be locally, for safety.

At the end of this i will also show you a very neat way to make money out of your apps with ads without using any ad service.

I already got some people who would like to pay for ads inside my app. It sums for about 500 shekels per month on the first month of release. 500 shekels are about 130-160 us dollars per month, a steady income with a 1 year contract.

Ok i finally realized how to do a decent string search no matter the file size. (In Hebrew, the English version was actually done the day i posted my earliest solution)
It took almost a month just to think about it.

Will show.my first app soon.