Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Work in Progress / Road to PlayBasicFX

Author
Message
Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 31st May 2013 10:12
Round UP,


PlayBASIC V1.64O Beta 9c - CRF tweaks (May 26, 2013)

Been looking into the CRF libraries the last few sessions. The tweaks started out looking at the loading process, but have progressed to cleaning up the communication protocol between libraries and the legacy VM, plus a few more optimizations. The clean up is a compromise really between the new and the old, but it removes some of the annoying set up work when sharing common resources between the VM with other libraries. But I'll get more into that in some future blog.

CRF formatted fonts have a couple of different flavors, namely 2, 8 or 32 bit. Font's created using the interval commands will be 32bit if your converting a bitmap font,or mostly likely 8bit it's it's a GDI conversion. While picking through some of the routines you can see's these a bit of generic meat on the bones so have trimmed a health slab of that fat out of some render combinations. The one that's showing the biggest benefit would be the 8bit mode when blending to colour. The previous edition seemed to be performing the blend every time a span was drawn. Depending upon the situation, this is generally unnecessary, so can be pre computed.

The result of the changes gives about 20-30% higher through put, which is more than handy.






Testing blend with colour in different depth surfaces






PlayBASIC V1.64O Beta 11 - Loading CRF's directly from memory (May 28, 2013)

While looking through the CRF library I've added a way to load the font from memory rather than disc. The following snippet loads a GDI font and saves it out into CRF format. The program then loads this file into a memory bank, then loads the font from the memory bank using the amended loadFont command. Allowing font data to be stored in pack files, encrypted, crunched etc. Ideally what i'd like to do, is implement some method where 'media' in general can be compiled into the exe's resource bindings. Which is only partially possible in legacy PB editions.






PlayBASIC V1.64O Beta 11b - Loading PNG images directly from memory (May 29, 2013)

So in keeping with yesterdays updating of the CRF loader, here we have a variation of the PNG loader. Just like the CRF stuff we pass the LoadImage (or any of it's variants) the address as a string with the & prefix. The only difference is that this time, the buffered image data needs a 4 header with the size of the image data in it. I'm not too keen on it, but it seems unavoidable.

Today's built only supports PNG's, since that loader was just fetching the data into memory to begin with, the other formats are a little more 'attached' to the disc interface.







Loading Map Blocks directly from memory

Since the Map library uses the image library to load media these days we're able to expose the functionality to loading blocks also.







To read all about the development of this upgrade check out PlayBASIC V1.64O Work In Progress Blog

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 11th Jun 2013 01:35 Edited at: 11th Jun 2013 01:38
PlayBASIC V1.64O Beta 12 - Loading BMP/TGA images directly from memory (June 01, 2013)

So tonight's little coding session adds load from memory support for BMP and TGA image formats. From the limited testing it's had tonight, it all seems to be working pretty well. When picking through the loader, the easy solution just seemed to be replace the 'file' fetching commands with an ambiguous interface. Which meant rolling a bit of extra set up code, but once that was done the decode logic is identical in both modes.

Just like loading PNGs from memory, the loader needs the size of the resource (in bytes) poked into the data header. This is really in an effort to trap some load possible errors up front, but existing loaders don't really bother with errors too much. Some formats will require it, others not so much.

Anyway, what all this now allows you to do, is create custom pack files with at least the image + font media in them.



PlayBASIC V1.64O Beta 13 - 32bit Power Of 2 and Black Optimizing (June 04, 2013)

Been taking a peek back into the texture map span rendering library, which is the core behind pretty much anything that's rotated. Can't remember now, but at some point we added a hand full of short cut routines for some of common render modes. So if the draw mode was 'simple' we call a dedicated routine to solve this. This avoids having to falling through the multi pass filler, which is pretty flexible, but can be eat up some render time.

Rotated Sprites are texture mapped quads in PlayBASIC, when we draw a rotated image/texture map, we're fetching pixels from texture in a none linear fashion. So we look up each texel (pixel in texture space) by interpolation down the polygon edges and across the strip. The coordinates might be anywhere in the texture though. Now since the texture can be any size, we need a multiply in the texel fetch to work out the where in memory the pixel actually is. Knowing this, we added some power of two render modes back during some other update. Now when the mapper routine knows the source texture is a power of two (width), it calls a version of the filler than uses bit shifts rather than mults. A bit shift is simply faster !

What this means is that if you're using image widths that are a nice round power of two, you'll get some extra speed for pretty much nothing. Not only that, if the mask colour is zero (black = argb(0,0,0,0)), there's some combinations of the span filler that use this knowledge to it's advantage. Since this allows the inner fill routine to avoid actually comparing texels with a user defined mask colour. Winning us back some extra through put.

OK.. This is all old news, so tonight's little challenge has been to try and peel back the 32bit routines as much as possible with some interesting results. This morning build of Beta 13 is rendering s the 500 (64*64) sprite test scene around 20% faster than beta 12 (and older builds) when the image is power of two and mask colour is zero. Not only that actually slightly faster across the board.


PlayBASIC V1.64O Beta 13 - Download (11,June,2013)

Here's what's looking to be the final beta for the V1.64O revision. We strongly recommend you test it with your programs prior to the final is released.

Get Download From BlOG Thread

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 7th Jul 2013 18:41 Edited at: 7th Jul 2013 18:52
Round UP of new additions in PlayBASIC,


PlayBASIC V1.64O Beta 14 - Palette Mapping Library (June 12, 2013)

Earlier this update we looked at some methods for mapping groups of 8bits out as palette mapped pixels. To test the theory, an example showed how we can draw mock up commodore 64 screen in 'hires'

mode. But the process is not just the realm of emulation, as such conversions also occur loading/importing image data, 2bit & 4bit windows bitmap files come to mind.

Since the initial tinker i'd been thinking of dropping some similar functions into a SLIB. So far there's only four functions, but they cater for the situations that come to mind. So we've got a function to take a byte and couple of functions to render splits of bytes/words as palettes mapped horizontal strip. There's not rect render as this point, but it's similar to something I was testing a number of years ago.

For those not sure what palette mapping is, well it's a bit of step back to the old days of 8bit graphics hardware. A classic 8Bit screen mode, has up to 256 unique colours on screen ( 2^8=256 ) . Most systems that had 8bit displays had 24bit RGB colours, much like today. The difference is that rather than the screen representation being an array colours RGB values like it is today, legacy hardware used a palette indexes. So each pixel was a index into the palette array. Each pixel index was one byte.

You can visual it like this,



So the hardware would read through the Screen() array, grab the each colour index value and use that value to read from the Palette to get output colour. ie OutputRGB =Palette( Screen(Xpos,Ypos))

Old systems used palette mapped images because they're very efficient (on memory and cpu). For example, to draw graphics in 8bit, the program only has to shift the colour index data in screen(), which in an 8bit screen mode is 1/4 of we have push around today in 32bit modes. The down side is that you're very limited in terms of colours. You literally only have 256 colours to choose from on screen at once and it's difficult to remap/make them at runtime. Some systems had ways around this (mutli plexing), but it's still pretty limiting.

However, there's a number of neat effects you can do with palettes, probably the one most people are familiar with is colour cycling. Which you'd often see used to convey motion in water/lava sprites/pictures in retro games. Ironically, now days we'd pretty much have to build an animation to achieve to the same thing.

Knowing this, figured one easy way would be to use a function that's designed to take 8bit/16bit arrays and render them as if they're a normal strip/image. So if you had a colour cycling fragment, you'd scroll the palette, then render the fragment to a regular PB image and hey presto colour cycling.


Here's an example of a routine to draw a palette mapped set of strips to an image. The 8bit image data is just set up as runs of ascending values with the palette set up to scale up from, rgb(0,0,0) to rgb(255,255,255), so we get a gradient of sorts. I've randomly changed some of the pixels and the first few colours to make sure it's rendering correctly.



Speed wise it's fine, the loop executes in about 1/2 a millisecond on my old desk top, attached is a picture. Was going to create something a little more interesting to look at but couldn't be bothered..



PlayBASIC V1.64O Beta 14 - Palette Mapping Library Cont (June 13, 2013)

Ok.. so dropped some more functions into the palette mapping library and back end. This is going to be pretty difficult to wrap your minds around, but in the demo bellow we're loading RGB mapped images, converted them to a global palette which is 16bit in size. This gives us a 2^16 possible colours that be used. The colours can still be 32bit. Now since we're using 16bit palette, we need to force the loaded images to be 16bit 656 format. We do this using the CReateFXimagfeEX command. Which allows you to create an image of any depth (pixel format), regardless of the screen mode. Then a conversion routine replaces the RGB colour with the palette indexes in the image.

Now if we create a 'screen' image that's the same depth as the palette mapped sprites (16bit), we can use the normal image rendering stuff in PlayBASIC to draw them to the conceptually palette mapped buffer. Once we're done rendering, we draw the screen out as a group of palette mapped strips. This gives us a 32bit end result, but we've halved the amount of bandwidth that's moved around in the picture. Plus you can modify the palette. Raster Bars, Colour cycling etc

Here's a demo showing the basic approach, the palette mapped scene is actually fading in/out in this demo. Fading a Palette mapped mode is much less work, since we only have to scale the palette

RGB's and not every single pixel.




So bellow we have a shot of what this looks like running.. In the bottom left hand corner you can see the palette mapped SHIP/BUBBLE images drawn directly to screen. The pixel data in them is no longer RGB's colour values, but INDEX into the palette array. So they look all crazy now when drawn to to the normal display..


PlayBASIC V1.64O Beta 14 - Palette Mapping Library Cont #2

Continued tweaking the library ideas so the following is more a proof of concept. Previously we had a pretty shallow palette (1200 colour or so), so I wanted to test out something a bit bigger. Soon found the Missile Attack backdrop which is 800*600. The only issue with converting it, is that the routine the palette maps image data isn't very quick (nested linear searching). In particular when the image is mostly unique colours like this one, so it can a take few seconds to map that image..

Drawing the scene is same as normal, it's a 16bit blit of the palette mapping background to the 16bit screen. Since there's no pixel format conversions it ends up as a pure memory copy.. What's interesting though, is that you can draw a good 10 layers of 100 transparent images over the top. Which can rotate also. What you couldn't do (at least without setting up the palette in very particular way) is blend or filter them. Since those actions would treat the individual texels in the sprites as RGB channels and not Indexes, so the mixing them would create all sorts of funky results.. I'll bet there's some very interesting effects that are just waiting to be discovered because of that however.

Since the palette data in the test scene is now pretty large (50K colours), sitting in a for/next loop tinting all those colours individually is a lot of dead weight. So one easy way around this is to the use the BlitImage functions to do the tinting for us. To do this, we make a image (32bit) (CreateFxImageEX), plot the palette colours in linear order (from 0 to $ffff). This gives us a chunk of memory that contains the palette that we blend with. So just make a second image (same size + depth) and then blit it across. This takes advantage of parallel mutilations available in the BlitImage functions. You could even thread it.


PlayBASIC V1.64O Beta 15 - Palette Mapping Round 3 (June 23, 2013)

Been messing around with the palette mapping routines in PB and in the engine, coming up with a better method is always pretty satisfying, in particular when you can get the vanilla PB implementation that runs quicker than the original brute force version in C. To map the mountain image the original routine was taking 16->17 seconds. The alternative version in standard PB does the same job in 3.7 seconds. Had a feeling the newer method would be faster, but wasn't expecting that. So by applying the rule of 10, then the same logic in C should run in about 370 milliseconds, which wasn't too far off, but It was more like 450.

But, there's a but.... it's actually not the same method. The original routine would map all incoming images into a global palette, where as the alternative method just works out the palette for this image. Which is handy for say image compression, but doesn't slot into what we already had and coming up with a variation on the same theme that was fast, wasn't initially forthcoming. Luckily while out on the bike, came up with a reordering/splicing idea to shuffle the new images unique colours into the existing palette and when combined with an idea to cull the original RGB colours as much as possible, the final solution is about 3-times faster again. Where it's now able to Palette Map the previous demo scene now in 130/140 milliseconds.

The interesting thing about this implementation is it gives you 2^16 unique 32bit colours within the frame, unlike an 2^8 screen mode in VGA/VSGA / AGA graphics hardware. Those with a little bit of creativity will no doubt discover ways of expanding the colour count way beyond that.



PlayBASIC V1.64O Beta 17 - Resource Binding Library (July 06, 2013)

So another session on and we're a bare bones implementation of data binding. To handle the binding we use the #AddResource directive followed by a literal string containing the file name (and path). It's pretty dumb as this point as it only allows files to be bound individually. This occurs during compile time currently so any time it sees the #AddResource directive it loads that media into memory. The load is loaded as is, it doesn't understand the file, it's just being considered as pure data.

Binding is one thing, using them other problem. To help out, i'm throwing together some high level functions to grab/copy resources in particular way. These will most likely be a [color=green]as string[/b], [color=green]as bank[/b], or [color=green]as image[/b] if it's a image media type. To query a resource we use it's imported filename. This is a bit iffy, but it fits perfectly into how this embeds, so ya get that.

The example code bellow, imports a text file into the program memory (and the final exe when built). To access the resource data in our program we use LoadResourceAsString$ function. The query functions use the index of the resource, rather than the name. There's FindResourceIndex(Name$) to find the index of the media in the table.

Example Usage:






PlayBASIC V1.64O Beta 17 - Resource Binding Library #2

Dropped a few more test functions last night, was going to add them all as internal bindings, but might split them in two and have the core stuff bound and the helper functions as a library. Since the helpers are extra bits and aren't always needed. So far we're got some functions that will load bound materials in as PB media as images and fonts. Since you can do this manually you can modify the resource before you use it. Which might be some form of simple decompression, decryption etc etc.

In this example we're binding the a text file, PNG and CRF font into EXE memory and then loading them directly on demand, without ever needing to save them out to disc.




Today's little task will be to hook into the LoadFont + LoadImage commands so they can find attached resources and automatically load them for memory for you. The frame work exists now so it should be fairly easy.. Should be.. but you never know.


Step #1 - Load Font Resources

So the first step is up and running. If you bind a CRF into the exe, the load functions will load it directly from the memory for you.



yes, it's that easy !



Step #2 - Load Image Resources (BMP/PNG/TGA)

The second step is up and running also. So if you bind a PNG,BMP,TGA image those images can be loaded directly from the resource buffer in memory. They never have to be extracted to disc.



Read PlayBASiC V1.640 Blog for downloads, pics, and more code tidbits..

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 1st Aug 2013 02:56 Edited at: 1st Aug 2013 03:04
Round


PlayBASIC V1.64O Beta 18 - Optimizing (July 25, 2013)

While tuning up the run time builders, noticed some chunks of code that could be better represented, by changing the input format slightly. The idea seemed simple initially but took most of yesterday to get working again, all because of one command declaration that was in a slightly different format.. Grrr.. But it's working now. The benefit is we remove a slab of function calls from the start up and some dead functions. Those changes and the updated pre-processor shaved another [red]28K[/red] off the retail runtime. So V1.64O is currently the same size as V164N3 (at the moment)

The plan this afternoon is to continue testing Beta18 and pick though some of the legacy commands looking for any easy performance gains. So far it all seems stable, but ya never know. Tempted to add some detection operations to the compiler for finding merge friendly operations, just to see what combinations are prevalent enough to make any difference. My suspicion is that it would be beneficial, but really depends on the type of code people use.. So one routine might benefit, where as others won't at all...


Edit#1 - Sprite Locals

While picking though the Sprite commands and noticed the sprites local commands have legacy string support in them. The following test just times the average cost of writing each data type to a sprites local data. The simple stuff like Bytes/Integers etc are about the same, but the string writing around 150 milliseconds faster.. So this code in the V164O beta 17 runs 5 FPS, while in today's build it's 35fps. Suspect there's a little more room in it also...






Edit#2 - Sprite Locals Peek/Poke

Have completely ripped all the legacy string thrashing from those commands and now it takes about 2->3 milliseconds to read/write 10000 strings into a sprites local bank. The picture bellow is from this second version of the test..

Code snippet removed for size. see blog


PlayBASIC V1.64O Beta 19 - Tweaks + Tuning (July 29, 2013)

Been picking through the command set while updating the doc's and making the odd fine tune where possible. While picking through the History noticed there's some commands like GetSpriteRect that weren't in the docs. This particular command was actually returning the wrong data, so it makes sense to address it there and then.

Spent most of yesterday looking for more 'speed up' conditions the compiler can silently make for you without much success. The problem with such opt's is they have to operate identically, otherwise there's a risk some legacy code will break. Have added a few more error traps to pick up when LABEL names within expressions. Since when those things occur it'd just pop a generic Syntax Error. They need to be a little bit more verbose..


Here's an example..



This is pop's an error on the Wombat = 45 line ,telling us the that Wombat is previously been defined as a label.


EDIT #1 - Popping error between operators and labels




It's still possible to sneak one through that, since the expression resolver tends to per-compute stuff out as literals, eg... [red]Restore Wombat[/red]



EDIT #2

Added more traps on label names in structural statements. Should catch a fair degree of those.. While looking at errors, have customized the command input parameter error statement, so it gives the parameter index in the error message as well what data type it's expecting.

eg
PositionSprite 1,x

Would spit out

'PositionSprite' commands parameter #3 is incorrect, expecting 'Integer / Float' Data type


Another one that you're probably hit upon at some point is a Expecting Equals error. Theres a number of situations where the pattern matching is expecting some user variable be followed by an equals token. But you'd even get this generic error when you forget to add some expression..


eg.

MyVariable

Now spits out something along the lines of

Expecting equals in assignment Eg. 'MyVariable = Expression'




PlayBASIC V1.64O Beta 19 - Prototype Pass #1 Tweaks (July,31,2013)

This is bit of an oldie but a goodie that's a by-product the skimming nature of the Pass#1 in the compiler. During Pass#1 in the compiler is only looking for Labels and function headers. These are needed up front so we can work out what input / return parameters they have before building code. This is what allows a function or label call to be used before the compiler has actually got to that chunk of code. Otherwise you wouldn't be able to, you'd have to provide a list of prototypes. Which can be rather tedious..

Eg,


Since the compiler works top to bottom, when it finds the HelloWorld keyword it searches the Function stack for the parameter pattern to match the expression with. if we didn't this we'd need to previously provide a prototype for function call.

The problem with Pass#1 is that it's a dumb skim over the input code at text level. So it's didn't look at the line of code, it just searched for the function/sub headers. All of the real parsing occurs during pass#2 in the compiler, since everything is tokenized on demand. This creates a bit of catch 22 with things like remarks blocks. Since the pass#2 is the only remark aware pass, if you wrap a remark pair around a function, then the remarks won't have any effect. Pass #1 would see the function header/footers, so the function prototype would still exist. Creating a bit of WTF moment.

This afternoon i've been updating the Pass#1 to at least skim for remark blocks. So far it only supports the /* to */ remark blocks, but RemStart to RemEnd


This example would parse differently in today's build than older versions. Today;s build skips the function block as when you try and compile this code, [red]TEST()[/red] is now undeclared and doesn't know what it do with so it errors..




#If / #EndIF Directives Limitations

Unfortunately, If you wrap an #IF / #ENDIF around a function /psub in the code and try to toggle this OF / ON in the code, then this won't work for the same reason. #IF statements and Constants etc only exist during PASS#2, the Test() function will be picked up during prototyped in PASS#1, but that slab of code gets skipped during pass two. This creates a situation where PB can't resolve the address if the call to the TEst() function.. So you get an error along those lines.




Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 9th Aug 2013 18:23 Edited at: 9th Aug 2013 18:24



PlayBASIC V1.64O _Retail Upgrade_ Now Available (9th, Aug, 2013)



The PlayBASIC V1.64O package includes updates of PlayBASIC Compiler, Release / Debug Runtimes, SLIBS, DOCS & IDE V1.17b.

The V1.64O upgrade was originally slated as quick bug fix, which then turned into a much longer project seeing a couple of major improvements and feature inclusions added. The most notable features with the Resource Binding/Loading Media From Memory, Palette Mapping support and queued blitimage threading modes.


*Resource binding - These features let programmers to attach media into the programs final EXE. The difference is that you can load this media directly from memory without extracting it to disc. Not all media types are currently supported, but you load BMP, TGA, PNG images as well as DLL files directly from memory (unique to PlayBASIC).

* Palette Mapping - These commands allow programmers to draw Palette Mapped styled effects ranging from colour cycling, flashing, raster bars, glenz vectors and many more with minimal effort..

* Threading / Multicore Support - The Blitimage command set now includes a queue based render system. The program can tell the library what effects to draw and then push this sequence off the main program onto a second CPU core.


There's really way too many changes to try and summarize it all down, As always, we recommend reading through the V1.64O WORK IN PROGRESS GALLERY for more insight in all the new additions.


Don't have a retail version of PlayBASIC ? and want to find out more information about PlayBASIC programming > please visit the www.PlayBasic.com and download the free learning edition.



Download

Download PlayBASIC V1.64O Retail Announcement (9th,Aug,2013)

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 23rd Aug 2013 10:15
PlayBASIC To DLL (Research Project)

With the completion of the V1.64O retail upgrade, our focus has shifted onto the way forward for the classic PlayBASIC platform. There's a number of options with equally as many positives and negatives. One of those options was building a translation tool to convert PlayBASIC byte code to machine code in the form of a DLL.

Such a tool would mean playbasic programmers can create plug in's that perform any brute force processing required. The programmer could then bind (and execute) the DLL functions directly from PB resource in memory, never needing to be extracted them to disc.

The project is already up and running, you can following it's development on our forums. You'll need to sign up & log in to do so. Bellow is a few snippets from the blog.



Work In Progress (The Dissassembler Short Cut)

This idea has been floating around for a while now, there's a number of older tech demos that convert VM instructions into x86 machine code at run time (JIT). The process is relatively easy, it's just those programs are doing it manually (in PB code). So it'd read the byte code and drop out an equivalent x86 code into some executable memory, then call when done as a viability test. In principal you could do the same to export binary as dll, but an easier option would be to spit out raw assembly an pass through an assembler and hey presto job done.

Ok, so what does this have to do with the dissassembly tool written last year ? - Well, that program runs through PB byte code and converts it back into a semi readable text file for internal debugging. The result isn't pretty, but the same raw process could be used to drop x86 assembly out in place of the command simulations. So it should be possible, Relatively easy actually to get it to output at least simple operational code. It'd probably required the user PLayBASIC code be set up in particular way, so the export and ID what functions are to be exported.

To export a function, the function name needs DLL_ like so,




The nice thing about using the classic PlayBASIC compiler as the initial stage of the process (apart from saving me lots of work), is that we get whatever benefits it applies to the input source in terms of optimizations. The output code wouldn't be comparable to hand written equivalent but obviously it'd give you some brute force muscle.

We've tested the above runtime running in PlayBASIC V1.64O compared to an equivalent assembly function and the classic VM takes about 90 milliseconds to poke (800*600) pixels and the Dll function takes about 3.5 milliseconds. So in this case, it's more like 25 times faster than the classic VM. Not only that, the compiled DLL is also 25% faster than one competitor and and 10 times faster than another.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 8th Oct 2013 05:56 Edited at: 8th Oct 2013 06:24
PlayBASIC To DLL (Convert PlayBASIC to Machine Code DLL's)

I've been quietly working on two sides of the tool in parallel, the translator tool, as well as making updates to PB so they can integrate side by side. It's pretty slow going as some things are added quicker than others, in particular commands that aren't actually commands in the legacy VM, which need to be wrapped in order to create an external interface. Most of the core operations work like this.

The current edition of the translator supports around 160->170 core commands as well as Integer, Float, String, Pointer , Arrays (int, float, string, used defined typed) and core logic. The great thing about using the PlayBASIC compiler to build the byte code, is the resulting code has lots of redundancies removed for us. Enabling us to export cleaner & faster executing assembly.

The assembly generation routines are a cross between cookie cutter and smart at this point. It all depends upon the operation, the current generator can only optimize output code for extension speed, when it notices friendly sequences. The excitingly thing about that, is the machine code is already routinely faster than our competitors.

Here's a few tidbits from the Work In Progress blogs.





PlayBASIC To Dll - One Stop Shop - (August 20, 2013)

It's 5:09am and the tool finally has it's first taste of automation. Previously, I had to compile the PlayBASIC code to byte code, then copy the output object code to the test folder, run the convertor, cut'n'paste the resulting source code fragments into the dll template, assemble and repeat.. over and over... Which gets old real fast ! So the goal tonight has been to get the program to a point where I can point it at a PlayBASIC source file (PBA file) and it'll call the compiler, build the byte code and do all the conversions itself. There's a couple of hacks for the time being, since the compiler will need a specific mode for this stuff, But all in all, it's pretty painless.

The build speed is pretty good, even though the test sources are very simple, the actual conversion and assembly stage is consistently executing in about 50 milliseconds (or less). The conversion engine only supports a single PlayBASIC source file at this time, it doesn't support includes (couldn't be bothered at this stage).. The resulting DLL will have all the code in the source in it, even if it's not in use. So ideally you'd use it to compile stand alone include files that perform some brute force task. The tool expects the functions you want to be made visible (exported) to have "DLL_" at the start of the function name. The tool rips this off, it's just a way to ID what functions you wish to export, without having to change the PlayBASIC compiler dramatically.

Once the DLL is built, you can not only link this to your application but bind it to be executed from memory (doesn't need to eb exported to disc). Size wise the resulting DLL's are pretty small, weighing in about 5K for the output of the following.








PlayBASIC To Dll - Fractals anyone ? - ( August 24, 2013)

Attached you'll find a nice shiny example of what the DLL convertor is able to do today in the form of fractal render. I've modified the code slightly to work around a few functions that aren't supported as yet (the original code is in the source code board), but the result is a pleasant 18->19 times the performance improvement. The demo is drawing 640*480 pixels with potentially 200 square roots per pixel, So yeah.... that's a lot of work for the runtime to try and brute force.

The convertor currently has a hand full of opt's it can make when translating / exporting code, but it has no real awareness of the register management, so it's hitting the variable heap a lot more than what it should (extra memory accesses). Even without bust a gut though, it's working pretty well for now on my 8 year old athlon system..


Made a few register tweaks before tea tonight, then ported the bench mark code to a couple of competitors. No surprises the PlayBASIC version runs 2.8 times faster than one, 4.5 times faster than the other... Oh dear, how embarrassing that must be for them...




PlayBASIC To Dll - Strings -(August 28, 2013)

Been working on getting strings working most of the afternoon only to run into a strange crash when two string are added together. On inspection one string was legal (Hello World) and the other was null for some reason.. It's funny how your mind focuses in on a segment of code of you think is the problem, only to find the issue a few lines above it. Turned out the function initialization was killing the string buffer. Once corrected, it worked as expected.

So far I've only got a handful of core operators hooked up, it's just a matter of joining the dots. Something that is interesting is the when we start talking string management, we actually get less benefit from translating to machine code than what you might think. Unfortunately there's this idea out there that machine language is the golden bullet, but really it isn't and never was. In regards to strings.. well, there's no 'instruction set' in your CPU for doing string operations, everything is just reading and writing arrays of bytes. So if we add two big strings together, then regardless of how this operation was called, be it from machine code or the VM, the string joining operation is taking 99.9999% of execution time here.

Strings are a notorious bottle neck of programming languages, knowing that, the PlayBASIC string engine is a very optimal solution, every effort has gone into making it as quick as possible. That's why I write string processing apps in it..

Here's a little something to bend your reality.. The function joins Hello World together 101 times. There's two version of the test, exported DLL version and the VM function. Bellow is clip from the current test code. So basically PB2DLL is pointed at this source and we get a nice shiny dll version couple of seconds later..




Yep, obviously the machine code version is going to be faster, but the interesting thing is by how much ? It'd be easy to assume it's going to be 5->10 times faster, where it's actually only around 35% faster.. Why, because it's spending most of it's time copying characters, not executing VM opcodes.

To put that in some real world perspective, DLL version is 12 times faster than one competitor and 18 times faster than another.



PlayBASIC To Dll - Typed Pointers Array Fields - (September 01, 2013)

Getting this working has been a detoured chain of events. The first problem was the disassembler didn't support most of the pointer opcodes I needed, so the first port of call was adding that functionality, just so I could translate it back into assembly. After adding the decoder, it's here we notice there extra additions in the array field writes with type pointers. Could have ignored this, as an extra opcode in output might not like sound like much waste here and there, but if that code is sitting in some brute force loop, then it's throwing away 1/3 of the operations performance for nothing. This wastefulness would then be translated to the machine code DLL too, so it'd just be extra unwanted wasted cycles having a negative impact when the code is within a loop.

The only way to solve such problems is fire up PlayBASIC and take a look at what it generates in particular situations. For some reason it was adding the structure displacement offset to the temp pointer register, adding the array offset, then doing the write. When all it should need is the adding the array offset, then do the write since the writes opcodes support displacement. So the offset is virtually free. Moreover, it didn't support literal array indexes, which can be pre-computed at compile time and represented as one displacement. So that was yesterdays little chore. The results are as expected as it's 30% quicker. I suspect there might be a few more situations like that hidden away also.

Tonight's mini session has been all about hooking up the assembly generation side, which gone relatively well really. Meaning it can produce working code from the following.



So it's getting to pretty familiar level of functionality, there's of course plenty of no no's just waiting to trap you. The main one that comes to mind, would be the lack of auto casting in some translated operations. Like if you have a FOR/NEXT loop, then it currently only supports Integer loop counters, where in PB you can have Integer or floating point loop counters. Same with things like function parameters. If a function expects an integer and you pass it a float, you can get away with this in PB, since the runtime is recasting the parameter on demand, but the translator tool doesn't currently support this. It will.. just not today.



(To get the most up-to-date info, you'll have to read the complete blog on our forums)

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 17th Oct 2013 06:51 Edited at: 17th Oct 2013 06:53
PlayBASIC V1.64P Beta10 Download

Surprise Surprise, BETA 10 of V1.64P upgrade is online and ready for testing. This beta has a lot of internal changes range from improved code generation and the internal command set interfaces which will be used for building external DLL's with PlayBASIC To Dll. So we're keen to find out what if anything has been broken during the update process NOT after.

Read more: PlayBASIC V1.64P Work In Progress

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 23rd Nov 2013 03:59 Edited at: 23rd Nov 2013 04:28
Some more excerpts from the PLayBASIC 2 DLL blog


PlayBASIC To Dll - Integer / Float & String Arrays Supported ( September 05, 2013)

This evening sees the translator support Variables (int, float & string), Pointers (all byte,word,int,float,string, UDT) and classic arrays in the form of Integer, float and Strings. These give it a much more complete feel, while still being pretty fresh. Since there's a lot of array related commands that it's got no clue about.

The Array library functions will have to re-written and split up to obtain complete support. Not really a bad thing since I can borrow the newer methods out of VM3 project code base, it won't mind - The problem with the VM command set it they're attached to the byte code. So to call VM DIM for example id' have to create a Byte code program write the tokens and call the VM... Which defeats the purpose entirely.

It's importantly to the keep everything using the same system so data can pass in and out of the external dll's without conversion. Another feature added tonight is the ability to return array handles. For example in the snippet bellow the function creates an array of our required size, fills it with some strings and then returns the handle of this array.





So when the above function is compiled to DLL, the caller code would then write this handle into an array and hey presto you've got performing the brute force work in the machine code.



So test is full of strings like this Hello World1Hello World1

This is really how this product will used, you'll first design your set of functions to be self contained, then you're when happy this them convert the entire slab of code to a DLL and replace it in your project with a the DLL version. It's unlikely you'll be able to access the main PlayBASIC command sets, so if you're expecting to able to move everything then forget it. It's not really worth it anyway.

If you have a chunk of code like this, then you'll gain absolutely nothing from translating it, since it's already 99.9999999999% machine code in the first place.




However if you store you image data in some 'funky' format, then that's the perfect scenario, you can write the loader to wrap around the image and simply the dll function to do the brute force work. Which is the how the fractal render example works.






PlayBASIC To Dll - Just Stewing (September 09, 2013,)


Came up with a solution for the TYPE ownership problem a few days back, the solution just replaces the internal bank indexes with pointers to field declaration structures. It's currently not fully implemented, but now it's just a matter of going through and converting the other operations that expect indexes over. What this change means is that either side can allocate the type and the pointer will direct the Free Type operation to the structure of how the fields are set out. The field can structure can be anywhere and isn't local to either side, which was the problem.

Another problem on the horizon is getting the around the 'attached' nature of the VM to the command sets, by that I mean coming up with some "easy" way to share at least part of the core command set with the externalized DLL's. One way is to pull a list of pointers from core functions, but often functions are hooked into the VM instruction set. Decoding the opcode and performing some action. Really not in the mood to rewrite slabs of it.. So still stewing on that one for the time being.

Apart from the regular grind, i've been having a sticky nose at how some other Basic's and some lower level languages implement certain things, namely arrays. While we already have array core up and running, which is easily ten times (or more) faster than the VM, but since arrays are dynamically created and managed in PB there's always some fixed overhead working out where in the memory the array is. Some BASIC's take their implementation from C and treat the array as a static structure. Allowing the base pointer of the data to be static, makes writing to the structure as simple as (pseudo code) mov [arraypointer+fieldoffset*4],Data

We can't match that with dynamic arrays, but you can if you think latterly and use typed pointers, you can almost recreate this in the exported DLL. What you do is pass the array in, get a pointer to the cell zero, then direct a local pointer to it.

Example,




If you exported the AddToArray function to your work horse dll, then you'll come close to doubling the performance, since the Pointer looking at the array data is computed once for the entire array and not every time like it would be if you just accessed the passed version (me())



PlayBASIC To Dll - Calling Internal Commands (September 23, 2013)

The easy intergration of the two sides has been something of a thorn in my side the last few weeks now. The sad reality is, it's just not easy, it's more a tedious. Building an interface to get function pointers is one thing, but the pointers aren't much use without knowing the input/output parameters of the functions. In fact they're useless There's a few ways around this, but it seems the easiest solution would be to build a standard interface with parameter table contained in the VM. Otherwise if you wrote a bunch of libraries there'd be command set declaration data in each of the lib's. The libs are pretty lean currently, like a 120K assembly of source code, assembles to about 32->36K (inside a second on my 8 year old system). Size wise that's peanuts, but really that's only possible if they share internals.

After much consideration over the last few sessions, decided to move ahead with the centralized solution. This morning we're seeing the first glimpse of life starting with the Bank commands. The bank commands are one of the smaller command sets, but they're used often, so they make an ideal starting platform. It's currently possible to call PB commands externals ie.. 'NewBank' ' CreateBank'. Since these are internal functions and not duplicates in the externalized DLL, the bank you create is visible in your PB code also. Some things are going to have to be tweaked in the command sets (or even wrapped) to give us a good selection of the commands, but it does seem like this hurdle may well just be no more.

This is example of what i'm talking about.



If the DLL version of your code had duplicates for the Bank commands, then the returned index in ThisBank would be useless when returned back in your main PB program. However this way, the function (DLL_MakeBank) is just like calling any other PB function, except it's now machine code. One problem is going to be trapped errors when calling internal functions. If the Dll function trips an error code, then your Main PB program will stop, but the position it stops at will be anybodies guess at this point. Since line numbers don't exist in the external DLL..




PlayBASIC To Dll - RGB Command Sets & Colour Crunching (October 01, 2013)

One of common misconceptions about compiling to assembly/machine code is that it's an instance fix for all things speed related. Which is.. well, a load of rubbish.. Unfortunately though this is something I hear routinely, often made by people who've never written a line of assembly in their lives. It's a shame programming has moved away from traditional fundamentals into this land of bloat ware and high level abstractions, but it is it what it is.

For example, examine the following block of code. Here we've using the RGB function (in this case) to pack some fields into a 32bit integer. All the RGB function is doing it masking and shifting the fields into place. The temptation is this to dump out some assembly that calls the RGB function from the colour library, which after a little dissassembly of some our competitors that's exactly what they do.



If the RGB function is a STDCALL, then to just call the function you've probably dropping 7->8 instructions to perform the call and return the result. These instructions don't do anything just change execution to the chunk of code you want, the RGB function. There's nothing wrong with it, it works and is fast to implement and will run much quicker than the VM, but it's not actually anywhere near optimal for this operation.

You can see this by just placing the instruction inside a loop.



If we give this function a span of memory where it has to loop 10,000 or 1,00,000 or 1,000,000 times. Then we're effectively multiplying the RGB call cost by the work done, so the ratio between what it does and how many steps it takes to perform that task is still rather large. One of the benefits of macro languages like C/C++ is they tend to inline everything that can and solve out redundancies where possible. If you look at the RGB call above. Then this operation has two redundant fields in it, we'd otherwise be passing these into our RGB function possibly millions of times for nothing.

This is the type of situation where I want to draw a clear definition between the code PlayBASIC 2 DLL produces and the easy option. The easy option is to just call the function from the colour library, the more challenging option is to try and catch such situations where it simplify such statements for you. For functions like RGBA(), RGBG(), RGBG(), RGBB() where you want to grab a field from the packed colour. Then it's a no brainier that it's going to be quicker to grab the colour value and shift and mask it on demand, rather than calling the function.

The following four lines of rather pointless demo code, output only 11 lines of assembly, which is about 30% of the code produced when it calls the actual functions.



We can apply this to RGB() statements also, so it'd look at the statement for any easy tweaks, if none are available it'll just call the library. If a field is literal, then it makes sense to avoid calling the function and pack the data directly.


Edit #1

Had to go shopping, so just getting around to dropping some parsing code in the translator and the results speak for themselves. If you compare calling the RGB operation with an in-lined version, the in lined version is almost twice as quick over 1,000,000 calls. On my old Athlon system calling the GreenFillMemory function above takes around 13 milliseconds to execute and around 7 milliseconds when it's inlined/opted which also produces smaller code.


Related Links

There are just a few tiny snippets from the private development blog, now it's entering the final stages we've set up a public one with some general FAQ at the start.

PLayBASIC To DLL Development Blog & FAQ

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 2nd Jan 2014 04:45 Edited at: 2nd Jan 2014 04:58
The following is another selection of blog posts from the development of PlayBASIC to DLL over the past months,



[large]PlayBASIC To Dll - ARGB & RGB Colour Crunching[/large] (October 02, 2013)

Rounded up the RGB + ARGB colour crunching routines after lunch. The generators looks at the parameters and tries to pre-compute any literal fields. This can make some calls reduce to as little as 3 instructions. About a 1/3 of what it costs the call a function let alone, perform it's action.

A function call like this Result=ARGB(255,255,G,255) would normally pass 4 fields onto the stack, call the ARGB function and then handle the result. But now it's solved in only 4 opcodes, assuming the parameters are integer to begin with, if not, they have to be recast prior.

Tested all the combinations and they're all faster inline. The following tidbit output less than 1/2 the code that it would have originally




Tested both methods against some competitors and it's 5.5 faster at one and 2.2 times faster than another. But I knew that already



[large]PlayBASIC To Dll - BASIC 2D Graphics Commands[/large] (October 04, 2013)

The weather has finally started to fine up, so i've been out doing such useful things as collecting fire wood and about training for the 100K challenge event. It's a shame my bike keeps breaking down. But between the day to day grind I've been dropping in some more command sets. The first was the intersection library. Which includes like things LinesIntersect and various other vector operations. While doing that found the translator would attempt to cast integers to floats in code, where the function wanted float parameters.

So an expression like Status=LinesIntersect(00,0,800,600,800,0,0,600), where all the parameters are literal integers, would spit out a int->float cast for each parameter, then push the temp floats onto the stack. Which is pretty wasteful when you consider the translator could do this operation at compile time, rather than making code that does it at run time. Trapping such situations just make it produce cleaner and ultimately faster code.

Been dropping in 2D graphics commands at lunch time today, these are primitive stuff line DOT, LINE, BOX etc. These commands are VM bound, which means they're actually part of the VM instruction set, rather than being nice easy functions I can point at. So to call them, I've got to write wrappers for most if not all of them, but the up side is that once the wrapping is done, calling from the command from external DLL is same as calling from the VM.

Here's a test code snippet. Running the VM as is (purely on the VM) gives about a 19->20 fps return on my Athlon system. If i compile the function DLL_FASTDOTFLL it runs about 55fps. Which is not too bad when you consider the equivalent loop in C/C++ is only 5 fps faster.




Of course the demo is pretty silly, we're basically calling a function 100's of thousands of times to draws strips of colour. It's more more efficient to lock the buffer and poke 32bits into it directly.

To really drive the point home, you do the same thing with BOXC and the VM version already run 230->250 FPS. Converting that DLL would give us some more FPS, but not a lot.






[large]PlayBASIC To Dll - File Streams, Surfaces & Trig Commands Go online[/large] (October 10, 2013)

The last day or so has been pretty productive, solved the issues with the PRINT/TEXT statements from the translator, added INK/PEN command set wrappers, added surface control command set (stuff like RenderToImage, lockbuffer etc), added Trig function library (cos/sin stuff) and finally got to the File Stream command set. I'm not sure of the exact number but that brings us it up to about 190->200 commands. I've tested the big stuff, but not
every single command, so there's bound to be few teething problems.

The file stream commands cover the sequential and random access file handling, and are another one of those counter intuitive areas where we see a number of programming myths come to the forefront. Disc access is an aspect of programming where as hardware improves so do the assumptions. People seem to forget that accessing data on optical devices imposes a seek time. Every time you grab a chunk of data, part (if not most) of that
access time is just in waiting for the device to find it.

Take a look at this code fragment. The function is literally spooling a file into memory bank byte by byte. Every time to read a byte, this action also includes a seek. If you test it with small file the function would performs OK, but the bigger the file the more seek overhead and slowing loading becomes.




Compiling the loop to machine code would make it faster right ? Nope, and when you test the two versions side by side, when loading the same file [color=green]100 times[/b], the machine code version takes [color=green]2100 milliseconds[/b], compared to [color=green]2245 milliseconds [/b]on the VM. Why?... Well because the speed of the for/next loop isn't the issue here. It's simply that every byte you fetch, has a seek time form the spinning hard drive. So the bottle neck is the optical device (the hard drive).

The solution is to change the size of the blocks you read from the disc. Reading Integers is effectively 4 times faster than reading bytes. Since there's only 1 seek per every 4 bytes. So it makes sense to that bigger the block you fetch, the more data will be burst from the device into memory in one hit. Hence why we have ReadMemory command. This command fetches a couple of K per hit (from memory) at a time which virtually negates the seek time impact upon the overall load time.

This version of the function can load the same file 100 times in 22 milliseconds.






[large]PlayBASIC To Dll - Maps[/large] (October 28, 2013)

It's been a busy few days in the real world but the mapping support is slowly getting done. Needed to rearrange the command blocking a little to make it easier to export. Which kind of feels like groundhog day but the end result is easier to deal with. Was necessary as some of the query functions had been blocked into a general set. Now mapping commands with a common purpose are in sub groups.

In exported programs, the actual DLL only initializes the command sets you use. So if you don't use banks say, then your dll doesn't need those hooks. With stuff like maps which are broken into 4 sub command sets, the same benefit applies. If you don't use collision or occlusion commands, then the exported dll doesn't need that block of hooks. Which helps make the exported libraries smaller. Previously if you used any map command the entire command set table would be exported. Just extra bloat for nothing.

Size wise the DLL are pretty small. In the current test project which is around 1800 lines for example and produces an 80K of byte code file. When we exporting this to assembly, it creates about 250K of assembly which assembles down to about 50K of DLL size. The DLL's don't need the data tables etc in them, so they should generally be smaller than the total byte code. The byte code segment is only about 10K though. The code distribution pretty similar, some stuff is smaller in byte code where others end up smaller in machine code. So it'll balance out.



[large]PlayBASIC To Dll - Select Statement Support[/large] (November 19, 2013)

Unlike most of the control statements Select/Case statement blocks are only now being implemented in the translator. PlayBASIC's support for Select Cases is rather unique. Generally select/switch statement blocks are explicitly literal, so cases must be constant. Which helps the compiler back end when producing the appropriate logic. The assumption most programmers make, is their low level compiler is building a jump table, but this actually isn't true. Visual C/C++ produces a range of different solutions for switch statements, as the input data has to fit the jump table model, even then, it often doesn't .

In the most BASIC languages (PB included) select/case block are generally nothing more elaborate IF/EndIF structures. The benefit being the 'select variable' can be cached on register unlike a block of user IF/Then statements. The caching removes most of memory access when falling through the structure. Converting this logic directly to assembly is a cake walk and will certainly perform very well. Hey, if it's good enough for visual C it's good enough for us

However, the PlayBASIC compiler & runtime times support Variables, expressions, floats and even strings in case statements. So you can really mix up the anything you like into them (within reason). This means the translator has to look ahead at any following case statements when producing code to try and suss out if a register can be cached or not. The current version doesn't bother with that for a minute, just wanted to get the structure up and running first.

Here's the current working test code, in this example we're building an Integer select on the A Variable with literal case matching.



The translator already includes generation logic to pre-flips literal case terms and sorts + recast case rows into order. So in the case 10,6,7,8,9 line, the literals are pre-sorted. If the set has more than 3 values it inserts a bounds check for you. So if the select variable is outside the range, it moves on without falling through this block of compares.


Edit #1:

Working on dynamic versions of the select statement block. The dynamic versions gives the exporter support for variables in case statements. The current focus is getting a good range detection without branching. Have worked out way, but it's behavior is different from the VM. The VM treats the range as inclusive ->inclusive, but the assembly version is currently inclusive->exclusive. Which might not sound like a big deal but the behavior then is different between the two.







[large]PlayBASIC To Dll - Flex Rotater[/large] (December 03, 2013)

This is something of a tech demo, much like the fractal render from a while back. The idea here being to take a bigger chunk of code (in this case an old Amiga demo effect) and and build that into a DLL library. The routine is an old z rotater styled effect where the spans are rotated and the result is drawn pixel by pixel to the screen. The render loop is interpolating 800*600 pixels and drawing them via the awfuly generic FastDot. Moveover the texture fetch is just reading a 2D integer array. So the code was slapped together out to work, rather than be fast.


Running the original routine on the PB VM returns about a 500 plus millisecond refresh on my 8 year old (single core) Athlon system. The first conversion to DLL cuts that to about 61-62 milliseconds. I suspect that once the array reading and fastdot are removed, it'll be possible to at least double possible triple that. The only trouble at the moment is the machine code version isn't acting the same the VM version. So there's some math operation not behaving the same way.

Bellow is an example of what the original effect running on the Amiga looks like.



Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 30th Jan 2014 05:54 Edited at: 30th Jan 2014 06:11
PlayBASIC V1.64P Upgrade blog (Nov 2013 -> Jan 2014)


PlayBASIC V1.64P Beta 17 - Operators and Precedence (November 28, 2013)

It shouldn't come as any great surprise that changes in this revision are to help with the PlayBASIC To DLL translator and it's ability to produce cleaner and better code generally. Building the translator means taking a fine tooth comb to the generated byte code and making sure the compiler outputs the leanest set of instructions possible to perform each operation. While looking over the bit wise/logical operators recently noticed they'd not been set to force an integer result. So if you had an expression fragment that contained a float, it'd return a float in the legacy VM. Fortunately this is simple to rectify as the parser has a switch to force an integer return. This is a bit of a none issue in most programs since the operators are generally only ever see integers on either side of them, but it can and does actually occur in some code.

Having operators return a float isn't an earth shattering problem, it just makes a simple operation messy and slower than need be when such things occur. The existing translator already supports operations upon integers, but it didn't handle mixtures between integers and floats. Forcing the compiler to produce an integer only result, helps the VM keep everything integer and translator produce cleaner machine code from the byte code without making multiple path ways.

While looking at the operators again in the compiler and translator, noticed that NOT seems have a precedence issue. For some reason it's evaluated before the math/compare operators in an expression. It should be the same as the other operators and occur after them.

So code like print not a=b is being evaluated Not A then result_of_Not_A=b , where it should do A=b then do Not Result_Of_Compare_A_With_B.

Moreover Not is implemented as a mask and Xor of the bit zero value on the right hand side. After a rethink, it'd be better implemented as a equality test with zero then xor. So code like Not 45, would actually be evaluated as Not (44<>0), Not 1 = 0. Where it'd currently return 1 since bit zero is OFF, it'll flip that off bit to on.

The bad thing about the NOT operator is that your introducing complexity into your expression for no real benefit, as result = (Not A = B) and Result = (A <>B) are functionality the same, expect the former is two operations and the latter is single operation. No big deal in a loop of 1 to 100, but a very big deal when iteration count blows out in both VM and Machine Code.

Live Edit #1:

Found the problem, the operator trap in the parser was missing the operation level screen. Once added, the order solves as expected.




Live Edit #2: Optimizer

The expression solver now contains logic to optimize & rewrite code fragments like the following. The optimizer also supports literal Not statements as well so that be evaluated in constant expressions.



Becomes,




So since we're saving an opcode every time, the object code is smaller and of course faster in both the VM and machine code. No brainier really.




PlayBASIC V1.64P Beta 18 - Shift Operators (December 02, 2013)

Just dropping in some more C styled operators with the << and >> which are equivalent of the existing LSR32 and LSL32 functions (which is what they are in assembly btw). Just makes dragging code between dialects that little bit simpler. There's no functionality difference between them.





PlayBASIC V1.64P Beta 18 - Updated Operator Codes For IDE ( December 02, 2013)

The compiler and IDE talk to each other when you first start PlayBASIC up. The discussion is pretty one sided, but the IDE basically asks PB for it's current internal command, constant and operator list.


The forum board syntax highlighter currently doesn't support them all as you can currently see in this snippet. (This statement will be obsolette when i next update the board, so they'll appear as per the IDE..)





Edit #1 - Not Equal Operator

It appears PB parser would only ID one version of the Not Equal operator, that being <> , where as the inverse also represents an inequality, but the parser didn't pick it up. With a quick tinker and it now supports >< also.


Live Edit #2 - C styled not Not Equal Operator

Dropped in the != this in for those who can't live without their c fix.. Same as <>and ><




PlayBASIC V1.64P Beta 19/20 (January 16, 2014)

The V1.64P revision has only had a few changes late last year, none of them were really user related (just the odd tweak for how PB and PlayBASIC 2 DLL communicate) , although tonight i've been looking over the GetArray and SetArray commands. These are a way a user can access arrays through a stub array. So if you create a stub (and array with no data of it's own) via MakeArray, you can copy the 'pointer/buffer' from any other array into the stub via getting that arrays index via GetArray and then setting it via SetArray. The array indexes are linear, so you can actually write code that runs through a block of similar arrays without having to pass them. The process is basically the same as how and array is passed into a function or psub anyway..





While working on PlayBASIC to DLL noticed the GetARRAY function can be pre computed at compile time, currently there's actually a library function that preforms the rather trivial calculation at runtime. SetArray is little more difficult as it's scope aware, which is fine for the VM environment, but a little difficult in the translated version, without piping all the compile time data into the DLL, which i'm not too keen on. Originally the SetArray would let you set and get between any array regardless of scope. I forget why it was changed, but have a feeling that could well get reverted at some point in the near future




PlayBASIC V1.64P Beta 21 - Data updates (January 27, 2014)

While building the data support into PlayBASIC to DLL, found a few issues with the VM data commands. The main one was the Restore command would allow a negative data pointer position. Which would pull data from off the table, since it's not clamped. Another change is the FindDATA function had some dead weight in it and was missing the ability to search from the current DataPointer position. This has been added as a switch in the position field, if you set the field to a -1 the command will use the existing data pointer to search from. The position field is optional now and defaults to -1






Test Code.



Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 26th Feb 2014 05:37



PlayBASIC V1.64O2 _Runtime Upgrade_ Now Available (26th, Feb, 2014)



The PlayBASIC V1.64O2 package includes updates of PlayBASIC Release / Debug Runtimes.

The V1.64O2 package includes updates the V1.64O runtimes, as they (and I suspect older version) could run into alignment issues when loading some compiled programs byte code. So if you have issues with your program loading, then try these.


There's really way too many changes to try and summarize it all down, As always, we recommend reading through the V1.64O WORK IN PROGRESS GALLERY for more insight in all the new additions.


Don't have a retail version of PlayBASIC ? and want to find out more information about PlayBASIC programming > please visit the www.PlayBasic.com and download the free learning edition.



Download

Download PlayBASIC V1.64O2 Retail Announcement (26th,Feb,2014)

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 8th Mar 2014 06:12
PlayBASIC V1.64P BETA #27 (Retail Compiler Only Beta) - (Avail for Registered Users ONLY) (8th,Mar,2014)

Beta #27 continues with it's PlayBASIC to DLL alignment so many changes related around cleaning up byte code / fixes mainly. But that's not the only thing this build includes, Beta 27 is first built to share the DLL interface with the internal VM. This means the VM and now call some commands directly without having to solve them. So far only the BANKS, SOUND and MUSIC sets have been converted to this form, but the idea is to convert as many of possible.

The advantage of sharing the DLL interface with the VM, is we not only get faster function calling from the VM but it helps reduce the size of the VM. Just converting 3 mid sized command sets pulls 16->20K from the runtime. I suspect we'll be able to trim a lot more when some of the bigger sets are done. At this point I just want to be sure it's all working as expected.


Read V1.64P Update Work In Progress gallery

Gunslinger
15
Years of Service
User Offline
Joined: 29th May 2007
Location:
Posted: 8th Mar 2014 09:05
Just asking, do i get this right?
After around 7 years, it's from V1.50 to V1.64.
So, estimated time for V2.0 is around 2035?

(Please don't take my post serious )
Chris Tate
DBPro Master
14
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 8th Mar 2014 11:43
I was talking about this stuff with my work colleagues. These projects take ages. It took 200 people to get La Noire finished in 7 years! Be it 2D or 3D, any animated creation takes ages.

It is good to be sure to start a project you are likely to want to stick with for years. A project that will interest you not only today but in the future is the kind of project that succeeds.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 2nd Apr 2014 18:57 Edited at: 2nd Apr 2014 20:08
wow... sorry, didn't notice the replies until now ! (Yep.. I don't read my own thread

Yes, it's taking a while. If we'd bumped PB version number for each update, we'd be well beyond version 2 and heading towards 4.

If you follow the history of what we'll call 'PB Classic' here (to make the DB->DBPro or AGK/AGK2 analogy), you'll see it's been steadily updating 2->3 times a year for over a decade now and for free. Many 'extensions' others sell, we give away. (but this is about to change) There's more going than meets the eye though, there's also couple of versions of the PBFX strands, which was first released internally back in 2007. Over that time though, a lot of FX'ism and code base has merged into standard PB, diminishing the need.


@Chris Tate,

Sorry, I don't actually read my thread.. Wasn't existing anybody else to either

Quote: "
I was talking about this stuff with my work colleagues. These projects take ages. It took 200 people to get La Noire finished in 7 years! Be it 2D or 3D, any animated creation takes ages.

It is good to be sure to start a project you are likely to want to stick with for years. A project that will interest you not only today but in the future is the kind of project that succeeds.
"


Yeah, it's an uphill battle most of the time. Often for me, it's not the actual coding part that eats through the time like candy, it's all the extra stuff that needs doing on top of that.

Few months a ago, worked out i'd spent around a year a my life writing documentation.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 2nd Apr 2014 20:05
The following is a selection of blog posts from the development of PlayBASIC to DLL (Machine code translator) over the past months.




PlayBASIC To Dll - User Define Types - Addressing the same ground. (December 15, 2013)


I've been testing the translation tonight hunting down those little gotchas in some bare bones type demos. Found one such issue where PB would crash on exit when a typed array created inside the dll was returned to the VM. Initially was thinking it was caused due to some difference between the Undim process in the VM, which is called when the VM exists. This was a close guess, but it turned out to be just the order in which some of the vm libraries release themselves. VM would release any bound resources such as DLL's before it release the array memory. This is no problem when all the array data is created locally in the VM, but it's a big issue when the arrays point to structure definitions inside the DLL. Which would cause access violations and hence the crash.

Today's build gets us to a point where the DLL side and VM side can create a type and other side can release it. This allows the flow of data from VM to DLL and DLL to VM more freely. Even so, i'd still recommend at least trying to keep the data life cycle management to one side or the other, but it's workable if you can't wrap your head around that. If this all sounds familiar it's because a few months back I had to rewrite how the VM deals with it's local types. In the older editions the VM maintains one central list. Which is fine in a world where 3rd party code isn't expected to plug into the VM, but bound DLL's need to be able to control data much the same way the vm does. There's other solutions of course, but altering it to support host structures seems the most workable.


All this messing around means you can finally do this, bellow we have a function that we'd export to Dll and bellow that is some code that accepts the returned structure..




The dll code just creates a junk array and fills it with some data, before returning the array handle upon completion back to the caller, the VM side.



The VM side is just running through the contents of the array and dumping them to the screen. Not much of an example, but it's fairly seamless. There's a couple of oddity that appear in the debugger if you looked at the contents of the k() array in the VM after it was created externally, but those seem from the DLL's dimension instruction not classifying the arrays structure. So it doesn't know the original 'name' of the type.


Edit #1 - Debugger Name Resolving

One problem i'd been having with structures passed back from a DLL was that if you clicked on them in the debugger the structure names would be wrong, but that seems to be now fixed.. As if you got to variables and click on k() array, it drops the info into the panel as normal.





PlayBASIC To Dll - Redim - Life cycle of arrays & LIsts ( December 18, 2013)

Pretty much all PB's data structures uses the same interface, that being the array interface. Most of the legacy code i'm ok with, some of it i'm updating to bring in some of the newer ideas explored in it's descendants. Haven't had much time to really chomp through it, but what I have been slowly working up the core routine with their support code. One such replacement routine was the ReDIM function, which is a lot more complicated than the DIM operation.

Dim is fairly dumb, it just creates some empty (formatted) space for the array. There's a little more to it than that, but conceptually that's about it. ReDIM on the other had has to rebuild the structure when the sizes are different. If you make an array larger, then it allocates some new memory and copies the old data across, the same goes for shrinking an array. It copies the existing data,not forgetting to deletes any excess. Which can be an easy way to leak resources if you're not careful. See-> Tips On Safe Resource Management:

The other thing i've been working on is the back end support for the linked lists. A link list is an optional extension to the array structure, so when a list is present the address resolver uses the structures internal current position. So far it's in a very bare bones state, due to how interdependent some of the back end actually is when dealing with lists. For example the list iterators are set up, but insertions don't work because that requires insertion management of the arrays size, hence the need for ReDIM.

But here's a general idea of where it was at last night..



I think it should be smoother sailing once all the little inter-dependencies have been ironed out.


PlayBASIC To Dll - Macros make the world turn faster

The back end assembly is built with flat assembler which has some very impressive macro support, most of which is way beyond the scope of the project and my understanding of it . However, macro's can be a very powerful approach to programming, in particular in assembly, as they allows the user to dramatically reduce the lines of code, where a common behavior is wrapped as a macro and used throughout the project. It's a bit like a function, except the macros are parsed out by the assembler at compile time. So all the code is inlined.

So what i've been doing is moving more of the code in the base template into macros. You could pretty much make macro's for all the core operations, which would simplify PlayBASIC to Dll's job by hiding how something works away in the template. The current build of the generator is pretty reactionary, so when it sees some known byte code pattern it spits out the same' template assembly chunk in reply. Such things could well be macro'd, allowing improvement to be make to the template and not only via the translator.





PlayBASIC To Dll - Data Statements (January 22, 2014)


There's only a few tidbits left in the entire conversion felt to do and DATA statement support is one of them. Initially I wasn't going to bother with them, in favor of resource binding, but local data is probably easier for most people to wrap their heads around. There's a few gotcha's with exporting data to our DLL's though and that's visibility. In the VM data is global regardless of where it's defined, so the data heap exists outside of the program code. Now if you compile a PlayBASIC source to DLL that has Data in it, now the question is who should have access to that data.

It's viable that the VM could retrieve any local data from the a bound dll and pull it into the global pool on the VM side, but then the dll's have to use the VM access. Which is about as painful as it sounds ! - I think i'd much rather that any data be local to it's dll, which makes a lot more sense in terms of encapsulation anyway. If the dll's shared data with the main program this could affect the users code in the VM or even in other bound dll's. Making it a slippery slope.

The DATA command set was re-written for the VM3 prototypes, but after looking over the replacement code, there's a few areas where it could be further improved. I think some of those changes would have to be made at compiler level to really see the benefits, but one thing that comes to mind would be replacing how the string heap is set up. If you have a lot of string data in a program and those strings share fragments/words, then there's some lost opportunities in terms of compression.

Ie.

Data "Hello World", "Hello World2"

The above data fragment would store two strings, since they're not a perfect match. But what it could do at build time is compare smaller fragments with the raw pool. So any matching runs would compress away, both still exist, it's just one is a part of the other.

Stuff like this doesn't make any significant difference to small programs, but as programs get bigger the amount of partial collisions increases.







PlayBASIC To Dll - Function Calling Cont. (February 03, 2014)

Yesterday was get function calling done day, but things didn't turn out how I'd hoped. In fact spent almost 12 hours on it, only to run into brick wall after brick wall. Was initially going to wrap up some functionality in the library to handle an emulated VM stack, which I know would work and would be a perfectly fine solution, but it just doesn't sit well with me. My preferred solution is to use the system stack, but that's where we hit a few dramas... PB's internal calling convention has type casting wedged into it. This makes sense in the VM, since it reduces the number of VM ocodes needed to execute a new scope. But it means that any interface has to be data type aware when pushing or popping. Macros to the rescue !

The solution was to write a set of tiny macros that emulate the behavior of the VM in assembly. Since they're macros, the code is inlined into our exported dll code. So there's no calling a VM function to do the task for us. The up side is it's about as fast as you can get it, the down side is it adds a little extra bloat to the amount of code that's produced around a function and a function call, but personally it's not enough to worry about.

So far, the caller only supports passing primitives such as Integer/Floats & Strings in and out. Multiple returns (inside PB) are supported, but I haven't actually tested to see if they work correctly as yet. Pointer and Array passing isn't currently hooked up..

Some test code:




In this program we see two different types of functions. The first one DLL_ExportMe is our public / exported function. This function is visible to other programs that load this compiled DLL. Since it's a DLL function, it uses the STDCALL calling convention rules. The second function TestFunction (which is called from the first in this example) is a local function to the dll. This function is not visible externally and uses the PlayBASIC's calling convention internally, so even if you have a pointer to this code, it's not callable externally.



EDIT #1 - testing multiple returns from functions

This seemed like it initially would work, but there was problem with the translator using a previously defined instruction in an unexpected way. Once that that was picked up it seems to work OK.

Recursion isn't currently supported though.






PlayBASIC To Dll - Range / ClipRange functions (February 21, 2014)


Added support for the integer Range instruction yesterday, you can't mix types at this point. For expressions that pass Range a mix of floats, it's probably going to be best solved by calling a function (the call size is smaller than my inlined code). Which is something I've been trying to avoid for maths functions where practical.

The ClipRange functions are borderline also, I suspect there's situations where the output code could be simplified more than calling a one size fits all function , like where the range terms are literal. If that occurs, it could output a compare block in it's place. Which would be faster at runtime !

ie.

Result=ClipRange(SomeValue, 100,200)

could become something like this (in BASIC styled logic)

Register=SomeValue
if Register<100 then Register =100
if Register>200 then Register =200
Result =Register



I could just take the easy way and cookie cut everything, but there's plenty of BASIC compilers that already do that..




PlayBASIC To Dll - For/Next Problems

It's not all rosey though as one of the problems i've been running into recently, stems from the very nature of the byte code itself. The VM uses self modification in places, which is very handy for removing decisions from certain opcode pairs. But can make translating the logic back something of a pain.

For/Next loops are one of those areas, where the compiler produces all number of combinations at the start the loop sequence, but these opcodes are generic, there's nothing special about them and they appear in other constructs as well, so detecting exacting what's going on isn't turning out to be that easy.. The result of which, is sometimes the translator misses the FOR loop set up code, when that occurs the for/next doesn't work in the exported DLL. I'm hoping there will be some rainbow moment where I can figure out a bullet proof solution, but nothing coming to mind at this point.

There's a few solutions, worse case would be adding bread crumb opcodes to the byte code in order to help any translator(s) process down the line. Another approach would be change how the VM works and use add loop start opcode which would make it easier to translate and possibly at runtime for VM also. I kind of favoring the latter, but we'll see..





PlayBASIC To Dll - GUI - Up And Running ! (March 28, 2014)


After a frustrating week of playing hard drive roulette, yesterday i've continued putting all the parts together. The build process is broken into two halves, there's the GUI side and translation engine side. Unlike other helper apps, these are two separate projects, which are combined using a slightly more customized version of the project loader/source merger. This version of the tool lets me wedge the Engine into the GUI easily and rips unwanted blocks of the code from the engine. The pre-processing tool works via looking for tokens inside comments. So the tokens mean nothing to PB, but the pre-processor tool just skims through looking token pairs and rips anything between them.


Example,



So the code (in this case a function) is included when engine when testing separately, but would be ripped when the engine is merged for inclusion into the GUI. I often use such approach to substitute methods in the final source codes for example. Once the engine source is built, I just include it into the GUI project and off we go. The GUI side holds the high level properties and just calls the conversion functions from the engine. To make the engine project output to the GUI rather than the odd print statement we just wrap up our own print function. So depending upon the context, the wrapped function either uses the GUI console or a print to screen.


When running the final program the user has to initially locate their PlayBASIC.exe compiler file. This will generally be in your My Programs or Program Files folder it's different between OS's. I can sniff this out only when install in the most common location, but can't if you install to a custom location. To find the file go to 'SETTINGS->CONVERSION' and click the locate button. The dialog is set up to only accept one file. It's just matter of you finding it. Once the location has been set, it remembers this path.

To convert a PB source (PBA file) we go to [green]FILE->CONVERT FILE[/green], this pops a file locate dialog where all you do is select the file. PB2DLL will take over from there, there's is one little issue though and that's what your DLL name is going to be. Unfortunately, you can't create the DLL then rename it later, that won't work as there's naming dependency. So the name has to be given to the DLL up front. There's a few ways around the issue i guess, the DLL name could become the source files name. This is fine for those people that give their files meaningful names, but unfortunately a lot of people don't. One way around the problem would to be search through the compiled byte codes constant table for a constant called "PB2DLL_NAME$" or something similar, another would be to use a DLLNAME_ prefix in a function/psub declarations . Either way the programmer can then simply add the name string to their source code and then not have to enter a name during translation. Which would get old really fast if your constantly editing -> building -> repeat..

So anyway, it's up and running.. just matter of tweaking everything now.





PlayBASIC To Dll - Pricing & Back End ( April 01, 2014)


PB2LL is something of a first around here, up until now all the helper tools have been freebies, PB2DLL is a commercial tool. The initial price will be for the early adopter version ! Which will be similar to what PlayBASIC retail current costs, so somewhere between $20->$30 US. The final will most likely cost around $50.

One of the most frustrating things when releasing new software is all the secondary stuff that's required. It's not just the software you have to write, it's the ordering infrastructure it requires as well. Will probably go with shareit again, but contemplating just using PayPAL. Much like the DTAB ordering process, the KEY creation side of the ordering process won't be automated. You'll most likely order, then your registration will be sent to you once the order has processed locally. Which is required, since the key builder process is written in PB. Shouldn't be a drama really, as we're unlikely to see more than a few sales.

So today's little chore has been working on the activation system library which is just matter of dragging some of the code around from older libs, was a little painfully setting it up, but the process seems to be working. With other tools i've build pre-processing tools to do the entire process, which is tempting, but again can't really see any great demand..



More Information

For the full picture/ screen shots etc see the PlayBASIC 2 DLL development blog

Chris Tate
DBPro Master
14
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 8th Apr 2014 19:27
Quote: "Few months a ago, worked out i'd spent around a year a my life writing documentation."


Yes I am starting to feel burden already, and I am just getting started.

It is interesting that it seems to me that it takes very long to explain to a computer how to do something in a programming language; and yet its seems even more difficult to explain to a human being how to do something in your mother tongue.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 12th Apr 2014 18:58
@Chris,

Quote: "Yes I am starting to feel burden already, and I am just getting started. "


ahh, so documenting your engine then ? Good luck - It's a pretty thankless task, only stuff that you ever hear is how can such and such not be in the manual.. yarda yarda..


Quote: "It is interesting that it seems to me that it takes very long to explain to a computer how to do something in a programming language; and yet its seems even more difficult to explain to a human being how to do something in your mother tongue.
"


yeah, that's so true.




Regular Blog Stuff Follows

The following is from a selection of blog posts from the development of PlayBASIC to DLL (Machine code translator) over the past months.


PlayBASIC To Dll - Joining The Dots ( April 08, 2014)

The GUI side of the app is looking fairly complete now (apart from some tweaks), wrote (largely cut'n'pasted) a history dialog the other day from one of the other apps. Turned it up a bit, one of my pet peeves is programs that only remember the last couple of files you worked on. The current solution has no max size, but really it's unlikely you'll ever have more than a page or so loaded. There's a clean up button to remove files that no longer exist on disk. So when stuff get moved or delete or whatever.

The other remaining GUI tidbit that was left over, was the batch / folder scanning stuff, which was just transplanted across from the Amos2BASIC converter. The batch processing stuff just scans the folder you provide looking for PBA files. A PBA file for those who don't know, is the PlayBASIC source code file. Each source it finds it'll attempt to build into a DLL. The translator expects the sources to be stand alone at this point, so they currently can't have #INCLUDE statements in them. Which is limitation, but It should be able to be overcome.

In order to get the best out of the tool, it's highly advisable programmers adopt a parent->child structure to their machine code accelerated projects. So your main project folder contains the game / tool source + media as normal, in here you'd also have any secondary library sources stored inside that main folder. Your libraries folder would include your PlayBASIC DLL projects sources. So if you have a couple of lib's you can use the build folder option in PB2DLL to run through and build everything for you in one go. The build process will write the created DLL's to those folders and ideally generate linkDLL wrapper code for them. So in your main source you just #Include those wrappers and your away. .

So if we had an imagery a game called "PONG" , we'd make a project folder called PONG and hopefully give it a meaningful project name such as PONG.PBP (PlayBASIC Project file), rather than the default 'Project.pbp' which the IDE will assign those lazy people. Inside that folder we'd create a MEDIA folder for all the games graphics and sound assets. This simple structure immediately helps keep everything more organized from just file management through our actual source code. If for some reason we wanted to add build a DLL library to help our pong game, we're create another folder inside the PONG folder. It could be called anything, but giving it a name such a DLL's or Libs, or Libraries or something, something that should be obvious to what the folder contains

Now let's say in your pong game you're loading media files that for of a different format, you can load them in your main source code, but there's some VM overhead in doing that. This would be an ideal situation for converting the loader/conversion code to a DLL. To do so, you start a new project in PB, then we save this project (with a meaningful name) into our games / dlls folder. So if our library was to be a packing library we might call it "Packer", where it's path might be PONG/DLL/PACKER/. Once we have a project, we'd cut'n'paste all the required code for the DLL functions to operate separately. So any structures arrays etc your packer functions need, need to be inside that project. The project should compile without any external requirements. Once you have that, directly PB2DLL to that folder/and source and translate it.


Once the functions we want are externally built into machine code, we need to convert our parent project to use the DLL version of the functions, rather than the original versions that might be in our code. In this case, we'd pull all the Packer functions of our game. We don't need them in the main source anymore since all that code is now hidden away in our machine code dll. However, if you're passing types between the sides you would still need those Types to be declared within your main source. PB can't get this information back from the DLL. Moreover the DLL is like a black box, so the only functions you can see inside the dll are those that you exposed in your code. To expose a function we rename it with the "DLL_" prefix. Only these will be exported. Any function not containing the prefix, will be included inside the final code but not visible from the outside, so we can't access it.




PlayBASIC To Dll - The last 10% is really more like 90% (April 09, 2014)

One of the many frustrating things about programming is the almost there sensation, where the last 10 or so percent of the project, feels like it takes more like 90% of the development time. Testing is notoriously difficult and not to mention time consuming, one of the challenges with a project like this, is the near infinite amount of input situations the program has to deal with. One little issue I've been having is getting PB2DLL to control and understand the responses from both the PB compiler and FASM. It's fairly easy trapping a compiler errors (Shown bellow ) from the PB and returning this information back to the user, but from assembler is still a bit iffy trying to sense of what it returns, which soon goes beyond the scope of the product. Obviously it makes more sense to trap errors during generation, rather than after assembly. Since it's entirely possible a broken translation source could assemble.

Another problem that's been appearing is with batched conversions, where on some sets of sources, it's happy to compile and convert them, but there's the odd program that doesn't like being compiled in a batch, they'll compile fine alone, but not in a group. So there's something wrong with the loader state not being refreshed, but just can't find the actual smoking gun.




PlayBASIC To Dll - Eureka, It's Finally Working (April 11, 2014)

There's a funny thing that happens when your chasing down bugs, where you'll follow a hunch down a rabbit hole almost infinitely, regardless of if there's real basis in it or not. You know it seems like this, so it must be this! Now the past week or so, I've tracking strange faults in the translator where compiling the same program twice in a row would work, then it'd fail when changing sources in a batch. Initially it seem GUI related, since the test code worked fine in the stand alone translator, but generally failed when you batched a folder out (translated a group of sources). There's a few obvious things come to mind, like the translators not initializing it's internal structures properly, or there's some unforeseen collision somewhere.

Initially focused on tuning up the initialization code, but the problem persisted.. So off I go looking for any collisions or potential issues between the two slabs of code, such as clash of variable scopes. Even found a couple of such issues, which looked potentially lethal, but turned out to be a bust.. Next logically conclusion, oh there's must some bug in PB that's at fault.. But right before diving into the runtimes, it seemed it might be nice to get a clearer picture of what the translator is looking at to begin with. If that code isn't valid, or has some bogus data in it, then there's possible smoking gun and wouldn't ya know it when comparing the byte code state being compiles, there's an odd similarity in where the errors occur in them, in particular in the map that flags byte code as data and to be skipped.

So off I go looking at the byte code loader function and hey presto, the function REDIM's the ignore table. This creates a problem as Redim preserves the contents of that array, the array is used in parallel to the main byte code to screen stuff out, so compiling the same program would work but if you compile program A then program B.. Program B has Programs A's ignore list applied to it, the more programs to built in sequences the of lists get merged creating all sorts of strange artifacts. But right on 5PM that's bee been fixed now and it's working nicely !

Haven't just been searching for bugs in the translator, I've also added more error trapping abilities. Earlier this week, added support to trap error codes form PlayBASIC compile process. This returned a line number / error number and error message, but it couldn't get the source line, until now. So when a compile error occurs, it'll dump that to the console also. Which should give you some idea as that's wrong with the source. Beyond that, there was no error trapping in the translator side of the tool as all. In fact if an error occurred, it'd still call the assembler and try build your DLL. PB2DLL has limit on the amount of errors it'll accept, unlike PB. This is because it's possible the error message you're getting is just a warning, so the code may assemble, but it may not also.. If there's too many errors though, it just considers this fatal case and aborts the process completely.

Anyway, prior to this week I was hoping to have the first commercial releases up and running by the this weekend, the delays this week make that very doubtful now, but it is just around the corner !



PlayBASIC To Dll - Testing - 1,000,000 Point Rotation Example (real time) (April 11, 2014)

Here's something for those die hard skeptics out there, is rotating / clipping and drawing a 1,000,000 points to the screen possible in PlayBASIC ? - Well, it's not running through the VM's, but it is if you translate your code to machine code via PB2DLL then it's easily within your grasp.

I remember back in the mid 90's counting clock cycles to try and get a few hundred, let alone a 1000 points rotated in a couple of frames from hand optimized assembly. You can do that easily in PB today (even running on the slowest VM), but it's mind boggling how far this has all scaled up.

Common Point Rotation / Dot Cube (PB2DLL Example)

The best news about this, is the current PB2DLL's translator has only a few redundancy tweaks in it, as yet I haven't added any arithmetic optimizations.



More Information

For the full picture/ screen shots etc see the PlayBASIC 2 DLL development blog

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 1st May 2014 08:39
The following is a selection of blog posts from the development of PlayBASIC to DLL (Machine code translator) and the parallel ungrade PlayBASIC V1.64P


PlayBASIC V1.64P Beta 38 - Loopy Loops (April 17, 2014)

After much head scratching it was decided that the VM's instruction set needed a few alterations with it's looping instructions. These changes are two fold, the parser now does a better job at producing cleaner input parameter data and the instructions produced are now more stream lined. Producing better code, is better for everything down the line from running the code through the VM and thus any translations. The translation stuff is really why these changes need to be made now, since the original parser would use generic instructions where some data was pre-computed. From the translators perspective, this made it very difficult to accurately detect those loops in a single pass.

Today's changes introduce some new high level instructions to the VM. These changes make it trivial to accurately pin point the set up code in a FOR/NEXT looping blocks. I've already added a decoder to PB2DLL which locates the new opcodes and builds a export string in the dissassembly. The detection and output are working fine, but it's here that I noticed there's a few extra moves creating into the output code than need be. Which mostly happens when a loop is using literals. In previous editions the parser would drop an extra move prior to the loop entry point. The same can happen in the destination value expressions also. These things are easily tweaked out, just hadn't noticed them before.





PlayBASIC V1.64P Beta 38 - For / Next Loop updates (April 22, 2014)

If you didn't already know, it's been Easter holidays here which has just come to a close, so only started firing up the compilers today. Last time I was working on the tuning the compiler and VM to use the new looping setup codes for FOR/NEXT loops. These changes have were translated across to PB2DLL when I was happy with the new structures. The changes are much easier for PB2LL parser to accurately detect what's going on in the program in regards to loops. But not only that, the new format allows us to make more optimizations to the assembly.

Tonight I've been back working on the parser in the PlayBASIC compiler, as when converting the code to assembly, there's often these extra move operations that would end up in the byte code first and thus translated assembly. Surprisingly it generally happened what a for loop's (start / end) were both literal. For some reason it'd drop them into temps and then pull them into the loop structures internal registers. Which appears to be some left over instruction set behaviors that are no longer necessary. Older versions of the VM worked differently than today, in particular when dealing with literals. Now they can be treated just like variables, so don't they need to be treated special ways.

Something you might not be aware of is that when a for/loop is started, there's a comparison between the Start and End of loop. If Start is bigger than the end, the execution jumps to the beyond the matching next.

For example,



So there's an indivisible screening wrapped around loop entry, which might look at like this in PB style code.



One of the new benefits of the FOR/NEXT replacement opcodes, is the export parser can detect constant behaviors and produce code accordingly. So the comparison behind the Start /End loop counters can actually be made at compile time when they're both literal. Removing those lead in comparisons from the output code. The same goes for the storing them in internal VM registers, saving more set up code.

Being able to trap literals helps with STEP loops also. When a variable is used in for/next/step loop, the looping structure has constantly query what sign the STEP, then clip the loop counter based upon it. If the sign of step is known at compile time, then the loop can be created with this knowledge ahead of time. Removing more redundancy from the final translated assembly (in PB2DLL)

PB is pretty flexible with FOR/NEXTS allowing integer and floating point counters with variable step directions in both. People rarely use floating point loops, but since the VM supports it, the exported assembly through PB2DLL should also. Dropped in the standard by one loop structure earlier today and have just started adding the variable STEP stuff with floats. It's painful purely because of the horrible fpu instruction set in x86 chips...










PlayBASIC To Dll - Tweaking Looping Constructs ( April 24, 2014)

The past day or so I've been wrestling with FOR/NEXT/STEP implementation in the translator of all things. Those reading the PB1.64P upgrade blog lately would no doubt be aware that the compiler/VM side has recently changed, the changes were explicitly made to facilitate an easier more accurate translation to machine code from the byte code origin. When porting the changes across to PB2DLL I noticed that it only supported Integer FOR/NEXT looping structures, where as the VM supports integer and floating point look structures

Floating point loops present a bit of a problem due to the FPU's rather awful implementation, but really that's everything floating point in these things. Writing the translator code isn't difficult, it's just that it needs extra steps when dealing with floats which feels rather awkward. Knowing this, invested some time into the PB compiler to make sure the compiler side cleans out any unnecessary steps when setting up the loop and gives enough information to easily work out how to set the loop and what kind of set up is needed. The decoder is able to pick up short cut situations where you used literals in your For Next Statement and cut down amount of screen code needed, in some cases on screening is need. Since the loops duration can be validated at compile time. It's those little things that help fine tune the performance of the final machine code.

The big question is when do we release ? - Well since both sides need to be in alignment for the resulting DLL's work, the original plan was to release them both together, but I think we might release PB2DLL first.. This will help further test the translation in real world situations. Of course that'll most likely mean that there will be a few updates made prior to the release, but there's not that can be done about it. It now just needs people other than me testing it.




PlayBASIC To Dll - Exporting LinkDLL binding code ( April 28, 2014)

Had a few short programming (testing mainly) sessions over the weekend. Up until now, I've been testing the translator from the PB IDE. So in order to run it, i've got to compile and run as normal (F5), but as release time nears, we need to be able to build it into a final EXE, but it's here we ran into a new problem, where the exported internal structures could stop the exe from running. I though this might happen, given how much of the PB VM internals have changed over the last 6 months. Tracking down the issue took a while, but was fairly easy to solve which is a good thing. So now I can build & run PB2DLL in exe form and it works as normal.

Once the build problem was sorted, i've been getting back to the GUI and internal exporter functions. For the first version(s) I'm not going to bother with adding switching over the assembly code generation/optimization, it'll just do it silently. There are times when that wouldn't be wanted, but can't foresee anybody running into such problems. Which means for the time being some of the GUI toggles have been ripped. The only option that is useful is the Export LINKDLL code option. This option gets the translator to build a chunk of the PlayBASIC source code that handles the function binding to the newly created DLL.

Linking to DLL's is something of a mystery for most people, basically all that happens is the PB compiler/VM create interfaces to external machine code that look and act like normal PB functions in your program. There's some limitations in current versions of PB as to what data type/structures you can pass into and out of them, but those will be tweaked out in time. In order to use our newly converted DLL's in our PlayBASIC programs, it stands to reason that we'd need to know how to write the LinkDLL binding code for the functions we wish to use. Which was true yesterday, but now the translator can build a template of the exported functions for you in PlayBASIC code. Obviously there's some limits in terms of data types that it currently supports, which stem from the PB compiler side not supporting passing PB internal structures through to external code. Why? - because nothing other than PB understands PB internal structures, making supporting that originally point less.

To test the LinkDLL exporting, added a function bogus functions to the common point rotation demo and here's what we get.



This code can either get cut'n'pasted in your project or #included. Either way once it's in and assuming PB can find your DLL (it's in the project folder) then your away in the machine code era!


What might be a good is to get PB2DLL to mangle the export names in the DLL. So if you have a function called "AddIntegers", the external name can be mangled into some random hash, such as 'ABHDGSNFSSF' - This would mean that exported linkDLL codes must match the build of dll exactly, as if the alias names don't match, windows can't local the machine code function and it won't run ! It'd only be helpful to prevent snoops from look at your function names and potentially working out the parameters.




[b]PlayBASIC V1.64P BETA #39 (Retail Compiler Only Beta) [/b] (30th,Apr,2014)

Beta #39 introduces a bunch of subtle but equally important changes, starting with the newly replaced FOR/NEXT/STEP parser and VM instruction set, there's also a few changes to the VM close down behavior. The close down changes are due to a strange issue i've been having with PB2DLL where the PB2DLL EXE would crash on exit when the program wasn't registered (worked fine once you entered your serial).

So i've been chasing the close down issue most of the day, it appears that some typed arrays can somehow contained data that wasn't a typed array container. How that's the possible is a mystery at the moment, but i've included some more screening in the clean up code to make sure it's releasing the array an not some of random bank of that data that something else was/is using. Getting this right is more important now than ever, thus is because of the way type deletion works in the runtime, so we have to make sure it's releasing data that makes sense, as if it doesn't it'll die.

The real question is how is the bad array data was/is getting into the arrays in the first place, which i'm not 100% on at this point. So i've added some debug messages to the debugger console (F7 to run, then click on the console tab) to help the community track where else these quirks put up.. Does it happen at runtime ? or only closing down the VM..

So get testing !!!!!!


[hr][b] Download[/b][hr]

[size=12pt]
Download PlayBASIC V1.64P Beta 39
[/b][/size]




More Information

For the full picture/ screen shots etc see the PlayBASIC 2 DLL development blog

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 3rd Jun 2014 15:45 Edited at: 3rd Jun 2014 15:46
PlayBASIC2DLL V0.99 Commercial Edition released!

Yes it's finally here! .. after more than few marathon nights the first fully fledged version of PlayBASIC2DLL is available to the community. PlayBASIC2DLL brings the power of native code to the PlayBASIC developer, all we can say is welcome back to the machine code age.

This initial release has been heavily discounted for the community down to $27.50 (US) from it's regular $49.99 US


What's PlayBASIC2DLL ?

It's a companion tool to the PlayBASIC programming language. The tool converts PlayBASIC Source code into machine code DLL's. Allowing the programmer to fully harness the process of their computers cpu.

The tool has been in development for the best part 8 months now so rather than me crudely attempt to paraphrase the blogs, I recommend you read the PlayBASIC2DLL blogs and check out some of the examples. ( 1,000,000 Vertex cube - Water Demo )



Order

You can order the product through shareit who provide all major payment processing methods from bank card, paypal, cheque, money order and more.

See->See Community Annoucement To Purchase PlayBASIC2DLL

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 24th Jun 2014 21:02 Edited at: 24th Jun 2014 21:10



G2D - 2D OpenGL library for PlayBASIC V1.64P PROTOTYPE

Release Date: 24th, June, 2014

PlayBASIC version: PlayBASIC V1.64P (Beta Edition 42 or higher).


[center]



[/center]

What is this?:

G2D is an alternative rendering library that uses the OpenGL API rather than the stock DirectX / Software interface that PlayBASIC normally uses. The command set is very minimal at this point, in fact only about a dozen primate operations are supported at all. Those that are, try to be functionality identical to the internal PlayBASIC equivalent command, while only drawing to the OpenGL screen.

G2D currently only works with PlayBASIC windowed modes, the library attaches the GL viewport to the PlayBASIC window. When we do this, both the DX and GL screens are being to drawn to the same window. You can get rid of the DX screen using the ScreenLayout command. The command gives you a rect (X1,y1,x2,y2) of where the DX screen should be drawn within the window. To get rid of it, just position it outside of the window.


You can download the library from maintenance board on our forums. This library was written in PlayBASIC V1.64P and converted to a DLL with PB2DLL

Attachments

Login to view attachments
Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 30th Jun 2014 11:02
Latest PB2DLL news

PlayBASIC To Dll - Breaking the one second mark for fractal render

After tweaking the floating point code generator just a little, we get an even more staggering result from the previous fractal test, which now is rendering inside a second for a 640*480 screen on my test machine AMD FX64 (3000+). The result gives us about 150 millisecond improvement over the initial test, so even the first versions of PB2DLL are surprisingly optimal, in particular when you compare those results to the primary competitors. PB2DLL is consistently 3.5 times faster than one and 5.2 times faster than another..

In this particular example the routine has a lot of dependency, by that i mean, calculations need to be done in a set order. I suspect if the generator could interleave some of the FPU instructions around integer instructions, we could go even further though. This is possible as the integer and float (FPU) sides of your CPU can actually run in unison. Instruction operations on either side can generally overlap other instructions also, providing the following instructions don't rely upon the result of a previous instruction. So we could interleave instructions that take a long time to execute around other instructions as long as they don't rely upon the result.


Note: The original code/demo is attached to an earlier post in the PB2DLL WIP thread, it's exactly the same code.





PlayBASIC To Dll V0.99g Released and Available For Download (30th,June,2014)

Here's revision G of PB2DLL V0.99 package. This revision is focuses on bugs and the more improvements to code generation engine. The main bug that this update fixes is the possible freeze when attempting to load a PBA that's has no exportable functions in it. It was a strange issue caused by the Replace$() function in PB, but that's been solved now. So if you select a source that has no exports, you should get an error message, rather than it hanging.


Code Generation:

Just like the previous build i've been working on improving it's detection and abilities to solve sequences of byte code operations in order and with in CPU registers. The solver can support integer and float data types mainly. SO it's looking for sequences of integer or float operations that stack up in sequence. When it finds one, it short cuts this series of instructions.

The Integer solver has three main additions, it support equality tests (= and <>) , Compare/Branches as well as better support for Left/Right hand operators. The latter allows it to better track where which side of the temp data came from, then remap the operator accordingly.

On the float side, the changes are a little different, the float solver can handle recasting results and some limited single parameter functions like Sqrt(), ABS() as well as the normal addition/subtraction/Divide/Multiplies stuff.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 31st Jul 2014 20:07
Some PB2DLL news


PlayBASIC To Dll V099h - Breaking 800 Milliseconds for fractal render

The last round of tweaks got the fractal demo running inside the one second mark, which as about a 250 millisecond improvement over the original base translation version. Breaking the second barrier was enormous and I was very very happy with those results, but knew there was a little more room to move. So yesterday I set my sights on increasing PB2DLL's serialization support to help avoid some unnecessary memory access. With these fairly simple changes we've smashed another 200 milliseconds from the V0.99g results.. Bringing rendering down to around the 780-790 millisecond range.

When compared to our competitors the results demostrate PB2DLL accelerated version is 4.1 times faster than one and 6.6 times faster an another. So in terms of high iteration pure math operations they're no competition at all..

8 year old Test machine AMD FX64 (3000+). Screen resolution: 640*480 screen




PlayBASIC To Dll - 1,000,000 Point Rotation Example (Remix)

You've probably noticed that dot cube examples is a particular favourite of mine when testing 'tweaks' to the Pb2DLL codes generation ability. It's not that it's a particular beautiful demo, it just boils down to some serious number crunching with a good dose of memory thrashing to boot. So it stands to reason if the generator can stream line any expressions better or avoid hitting memory unnecessarily, then those tweaks will show up as better performance.

For the last few sessions of PB2DLL, I've been focusing upon improving the serialization detection as well as embedded this behavior around more of the common functionality. For example, if you call function that returned an Integer/float, then it scans ahead looking for any sequences it can clean up for us. So if we had a bit of code like AlignedToGridMouseX = MouseX() / GridWidth, then previous versions would have hit memory when pulling the mouseX() function, now the result lives in a register and would be divided on chip, before finally being written out to memory.

Performance wise today's build of PB2DLL gives us around a 4FPS boost, going from running at 11FPS on the test system to 15FPS. In millisecond terms, that's about a 24 millisecond improvement. The demo runs in about 66 milliseconds now, where as it was running in approximately 90 milliseconds before.





PlayBASIC To Dll V0.99h Released and Available For Download (11th,July,2014)

Here's revision H of PB2DLL V0.99 package. This revision continues to focus on finding those left over core instructions that weren't supported in the translation, as well as improving the quality of the code produced. One of the operations that was missing was reading from a handle from a type structure. This is important as we can use this to detect what cells within an array are populated are what ones aren't.

eg.





Code Generation:

This version supports better stream lining of expressions of core instructions onto the CPU from both the integer and floating point sides. The goal of such changes it avoid hitting memory were possible as well as replacing common operations where it makes sense. One of the recent additions can now trap temp registers that appear following internal command set function calls. So if you called some internal command function that returned a floating point value, the generator will try and sequentially solve any following instructions.

Eg.

x# =GetSpriteX(SomeSPRITE) / GridWidth

So some code like this now leaves the result of the function call on the fpu and performs the division on the fpu also. Which gets rid of some temporary memory thrashing that the runtimes have to do, but our machine code dll's don't. This means that generally if your expression stacks well, you'll get very solid performance from it.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 20th Sep 2014 20:42
PlayBASIC V1.64P - Building Release

Well... after something of a long delay I've started building the all the release versions for the V1.64P update. There's not enough time tonight for a release, so suspect that'll come later today probably Sunday night. Was planning on importing a few more VM changes, but they can wait until the next round of updates. This version will sync up with current PlayBASIC2DLL releases, so there's no need to update that at this time.

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 21st Sep 2014 23:07 Edited at: 21st Sep 2014 23:11



[hr]PlayBASIC V1.64P _Retail Upgrade_ Now Available (22nd, Sep, 2014)[hr]



This release updates any version from PlayBASIC V1.63w2 to PlayBASIC V1.64O2 (retail) to the current retail version of PlayBASIC V1.64P. If your current version is older than PB1.63W (pre June/July 1997), then you may need to install patch PB1.63w2 prior to this one ! (if you haven't already)


The V1.64P presents a huge change to the underlying technology that PlayBASIC programs execute upon. These changes range from high level compiler optimizations/bug fixes/omissions down to how almost every internal commands is called, which not only give us faster program execution, but it allows internal PlayBASIC command sets to be called from the external DLL's built via PlayBASIC2DLL tool.

Programmers using PlayBASIC2DLL in combination with PlayBASIC V1.64P can now write truly modular applications in high level PlayBASIC without ever having to learn low level C/C++ or PASCAL programming. You can write your functions and compile it a DLL using PlayBASIC2DLL. Meaning PlayBASIC programmers can build their own command sets /extensions and perform all brute work number in highly optimized machine code DLL's.


There's way too many changes to try and summarize it all, so we highly recommend carefully reading through the V1.64P WORK IN PROGRESS GALLERY for more insight in what new additions are hidden away in this release.


Don't have a retail version of PlayBASIC ? and want to find out more information about PlayBASIC programming > please visit the www.PlayBasic.com and download the free learning edition.



Downloads

Download PB1.64P Retail Upgrade (22nd,Sep,2014)

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 2nd Oct 2014 10:09 Edited at: 2nd Oct 2014 10:12



G2D Prototype #4 Released - 2D OpenGL library for PlayBASIC V1.64P PROTOTYPE

Release Date: 2nd, Oct, 2014

PlayBASIC version: PlayBASIC V1.64P (Retail or higher).


[center]



[/center]

About:

Today we've released build #4 of the G2D library on our forums, it includes more Sprite / Image and Map commands. For stuff like maps all we needed is a way to render a map to the GL screen, which is what the g2dDrawMap function does. The only difference to the internal command (DrawMap) is that the g2D version currently requires a texture index to be passed to it as the block graphics. (See previous blog post about that)

Command wise, this build includes some new sprite commands such as AlignSpriteToSprite(parentsprite,limbsprite,limbx,limby,limbangle#) and the

x#=GetAlignedSpriteCordX(thissprite,cordx) (there's also a Y version) which are used to compute the location of a point that's attached to a sprite (rotated/scaled / offset. Extremely handy for gun turrets, character limbs, particles that sort of thing.

There's also a few new Image functions such as.

MakeAlphaChannel(imageindex,maskcolour) ; This function builds an Alpha channel for images that original use MASK COLOUR based transparency. The image can then be converted to a texture use us in a Open GL scene.

g2dConvertToTexture(index) ; This function creates an FX/AGX image to a GL texture for this image.

What these functions allow users to do is convert existing image media so that doesn't contain alpha channel into a form that can be used with OpenGL


While updating the test / demo with the dll version I just noticed the G2DDrawOrderedSprites function seemed to be broken in this build, not too sure why, the g2dDrawAllSprites is working.


You can download the library from maintenance board on our forums. This library was written in PlayBASIC V1.64P and converted to a DLL with PlayBASIC2DLL

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 4th Nov 2014 12:37
PlayBASIC V1.64P2 BETA #01 (Retail Compiler Only Beta) - (Avail for Registered Users ONLY) (7h,Oct,2014)


Beta #01 addresses a few bugs in the sprite commands as well as addressing pass #1 parsing issue that wouldn't allow comments after Function/Psub prototypes. While single comments are supported now, comment blocks aren't at this time.

Eg..





PlayBASIC V1.64P2 BETA #02 - Pointer Corrections

Somewhere during V1.64P (or perhaps later) a number of odd parsing issues have appeared with pointers. Pointers are internally one of the foundational elements of the runtime, the issue has been with the higher level exposure of these. In terms of the parser they've always been (and still are) something of after thought which apparently makes them even more susceptible to parsing changes. Fixed a couple of odd problems that seemed to occur during math operations between them as well as de-referencing (peeking) them..

So code like this is working again..





PlayBASIC V1.64P2 BETA #02 - Pointer Corrections Continued

Last week was full of fun, if not wasn't out on the mower or wrestling with our tax return then i was hunting parsing issues concerning pointers. Fixed a number oddities earlier in the week, only to discover those fixes broke other seemingly unrelated code later in the week. Initially it seem unrelated, but upon close inspection there was generation common function that could leave the expression in the state that following generators would make incorrect assumptions upon.

One such case occurred, when the parser was resolving user defined type array fields, which would call this function and occasionally get back some strange data. Making them do everything from function correctly to completely fail. I've actually noticed something like this oddity before in the byte code diss-assemblies, but was unsure if that an issue in the compiler, runtime or dissassembly. The program ran so one would assume the latter, but apparently not. Which is fairly rare really.

The following is the current test code that's running in the latest Beta #2 which i'll uploader later



Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 23rd Nov 2014 04:55 Edited at: 23rd Nov 2014 05:13



G2D - 2D OpenGL library for PlayBASIC V1.64P
PROTOTYPE #8


Release Date: 22nd, Nov, 2014

PlayBASIC version: For use with PlayBASIC V1.64P (Retail or higher).


[center]



[/center]

What is this?:

G2D is an alternative / replacement rendering library the uses the OpenGL API rather than the DirectX / Software interface that PlayBASIC normally uses. The command set is evolving constantly and already supports a range of core 2d primitive drawing functions (Dots, Lines, Boxes,Polygons) through to Image, Sprites, Maps, Fonts / Text, and many the most recent addition is the programmable particle library. These functions try to be functionality identical to the internal PlayBASIC equivalent command (where one exists), while only drawing to the OpenGL screen.


This library was written in PlayBASIC V1.64P2 and converted to a DLL with PlayBASIC2DLL



Latest Blog Posts:

The following is a selection of development blog posts, note the posts may mention pictures / source codes / video that may not be shown in this overview.


G2d Blog - Particles and all the small things (October 24, 2014)

Been thinking about adding particle / entity layers to the G2D library. Personally I tend to find particle libraries can be a hit hit and miss, some things are easy to do and others fall outside the scope of the solution. Although the same can be said about any library really.

Particle effects are useful in many situations, but there's a handful that come to mind such as smoke / fire / thrusters / explosions, so they're worth spending a bit more time thinking about, rather than jamming in the same old same old solution in.

Control wise different effects need different controls per particle, stuff like controlling speed / gravity / scale / rotation / image / Randomization etc could all be useful in various situations. So supporting a cross section of controls seems like the way to go, but this does make initializing them potentially harder for the user. Not a huge fan of functions that require 20 parameters though, obviously such things could be wrapped down, but still you soon end up with a lot more set up, than solution.

In recent years my favorite solution for such problems is a type of generalized solver. Where the user can define a set of operations which it runs it's internal logic over the data set. In particle terms the user builds a type of particle script (for want of better word) and the solver would apply this logic to each particle in the set. So the user isn't really hands on in the process, but they have control over how they move / rotate / animate and when they die.

Even with such customizable control there would still be situations where the user need to roll their own.. But ya get that !



G2d - Particles Script / Controller Cont. (November 03, 2014)

The awfully static picture bellow doesn't really give you much of sense of the demo but anyway, the execution engine is taking shape. You can think of as a special purpose virtual machine for running basically vector operations, a bit like the vector stack stuff from a few years back. The code are thus far 3d vector only, but the idea is to support 1D,2D,3D vectors and possibly some bare bones integer controls. Don't wanna have too many opcodes and too much logic getting in the way of the particle execution engine though.

In this demo the controller runs the Particles.NEW() function upon creation which is then updated by the Particles.UPDATE() function. The FREE() code isn't used/defined in this demo, not too sure how the language will control the particles state. Currently the execution engine returns an ALIVE/DEAD[/b] (true/false) state on this object, if it's false it's died and needs to be reclaimed.

In terms of instructions so far we have [b]SET
,MOVE,ADD,SUB,DIV,MULT, RANDOMIZE and CMP. Which gives us just enough controls to create and explosion with gravity. It's not exactly BASIC, but with a little thought/reshuffling it seems like reasonable start. You won't be able to 'loop' in script though, so the logic will have to generally solve in order. Should be able to add some type of conditional branching (IF/ENDIF) styled statement, but it'd be nothing like BASIC..

Anyway.. here's an approximation of the controller code in our mock up script that creates the explosion attached (Running in just PLayBASIC for now) .






G2d Blog - Particles Script Compiler - Local Rotations/Scaling/Animations (November 09, 2014)

Even though today was my birthday, still found time to add a branch (IF/ENDIF) block into the script. This means the script can make comparisons and make simple decisions upon on the result. Unlike the previous versions of the library the particle will only return a 'dead' state when the script hits the DIE() operator. Which is basically and END statement. So script is exited and the status set to dead.

So while still rather crude, we've basically arrived at a frame work that can do most common place particle effects. What we're missing is some way of spawning them relative to a parent. The current idea for that, is the user set the current parent (passes a structure to the library) and then the new() operator can pull useful stuff from, such as it's position, direction / rotation, speed. So if you had a ship that rotating, all you'd do is set the parent structure with that info, then any and all particles generated at that point will use that as they're origin. Allowing you to attach a thruster, spawn related sparks/explosions that sort of thing.

The current Particle VM is a bit naff, I think what i'll do is wrote a post processing converter, which will convert the instruction calling to a more optimal method within the final DLL version (PB2DLL), which would easily be using some type of ON VARIABLE GOTO/GOSUB structure. Which yields a fixed cost per instruction. Moreover I can use the same registering caching method that PB+PB2DLL use when writing opcodes, would allow the compiler to sub faster operations, or use packed methods. So the scripts should perform very well.

Anyway here's today's little picture.. It's show the particles using different frames, rotations, scales, speeds etc etc... Check the source code to







G2d - Programmable Particles - Controller / Emitters (November 14, 2014)

Wrote the emitter commands yesterday, the functions turned out pretty much like what's outlined above, with a few subtle changes. The emitter is the controller or owner of any group of particles. All particles within the group share common attributes, such as the spawning origin, logic scripts and the texture they use, but each emitter is unique. The user controls the spawning frequency (particle emission) updating and rendering. So you have complete control over if/when something is updated, and if it's rendered. There's no particle level access at all really, but a function to return a pointer to a particles locals could be useful, or perhaps a function to alter the locals of the set. Which is useful for adding/changing some global force, like wind for example.

The scene bellow is running in a new hacked together testbed. In the original demo, the code was man handling each particle through a list, now we've just managing the emitters. Basically telling them to emitter new particles, when to refresh and when to draw. Since they're all on screen we're updating all of them all the time. If and emitter was to do out of view, then of course you'd suspend updating/rendering it.


Here's an animation created from frame work running in standard PlayBASIC, with any luck the damn thing will play... as it doesn't here !






G2d Blog - Programmable Particles DLL version (November 20, 2014)

Up until today, the particle library only existed as a set of PlayBASIC functions and Psubs. After tonight though, the library has made it's way into the G2D library and now being compiled to machine code DLL using PLayBASIC2DLL. There was few gremlins with the translation initially and a few spots where I'd been lazy and not manually allocated my types. Which would make it crash during the script compile, but those seem to be all sorted now.

For me, the most interesting thing about this isn't actually the rendering at point, it's how fast the Particle VM will execute the byte code, as if there's too much overhead in the particle VM, then there's no much use in them being programmable. But any fears I may have bad were soon squashed. With the particle scene controller being able to run 10,000 scripts at over 60fps on my 9 year old Athlon system. Which can be seen in the rather boring screen shot bellow.

The library can't render the particles to the OpenGL screen as yet, but even drawing the same FX screen yields a 10+ times performance boost in 4K particle heavy scene, rendering the particles as FX images with alpha addition. I'd say that with batching them as quads in GL, rendering should make mince meat of it really. While in reality it might be able to process 10K particle scripts, I'll be more than happy with rendering 1/2 of that at a reasonable rate. But that's Fridays little challenge


G2d Blog - Programmable Particles - 17000 particle test scene.(November 21, 2014)

Well that was much ado about nothing, as any fears that running particles through our custom particle VM would be too slow, should now be completely and utterly smashed with the library returning better than expected performance. The test scene bellow is shown in two states, the scene is made up of a number of elements again all cut and pasted together form the basic 2d demo that comes with the library. So in this scene we have some sprites an mock up map and depending on how fast you can press the ENTER KEY anywhere between 5K->17K live particles... Obviously the demonstration is over the top, but it does prove how efficient the PB Compiler -> PB2DLL process actually is. Which is VERY !

Test Machine: AMD64 3000, GF6600 ( 9 years old)


Download:

The current build of the G2D library is available through PlayBASIC maintenance area on our forums.


note: Unfortunately the Blog posts are best view from the BETAFORUM, as the two versions support different colour tags.

View thread in Betaforum

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 1st Dec 2014 11:04
PlayBASIC2DLL V0.99I BETA #05 Download (1st, Dec,2014)

Beta #05 contains the latest code generation updates for PlayBASIC2DLL tool. Most of these updates improve the quality of the code created when using pointers through to internal function calls.


See PlayBASIC2DLL Dev Blogs for downloads

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 29th Dec 2014 17:00



PlayBASIC V1.64P2 _Retail Upgrade_ Now Available (29th, Dec, 2014)



This release updates any version from PlayBASIC V1.63w2 to PlayBASIC V1.64P (retail) to the current retail version of PlayBASIC V1.64P. If your current version is older than PB1.63W (pre June/July 1997), then you may need to install patch PB1.63w2 prior to this one ! (if you haven't already)


The PlayBASIC V1.64P2 package includes updates of PlayBASIC Compiler, Release / Debug Runtimes, SLIBS, DOCS (HELP FILES) & IDE V1.17b.

The V1.64P2 release is a relatively minor update to address functionality broken during the massive V1.64P update. Beyond those bug fixes, there are of course more improvements to the compiler & runtime command sets and debugger.

The compiler / runtime time changes are generally improvements to the quality of the byte code the compiler produces within certain circumstances. One that comes to mind, was the ability to completely optimizing out literal fields when querying array structures from typed pointers. So code like the snippet bellow runs better in V1.64P2 than previous versions. Which in turn helps PlayBASIC2DLL produce even faster machine code versions of such routines. So that's a big win !



Such compile time optimizations are useful when we want to iterate through structures of data, so rather than bumping the ME pointer between integer fields, we could unroll the inner loop to process groups of integers and access fields at fixed offsets then bump the ME pointer. Which is a more efficient approach generally.


Changes to commands sets that users will find the most useful, can be found in variable Alpha Blending situations, generally filling surface routines are now twice as fast when performing variable alpha blending in this version, there's also changes to the AFX alpha blending logic which screens the alpha channel value for pixels that are fully translucent or opaque, in either case those are handled with a special case. The performance benefit really comes down to how many pixels in the image need to be alpha blended. But the standard tests had a 10->20% improvement without change..


In terms of Debugging, there's two new subtle changes in the compiler runtime. The first, is the new ability of the runtime to intervene when some pointer opcode attempts peek/poke from a null pointer. Which should generally return a runtime error if that occurs like in the example bellow.




The other runtime/debugger change is a mode in the debugger to Enable/Disable automatic type creation when executing a program. The PB runtimes default behavior is to automatically allocate a type if you're writing to a structure that's currently empty. Generally this is handy behavior as it saves you some lines of code, but can be an issue when exporting your programs to DLL since PlayBASIC2DLL doesn't support this behavior. So if you writing library code that you're going to build a DLL of, you'll find this invaluable !


Don't have a retail version of PlayBASIC ? and want to find out more information about PlayBASIC programming > please visit the www.PlayBasic.com and download the free learning edition.




Downloads

Download PlayBASIC V1.64P2 Retail Upgrades From Forums

Kevin Picone
20
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 23rd Feb 2015 06:42
2014 was the year of conversion for PlayBASIC, as will be 2015.

Last year we introduced PlayBASIC2DLL, a tool at allow PlayBASIC programmers produce high quality, compact and very fast machine code binaries directly from PlayBASIC source code.

This year we'll be doing much the same, but focusing on our web translation technologies. Although very different targets, the principal approach remains much the same, the user (you the programmer) writes code in PlayBASIC once, which is passed through a translation tool for create the output code for the target. The method doesn't require plug in's or bulky interpreters written for the client side, so it's about as direct a translation of your code as you can get.

Have a nice day..

Login to post a reply

Server time is: 2022-09-28 10:39:29
Your offset time is: 2022-09-28 10:39:29