I'm not too well versed in Dark GDK, I'm afraid (so I can't give actual code), and it looks like you got it mostly working.
However I wrote some similar code for DBC a while ago and figured that I would post the breakdown of the math, in case working out the kinks in your code ends up becoming more trouble than its worth. This system tracked objects and placed them on a radar. If you think about it, the math would be nearly identical up until the display portion(essentially, now the radar is the screen size), so I'll give you a run down of the math. It would be helpful if you are familiar with vector algebra, but if you aren't, I'll try to explain things.
What you need for setup:
-Unit Vectors (a 3D vector that is 1 unit long) for the player's LOCAL X,Y,Z orientation (Essentially, the local Z axis is the player's heading, x and y are to the right and straight up, respectively).
-Construct a vector from the player to the object being tracked (I would suggest storing the data as a unit vector and a distance). This will be called the Tracking Vector.
-Perform the DOT PRODUCT on the Tracking Vector and the Heading (local Z axis) Vector. This will project the tracking vector onto the heading vector. This will be referred to as R.
-The angle between these two vectors can be found by: THETA=acos(R)
-This angle is only MILDLY useful, in that it is rotation about a non-standard axis, meaning that on it's own, it gives us nothing. The next step is to find that axis.
-To find the axis, perform a CROSS PRODUCT on the vectors (since order matters, lets do HEADING x TRACKING). Make sure you are using the UNIT VECTORS, otherwise your resultants will be in the right direction, but your magnitudes will be off (which is important). The cross product will return a vector that is perpendicular to the two vectors being crossed. This is the SAME AXIS THAT THE ROTATION IS ABOUT.
Why this works: When you dot the tracking vector onto the heading vector, you are essentially constructing a right triangle. The base is the heading vector and has a length of whatever the dot product returns. The hypotenuse is the tracking vector and has a length of whatever that length is. Viewed on a piece of paper, the math is straightforward for the angle. Unfortunately, in the program, we don't know what the orientation of that paper is. The cross product gives us a vector that is perpendicular to the paper. Thus, the angle of the between the base of the triangle (heading) and the hypotenuse (tracking) is an angle about the perpendicular vector.
-One thing that you might have not been exposed to is the concept that rotations can be expressed as vectors. When drawing them, they typically get a double arrow or have a circle drawn around the line to denote it as a rotation vector, rather than a linear one. Thus we will do the same with our rotation vector. It has the unit vector that was returned from the cross product (lets call it the AXIS VECTOR) and a magnitude of THETA. Multiplying Theta across the components of the unit vector will give us the components of the vector itself. Lets call this the ROTATION VECTOR
-Finally, perform the DOT PRODUCT of the ROTATION VECTOR and each of the local axis vectors. This will yield numbers showing how much of the angle is around each axis.
-Output this data to the screen. In my code, I used the Pitch offset for the Y coordinate of my display and the Yaw offset for the X coordinate. I divided each by 180 and then multiplied by a constant to position the dots at the correct location on the screen. The dividing by 180 came about because I was using a circular radar and the radius wasn't set in stone yet (I was toying with things), so by dividing it by 180 (the maximum angle value around any axis in this case), I got a fraction, which I could multiply by any angle to make it display correctly.
Some additional Vector Algebra Notes:
Unit Vectors: Basically, a unit vector is any vector with a length of 1. Any vector can be converted into a unit vector quite simply (shown below). Futhermore, any vector can be expressed as a scalar (non vector) magnitude and a unit vector denoting direction. I will "A" to show a normal vector and "a" for a unit vector. I will use L as the magnitude of vector A.
Note that to find the magnitude, you just apply the distance formula to the vector.
The Dot Product: The dot product returns a non vector quantity (also referred to as a scalar quantity). Here is how it is performed (I will be using a WEDGE notation style, such that a vector is expressed as V=<x,y,z> where vector V has components x,y, and z).
The CROSS PRODUCT: The cross product is a more complicated operation than the DOT PRODUCT. Typically, Matrices are used, but I will skip through all that and show you the end result. Note that the CROSS PRODUCT will return a VECTOR quantity, rather than a scalar quantity.
AxB = <Ay*Bz-By*Az, Bx*Az-Ax*Bz, Ax*By-Bx*Ay>
Note that in a 2D system, the X and Y components of the cross product result are 0 (since Az and Bz=0).
Hope this helps! If you have questions, ask, though to be honest, I don't frequent this forum much, but I'll pop in over the next couple days to check.
"Time...LINE??? Time isn't made out of lines...it is made out of circles. That is why clocks are round!" -Caboose