I'm having trouble working out how to recalculate 2D sprite sorting / draw order based on camera rotation.

In the example below:

Screen 1 shows a non-rotated typical "top down 2D" scene, where the grey objects can be sorted based on their screen Y position, drawn from top to bottom, so in this case B, A, C.

Screen 2 shows the situation after the player/camera rotates clockwise, with the original screen up vector (green) and the new up vector (red). In this situation, sorting by object Y position will not work, because A should now be drawn after C.

What I need to do is to sort the objects not only by their Y position, but also relative to the new up vector and/or camera rotation, but I'm unsure how to handle this. Appreciate any help you can give!

You should be able to sort consistently by a single vector element (x or y, whichever matches the conventions you're using) regardless of orientation if you transform the object positions into the local space of the player/camera. In local space, things will always look like in your first image, with the player/camera pointing 'north' (or whatever the case may be).

I don't know how you're rendering, but rendering often involves transforming geometry into the local space of the camera anyway, so you may already be doing that.

If you need to do it manually, you can perform the transformation using the inverse of the player's world transform. If you don't need the full transform, but only a single vector element for sorting, you can instead just subtract the player position from the object position and dot the result with the player's forward vector. (If you don't need to know exactly where the objects are along the forward axis relative to the player position, you might even be able to skip the subtraction step.)

I think just sorting by squared distance has the same effect. It would probably be cheaper than rotating all your objects.

Thanks for the hints, I'm unfortunately just doing a basic prototype using Godot, so not dealing directly with any geometry or transformations manually at the moment. I may have explained poorly in my first post so I'll try and clarify:

• The red "dot" in the middle is the player sprite. The camera is centred on the player in the example, but doesn't have to be.
• The player can move  forward, backwards, left and right through the world, past various other sprites ("A", "B", and "C")
• In reality these other sprites (and the player sprite itself) could overlap each other (i.e. one sprite is "in front" of something, or "behind" something, shown through correct draw order)
• When the camera rotates, it changes the viewing angle such that objects should appear "in front" or "behind" differently. (For example, with no rotation, say the player sprite is below ("in front of") A, so should be drawn after (basic Y sorting). Camera rotates 180 degrees. The player should now be above ("behind") A, and so drawn earlier.

I understood the theory there of transforming the sprite positions into a different space so that their y's would be relative to the rotation, and tried a few of the points you suggested Zakwayda, but can't seem to get the right results. I wonder if you could explain how the camera position is related to the sorting? I thought only the rotation would be relevant to it, because no matter where the current camera X/Y is, the draw order would be the same, and only change when it's rotated. 

I figured this one out now, thanks Zakwayda for pointing me in the right direction, I just rotated the sprite positions by the inverse of the cameras rotation and this gave me what I was looking for. I must say that as I'm using the built in Godot functions (e.g. Vector2.rotate(-camera_rotation)) I don't fully understand the maths behind it, so I should read up on the actual calculations to understand this better.

7 hours ago, Prototype said:

I think just sorting by squared distance has the same effect. It would probably be cheaper than rotating all your objects.

Whether the difference matters or not depends on the context, but sorting by squared distance won't necessarily give the same sort order as sorting along the forward axis. Also, as I noted earlier, you don't have to perform a full transform if all you need is sort order - you can get by with just a vector subtraction and a dot product (and maybe even just a dot product, depending), which would have roughly similar cost as taking the squared distance.