I think I found an answer: I physically separate the objects based on penetration depth

It seems to work ok atm

I'm not using Unity, I'm writing it in C++

10 hours ago, JoeJ said:

Interesting (and new to me, but it already helps).

What would be the simplest general problem you can think of where this is useful so i can practice that?

Rotation/scalinf/shifting of equations is just a basics of linear algebra that widely used into analitics and computation geometry field. It is a main tool used to perfom objects transformations into 2D/3D and other D spaces.

10 hours ago, JoeJ said:

I want to look at how quadric error metrics for mesh simplification works anyways, but if you know something simpler please let me know...

Im not used mesh modeling since seen solid body modeling at 2003 . Hybrid curved body representation always have a glide surfaces that can be transformed to meshes with preordered complexity an precious normals and surface UVs/tangents and so on.

@Scouting Ninja

my main reference book is 'Essential Mathematics for Games and Interactive Applications' + information from the internet

I considered using 'Bullet SDK' for my physics but the documentation was lacking so I decided to build my own physics engine