In game development, we need to design a turning system. When the angle is less than 80 degrees, it will rotate directly and rotate 80 degrees when the angle is greater than 80 degrees. Known (A1, B1, C1), (A2, B2, C2) how to calculate (x, y, z)

	vec curDir = a1.Unit();
	vec targetDir = a2.Unit();
	float dot = curDir.Dot(targetDir);
	float angle = acos(dot);
	if (angle > maxAngle)
	{
		// calculate axis and angle rotation between directions
		vec axis = (curDir.Cross(targetDir)).Unit();

		// limit angle if necessary
		quat rot; rot.FromAxisAndAngle (axis, maxAngle);
		targetDir = rot.Rotate(targetDir);
	}
    vec result = targetDir;

Instead of creating a quaternion to rotate the initial direction, you could just construct the result using

sin(maxAngle) and cos (maxAngle) times the basis vectors 'axis' and 'curDir.Cross(axis)'.

But i would need to do trial and error to figure out which trig function to use with which vectors, and what signs give the expected result, so i was lazily using the quat.

At first I was confused by the question because your picture kind of looks like a 3D right angle corner.  In any case here's how I might do it:

Take the cross product of (a1.b1.C1) and a2,b2,c2) and normalize the result. This will give you a vector that you can rotate stuff around.

From that vector and the rotation angle you wish to use, in your case that would be Max(Angle, 80), you could build a quaternion or alternatively a matrix that you can use to rotate any points. So for instance if you fed your point (a1,b1,c1) though your matrix or quaternion you would get (x,y,z), assuming they are the same distance from your rotation axis.

There are libraries for matrices and quaternions available, but if you need to know how to build your own I cam post some minimal code.

Thank you very very much.Recently some problems have been encountered, and the knowledge of 3D mathematics is found, but it is regrettable that I am a novice to 3D mathematics but are eager to solve the problem, and the knowledge of 3D mathematics is still not clear. Thank you for reminding me of calculating normals and rotating with Quaternion. Thanks again.

My English is very poor.Sorry╮(╯▽╰)╭