Given the up vector of (0, 1, 0) and a direction vector (such as 0, 0, 1 being forward of 0, 0, -1 facing left) how could I construct a rotation vector (Euler angles) from the direction vector?

This sounds like a math homework question but it is not. I tried to break it down into sections that I could dissect to solve this. I could not though

It's clearly not homework because the question is wrong : )

If you have given current and target directions, you don't need an up vector. (Just calculate axis and angle from the directions to get the rotation).

If you have given target direction and up vector, you don't need a current direction. (Just construct orthogonal frame from the two given vectors to get an orientation).

Finally, i think ‘rotation vector’ means axis times angle (concept e.g. used in physics to represent angular velocity or torque), while Euler angles results in 3 rotation in order (mostly only used for user interface purposes)

Probably you do not want to use Euler angles. Why do you ask for them?
And think again about what you need, or describe what you want to achieve. Also, what tools do you have? (Matrix library, quaternions…?)

What I have is a object going from one point to another point over time. When it decides to go to another point I want to orient the object in direction it is going.

What I have is my own engine and math library. I thought it would be possible to take the direction the object is moving and form a rotation vector. That way I could plug the rotation vector into my transform matrix. My transform matrix a Translation vector, Rotation vector and Scale vector. I thought I would need the vector as Euler angles for my transform construction

judeclarke said:
What I have is a object going from one point to another point over time. When it decides to go to another point I want to orient the object in direction it is going.

This reminds me on an older post, where the question was to make a turret rotating towards a goal. I add my proposed solution below. (Originally this also had limits so the turrets orientation is restricted - i have removed this code sections and hope i did not mess it up somehow.)

I guess it's pretty confusing so here a description on how it works. Your problem is similar to the process of constructing a ‘look at matrix’, which would be easy. But doing the rotation gradually over time makes it harder.

1. Calculate the axis and angle between your objects forward direction and the vector from object center to the target point.
   Notice that axis and angle is the most natural way to imagine a 3D rotation. Spread two fingers and imagine the axis one finger would need to rotate so it matches the other.
   Now imagine a triangle from the (almost) common root of the fingers to the two finger tips. You get a triangle, and the axis of rotation is the same as the triangle normal. (The rotating finger remains on the plane of the triangle).
   So it becomes obvious how you can calculate axis with cross product of the two vectors and angle from their dot product.
2. relate the angle to some maximum speed you want the object to turn, and reduce it if necessary. (I also added some smoothing so it behaves more natural)
3. Convert your axis and angle representation into a rotation matrix or a quaternion (like i did below), so you can apply it to your objects transform data.
4. Rotate the object, and optionally constrain its up vector if desired. I did this by constructing orientation matrix manually just like usual when we need a ‘look at’ matrix.
5. repeat all this in the next frame until the object is nicely ‘looking’ at the target.

If you're new to 3D rotations as it seems, this is not easy to begin with. But it's often necessary, so worth to learn. Expect to invest some time.

judeclarke said:
My transform matrix a Translation vector, Rotation vector and Scale vector.

It sounds like you have a SRT representation of transforms. (scale, rotation in euler angles, translation - given as 3-vectors)
This is useful e.g. for a scene graph in a game editor, because a human can edit the number intuitively.
But it is not optimal for a game runtime. Becasue euler angles are hard and slow to work. Aslo notice SRT is not yet a matrix.

The typical solution is to convert your SRT hierarchy to 4x4 transfrom matrices (you likely do this already so you can render your stuff),
then use the matrices to solve problems as discussed, and finally convert results back to SRT if needed.

When working with 3D rotations it is key to understand that matrices already contain front-, up-, and right-(or left) vectors to define orientation, which makes things easy that would be super hard when using euler angles.

{
	struct Turret
	{
		// orientaton convention: X axis: front (turret shooting direction), Y axis: up, Z axis: side
		quat orientation;
		vec position;
		
		quat mountOrientation; // e.g. the ground or a vehicle frame, we use it to define up vector and angle limits

		Turret ()
		{
			orientation.Identity();
			mountOrientation.Identity();
			position = vec(0,0,0);
		}
	};
	static Turret turret;

	static vec target (10, 3, 0);
	ImGui::DragFloat3 ("target pos", (float*)&target, 0.01f);

	vec curDir = turret.orientation.Rotate(vec(1,0,0));
	vec targetDir = vec(target - turret.position).Unit();

	float dot = curDir.Dot(targetDir);
	if (fabs(dot) < 1.0f - 1.0e-5f) // otherwise axis would be result of division by zero
	{
		// calculate axis and angle rotation between directions
		float angle = acos(dot); // Notice this is always positive (the direction of the axis defines if we rotate left or right)
		vec axis = (curDir.Cross(targetDir)).Unit();

		// optional: some damping to make it smooth. This simple approach assumes constant timestep and needs to be adjusted to this value. (for 60 fps try 0.3)
		const float damp = 0.03f; 
		angle *= damp;

		// limit angle if necessary
		const float maxAngularVelocity = 30.0f / 180.0f * 3.14f; // e.g. 30 degrees per second
		const float timestep = 1.0f / 500.0f; // i have 500 fps
		float maxAnglePerUpdate = maxAngularVelocity * timestep;
		if (angle > maxAnglePerUpdate) angle = maxAnglePerUpdate;

		// finally rotate turret
		quat rot; rot.FromAxisAndAngle (axis, angle);

		if (0) // no upvector, ok for a spaceship
		{
			turret.orientation = rot * turret.orientation;
			turret.orientation.Normalize();
		}
		else // using upvector, turret side axis remains on the up plane defined by mountOrientation
		{
			const vec globalUp = turret.mountOrientation.Rotate(vec(0,1,0));
 // if turrets already aligned to up vector, could just use vec(0,1,0) and no need to rotate relative from its mounting base.
			vec newTurretDir = rot.Rotate(curDir);
			if (fabs( globalUp.Dot(newTurretDir) ) < 1.0f - 1.0e-5f) // turret would be clueless if pointing exactly up / downwards
			{
				vec side = vec(globalUp.Cross(newTurretDir)).Unit();
				vec localUp = newTurretDir.Cross(side);

				matrix temp;
				temp[0] = newTurretDir;
				temp[1] = side;
				temp[2] = localUp;
				temp.Rotation()->ToQuat(turret.orientation);
			}
		}
		
	}

	// visualize
	RenderLine (target, turret.position, 0.5f,0.5f,0.5f);
	RenderQuat (turret.position, turret.orientation);
	RenderQuat (turret.position, turret.mountOrientation, 0.2f);
	RenderPoint (target, 1,1,1);
	RenderCircle (1, turret.position, vec(0,1,0), 0,1,0);
}

Thanks that is helping. I am starting to understand this more and the 3D math I worked with a long time ago is making more sense again. With the scenario you have below though, what happens when you are looking forward (0, 0, 1) and then immediately look backwards (0, 0, -1)? Doing Current (dot) New results an angle equal to 3.14. Did I interpret this wrong?

Vector3 currentDirection;
currentDirection.x = transform._31;
currentDirection.y = transform._32;
currentDirection.z = transform._33;
currentDirection.Normalize();

F32 dot1 = Vector3::Dot(currentDirection, direction);
F32 angle1 = acos(dot1);
Vector3 axis1 = Vector3::Cross(currentDirection, direction);

Vector3 rotation = axis1 * angle1;

judeclarke said:
what happens when you are looking forward (0, 0, 1) and then immediately look backwards (0, 0, -1)?

I see i forgot to handle this case. Angle is PI, but the axis is zero and can't be normalized.

You could use any axis that is perpendicular to either (so both) directions, e.g. using a function like this:

const sVec3 ArbitaryTangent (bool normalize = true) const
	{
		const sVec3 &norm = *this;
		sVec3 ref;
		float x = norm[0]*norm[0];
		float y = norm[1]*norm[1];
		float z = norm[2]*norm[2];
		sVec3 temp;
		if (x<=y && x<=z)		temp = sVec3(1,0,0);
		else if (y<=x && y<=z)	temp = sVec3(0,1,0);
		else					temp = sVec3(0,0,1);
		temp = norm.Cross(temp);
		return (normalize ? temp.Unit() : temp);
	}

I don't understand what you mean by

You could use any axis that is perpendicular to either (so both) directions, e.g. using a function like this:

How does this function fit into the context of the code and algorithm mentioned above?

Just the expression “rotation vector” rubs me the wrong way. Rotations do not form a vector space: They form a Lie group. The tangent space at the identity is the space of angular velocities, and that one is a vector space.

[/rant]

alvaro said:

Just the expression “rotation vector” rubs me the wrong way. Rotations do not form a vector space: They form a Lie group. The tangent space at the identity is the space of angular velocities, and that one is a vector space.

[/rant]

I realized I should have said “Euler angle from direction vector” in the title but I am unable to edit it now

judeclarke said:
How does this function fit into the context of the code and algorithm mentioned above?

Now i see i did not forget to handle the case : )

It's in those lines:

	float dot = curDir.Dot(targetDir);
	if (fabs(dot) < 1.0f - 1.0e-5f) // otherwise axis would be result of division by zero
	{
		// calculate axis and angle rotation between directions
		float angle = acos(dot); // Notice this is always positive (the direction of the axis defines if we rotate left or right)
		vec axis = (curDir.Cross(targetDir)).Unit();

if curDir and targetDir lie on a similar line (independent if they are almost the same or opposite directions), tha absolute valuie of their dot product is close to one, and in this case my code does nothing.
So this check handles both 0 and 180 degrees cases.

But that's not always what we want. If we still want to rotate one direction to the other,

e.g. (0, 0, 1) to (0, 0, -1) like in your example, we could use (0,1,0) or (1,0,0) for the axis, or any other unit vector with z being zero, and an angle of 180 degrees.

The resulting rotation always succeeds to align the given two directions, no matter which axes we choose. It only has to be perpendicular (or tangential) to the line formed by the given two directions.
But it is important to notice the resulting orientation after the rotation is different for each of those potential axis! Just the two directions match.

So it is often not clear which axis is better than another, so we just construct any tangential axis direction, which could be seen as a random solution in some way.
It wood look like this, for example:

float dot = curDir.Dot(targetDir);
if (dot > 1.0f - 1.0e-5f) // directions already match - do nothing
{
	return;
}
else
{
	vec axis;
	if (dot < -1.0f + 1.0e-5f) // special case: current and target directions exactly opposite of each other
	{
		axis = ArbitaryTangent(targetDir); // pick any axis perpendicular to both
	}
	else
	{
		axis = (curDir.Cross(targetDir)).Unit();
	}
	
	float angle = acos(dot);
	//…