Right, so the rotations are fixed. The issue lie in how the offset rotation was added to the base input.
In short, this is how it works: MainCamera is the one that renders the image. BCamera is the desired position and rotation.
We use a lerp to make MainCamera's position the same as BCamera over a second or so, and make it move to that position in a panning motion.
We can do the same for the rotation, but that caused problems.
Below you ca see the code for the old rotation calculation, which multiplies two quaternions ( the Storage.SumQuaternions() function)). This is because, depending o n the character, MainCamera may have to be upside-down (You have to turn the phone), and so it gets an offset on top of the new position and rotation. That however messes up the movement, and it gets shifted 90 degrees on the Y and Z axis for some reason.
To fix and make the system way more stable I instead placed the offset before the Lerp even happens, so that it's like the BCamera transform was upside down to begin with. That's the endRot variable that you can see.
Now on to fix annoying transparency with the outline shader. Cheers.