I am trying to recover the 3D orientation of the three axes of a building in an image.
My current approach is to manually annotate the three vanishing points corresponding to the three axes, estimate the camera calibration matrix K, and then compute each column of the rotation matrix from a vanishing point by normalising K^{-1}v_{i}, as outlined in the appendix of this paper.
Although this seems to work, the computed columns of R are not completely orthogonal, and I suspect the problem would be even more pronounced if the vanishing points are more noisy.
I could fudge it by decomposing the estimated R using SVD and setting all singular values to 1, but I would prefer a more principled solution. Another line of thought is to solve it as a constrained optimisation problem that seeks to minimise projection error on the vanishing points.
Is there a better way of computing the rotation matrix from K and the vanishing points, that simultaneously take into account of orthogonality and column unity constraints?
Thanks in advance.