Let $f,g:X\to Y$ be maps between two closed oriented $n$-manifolds. When $Y$ is an infra-solvmanifold, necessary and sufficient conditions are given for the equality between the Nielsen number $N(f,g)$ and the Reidemeister number $R(f,g)$. The proof makes use of certain residual property of virtually polycyclic groups and the following factorization result: let $\pi$ be a finitely generated torsion-free virtually polycyclic group. For any finitely generated group $G$, there exists a finitely generated torsion-free virtually polycyclic group $\bar G$ and an epimorphism $\epsilon: G \rightarrow \bar G$ such that for any homomorphism $\varphi:G\to \pi$, there exists $\bar {\varphi}:\bar G \to \pi$ such that $\varphi=\bar {\varphi}\circ \epsilon$.