We propose a novel unsupervised word alignment method that uses a constraint based on Inversion Transduction Grammar (ITG) parse trees to jointly unify two directional models. Previous agreement methods are not helpful for locating alignments with long distances because they do not use any syntactic structures. In contrast, the proposed method symmetrizes alignments in consideration of their structural coherence by using the ITG constraint softly in the posterior regularization framework. The ITG constraint is also compatible with word alignments that are not covered by ITG parse trees. Hence, the proposed method is robust to ITG parse errors compared to other alignment methods that directly use an ITG model. Compared to the HMM, IBM Model 4, and the baseline agreement method, the experimental results show that, in word alignment evaluation, the IBM Model 4 with the proposed ITG constraint achieves the best performance on the Japanese-English KFTT and BTEC corpus, and in translation evaluation, the proposed method shows comparable or statistically significantly better performance on the Japanese-English KFTT, Japanese-English IWSLT 2007, and Czech/German-English WMT 2015 corpus.