The castle of Passy-les-Tours (private property) has been listed as a historic monument since 28 June 1927.
Built on a barlong plan of about 50 m on each side flanked by four round towers, the castle was probably built at the end of the 14th century. In 1367 we find a Jean de Beaumont lord of Pacy. The castle is bought in 1400 by Jean de Chevenon, then adviser of Charles VI of France, and claimed richest squire of the kingdom, it is surely him who modifies the keep. With a height of 25m it allowed to monitor the main access to Charité-sur-Loire: the current N151. Its architecture is inspired by the castle of Vincennes and its imposing towers give their name to the village.
During the Hundred Years' War, from 1422, it was occupied by Captain Perrinet Gressard who stood up to Joan of Arc during the siege of La Charité-sur-Loire in 1429.
Heavily damaged by the Wars of Religion, the castle became the property of the Marquis de Vergennes, a politician and diplomat, in 1782, who did not restore it any more than his successive owners did. On the contrary, its stones were used for other constructions