Having fallen under French rule in 1499, the Duchy of Milan would be fiercely contended by the Habsburgs, the French and Sforzas for the following 30 years. They were difficult years for both the city and the castle. During the French occupation the Filarete Tower, which had been repurposed as a munitions store, exploded causing damage to the surrounding walls and killing the castle governor.

When Francesco II Sforza, il Moro's second son, took power the castle was repaired but remained without a tower over the city-side gate. On the 3rd March 1534 it was the setting of the marriage between Francesco II and Cristina, the daughter of the King of Denmark.

Following this grandiose event, the building would never again serve as the ducal residence. The last of the Sforzas bequeathed the Duchy of Milan to Emperor Charles V.