Between 1360 and 1370, having become the Lord of western Milan, Galeazzo Visconti II built a fort straddling the medieval wall and encasing the postern of Porta Giovia or Zobia. His successor, Gian Galeazzo, added barracks for the troops in 1392. The two parts of the building were divided by a medieval defensive structure, the so-called dead moat and would only be united under the reign of Filippo Maria, the last of the Viscontis. It was in this period that the castle, the largest of those built by the Viscontis, was adopted as the family's official residence. At the time, the square plan castle was 180 metres in length, fortified with four square towers and delimited by a defensive wall. The uncultivated fields to the north-west were transformed into a large garden known as a “zardinum” or “barcho”.

The last member of the dynasty spent a solitary existence here, confining himself with his court within his immense abode, where he died leaving no heirs. His only daughter, Bianca Maria, legitimised by Emperor Sigismund in 1426, was married to the military leader Francesco Sforza, upon whom Filippo Maria Visconti had relied to defend the Duchy from the Venetians.

In 1447, upon the death of Filippo Maria, the people of Milan proclaimed the Ambrosian Republic and partially damaged the residence of Porta Giovia, the emblem of Ducal power.

The grey Serizzo stone batters in the dead moat and on the external walls of the Rocchetta Keep and the Ducal Court are today all that remain of the Visconti fortress.