At the top of the hill of Puig de Sant Andreu, in what was once the sacred enclosure, a castle was built in the Carolingian period of which parts of the wall remain, and the bases of the four towers that stood at each corner. This medieval castle was relatively small, with a trapezoid plan, and two ditches to reinforce its defences. According to the scant surviving Catalan documents from the 9th and beginning of the 10th century, the castle was known as Castellum Vellosos (it was mentioned by Louis the Pious in 834) and is one of the oldest military constructions in the Empordà region.
The erection of a medieval castle in this location must be related with its elevated strategic position, as it could be used to observe a large expanse of territory. To the south of the castle there is the so-called Carolingian Tower, which was built on the foundations of a former Iberian round tower, and from which it is possible to observe a large swathe of the surrounding countryside.
In the 16th century, the sanctuary of Sant Andreu de l’Estany was built over the remains of the abandoned castle. The nave of this chapel today forms part of the architecture of the museum facilities dedicated to the site.