The building was once owned by the Holy Roman Emperor, Carlos I, who sold it to Arias Pardo de Saavedra, Mariscal de Castilla, for the enticing sum of 42 cuentos and 24,572 maravedíes, along with a stipend of 3,000 and, inevitably, 7 sheep per year.
The four towers were added in 1697.
The Spanish dictator Francisco Franco briefly employed it as his residence at the end of the 1930s.