The palace was founded by Juan de Valvidares and his wife, Cecilia Estrada. Both their families were the greatest landowners of the region at the time.

The coat-of-arms that presides over the front of the palace is one of alliance of the surnames of Cecilia Estrada's parents, don Bernardo de Estrada y Nava, of Bimenes, and doña María Fernández de la Rosella, of Langreo. The individual coats of arms are described, from a heraldic point of view, in tha following way:

The surname Estrada, of which one of the branches of settled in what is now the Council of Bimenes, has its origin in a locality close to San Vicente de la Barquera, which in those times was part of Asturias, more concretely to what was known as "The Asturias of Santillana" or what is the same, "The mountains of Santillana". The surname as "way" or "road" is probably derived from what was the road par exellence at the time, "The Road to Santiago", one of the main roads travelled by the faithful in their pilgrimage to the tomb of the Apostle Saint James, in Santiago de Compostella, in Galicia.

The word estrada is derived from the latin strata and is in active use, with some evolutive changes, in many languages with latin roots, including Spanish. It is the equivalent of "road" and this equivalency is maintained in the chapel of the Estradas, dedicated to "The Virgin of the Road".

One of doña Cecilia's brothers, don Gerónimo de Estrada Fernández de la Rosella y Nava received a commission from His Majesty King Phillip V of Spain, to perform certain services to the Crown in New Granada, what is now Colombia. At the end of his commission, don Gerónimo was rewarded with the title of Marquis of the House of Estrada, which was granted on the 13 of October, 1704. He married doña Juana de Angulo, natural of Cartagena de Indias and when he returned from overseas he fixed his residence in Cádiz. He continued his life in Andalucía where his descendants still live. However, apparently it was his wish that, upon his death, he be buried in the land of his birth, and following those wishes his remains now rest under the main altar of the Chapel of the Virgin of the Road, beside the palace. In two tombstones that evoque his memory at each side of the altar, one can read the family motto: "This is the House of Estrada / founded on this rocky outcrop / older than Velasco / and to the king I owe nothing".

His son, Diego de Estrada y Angulo, maried in 1712 to María Panés Viganiego, of genovese ascendancy and related to the Marquis of Villapanés, title that was inherited by his great grandson Juan de Dios de Estrada y Villalón, born in 1773. The son of this last one, Juan Antonio de Estrada y González de Sepúlveda, inherited the title of Marquis of Torreblanca of Aljarafe and was named Grandee of Spain (the equivalent of a peer) in 1865.