In the 19th century the island was an object of dispute between the Naval Ministry that wanted it for a rescue station, and the Finance Ministry that wished to set up a customhouse. Called at the time Rats Island, it was entrusted to the Finance Ministry. In 1881 the customhouse started to be built, a design of engineer Adolpho José Del Vecchio. Soon after, the island received the visit of D. Pedro II. Delighted with the superb view of the bay, the Emperor is said to have defined it as “a delicate jewel box, worthy of a dazzling jewel”. Thus Del Vecchio, admirer of the gothic, created a castle inspired by the 14th century French buildings. His design was awarded the Gold Medal at the Escola Imperial de Belas Artes (Imperial Fine Arts School) exhibition.
On 27 April 1889 the building was inaugurated in the presence of the Emperor who was transported to the island on the famous Galeota Imperial (Imperial Galley), currently displayed in the Espaço Cultural da Marinha. The highlights of the castle are the exceptional carved stonework, by Antônio Teixeira Ruiz, the floor of the turret in mosaics made of over ten different species of hardwood, by Moreira de Carvalho and the beautiful work in cast-iron by Manuel Joaquim Moreira. Also remarkable are the wall paintings by Frederico Steckel.