The core of the castle was built in the half of the 14^th century as a watch, a defense and a housing tower. The castle had a  very good location. Its task was firstly to defend the trade route, which connected the two towns: Sušice and Horažďovice and secondly, to defend gold washing along the river and in the gold-bearing river Otava.

The first written report we possess, goes back to year 1380, when the legal owners were members of the most significant and influential Bohemian noble family - "The Rosenberg family". The family Švihovský of Rosenberg bought the Castle or gained through matrimony from the noble family of Velhartice sometime during 14^th century. At the turn of the 15^th century the Castle was rebuilt and fortressed. Since then it has become one of the most massive and safest castles in Bohemia. It happened most likely because of battles between supporters of king Wenceslas IV and members of the seigneurial unity, to which the owner of the Castle, Břeňek Švihovský of Rosenberg, also belonged.   

Jan of Rosenberg obtained the whole estate with the Castle in year 1407. Jan (John) of Rosenberg was known as a political public figure and later as an enemy of the Hussite movement.That is why the Hussite armies twice besieged the Castle and twice seized, namely in 1420 and 1421.  Jan Žižka lost his second eye in the second siege of the Castle. This event put Rabi Castle on the map.

The Castle passed to Wilhelm Junior of Rosenberg after death of John of Rosenberg and then the Castle fall to his nephew, Půta II Švihovský of Rosenberg. Půta Švihovský was enormously rich and his fortune and high rank in society (highest provincial) enabled him to rebuild magnificently his Castle. The reconstruction conducted skillful architect, Benedict Rejt. Massive walls with advanced artillery bastions surrounded the Castle. Půta was very wealthy, but not his sons any more. His sons, who had the Castle since 1504, were unfortunately indebted and were forced to sell off their assets. In 1548, they had to sell even the Rabí Castle.

In the following times, many holders were living in the Castle - Henry Kurcpach of Trachenberg, Diviš Malovec of Libějovice, Wilhelm Rosenberg. Last important possessor was Adam Chanovský of Dlouhá Ves - his family had the Castle until beginning of 18^th century. After Thirty Years War went the building to noble family of Lamberg, but the Castle was already desolated in this period.

The final destruction caused a fire in 1720, afterwards served the Castle as a source of building stone for inhabitants of villages. The aristocratic family of Lamberg gave the Castle to "Society for the preservation of artistic, historical and natural monuments" in 1920. The symbolic price was 1,- Kč. In this time, was the Castle repaired and rescued against total collapse. In 1945 is the Castle nationalized and in 1978 is proclaimed as National Cultural Monument.