The town dates back to the 5th century, although Prehistoric evidence suggests earlier Bronze Age and Roman Settlements in the area. In 1177, the Lord Rhys of Deheubarth constructed a castle in the town, which sat at the very edge of his lands in order to resist Norman invaders, and in 1194 he rebuilt it, making it stronger and more durable to invasions. This castle was known as the Castle of Gwrtheyrnion, and like many castles in Wales passed hands between various Welsh princes and lords and the Norman invaders innumerable times until it was destroyed by fire in 1231 by soldiers from North Wales.
The centre of the town forms a crossroads between important routes from north to south and east to West Wales, and is marked by the clock tower which was an important 19th centrury staging post on the famous coach road from Aberystwyth to London. During the 18th and 19th century sheep and cattle drovers crossing the Cambrian Mountains on their journey towards the English market towns of Banbury, Hereford and London would often stop at Rhayader for lodging and provisions.
Its strategic position in the heartlands of Wales and at a crossroads meant that during the nineteenth century, no fewer than six toll gates were placed on the roads in and out of the town. This made journeys bringing animals into market expensive and became a real burden on the hard lives of the poor. When prices of stock fell and harvests were poor, these tolls became an impossible burden. As a result, between 1839 and 1844 the area witnessed mass rioting - known as the Rebecca Riots when local people, angered by the increasing financial pressure followed a group of local tenant farmers and workers who dressed as women, known as Rebecca and her daughters. It is thought that the idea for disguise came from the Bible. In Genesis Rebecca recommended that some "possess the gates of those which hate them".