In 1177, Henry's oldest child Henry the Young King rebelled over control of lands allotted to him. The 1st Earl of Norfolk supported the rebellion leading to Orford Castle becoming heavily garrisoned. Upon the collapse of the rebellion, Henry ordered the confiscation of Framlingham Castle.

Orford Castle was not as important after Henry's death in 1189. The castle was captured by Prince Louis of France who invaded England in 1216 at the invitation of the English barons who were disillusioned by King John

In 1280 under the rule of Edward I Orford Castle was granted out and eventually sold. This marked the beginning of the end of Orford Castle, as the estuary of the River Alde silted up and trade declined, reducing the importance of the castle as the centre of local government.

Orford was a borough from 1579 to 1886 with parliamentary representation between 1483 and its abolition under the Reform Act 1832. There was also an Earldom of Orford, the most notable of whom was Sir Robert Walpole, the first Prime Minister of Great Britain.