Charlton Castle was once the home of Sir John Charlton, the ‘beloved valet’ of Edward II who was appointed Governor of Ireland in 1338 and served as King’s Chamberlain. Records of a property on the site exist from at least 1174, when a house was apparently leased to the Charlton family by the abbot of Lilleshall Abbey. Sir John, who was knighted around 1307, appears to have inherited the estate at the same time his younger brother Alan succeeded to the family seat of Apley Castle, near Wellington.
In 1316, Charlton was given license by the Crown to crenellate, or fortify, the house as part of a series of improvements that may have been instigated by his wife Hawys (the daughter of Owen ap Griffith – the last Prince of Powys), to whom Charlton was married in 1311. Sir John later received permission to celebrate mass in a chapel on the site in 1341 but, by the time of his death twelve years later, Charlton Castle was said to be worth next to nothing and it seems unlikely that he spent much time there.