The Frasers probably come from Anjou in France, and the name may derive either from Fredarius, from Fresel or from Freseau. It has also been suggested that they descend from a tribe called Friselii in Roman Gaul, whose badge was a strawberry plant.

They first appeared in Scotland around 1160, when Simon Fraser held lands at Keither in East Lothian. About five generations later Sir Simon Fraser was captured fighting for Robert the Bruce, and executed with great cruelty by Edward I in 1306. His cousin, Sir Alexander Fraser of Cowie, Bruce's chamberlain, was the elder brother of another Sir Simon Fraser, from whom the Frasers of Lovat descend. He married Robert the Bruce's sister, Mary, who for a time was imprisoned by the English in a cage hung from Roxburgh Castle wall. Simon's grandson, Sir Alexander Fraser of Cowie and Durris, acquired the castle now called Cairnbulg and the lands of Philorth by marriage with Joanna, younger daughter and co-heirness of the Earl of Ross in 1375. Eight generations later, in 1592, Sir Alexander Fraser of Philorth received from James VI charters creating the fishing village of Faithlie, which he had transformed into a fine town, also improving the harbour, which became a burgh of regality and a free port, called Fraserburgh. He was also authorised to found a university in the town, but the scheme was short-lived, falling victim to the religous troubles of the times. When plague hit Aberdeen in 1647, staff and students from King's College were evacuated to the town for two years, but now a street name and some lettering carved on a wall are all that remain of the early seat of learning.

The eighth Laird of Philorth also built Fraserburgh Castle, later the Kinnaird Head Lighthouse, and in doing so bankrupted himself, being forced to sell the Castle of Philorth, which passed out of the family for over three hundred years until the nineteenth Lord Saltoun bought it back in 1934.

The ninth Laird married the heiress of the Abernethy Lords Saltoun, and their became the tenth Lord Saltoun. The present chief, as well as bearing the undiffenced arms of Fraser, has also a 'grand coat' which quarters the arms of Abernethy, Wishart and Ross. The tenth Lord Saltoun was severely wounded at the Battle of Worcester in 1651, but survived thanks to his servant, James Cardno, who rescued him from the battlefield, hiding and nursing him until he finally got his master home to Fraserburgh. In 1666 he built a house a mile from Fraserburgh which he called Philorth House, where the family lived until it was burned down in 1915. Sir Alexander Fraser of Durris was personal physician to Charles II. He was educated at Aberdeen and soon acquired a reputation for general scholarship but particularly in medicine. He accompanied the King throughout his campaign in 1650, but seems to have been unpopular with some of the more extreme Covenanters because of his progressive scientific opinions. After the Restoration in 1660, he sat in the Scottish Parliament although he was still prominent enough in court circles to feature in the diaries of Samuel Pepys. He died in 1861. The family took no part in the Jacobite rebellion.

The sixteenth Lord Saltoun commanded the Light Companies of the First Guards in the Orchard at Hougoumont on the morning of the Battle of Waterloo in 1815. It was he who, later in the day, first noticed the Imperial Guard emerge from the Hollow where they had been hiding all day, and drew the Duke of Wellington's attention to them. The nineteenth Lord Saltoun was a prisoner of war in Germany for most of the First World War. He became a member of the House of Lords from 1936 and devoted himself to numereous public works, and latterly to promoting the Royal National Lifeboat Institution (RNLI). He died in 1979 at the age of 93.

Other branches of the family also prospered. Andrew Fraser of Muchalls was raised to the peerage in June 1633 with the title, 'Lord Fraser'. He completed the early work on Castle Fraser which stands south of the River Don near Inverurie. This magnificent castle, entirely reminiscent of a French chateau, is now fully renovated and in the care of the National Trust for Scotland. The peerage conferred on Andrew Fraser fell dormant in the eighteenth century. The nineteenth Lord was succeeded by his daughter, Flora Fraser, twentieth Lady Saltoun. She is married to Captain Alexander Ramsay of Mar, a great-grandson of Queen Victoria, and he and Lady Saltoun are officially members of the royal family.