James Douglas was born in 1803 in Demerara (later part of Guyana). His father was John Douglas, a Scottish planter and merchant from Glasgow, in business with three of his brothers. The Douglas family were part of "the business elite of Scotland". His uncle was Lieutenant-General Sir Neil Douglas, Commander-in-Chief, Scotland. His aunt was Cecilia Douglas, a notable art collector. Through his paternal grandmother, James Douglas was related to Sir Andrew Buchanan, 1st Baronet, diplomat.
His mother was Martha Ann (née Ritchie, later Telfer), who was a Barbadian Creole. Ritchie was classified as free coloured, which in that time and place meant someone of mixed African and European family history, who was not a slave. According to the Dictionary of Canadian Biography, the couple had three children together (Alexander, born 1801 or 1802; James, born 1803, and Cecilia, born 1812), but never formally married.
In 1812, John Douglas returned to Scotland with his children and put James into school at Lanark to be educated. John married Jessie Hamilton in 1819, and had more children with her, making a second family. James went to school or was taught by a French Huguenot in Manchester, England, where he learned to speak and write in fluent French, which helped him in North America.
Source: https://en.wikipedia.org/wiki/James_Douglas_(governor)