Ballantrae was once noted as a smugglers' haunt. In early times the smuggling of tea, tobacco and brandy formed one of the staple industries of the village.

Large pirate vessels called Buckers were accustomed to lie in the bay and discharge their cargoes at Ballantrae. The arrival of one of these vessels was the signal for the smugglers, who were called lintowers. They numbered about one hundred men, armed with cutlass and pistol ready to fight if resistance was offered them in the discharge of their illegal calling.