Lord Michael Heseltine was one of the defining figures of politics in the 1980s and 1990s.

Now 92, the former defence secretary and deputy Conservative leader is often described as the greatest prime minister Britain never had.

Yet he believes his lasting legacy may be something rather quieter – the arboretum he developed with his wife Ann in Oxfordshire, now home to hundreds of trees, shrubs and a national collection of snowdrops.

His new book, From Acorns to Oaks, explores the arboretum and his two other passions – politics and business.

Lord Heseltine discussed all three during an In Conversation With event at St Andrew’s Church as part of the Farnham Literary Festival on Friday, March 6.

Explaining the inspiration for the arboretum, he reflected on the fleeting nature of political fame.

“Who remembers any politicians?” he asked the audience. “You remember a few prime ministers a hundred years ago, but that’s about it.

“The arboretum in the Midlands is something where people will walk around and say ‘who did this?’ I’ll do my best so there’s some way they can find out it was me.”

During a wide-ranging question and answer session he reflected on his six decades in politics, including his resignation from Margaret Thatcher’s government during the 1986 Westland affair.

“Do I regret it? Yes. Do I believe I was wrong? Absolutely not. Would I do it over again? Yes I would,” he said.

He acknowledged the episode damaged his prospects of ever becoming prime minister. Asked whether he would have liked the premiership, he replied: “Of course. What is the point of climbing the ladder if you don’t take the final step?

“I had a fantastic career and I loved the jobs I was doing. But I would have liked to have had a go at the top job.”

An ardent Europhile, Lord Heseltine also criticised the decision to leave the European Union.

“Brexit is a disaster in every conceivable way,” he said. “It is seriously damaging our economy and seriously damaging our international power base and frankly it is an insult for this country to be on the margins of Europe when we should be at the centre of it.”

He was equally blunt about some of the leading figures behind Brexit.

“Nigel Farage is a fraud – it is quite extraordinary the way people have fallen for it,” he said.

Asked about former prime minister Boris Johnson, who later succeeded him as MP for Henley, he said: “I know Boris well, he took over my constituency, and he’s a charming man.

“If he was here now, he’d have you laughing in fits and rolling about in the aisles.

“But he’s a man without a shred of moral integrity whatsoever.”

During the discussion he also spoke about his long-standing campaign for directly elected mayors in major cities.

“Why would a minister want to see his power spread around?” he asked.

“Do local councillors want mayors to have more power than them? Do local Members of Parliament want mayors who are more important than them?

“The whole system is geared towards preserving the status quo.

“But turkeys don’t vote for Christmas,” he added.

Lord Heseltine also reflected on how the internet has transformed political life.

“When I was in the House of Commons we used to get letters from constituents – 10, 20 maybe 30 letters a day,” he said.

“Now with the internet, the ability for any member of the public to circulate letters to any Member of Parliament is so evident… the flow of public pressure and the speed with which events change is something I never experienced, but I can see what a problem it is.”

Despite that, he suggested the nature of politics itself has changed little.

“There’s nothing new in politics. Read Shakespeare, it’s all there – and I bet he got it from the Greeks,” he said.

“It’s not the easiest place to be, it’s very demanding. But they used to cut people’s heads off – I certainly would have been executed in the past.”

Returning to the arboreal theme, he was asked what kind of tree Sir Keir Starmer would be.

“It would have to be a willow – blowing with the wind,” joked Lord Heseltine.

Farnham Literary Festival continues until March 15, with more information on farnhamliteraryfestival.co.uk