The businessman and podcast host survived five days on the ITV reality show before he was evicted after a public vote, as he was nominated alongside The Real Housewives Of Cheshire star Lauren Simon.
In the Friday episode, Goldsmith, 58, said that he had not found “true peace” during one Big Brother task.
Gary Goldsmith. Victoria Jones/PA
“I think peace is something always to be striving for,” he said.
“I thought I was having a really cool life until something hits you. One day I was Gary, a businessman having a laugh with a great family in a good place.
“And the next minute I’m a national villain. If any of you thought to Google me, it is just shit, but it’s not me.
“So, I don’t really know what true peace is if I’m honest.”
We need your consent to load this Social Media content. We use a number of different Social Media outlets to manage extra content that can set cookies on your device and collect data about your activity.
The brother of Kate’s mother, Carole Middleton, was fined £5,000 and given a community order in November 2017 after admitting he assaulted his wife in a drunken argument.
Goldsmith’s inclusion on the show was previously criticised by Women’s Aid, who advised the programme to consider his appearance.
In a statement from the show, Goldsmith previously said: “I’m not perfect and I made a mistake, a personal row seven years ago which was publicly exposed that I’ve worked endlessly for seven years to put right.
“The offence is long since spent, however my regret continues.”
Bradley Riches, one of the contestants in this year’s Celebrity Big Brother, has said his biggest regret was not being proud of himself(Ray Burmiston/ITV)
Elsewhere, Heartstopper star Bradley Riches said his “biggest regret” was not being proud of himself for being gay and autistic during his teenage years.
And former X Factor star Sharon Osbourne chose Dragons’ Den star Levi Roots as the housemate who inspired her the most, because he had found “serenity”.
Roots said he has not managed to heal wounds from when his two children came out against him in the media, and “in the back of my mind I’m waiting for my two girls to knock on my door”, he added, as he became emotional.
Also in the episode former X Factor judge Louis Walsh and Simon were berated for breaking the house rules for “speculating or guessing on who has nominated whom or who might nominate whom in the future”, Big Brother told the housemates.
Zeze Millz clashed with Real Housewife of Cheshire Lauren Simon (Ian West/PA)
It was disclosed to the wider group that Simon had said YouTuber Zeze Millz “hates me”, and they later had an argument.
Walsh and Simon were placed in the “wheelie bin of shame” and had to write a letter of apology to their fellow housemates.
Celebrity Big Brother continues on Friday at 9pm on ITV1 and ITVX.