Shadow justice secretary Robert Jenrick has defended remarks he made in March about the Handsworth area of Birmingham, calling it “one of the worst-integrated places” he had ever been to.

In a recording reportedly made during a dinner and published by the Guardian, Jenrick said he had not seen “another white face” in the hour and a half he spent in Handsworth filming a video about litter.

Jenrick stood by his comments on Tuesday, saying he had no regrets about the language he used.

Labour Party chair Anna Turley criticised Jenrick, saying his comments reduced “people to the colour of their skin”.

Handsworth’s Independent MP Ayoub Khan said the remarks were “not only wildly false but also incredibly irresponsible”.

Conservative leader Kemi Badenoch said she did not know the context of the recording but that Jenrick may have been “making an observation” about his visit to the area.

“I wasn’t there so I can’t say how many faces he saw, but the point is that there are many people in our country who are not integrating,” she told BBC Breakfast on Tuesday, adding she was “very worried about what is happening in Birmingham”.

The authenticity of the recording at the Aldridge-Brownhills Conservative Association is not disputed by Jenrick’s team.

In the recording, he goes on to say: “That’s not the kind of country I want to live in. I want to live in a country where people are properly integrated.

“It’s not about the colour of your skin, or your faith, of course it isn’t. But I want people to be living alongside each other, not parallel lives.”

Asked on BBC Radio 5Live on Tuesday whether he regretted the comments made in the recording, Jenrick said: “No not at all and I won’t shy away from these issues.”

“It’s incredibly important we have a fully integrated society”, he said.

“It’s a very dangerous place if we have a country where people are living in ghettoised communities, where people are not living together side-by-side in harmonious communities. We’ve seen the damage that can do in our society,” he said.

“We’ve had major failures of integration in this country for my whole lifetime. We’ve got to fix it, and that’s the comment I was making in Birmingham the other day.”

Responding to the recording, Labour’s Turley said: “This weekend Kemi Badenoch said she stood against a politics that ‘reduces people to categories and then pits them against each other’.

“Robert Jenrick in his leaked comments reduces people to the colour of their skin and judges his own level of comfort by whether there are other white faces around. His comments clearly cross a red line that his leader has rightly laid down.

“People of colour should not have to justify their Englishness, or their Britishness, or their presence in this country, to Robert Jenrick or anyone else.

“Robert Jenrick needs to urgently explain himself and why these comments are in any way compatible with what his party leader said.”

Asked if the number of white people seen in an area is the right measure for integration on BBC Radio 4’s Today programme, Badenoch said: “The right measure for integration is that people don’t care what people look like.”

She added: “We are a multiracial country. That means we have to work harder to bring people together.”

She said Jenrick was “making a point which I don’t have the context of”.

“I think we should look at these things in the spirit of what was intended, which I believe knowing Rob and hearing him speak, is that he wants, as I do, a country that is well-integrated”.

Jenrick is due to address the Conservative Party’s annual conference on Tuesday, when he will set out plans to put ministers in charge of sentencing policy.

Khan told the Guardian that Jenrick had “misrepresented a storied and diverse community, awkwardly distorting the product of an all-out bin strike to fit his culture-warrior narrative filled with far-right cliches”.

Former Conservative Mayor of the West Midlands Andy Street told BBC Newsnight: “Putting it bluntly, Robert is wrong.”

“Handsworth, it’s come a hell of a long way in the 40 years since the last civil disturbances there and it’s actually a very integrated place,” he continued.

Street also rejected Jenrick’s recorded comment that Handsworth was “the closest I’ve come to a slum in this country”.

The former mayor noted the “incredible hope, optimism and people taking part in education which is based around British values and thinking how they can make a contribution to the future of their region their city and their area.”

“That is not a definition of a slum,” the former Conservative mayor said.