The latest incident is the second time this week Mr Charles, 72, has been refused a taxi.

On Wednesday, just moments after he was named Victorian Senior Australian of the Year Mr Charles was refused a taxi in Docklands in Melbourne unless he paid upfront.

Mr Charles' manager Patrice Capogreco, who was with Mr Charles on both occasions, said the latest incident left him "very distressed".

"He is seeking legal advice on taking further action against the drivers involved as well as the taxi company," Ms Capogreco said.

"He just wants to see some action now."

Ms Capogreco said the latest incident happened at Melbourne Airport at 3pm when the pair were returning from Adelaide, where Mr Charles gave a keynote speech at a Jobs Australia event.

"We were in line at the taxi rank and the taxi attendant allocated us a bay," she said.

"We went to the bay where a taxi was waiting. Uncle Jack was walking ahead of me, and as Uncle Jack approached the taxi ... he was about to open the door ...it took off."

Ms Capogreco said the attendant was horrified and "very quickly" arranged another taxi.

But, she said, the incident had left a mark on Mr Charles.

"Uncle Jack is such a resilient, strong man, but this has really knocked him for six," Ms Capogreco said.

"Twice in three days - it is unacceptable for him, for anyone."

She said Mr Charles believed he was refused the cabs because of his race.

"Uncle Jack knew on Wednesday night that the taxi won't stop for him but will stop for me," Ms Capogreco said.

"This afternoon it was Uncle Jack who approached the cab, and it took off.

"If it was me, would it have taken off? No, because it is racial discrimination."

She said what made the matter worse was Mr Charles' stature in the Aboriginal community.

"He is a role  model," Ms Capogreco said.

"He continues to go on even when he is knocked down like this.

"He will be back on a plane again on Sunday to Sydney to do more filming. He is not going to let this stop him, but it certainly had a large impact on him this afternoon, as it did on Wednesday afternoon."

Taxi Service Commission chief executive Aaron de Rozario said no law or regulation in Victoria gave taxi drivers the right to pick and choose fares based on race, colour, national or ethnic origin or immigrant status.

"This type of behaviour is unacceptable and illegal," Mr de Rozario said.

"The TSC is investigating Uncle Jack's case as a matter of priority. A complaint of racism could lead to a driver having their accreditation cancelled."

Aboriginal and Torres Strait Islander Social Justice Commissioner Mick Gooda said it was "shameful"  that someone who worked so hard for reconciliation could be treated so badly.

Mr Gooda said he wanted Mr Charles to know that the community was behind him.

"We all support him and want him to be vindicated in taking a stand that the people who have done this should be made accountable," he said.

"There is a lot of anger in the Aboriginal community - here's one of our leaders, icons being racially discriminated against.

"I hope Uncle Jack follows through with a complaint to the Australian Human Rights Commission or to the Victorian Rights Commission."