Former prime minister Malcolm Fraser called for Australia to skip the conference, which is scheduled to be held in November, following reports of the torture of Tamil civilians and government-sanctioned abuse of journalists, judges and opposition politicians.
"No, we shouldn't [attend]," Mr Fraser said.
"We should have been arguing at the Commonwealth conference in Perth that Sri Lanka was an inappropriate place to have the conference.
"From all the reports that we're getting, there is still continuing human rights violations in Sri Lanka."
But Senator Carr says he has seen no evidence of abuse.
"Any suggestion of a boycott would be counter-productive. It would simply isolate the country and render it defiant of international opinion," he told Lateline.
"Our challenge is to keep the pressure on to see there are further improvements, especially directed at reconciliation in the north.
"People ... in the north, they've told me they have seen former Tamil Tigers - that is fighters using terrorist means - are now being rehabilitated, being employed, gainfully employed, being reintegrated into that community."
Canada's prime minister Stephen Harper says the Sri Lankan government's claim that it never deliberately killed any Tamil civillians - only Tamil Tiger rebels - is blatantly untrue.
He warned that, unless changes occur, he would not travel to Colombo for the meeting.
But Senator Carr says all other Commonwealth countries are expected to attend.
"Apart from Canada, I can identify no other country in the 55 member Commonwealth that would not be represented at Colombo. I'm not aware the Canadians have made a final decision on that," he said.