Harriet Harman and Patricia Hewitt failed to take action to oust the P aedophile Information Exchange (PIE) because they "didn't want to rock the boat", its former chairman has claimed.
Tom O'Carroll said Labour's deputy leader and the former health secretary did not support PIE but "didn't even try" to cut its affiliation to the civil liberties organisation they worked at for the sake of their careers.
Ms Harman has said she has nothing to apologise for over the involvement of the National Council for Civil Liberties with the paedophile rights campaign but expressed "regret" that there had ever been a link between the two.
O'Carroll sat on an NCCL gay rights sub-committee in the late 1970s and said there had been no active attempt to remove him.
He told BBC Radio 4's Today programme: "Really they didn't do much to oppose PIE's presence in my view because there were these other liberal forces, or radical forces, within NCCL.
"The support didn't come from Harman and Co but it was there. The Gay Liberation Front was very radical and at that time Harman and Patricia Hewitt couldn't just kick out PIE. Well, they could both try but they didn't even try and the reason they didn't try is they didn't want to rock the boat because their careers within NCCL depended on them not rocking the boat too much."
The former PIE chairman told the programme he never met Ms Harman but met Ms Hewitt briefly at a conference in Sheffield in 1978.
He said: "I spoke to her when we were just going up a floor or two in a lift and she was somewhat frosty.
"I said something to her, I think as a pleasantry, and she didn't take it that way. She said that I'd been rude to her afterwards, which I hadn't been.
"My impression was that she simply had great distaste for being in the same lift as me."
Former culture secretary Tessa Jowell said Ms Harman and her MP husband Jack Dromey, who also worked at NCCL in the 1970s, had been " completely consistent" in their opposition to PIE and insisted there was "not one shred of evidence that they gave any comfort to this revolting organisation".
She told Today: "There's no question of post-hoc justification here. The evidence shows how vigorous Jack Dromey was in his attack on the Paedophile Information Exchange.
"Harriet's work at NCCL didn't bring her into contact with them at all."
A Labour MP has called for an investigation into PIE after concerns that it could have received public money.
Tom Watson says he was contacted by a former civil servant who saw documentation suggesting the organisation received taxpayer funding under Margaret Thatcher's Conservative government.
"I was contacted by a former Home Office civil servant who told me that he saw a document that suggests the Paedophile Information Exchange got public money," he told the Daily Mail.
"The document was recommending approval for funding during the Thatcher government. My source, who does not wish to talk to the media, said it appeared to be a re-application for funds.
"He could not be certain whether or not it had been funded by a Labour government but he thought this was possible."
Mr Dromey said in a statement: " It is no surprise that a convicted paedophile, the like of whom I took action against during my time in the NCCL, should choose to smear me.
"The record is clear. I took on PIE and, when I was elected chairman, defeated by a massive majority at the 1976 NCCL annual conference a loathsome motion calling on the NCCL to support the so-called rights of paedophiles.
"Indeed my stand was denounced in a leaflet distributed by PIE to the delegates to the conference.
"Quite simply, no one will believe a man like Tom O'Carroll."
Deputy Prime Minister Nick Clegg said it was a matter for Ms Harman whether she chose to say sorry - but he "strongly supported" the apology from Shami Chakrabarti, the head of successor body Liberty.
"She (Ms Harman) has explained why she has chosen not to apologise," the Liberal Democrat leader told a press conference in Westminster. "She says it was not her decision, the association between this abhorrent organisation and the NCCL.
"I certainly strongly endorse, I strongly support the fact that Shami Chakrabarti, the current leader of the successor organisation to NCCL, has apologised for any association in the past between the NCCL and such an appalling outfit as PIE.
"But you will have to ask Harriet Harman."