Mr Zuma has already survived seven no-confidence votes.
But the opposition hopes some ANC MPs will side with them as the vote is secret for the first time.
The parliament's speaker, Baleka Mbete, made the decision to hold the vote in secret on Monday, after opposition parties took the case to the Constitutional Court.
In order for the no-confidence motion to pass, at least 50 out of the ANC's 249 MPs would need to vote against the president - leading to opposition parties pleading with ANC MPs to "vote with their conscience".
"Today our choice is between right and wrong; between good and evil," the Democratic Alliance's Mmusi Maimane said, while Andries Tlouamma, deputy president of the Agang party, warned if Mr Zuma survived then South Africa was "on a highway to hell".
Meanwhile, Mangosuthu Buthelezi, of the opposition Inkatha Freedom Party, said President Jacob Zuma has "trampled on the constitution" and has "sold the country to the highest bidder".
But Defence Minister Nosiviwe Mapisa-Nqakula told the gathered MPs the vote was "akin to a coup".
"My conscience tells me we all need to respect the voters who brought us here," she said, urging MPs to wait until the 2019 election, when the people would make their voices heard.
Mr Zuma will not be standing in 2019.
How does the vote work?
Following a 90-minute debate, the House will adjourn to prepare for the vote.
A bell will then be rung and MPs called in alphabetical order to collect their ballot papers. There are three options on the ballot paper: Yes, No or Abstain.
After each MP has voted, the ballot boxes will be sealed.
A representative from each party will be called to witness the count. The counting room will be guarded by the Sergeant-in-Arms, under the supervision of the party representatives and Speaker.
Finally, the bell will be rung again and the speaker will announce the results.
Criticism of Mr Zuma has increased this year, after he fired his widely-respected finance minister, Pravin Gordhan, in March.
This came amid accusations that the president had become too close to the wealthy Gupta family, who are accused of trying to influence politician decisions - including the sacking of Mr Gordhan.
Mr Zuma and the Guptas have denied any wrongdoing.
Outside of parliament, rival protests were being held in Johannesburg, Pretoria and Cape Town, where the vote will take place.
Johannesburg police are on high alert ahead of the vote, the city's Councillor for Public Safety, Michael Sun, said in a statement.
He has faced numerous corruption allegations, all of which he has denied.
His critics also say he has mismanaged the economy and failed to raise living standards.
Zuma's legal woes:
- 2005: Charged with corruption over multi-billion dollar 1999 arms deal - charges dropped shortly before he becomes president in 2009
- 2016: Court orders he should be charged with 786 counts of corruption over the deal - he has appealed
- 2005: Charged with raping family friend - acquitted in 2006
- 2016: Court rules he breached his oath of office by using government money to upgrade private home in Nkandla - he has repaid the money
- 2017: Public protector said he should appoint judge-led inquiry into allegations he profiteered from relationship with wealthy Gupta family - he denies allegations, as have the Guptas
Although allegations of corruption have dogged Mr Zuma for years now, a growing number from within his party have publicly called on him to step down, the BBC's Milton Nkosi in Cape Town says.
Some, like ANC MP Makhosi Khoza, have even received death threats for speaking out against the president.
She said that a vote for the motion will not be a vote against the ANC, but a vote against corruption.
But our correspondent says that Mr Zuma still enjoys a considerable amount of support from his party - the question now is how many of his MPs are prepared to take a stand against him.
Mr Zuma is due to step down as head of the ANC in December, ahead of the 2019 general election.
He has endorsed his ex-wife, Nkosazana Dlamini-Zuma as his successor.
Also vying for the leadership is Cyril Ramaphosa, a former trade unionist and one of South Africa's wealthiest politicians.