Israel’s military however stressed that a “significant force” will remain in Gaza, with the defence minister saying troops had to prepare for future operations.
The pull-out is being interpreted as tactical, rather than a sign the war may be moving closer to its end.
Khan Younis has been under Israeli bombardment for months – the city and surrounding area are largely destroyed.
One man, now living in Rafah, tells BBC News he is “filled with hope as my neighbour in a tent from Khan Younis was able to return home”
Meanwhile, international pressure for a ceasefire is mounting as negotiations between Israel and Hamas continue in Cairo
The US warned last week that its ongoing support for Israel was dependent on “specific, concrete steps” to boost aid and prevent civilian deaths.
Israel’s military says it is reducing the number of its soldiers deployed in southern Gaza, leaving behind just one brigade in the area.
In a statement on Sunday, it stressed that a “significant force” would remain to operate in Gaza.
The withdrawal from the southern city arrives as Israel is facing growing international pressure for a ceasefire, but the pull-out is being interpreted as a tactical move rather than a sign the war is coming to an end.
Earlier, the Israel Defense Forces (IDF) spokesman Lt Col Peter Lerner told the BBC troops would rotate out of the area because their mission in the southern city of Khan Younis, which has been under bombardment for months, had been completed.
“The war is not over,” he has said. “[This] is a decline in the forces but there are more operations that need to be conducted.
“We need to dismantle Hamas’ capabilities wherever they are.”
The withdrawal of troops from Khan Younis comes as Israel faces growing international pressure to de-escalate the conflict.
The Israeli defence minister said soldiers had left the city, the largest in southern Gaza, to prepare for future operations against Hamas.
It is still not clear what this could mean for a possible incursion into the overcrowded town of Rafah, where more than a million Palestinians are sheltering.
Israel says an offensive there is essential to defeat Hamas, but the UN and the US have warned that a major operation could lead to a humanitarian disaster. In Cairo, efforts to agree a ceasefire deal continue.
