The 64 year-old Mosman Park man set out from the shore near the Indiana Tea Rooms just after 8:00 am on his daily swim.
He was heading out to a buoy several hundred metres from the shore when he disappeared from sight.
A friend and several other swimmers went into the water looking for the man but could find no sign of him.
Senior Sergeant Denise Grant says the friend quickly alerted authorities.
"He comes down with a friend and his friend was waiting on the beach and he saw him swimming out there and then he's gone to enter the water and then he couldn't see him anymore," she said.
"He was a very strong and a very good swimmer apparently."
Sergeant Grant says the man's bathers have now been found.
"Police divers have located a pair of black speedo bathers that have been positively identified by our missing person's wife," she said.
His wife and members of his family are at the beach.
Sergeant Grant says there is nothing to confirm a shark attack is behind the disappearance. Read More