Affleck held hands with Sam—who was barefoot and wrapped up in a towel—while leaving the local Pacif