Their physical power does scale as they get stronger, but is definitely outpaced by ki. I mean cmon, otherwise Goku would be able to blow up a planet with his pinky finger. A punch that missed the opponent but hit the earth would send it hurtling into the sun at light speed.