Would life be impossible on a desert planet?

Not necessarily. A desert planet could still be habitable at half the distance between the Earth and the Sun.

Desert planets have less greenhouse effect due to less water vapor in their atmosphere, which means they are less susceptible to their distance from their star.

They are in fact able to absorb from 170% to 58% of the energy that the Earth receives from the sun, without overheating or freezing.

A lot also depends on the axial tilt. The greater it is, the higher the average temperature of the planet.

Furthermore, depending on the angle of tilt, most of their water would likely collect around the poles and/or in relatively distant and isolated oases.

This would imply that, at the planet’s poles or near bodies of water, could live very different and only remotely related life forms.