A task killer won't do any harm if you are using it manually, but you shouldn't need to since android kills processes automatically if memory is needed. Of course this does take a small amount of time and may be experienced as slightly slower response times, however, it shouldn't happen very often (although it probably happens a bit too often on the desire since a lot of memory is taken up by the HTC sense crap).
If you are using a task killer to kill automatically, then it will impact negatively on your battery life. When android threads are sleeping (i.e. when you aren't using them and they aren't auto-updating) they use no resources. This means when the screen is off the processor is also sleeping and using minimal power.
If a task killer is running, it will be waking the processor regularly (thus using power) to check for and kill threads that aren't using any power.