The Elves already can't regain numbers with the same speed like humans, so why would they wait for the Empire to fully regain its strength?

The Elves already can't regain numbers with the same speed like humans, so why would they wait for the Empire to fully regain its strength?