It may seem like you're eliminating color but in every over the hill flower there is brown. Removing the brown immediately clarifies the flower color and you'll be spurring the development of new buds, too.
As with cutting back annuals in containers, you will be more confident with your snips once you make a positive I.D. -- what's a bud, what's a seed pod.
Blackeye Susan vine (Thunbergia alata) is one of those plants with just-forming seed pods similar in appearance to half-grown flower buds.
Blanket flower Gaillardia