Woody weeds insinuate themselves in hedges and shrub beds.
In winter the culprits are very apparent when they're deciduous within evergreen hedges (right). You can also tell one deciduous plant from another by differences in buds, bark color, branching and height.
Roust the interlopers before budbreak so the rightful occupants will have full sun during spring.
It's probably not possible to completely remove these weeds in one go, since to dig their roots could do serious damage to desirable plants. Instead, cut the offenders at ground level, mark the stumps and keep coming back every two or three weeks to cut off any shoots they produce throughout the year. Over a season or two even the most entrenched weed tree will run out of starch if it's cut every time it makes a comeback.
Alternatively, you can cut the weed trees, then slow their new growth or kill them outright with a systemic herbicide. Use a small paint brush to dab the freshly cut surface of the stump with a brush killer such as Brush-B-Gon®. (Use it with care, in concentrated form; more detail about this and also some non-chemical killing tactics in Growing Concerns Issue 763).
You missed it during the past two growing seasons, since it first poked its head above the cranberrybush viburnums, but this elm tree (the chimney points to it!) should grab your eye now when its bark color and branching say "I'm different." Go cut it out before budbreak!
Make no mistake, these weed trees and shrubs are killers. Once they're tall enough to cast shade on the bed's rightful residents, the killing progresses quickly. Notice that the cranberrybush viburnums nearest this interloping box elder (in its third year above the viburnums) are shorter and have less fruit. Those are just the preliminary consequences of increasing shade.