For the other gods decided that Frigg's astonishing act of coercion was a _great_ opportunity for some harmless fun. They started hurling objects at Baldur, knowing that because everything (almost) in the world had promised not to hurt Frigg's golden boy, they could chuck rocks and knives and arrows at Baldur all day and nothing would happen. "Oh, you're a sword, huh? Sharpest in the world, eh? Could cut *anyone's* head off in a blow? Well, joke's on you, because I'm about to do--THIS!"
Needless to say this final piece of egotism finally got to #Loki, and they decided they would show just how flimsy Frigg's "safety" was.