Question for you constitutional law-types

If the Elks, Boy Scouts, VFW or NRA wanted to hold a meeting at my business I could easily tell them no, as they have no right to organize on my private property.

Why then can a labor union legally use my private property to organize against me?

Shouldn't the First Amendment's protection of my freedom of association, which include the freedom not to associate, from infringement by government action allow me to tell a labor union to ram it sideways?