Is it even possible without changing browser preferences?
Q? How can I open a page over https in a popup window/modal overlay from a non secure site without getting security alert?
I would very, very much hope that it would be impossible. the whole point of HTTPS is that you serve everything from a secure server.
Assuming the "security alert" you're referring to is the one that some browsers display when you move from a secure page to an insecure one, then you have absolutely no control over that.