For all intensive purposes I'm assuming that you've downloaded a freely licensed movie and not something that would be considered illegal. While I don't use a Mac, I know one program that you'll be able to use since it's cross-platform -
VLC. It'll play pretty much anything you can throw at it, and I've not found something it wouldn't recognize.
If you want to move them into iTunes, you'll have to do a conversion to a format it recognizes. A quick way to do this would be through
Handbrake. Basically drag the movie file into Handbrake and select a conversion for any of the Apple based products and you'll be able to import and play the movie without issues.