the factory plastic ones are easy...get a punch and a small hammer and tap it out. and it depends on what sights your putting on, you may need a file to get it on.
use a brass punch if you can find one, which you wont unless you have a smaller hardware store such as an Ace hardware. Otherwise you could use something with a rubber tip. if youre just removing and putting the factory sight back on i used a regular nail punch. didnt leave any marks