are you creating the field in Suitelet or on a form, If on the form, you might wanna have a look at the field Validation & Sourcing subtab and see how to go from there, otherwise you can script it.
Should not be that difficult. Just use the client script and check the File.type if matches the types you want to insert and return false if that is not true. Or else, you can use a inline html field and do this using jQuery