There is a plus button next to a list record fields that allow user to add item to that list, how we can remove that or disable it?

Give the user's role view only permissions for that list.

@kennkerp thanks, but there is no permission for that user for that list. there is permission for the record but not for the list. and we can't change the permission for that record to view