I have a custom recordset that I am trying to search for the next available record that has a field that is blank. In SQL, I would use TOP or MAX or MIN... How can I do this in a suitescript search (v2)? So that I do not get thousands of records?

you don't really need to, both of the main search mechanisms (ResultSet and PagedData) have ways of controlling how many results you get at a time

yes, but I have specified .runPaged({"pageSize": 1}) on my request and I get the LAST record in my table instead of the first. even though I tried sorting on both the internal id and another record ASC

if you are desperate, you can try using summary columns