No, you do not. You can return your books to any of the Library's branches including Main Library, Chinese Medicine Library, and Shek Mun Campus Library.
In addition, you can also return books by dropping them in any of the Book Drops on all of the University's various campuses.
Note that the above does not apply to Course Reading items and HKALL items. Such items need to be returned directly to the User Services Counter.