La Vila Vella castle walls surround the most charming part of the city, the old town.