With our backs to the castle, we headed back down the Royal Mile. The Royal Mile has tons of shops and restaurants along the way and at the bottom is Holyrood Palace.