1. When your door is closed, the sides keep out drafts by being wedged between strips of weatherseal. Over the years this material wears down and air can seep in. It is relatively inexpensive to have these strips replaced.
2. If your weatherseal is OK and you still feel a draft from the right or left side of the door, open your door just a smidgen and look at the light gap between the door and the jamb. If it is uneven from top to bottom, the door will not be sealing well on both sides. An inexpensive roller adjustment will even out the gap.
3. Most sliding patio door systems have at least 50 square feet of glass that gets cold even if it is new Thermopane. Cold temperatures will radiate off this surface and convection can make it feel like a draft. Good window treatments (lined drapes) can create a barrier to keep you comfortable. Plastic vertical blinds just don't do the trick.