I was 33 when I had my aneurysm repaired along with my valve, about nine and a half months ago. I did not live a very active lifestyle but I did partake in a few sports in high school plus practiced martial arts for eight and a half years. In the past twelve years before my surgery, I did not really partake in a whole lot of exercise other than hiking and camping on occasion. In the nine months prior to my surgery my ascending aorta was 4.65 cm and less than nine months later it had grown to 5.6 cm!!!--all the while I was not exercising at all.
Those nine months I was completely sedentary.
I suppose the moral of the story is that if it is going to grow, it will, beyond your control. Granted, during that nine month time I was under tremendous stress in my life, more than I had ever experienced, but, as far as activity was concerned it was minimal. My Cardiologist always told me to refrain from lifting heavy weights my whole life (well, since I was a teenager) and I obliged very diligently. This did not prevent my aneurysm from growing.
What I believe it may have prevented is it growing _sooner_ though, in the end, it grew anyways when I was not doing much at all. You can't live your life sitting around worrying about it getting larger. That will tear you up inside. Listen to your doctor, and, most importantly, listen to yourself and your gut feelings.
However, do not let wanton worrying consume your time. Life is too short to live that way. Besides, if you ever do find yourself truly in the waiting room knowing in a month and a half that you will be having surgery, the stress can tear you to pieces and eventually, through a trial such as that, if you hadn't learned it by then you will eventually come to peace with the inevitable (if it is such the course you are destined to take).
Take my word for it--come to peace with it now. If your doctor says you can live a normal life, do so! Don't let it hold you back because life is too short to worry every day.