With your lifestyle, you well could be atheletic enough to have had the high EF. There are a lot of people who think they're active enough to have an EF at that level, but don't really understand the level of physical dedication required to maintain that. It certainly sounds like you're one of the few exceptions, and I apologize for taking the more likely route in my reply.
It must be a real blow to you to watch the capacity of your heart dwindle, especially considering the care you have taken of it all these years. At least be cheered that the care you have taken of it should help you rebound more rapidly and fully than if you had jockeyed a desk for years.
I would ask you to be very careful in one aspect: those who are extremely fit frequently experience no symptoms that they notice, and most cardiologists are heavily weighted toward the symptomatic patient. Don't ignore anything that can be considered a symptom.
Your loss of EF may still be the downward result of an abbreviated ventricular cycle, based on your initial high EF. If you are enlarged enough to be treated with drugs for verticular hypertrophy, it would definitely make sense to look for a second opinion. It's not about avoiding the surgery at all costs. It's about picking the time when surgery becomes the greater good. Not all cardiologists are good at that (can I hear an "Amen" from the assembly?).
If your ventricle becomes too enlarged, you risk asymmetrical ventricle recovery, atypical muscle movement, leakage of other valves from distortions caused by the hypertrophy, permanent enlargement of the atria (again from distortion), persistent arrhythmias (including AFib from misshapened and enlarged atria), primary pulmonary hypertension, and other types of permanent damage.
I'm not saying you need to run to the surgeon immediately, but you need a second opinion that's not tied to this doctor or his group practice, if he has one. You are not as likely to produce symptoms. You need someone who can read between the lines.
Very best wishes,