Short term disability providers allow (and pay) six to eight weeks for a normal recovery. They wouldn't do that if they didn't feel they absolutely had to. I had a great and fast recovery, but I started back with half-days at six weeks, going to full time at eight.
You'd be amazed how much stress burden you have at work - including for an "administrative" job. Going back is a lot more than just considering physical effort. Shame on your doctor, Leah!
Don't rush it, because you can pay an unseen price for it in arrhytmias and other problems. This surgery is no joke, and recovery is more than having the external wound close and being allowed to drive again.
How much could you be worth to the people at your job, if they think that you starting back two weeks earlier is more important than your full recovery or your future health? It's not a sign of how important you are, it's a sign of how little concern they have for you.
Take care of yourself. Don't expect the people you work with to do it. And don't believe that they will think twice about any personal difficulties you develop, if you start back early. They'll just say, "Well, he shouldn't have come back so soon, if he wasn't ready."
You are more than a thrall to your job, and you shouldn't measure your worth by your sacrifices there.
Best wishes,