I think that may have something to with it, but that 1971 season undoubtedly took a toll on his arm. Sort of like Denny McLain a few years earlier. Three-hundred plus innings in one season tends to turn one's arm into a side of beef.
Some guys were able to do it, but yea it was tough to be a pitcher in those days.
I'm not necessarily disagreeing with you guys, but...Blue was also a dominant HS QB. Hell, he actually loved football more than he did baseball. But given the era he played, he opted to go the baseball route because that was both the earliest way to a pay day (and a fair shot at being paid at all). His father had passed away I believe in his senior year of HS. When you do the math, yes, he got drafted and then started earning a paycheck as a minor leaguer, but what was also likely not lost on him (Notre Dame, Purdue and Houston all recruited him hard to play QB) was that there just weren't any black QBs being given a shot in the NFL. Hell, a decade later, Warren Moon had to ship it to the CFL to prove himself by destroying that league for a half dozen years before being given a shot in the NFL.
You have to wonder if he carried that chip on his shoulder during his confrontations with Charlie Finley whenever the A's owner tried to lowball him.