For better or for worse, the quarterback touches the ball on virtually every play and he will get either all the credit or all the blame. Lest we forget how polarizing Joe Flacco was amongst the fanbase despite being an integral part of the success of the franchise for a decade.
Albeit, as TwentyThirtyFive said, there's the race factor with Lamar and unfortunately, that does cloud people's judgement; subconsciously or otherwise.
Maybe I'm not remembering correctly, but Joe only became a topic of extreme debate, after his extension. And that debate was is he or is ne not elite. I'm honestly afraid of what Ravens discussions will look like, if he gets the deal he wants. I may have to step away from this part of BSL for a few years