I think the issue is less about the number of monitors you use and more about the number of contexts that you maintain on those monitors:
It is possible to configure a monitor arrangement to act as a single desktop area and tile application windows across all of the monitors or at least arrange windows from the same application across multiple monitors - e.g. a designer may have their main document on their largest display and their digital asset browser on their secondary display.
So, in this setup, the displays represent a wallpapering of a single (primary job) context. Switching context involves key-combos, gestures or shutting down the context and starting up something else instead.
I've seen many multi-monitor setups that have multiple applications arranged over a number of monitors, for example:
Primary Display: 3/4s browser, 1/4 for 'task list' (start bar/dock, etc)
Second display: 1/2 email, 1/4 music player, 1/4 Social Media aggregator
There's nothing inherently wrong with this setup, but it allows multiple contexts to compete for your attention. For example, you're researching some topic and a mail comes in about something else. You open the mail.. and you're distracted for a while. You eventually get back to the research then see there's some buzz on your social feeds that pulls your attention.
In all the studies, articles and books on productivity I've read, "context-switching" is the biggest threat to productivity. Sure, some jobs require switching between multiple high-priority items, but that's usually a concerted effort in between larger periods of focus.
In conclusion, multiple displays only impacts your productivity if you fill them with items which will cause you to become distracted away from your goals.