It was a combination of many factors that, when added together, meant Sega was having trouble turning a profit or keeping developer attention on their system. Please let me ‘set the stage’:
In the Japanese market, developers were kind of pissed Sega pulled the rug out from under them since the Saturn was still a reasonably successful system over there. It was definitely in Second Place, but doing well enough that several companies kept putting stuff out for it and felt it was worth their trouble. Adding the Dreamcast to the mix cut into that, since you now had three active systems on their market: PS1, Saturn, Dreamcast (and the Saturn and Dreamcast effectively competed with each other!)… four if you want to count the N64, but it was starting to fade around this time anyway. So that meant Sega suddenly lost a lot of good-will with publishers and developers; look at Nintendo’s history to see how well it turns out for any company that does this. Sega had put themselves in a very bad spot for one of their two primary markets.
In the west, this was not such a big deal. The only serious competitor in the market at the time was the PS1. The Saturn was a niche system over here, so western companies wouldn’t mind seeing it go. The N64 had a decent run but was stalling around now over here too. So the western market was ripe for a new system, and the Dreamcast came at the exact right time to take advantage of that. You could see it in the system’s massive early success, and at first everything was fantastic for Sega over here. Well, one out of two isn’t bad; you can sustain a system on one primary market if you do it right, and having US/Canada buying your stuff could be enough. However, Sony was aware the PS1 wouldn’t last forever and had some near-future plans of their own.
We all know what that plan was: The Playstation 2. It offered comparable (perhaps not identical, but basically ‘in the same league’) specs to the Dreamcast, Sony hadn’t pissed off nearly as many publishers/developers, and they had a trump card since the PS2 would also serve as a reasonably affordable DVD player. Remember, this was back when DVD was trying to break out as a format; people wanted what it offered over VHS, but getting a DVD player wasn’t very cheap or convenient back then. So when the PS2 showed up just a year later, it had something for both primary markets; companies in Japan were quite eager to jump onboard with ‘Sony + Better Hardware = We Can Get Away From Sega’, and western consumers now had a bet-hedger; even if the PS2 sucked (which it most certainly didn’t), they would still be left with an okay DVD player. On top of that, microsoft was making rumblings about jumping into the fray with their Xbox (which only the west cared about, as history showed). They arrived just a tad too late to truly trade blows with the Dreamcast, but it was one more little thing on top of a pile of problems Sega was already facing.
All of this results in Sega picking the best and worst time to make their move. There was a roughly 8-14 month window in which the market was open to a new system, and they hit that just great. The problem was a lack of futureproofing; they could not directly compete with what Sony was offering, and Sony had a much bigger manufacturing base to work with as well. People quickly lost interest in the Dreamcast, and a simple look back bears that out: Its sales suddenly plunged to the point where gaming stores were having clearance sales to get rid of old stock (being younger and less financially well off at the time, I thought this ruled; I loaded the hell up on great DC stuff super cheap), while the PS2 went on to be one of the most successful systems of all time.
You could say both Sega’s business decisions and the PS2 killed the Dreamcast; if it hadn’t come about then the DC would have continued to have a great hold on the western market and more Japanese companies probably would have warmed up to it eventually. On the other hand, the circumstances of the time made the PS2’s creation inevitable; the Dreamcast merely lit a fire under Sony’s ass and said “Hey, you have real competition now. If you’re holding any cards worth playing, play them very soon.” I speculate that they brought the PS2 out a tad sooner than they otherwise would have due to this, but it was going to be released regardless.
As a business venture, the Dreamcast was doomed from the start but Sega had no way of knowing at the time; I can only say that in hindsight. As a system… well, obviously we all treasure it for its creative ideas and great games.