An update on the chat issues
EDIT: Another update in the second post.
Hi guys!
I just thought it was time to give you a bit of an update on the issues with the chat system that some of you are experiencing; what exactly is happening, and what we're doing to solve it.
First, some background. Before we did the Single Server upgrade of the Dreamworld engine, the dimensions were completely separate on the server-side. Each dimension had their own set of server processes "driving" it, including one chatserver each.
The idea behind the Single Server upgrade was that the pool of servers would be completely shared between all the different dimensions, meaning a dimension in the new system is much more of a virtual concept. This technology is what makes the cross-dimension PvP matches, joining your friends in other dimensions, and (most relevant here) chatting to everyone through tells to players on other dimensions, cross-dimension cabal channels, team chat channels, and so forth, possible.
To properly implement this chat functionality, the technology for the chat backend had to undergo a complete upgrade. While the frontend (what you see in the client) may look similar to previous Dreamworld engine games, the backend is now completely different. As there's too much traffic for a single server to handle, the load has to be distributed: We structured a network of chat servers that all talk to each other, to the clients (you), and to the game servers.
So, to the problem itself. At some point after we "went live" with the game, some of the servers would suddenly stop talking to the other servers for periods of time. Unfortunately, this means that any player that connects to a problematic server while it's not communicating won't be able to chat, as the chat servers need information from the game, which they can't get. Players already connected to the server can also only talk to each other during this. The server will eventually recover, but it takes some time before everything gets synced up.
We never saw this internally while testing the technology, or during the various betas. And while the first beta weekend had an issue with the chatserver being unavailable for a time, the cause wasn't the same, so there wasn't much we could do about this ahead of time.
Again, the problem with solving this is that it's very hard to reproduce the issue in a controlled environment where we can debug it. This means that everything takes a lot longer than we'd like in terms of pinpointing problems and testing out solutions. At this point, we are working on several fixes for this, and some backup plans, as fast as humanly possible. We can't thank you (our players) enough for your patience while we repair this, and hope that you won't have to wait too much longer!
I can assure you we have our top men working on it. Top men.
Vinterstum
Lead Programmer
UPDATE:
We applied a hotfix to the chatservers a couple of hours ago, which so far seems to have resolved most of the issues.
