Screenshots without logs… again…
well, what can we see ? We can see small packet loses at the begin of the trace, in the network of you ISP
I’m fairly certain if you disabled EU locking and therefor forced more players to play across the pond on NA servers you’d be hearing about this issue tenfold.
Increasing the assets in a match directly correlates with this issue, maybe you should look at increasing server resources when you release updates that increase sticker count or the like. It was not this bad before the recent update that did that. Every other update that increased environmental effects (and server load) has resulted in spikes of this very same issue.
And it makes perfect sense that loss would compound this issue. I have a gigabit connection, 1% loss should not be causing complete desync. I can still input commands to most of these games but the data the server sends me is janky and completely out of sync.
What protocol does the in game net test use? When I use -ping to any of the servers I have far less loss than your in-game net test reports. Right now I run the in game test and it says 90% loss on 184.105.213.137 port-channel.14core2.phx2.he.net, yet cmd ping shows 0% loss on the same node. Curious.
Like, have you even BOTHERED running traces from your end to the players/customers?
1 Like
I am literally running these at the exact same time:
The damn test says I have 10% loss to my router ffs
1 Like
O, net log with test connection! Very good!
Let’s watch UDP part of result:
1: 95.211.237.144 wheel85lw-nl.crossout.net : 1350b 141 162ms 2%
2: 95.211.237.144 wheel85lw-nl.crossout.net : 1215b 140 159ms 2%
3: 95.211.237.144 wheel85lw-nl.crossout.net : 891b 141 158ms 0%
..
7: 95.211.221.75 wheel80lw-nl.crossout.net : 1350b 142 159ms 2%
8: 207.244.92.140 wheel05lw-us.crossout.net : 1350b 142 90ms 1%
9: 207.244.92.143 wheel10lw-us.crossout.net : 1350b 142 85ms 1%
As we can see:
- There are UDP packet loss at all servers from list.
- Packets with sizw near 900 bytes walk without loss.
What we can say about this:
- Most probable a one of routers between your PC and our servers drop packets with size more than 900 bytes (MTU ~ 900). But may be an other reasones, for example, too big trafic at some routers.
- Minimal packet size for Crossout is 1200 bytes.
- Sorry, but, most probably, our servers isn’t a reason for your problems.
Also you can read this article for more information How to report connectivity issues - Bug Reports / Server connection - Crossout
But does this explain why I can still input commands to the server while the stuff I see on my end is still utterly unplayable?
What node is the one dropping packets? Who do we contact to :deep inhale: actually solve the issue?
1 Like
Nobody know. We can’t know. It’s a Internet. Sorry.
Try to select various regions in the game settings.
I’m not sure you still understand exactly what the issue is, like the server will accept my inputs but I see nonsense. Of the many many times this has been happening, I found two good examples that show when this happens. The examples have the view that I have (the youtube links) and I supplied the timestamp for the very same games my clanmate streamed on twitch.
I had two desyncs in Levi cw, here’s the first 1: https://youtu.be/uP56cpCOrto
It matches up with my clanmate’s stream at Twitch
At 2:53:40 on the twitch stream you can CLEARLY see the server accept my driving inputs as I am literally driving up the map as I watch Crown’s stream. Yet if you look at my youtube video you’ll notice that I see a game state that is completely wrong.
Number two
I started recording about 15 seconds after the start of this match: Twitch
You can see the same thing happening from my perspective, I see nonsense but the server accepts my inputs and I can still drive. I cannot aim or see what I am doing though. https://youtu.be/sJSpYmpq02M
Desync in knight’s mayhem where I make it out of the loading screen, but the game state I see is still completely wrong.
I’ve been testing other online games in the ways that I know how to, and I have no where near the same issues. I highly doubt this is a problem with my ISP unless the way that XO handles traffic is wildly different. I’m looking to see if I can get more information about why UDP might be dropping for XO but when I talk to my ISP and they run tests they don’t see anything wrong.
There is also the correlation with every time a game mode with more assets, or an update puts more assets, or you can have more stickers (assets), this issue gets much worse. I get that loss has a negative impact on connection quality, but the amount of loss that you claim that I have with this game appears to be exclusive to this game. And that makes no sense.
2 Likes
For posterity:
I’ve had this issue through different ISPs over the course of a few years.
1 Like
If the problem is UDP loss, you should illuminate people on how to possibly fix it. If it’s firewall rules or whatever.
It still doesn’t make any sense that all kinds of other online games are perfectly viable and yet Crossout suffers so badly if someone has as you say “problem with your internet”
1 Like
You’d have to contact each server that your passing though on the internet. It would not be a stable fix but you could do a whois on each server showing you a packet loss. The problem is on the internet your connection between servers will shift dependent on traffic. It’s not a stable connection 1 to 1 every day connection.
Example from ip numbers you listed in the logs:
Contact details are on that listing.
Leaseweb is a datacenter and backbone provider that XO uses. People have had issues with their servers for years.
Most of the other nodes on my route are owned by Hurricane Electric.
I am a part owner of the ISP that provides internet to the small town that I live in.
Believe me when I say I’ve been working on this issue. It’s an issue only native to XO. Every other online game I have ever played has no where near the UDP loss that XO has. This is a game specific issue and I’ve been trying to figure out why with basically zero help from Skula.
I haven’t seen anything that refutes my hypothesis that the game just has too many assets to synchronize between client and host, and any loss exacerbates this issue. Of course people like Skula don’t see it, they live very close to their servers. Even on NA servers I have been having this issue (I live in the United States). Servers that are farther away with more loss/late packets are basically unplayable.
Every. Single. Time. That XO pushes an update that increases the amount of assets in a game this issue gets meteorically worse for every player that I know. The new sticker update came along and bam, servers like wheel80-88 explode with regularity in Clan Wars and entire teams get disconnected. Yet Skula is a broken record and refutes that they could do anything about it. It’s very tiring.
2 Likes
Oh my god.
I can’t believe this. Skula, what am I looking at?
This clearly says one thing:
And I can watch the traffic, and they uhh, don’t use those ports?
What is going on?
Seems to me that it would be logical if XO’s network test function used the same ports as the game does for more accurate results. The ports are ALL OVER THE PLACE though and they don’t match your advertised ports on your website. If you can clean up your ports and make it clearer maybe people can go about making accurate firewall rules.
1 Like
I know these are diff protocols but seriously, if your UDP is reporting 90% loss at a node and I can ping it with TCP and get 0% loss something is messed up. Meanwhile the server IP shows 0% UDP loss but wont accept TCP pings.
Your claim that XO uses an MTU of 1200 is wrong. Some of your servers are sending packets over 1200:
There really should be trace going on with the network test that is from server → client as well. Having only client → server doesn’t help us that much when I can already verify that the server is receiving my packets (since I can drive when I’m de-synchronized). My guess is that because y’alls ports are sloppy spaghetti plenty of ports coming from the server get refused because they look like attacks.
1 Like
Yes, count of assets is a key difference from other games. Every car part, every bullet must be synchronized: position, speed, torque, acceleration, health point.
And we can’t decrease count of assets.
Give us Ex Machina MMO! Assets number probably gonna increase instead.
There are other ways to alleviate the issue. Your refusal to acknowledge this is disparaging. It is a hindrance to the entire Crossout community.
You still haven’t answered my question on why your port list doesn’t match what actually goes on.
1 Like
It would be more helpful if you just listed these ways to alleviate the issue as a game company will more than not likely list how they do stuff via restrictions on discussing internal IP (intellectual property rights) How would you address the bottle neck in the data flow?
I can only do with the info that I have.
I know their ports are a mess. I know my data reaches the servers just fine. I know their network test only tests one side of the route (and since I know my data reaches the server, this makes the network test rather useless).
Given that I get soft-locked in the loading screen, or I rubberband in spawn eternally, I can deduce that the server sends me the wrong information. Every time the de-synch occurs, the received bandwidth increases by quite a bit, sometimes up to 5x what it typically is. The upstream sees no noticeable change.
How do they fix this? I can only guess how their stuff resolves the gamestate between client and host. I am not privy to that information.
Could they offload non-essential assets (paints, stickers, shell impact markers, etc)?
Could they increase server resources per match based on assets?
Could they relax parameters around the timing that the servers send packets?
Could they add redundant measures for hashes/checksums?
Could they force the player past the loading screen if the clock passes a certain time?
Maybe they could match their port forwarding list to the ports actually used
Could they make sure their servers have the right firewall rules to make sure client traffic doesn’t get rejected? Looking at you, wheel88.
Could Skula run a trace from different servers to a player like me who has difficulty receiving the right packets at the right time to get a better and clearer idea of what the actual problem is instead of saying “it’s you internet” over and over?
I literally don’t have enough information to solve this for them for free.
It adds insult to injury when Skula constantly dismisses that anything could be done on Targem’s end. Especially when I’ve seen them do things to fix this issue in the past. Remember when destructible objects were added? This issue exploded. The forums were blowing up about it. They fixed it in a weeks time!
I have literally offered to add Skula on discord so we can work together on this issue in real time. If anyone is the problem here, it is not me.
1 Like
Skula’s just trying to do his job though and he speaks multiple languages sometimes that plays into how things are said. Often even streaming traffic going on at the same time in a house while people are playing can have a huge impact even with a high tier plan from an ISP. Can you confirm what’s going on in your network as a whole when the problem events happen. Also are you hardwired or are you using wireless? What can we confirm on your side to rule out it’s not playing into the issue? I’ve noticed a few tvs streaming have a noticeable effect on the lag I encounter in the game.
As far as the issue with destructible it was an identifiable issue that could be fixed that’s why it got fixed fast. Network and internet issues between multiple servers is a lot harder to track down.
Which ports are wrongly listed? This is something if pointed out clearly can be resolved fairly fast.
@1914000 Is there a way to check from the servers to delraith’s IP? Ruling out the other side of the issue as pointed out below? Can that wheel88 issue be checked too?
Understandable out of the information that you have how could we best fill out a report on the new issue tracker to check for how many other users might be effected by the same issue? If a lot of people are having the same issue it will get elevated as a priority.
Can we work out some of the terms your using to make sure they are translatable “softlocked, rubberband in spawn” (just a few that I’d point out)?
@1914000 Can you help us any on making this more trackable as an issue when reported? Are there things that you would like to see included?