Sky Global Holdings Posted March 15 Share Posted March 15 the arcade server is acting wonky does anyone know what is happening? Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 15 Share Posted March 15 Yes, I deployed a new change to deprecate the event manager which inadvertently created a race condition. I have pushed code to fix the issue and am testing it out now. Thanks! Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 15 Share Posted March 15 Also FYI, I have temporarily paused both world, I believe the fix is working however the tasks themselves are backed up and I have to let them run their course, so will keep things paused until then. Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 15 Share Posted March 15 World timer for B IX was re-activated and running normally. Arcade I will remain paused for the time being. Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 16 Share Posted March 16 Arcade I is unpaused, so far I'm not seeing any issues but will keep monitoring so it appears letting the tasks time out resolved the issue. Thanks! Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 16 Share Posted March 16 For context, as runaway tasks were executed, this caused the infrastructure container CPU utilization to spike (green line, yellow line is the ephemeral service container) due to all the database operations. This caused multiple issues including failure of multiple tasks and even some service calls (mostly due to database errors related to blocking calls), caused task execution times to be excessive (new day tasks were running in the realm of 30 minutes, whereby they usually take about 30 seconds) and eventually crash entirely. The root cause was a race condition in the way that the new day task handled arcade world (since it advances by 7 days rather than 1), causing more new day tasks to be scheduled than were supposed to. This was not the issue in previous event manager due to a check for a scheduled event that prevented it, but this check did not work as expected in new flow. Quote Link to comment Share on other sites More sharing options...
Sky Global Holdings Posted March 16 Author Share Posted March 16 ok thank you Quote Link to comment Share on other sites More sharing options...
rohithbala Posted March 16 Share Posted March 16 @sviridovt I think something happened again. Servers are acting weird and I can't get a single passenger for my new flights. Quote Link to comment Share on other sites More sharing options...
rohithbala Posted March 16 Share Posted March 16 Never mind, the servers are working. prob a slow internet connection on my end. Quote Link to comment Share on other sites More sharing options...
sviridovt Posted March 16 Share Posted March 16 7 minutes ago, rohithbala said: @sviridovt I think something happened again. Servers are acting weird and I can't get a single passenger for my new flights. There was a deployment that happened so some wonkiness is expected immediately following. Let me know if you're still having issues. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.