Pending Over 1k users

[lbenicio]

hey folks,

I just noticed that when you get over 1k users distributed across over 50 instances the server starts failing to handle this amount of users.

am I missing some configuration?

I put on a server with 4 cores / 8threds and with 12gb/ram... I'm using ~3gb ram and htop shows CPU load below 50%

do I need to get a bigger server? is it related to I/O of unix? like it can't handle 1k connected socks?

cheers

 

[farhadhelix]

hey folks,

I just noticed that when you get over 1k users distributed across over 50 instances the server starts failing to handle this amount of users.

am I missing some configuration?

I put on a server with 4 cores / 8threds and with 12gb/ram... I'm using ~3gb ram and htop shows CPU load below 50%

do I need to get a bigger server? is it related to I/O of unix? like it can't handle 1k connected socks?

cheers

hello

whats exact behavior of server ? do you see packet loss on IN or OUR in client connection info when this happens?
describe exactly so i may be able to help you .

and you may have 4 cores but there may be a limitation on total CPU HZ usage within your vps provider

 

[WolverinDEV]

In general TeaSpeak (and ofc linux) can handle way more than just 1k users.
But 50 instances seem (for me at least) overloading the server a bit. Specially when every instance is quite full.
Out of interest, whats the load of your server? (The values in the attached screenshot)

 

[lbenicio]

hello

whats exact behavior of server ? do you see packet loss on IN or OUR in client connection info when this happens?
describe exactly so i may be able to help you .

and you may have 4 cores but there may be a limitation on total CPU HZ usage within your vps provider

yeah, that's mostly the problem. Voice lagging, move channels lagging, packet loss... sometimes a bunch users are "kicked" out of nowhere.

I'm using a dedicated server with dedicated cores. I don't believe they are throating my cores.

I didn't check lot of users to see if they are losing packets IN or OUT. I just set up event for users being kicked and started to receive complains about lagging (and i noticed that i just reached 1k users at nght hours)

---- Automatically Merged Double Post ----

In general TeaSpeak (and ofc linux) can handle way more than just 1k users.
But 50 instances seem (for me at least) overloading the server a bit. Specially when every instance is quite full.
Out of interest, whats the load of your server? (The values in the attached screenshot)

i just get over 1k users at max 2-3h/day (at night). now im with 400 users connect and they are: 0.20 0.23 0.31

 

[REDOSS]

Describe in more detail what version of the server you are using, and provide a little more data.
1. Provide a screenshot of the command htop
- Set the display options "Show custom thread names". F2 -> Display Options -> Show custom thread names -> F10.
2. Provide a screenshot of the command top
3. Provide server configuration file, file config.yml
4. A screenshot of the YaTQA will not be superfluous, where the generalized statistics of your instance are displayed.

It will not be superfluous to diagnose your network, both from the server side and from the client side.

---- Automatically Merged Double Post ----

i just get over 1k users at max 2-3h/day (at night). now im with 400 users connect and they are: 0.20 0.23 0.31

This is a small CPU load, and there should be no problems... when the problem recurs, try to provide what I wrote above.

 

[mkll11one]

I have the same problem, but I'm not running multiple "instances" just 1 (on 1 machine - as for example!)

1) I running with: 38 virtual-servers aka max-users: 1380 - Online right now: 209-215 (ish!)
2) Around 779 channels.
3) CPU is always 87-150% - but depends on how many connected. [TeaSpeakServer is limited to set using 4 cores - As an example!]
4) The version of OS: Ubuntu 19.10 // Version of TeaSpeakServer: 1.14.11-beta8 (I don't have updated it atm!)

config.yml with voice only [No web/MUSIC-bots!]

threads:
  ticking: 2
  music:
    execute_limit: 5
    execute_per_bot: 1
  web:
    io_loops: 2
  voice:
    events_per_server: 2
    execute_per_server: 2
    execute_limit: 5
    io_min: 2
    io_per_server: 2
    io_limit: 5
    bind_io_thread_to_kernel_thread: 0

htop screenshot:

 

[REDOSS]

It is interesting... the screenshot shows 14 io processes, a in the config.yml io_limit: 5
Please show the result of the command top

You can also enable tree (F5) display in the htop command to facilitate perception.

 

[lbenicio]

Yeah, i will wait for the load to print the stats u asked for.

i have 2 friends struggling with the same problem, but they have more users and instances than me. They all said they found no way around and had to switch back, that is why i started this thread.. i had this conversation with than like 2-3 months ago, and now im starting to feel that will affect me as well but i would prefer set 2 servers instead of a switch).

From what i understand, Linux should handle well like 2k connected users? at least with rails i can handle 2k connections/second just fine on a single server with 2 cores / 4 threads (dedicated).

what they all say is that when you get a lot of users connected at the same time server start failing, even crashes (not like a crash log, but users cant connect anymore).

As this appear to be starting only on peak hours, i think i have time to figure a way out (my best guess now is setup 2 servers instead of 1 with 1k+ users)

most of my instances are pretty empty.. like <10 users connected.... i have like 10k slots provided but around 500 used at all day and at peak hours around 1k slots used

 

[mkll11one]

It is interesting... the screenshot shows 14 io processes, a in the config.yml io_limit: 5
Please show the result of the command top

You can also enable tree (F5) display in the htop command to facilitate perception.

There >>>

 

[REDOSS]

There >>>

Spoiler

View attachment 2484

everything looks good... the only thing I don't understand is why so many io processes are open.
although in the config.yml is set to io_limit: 5

@lbenicio
I have an instance with 800 clients that is running right now, but I am not experiencing any problems.... but your feedback will be very helpful, and I hope this problem can be resolved if it is present in the latest version of the server.

 

[mkll11one]

still, it's strange.... although in the config.yml is set to io_limit: 5

Do I need to change it? or just stand as io_limit: 5

 

[REDOSS]

Do I need to change it? or just stand as io_limit: 5

Leave)) I'm interested in hearing Markus's comment) I think as soon as he has time, he will definitely leave a comment.

 

[farhadhelix]

i have over 1700 users and i have don issue with that. your problem is your network or your cpu. i suspect cpu.

you mentioned you have dedicate server? its a dedicate server with linux installed on it or you have some virtulazation on it?

 

[mkll11one]

i have over 1700 users and i have don issue with that. your problem is your network or your cpu. i suspect cpu.

you mentioned you have dedicate server? its a dedicate server with linux installed on it or you have some virtulazation on it?

i9-9900K here! =) No virtualization =)
1000Mbit (Unlimited Traffic). =)
Aka Hetzner =)

 

[lbenicio]

i have over 1700 users and i have don issue with that. your problem is your network or your cpu. i suspect cpu.

you mentioned you have dedicate server? its a dedicate server with linux installed on it or you have some virtulazation on it?

yeah, they are virtualized using KVM but the company assure full core access up to 3.9GHz/core with Intel Xeon Gold 6140 / 6230

network card with 1gbit/sec with 70tb/month

 

[WolverinDEV]

It is interesting... the screenshot shows 14 io processes, a in the config.yml io_limit: 5
Please show the result of the command top

You can also enable tree (F5) display in the htop command to facilitate perception.

Well the IO threads are really interesting^^

yeah, they are virtualized using KVM but the company assure full core access up to 3.9GHz/core with Intel Xeon Gold 6140 / 6230
network card with 1gbit/sec with 70tb/month

As I remember correctly you've a lot of TeaSpeak server instances instead of one a bit bigger instance right?
I really expect that may the amount of instances are the cause for the problem (but thats not for sure!)

Ad you said, having a load of 0.2 should be totally fine, but it would be interesting what the whole machine load would be. Since even thou you may only have that less the machine is may used by other users.

Is there anything in the log? I suspect some warnings about delays?

Are you using the default config.yml?

PS: Sorry for the late delay, but had to debug a really nasty WebRTC issue which took me 8hr strait with full concentration.

 

[eduardoroeder]

Just a follow up to say that this is the same issue I had couple months ago... Still have. The solution for me was to split up into multiple machines, since no actual fix happened for having more than 1300 users in a single server...

Btw, have over 5k users in the same machine on TS3 server without any problem, while if I let the TeaSever go over 1300, TeaServer connected users start lagging.

 

[WolverinDEV]

Just a follow up to say that this is the same issue I had couple months ago... Still have. The solution for me was to split up into multiple machines, since no actual fix happened for having more than 1300 users in a single server...

Btw, have over 5k users in the same machine on TS3 server without any problem, while if I let the TeaSever go over 1300, TeaServer connected users start lagging.

Few months ago? I guess it might be better now (depending on your vs a lot better) with the 1.4.12 connection statistic performance improvement.
But I'm actually really surprised that TeaSpeak seems to start lagging after 1.3k users. I know some instances which have a lot more (2k~3k users) and don't have issues at all

 

[eduardoroeder]

C'mon @WolverinDEV... I've created at least 3 posts about this... Said more than 5 times in other threads. Don't be surprised lol. You even had my db for a couple of times, sent me different sets of configurations...

But for now, I've updated to the latest version but didn't manage to get past the 1.3k limit yet to vouch for that. Isn't quite straight forward to keep moving snapshots between TS3/Tea since some permissions always mess up...

 

[WolverinDEV]

HAHA Well I know you, but I also know some others so I'm a bit surprised by these differences. Not the actual fact.

Yes, permissions is a thing, but since TeaSpeak has a lot more which are sometimes not really derivable from the TS3 permission set....