• Hey Guest, we're evolving the future of TeaSpeak.
    You're invited to join the discussion here!

Pending Voice freeze on 70+ users on channel

eduardoroeder

Fanatic member
Premium User
I've got this setup in Digital Ocean.

After 60+ players on a channel the whole Server starts gettings a voice freezing every couple seconds. Anyone experiencing the same thing?

---- Automatically Merged Double Post ----

I can send the logs, but I don't know what is causing. It can be the bandwidth or other things as well. I didn't notice any packet loss... But the ping variation went from 1 to 200-300, with most of time being 20-40
 
Last edited:

farhadhelix

TeaFanatic
I've got this setup in Digital Ocean.

After 60+ players on a channel the whole Server starts gettings a voice freezing every couple seconds. Anyone experiencing the same thing?

---- Automatically Merged Double Post ----

I can send the logs, but I don't know what is causing. It can be the bandwidth or other things as well. I didn't notice any packet loss... But the ping variation went from 1 to 200-300, with most of time being 20-40

hi . did you check your vps CPU usage when this happens? post the TOP command output here if you dont know what it does
 

eduardoroeder

Fanatic member
Premium User
The CPU was ranging from 10 to 100%, it continued even after I closed the CPU-consuming programs. RAM had about 6,5GB free. Forgot to mention it because I was looking deep into the problem hehe

I've switched hosts and it still didn't happen yet. But the same situation didn't happen aswell, so can't confirm that this is a network related issue (looks like it is tho).
 

eduardoroeder

Fanatic member
Premium User
No. TeaSpeak was low in CPU usage. Just as a normal instance running.

By "it continued" I meant the symptom, not the 10%-100% CPU usage. After I closed those programs the CPU shrinked to it's default usage for a standard TeaSpeak server. Same thing happened with RAM.
 

yamano

TeaSpeak Team
Staff member
TeaTeam
If you're using the teaspeak server v1.4.1-beta1, downgrade it to v1.4.0-beta or v1.4.1-beta ;)
 
Last edited:

WolverinDEV

TeaSpeak Team
Staff member
Administrator
If you're using the teaspeak server v1.4.1-beta1, downgrade it to v1.4.0-beta or update it to v1.4.1-beta2 ;)
Use 1.4.1-beta0
Beta 1 and 2 are fucked up currently

---- Automatically Merged Double Post ----

may use beta 3 :D

---- Automatically Merged Double Post ----

Well the main issue is (i think) 70 ppl in one channel and a lot of them talking.
Depending on your config.yml and your instance you should may change some settings.

As a question: How many VS do you have with how many users?
As well how many cores & threads does your VPS have?
 
Last edited:

eduardoroeder

Fanatic member
Premium User
Like I said on the main topic, I've now changed the host provider just after those issues and they seem to have disappeared.

All the TeaServer described here are the 1.4.1-beta (no 0, 1, 2 or 3)

I had initially a VPS on Digital Ocean with 3 shared CPU core, 1gb ram, running TeaSpeak with 7 virtual servers, 300 slot each, 100~ clients/VS + 7proccesses which peaks to 100% cpu usage several times per minute, having about 100mb ram left and then 4gb swap space. This was working fine for 2-3 weeks.

I needed to open more of those proccesses (it's a bot instance which uses 80-110mb ram each, and extensivelly does some crawling on a game site to catch information and display on the TS3 server channels), so I decided to make a upgrade: I switched on 8/11/2019 to a VPS with 2 Dedicated CPU core + 8gb RAM so i could increase my clients amount. I proceeded then to add one more Virtual Server with 400 slots, with more or less 280 people connecting. Plus one more Bot instance. This made my CPU usage to go almost always over 80%, going some times to 100% (looked like it was the same behaviour as before, but a little stronger). In this situation, I joined all server instances to watch out what were happening: on 4 of those 8 instances they were doing some kind of meeting. One server had 200 player on one channel and 80 on another; second server had 80 players on a single channel; third and fourth servers had 50~ players on a single channel.

Sometimes it just lagged one instance, other times all of the instances (even the other 4) lagged also.

I further tryied to resize my VPS from 2 dedicated to 4 shared CPU cores, and from 8gb ram to 6GB ram, but it didn't fix aswell.

Then I started looking that this happened just after this last instance joined. I talked to them and they had some similar situations on the last 2 hosting providers. I couldn't be that a simple server cant handle 200 players on a single channel, since I've done this several times on other people TS servers and never had any trouble at it.

So, it could be two things: either someone was bugging the server (DDoS, some kind of invalid packet, some kind of exploit, idk) or the server network could not handle everything together, since the more people in one channel the more bandwith the server needs. 200 players accross 100 channels (2 players in each channel) requires little-to-no bandwith, while 200 players in one channel requires massive bandwith.

Then I decided to change the host. In couple of hours I switched to OVH (where my concorrent hosts their TS and never had this kind of issue, and where I know they have DDoS protections) and the problem vanished. Just copied the DB, reinstalled everything etc and it was all good. The whole weekend it's been some meeting with well over 100 players on some servers and yet I(they) couldn't notice any voice lagging.

Probably it isn't related to TeaSpeak, but yet it can still be.
 

yamano

TeaSpeak Team
Staff member
TeaTeam
OVH is a good, if not the best option to a host provider!
I'm a OVH client since 2004/2005 ;)
 

REDOSS

TeaSpeak Team
Staff member
TeaTeam
OVH is a good, if not the best option to a host provider!
I'm a OVH client since 2004/2005 ;)
But in any case, 50, 100, 200 clients in one channel it requires good channel bandwidth and good processor. Most likely, his previous hoster does this not provide.
 

eduardoroeder

Fanatic member
Premium User
But in any case, 50, 100, 200 clients in one channel it requires good channel bandwidth and good processor. Most likely, his previous hoster does this not provide.
Bandwidth of course, but processor didnt play a thing there, since I tryied turning off all the "bots" while the issue were happening, lowering the CPU usage down to ~15%(or less).

---- Automatically Merged Double Post ----

I also set up the default channel codec to Opus Voice with 5 of quality
 
Last edited:

WolverinDEV

TeaSpeak Team
Staff member
Administrator
Well I love to see such detailed investigation.
Normally users just blame the software for errors :D

And you really got the main point.
100 users spread across a lot of channel make less traffic than 100 users in one channel.
Simple formula: 300bps * (<user count> - 1) ^ <concurrent speakers>

Another big thing if you're having servers of that scale would be to play around with the thread settings within the config.yml :)
The second option which could be adjusted is the encryption. On this scale it makes a huge difference if voice packets get encrypted or not.

Some further questions:
How many users do you have as well what is the resource usage?
CPU, RAM, Traffic and ofc which CPU you're using.

And as a takeaway for me:
I never thought really that channels with 100+ users had a real use case, but I might should do some code optimizations to improve performance at that scale. Specially within the IO handler I've some ideas, but I'm afraid to apply them because this could have some quite unexpected side effects.
 

eduardoroeder

Fanatic member
Premium User
Well I love to see such detailed investigation.
Normally users just blame the software for errors :D

And you really got the main point.
100 users spread across a lot of channel make less traffic than 100 users in one channel.
Simple formula: 300bps * (<user count> - 1) ^ <concurrent speakers>

Another big thing if you're having servers of that scale would be to play around with the thread settings within the config.yml :)
The second option which could be adjusted is the encryption. On this scale it makes a huge difference if voice packets get encrypted or not.

Some further questions:
How many users do you have as well what is the resource usage?
CPU, RAM, Traffic and ofc which CPU you're using.

And as a takeaway for me:
I never thought really that channels with 100+ users had a real use case, but I might should do some code optimizations to improve performance at that scale. Specially within the IO handler I've some ideas, but I'm afraid to apply them because this could have some quite unexpected side effects.
Well, after the whole month the issue appeared again, but just small enough to still be usable. I've got 20 virtual servers with 1800 users connected accross all them. I've got a 4-core + 8gb ram VPS. What should I tweak in the IO handler as you said you've got some ideas? The encryption is disabled.
 

latters

Well-known member
Well I love to see such detailed investigation.
Normally users just blame the software for errors :D

And you really got the main point.
100 users spread across a lot of channel make less traffic than 100 users in one channel.
Simple formula: 300bps * (<user count> - 1) ^ <concurrent speakers>

Another big thing if you're having servers of that scale would be to play around with the thread settings within the config.yml :)
The second option which could be adjusted is the encryption. On this scale it makes a huge difference if voice packets get encrypted or not.

Some further questions:
How many users do you have as well what is the resource usage?
CPU, RAM, Traffic and ofc which CPU you're using.

And as a takeaway for me:
I never thought really that channels with 100+ users had a real use case, but I might should do some code optimizations to improve performance at that scale. Specially within the IO handler I've some ideas, but I'm afraid to apply them because this could have some quite unexpected side effects.
I have clients reported this problem.

I am currently a hoster that has a lot of clients and growing, and all sorts of things happen on servers ...

Would you have a suggestion of what to optimize in config.yml to optimize this?
Another thing, how do I disable voice encryption?
 

eduardoroeder

Fanatic member
Premium User
I've found the link yesterday, so I've already changed some settings in order to prevent that stuff. It looks like it's better but still happening after tweaking. Can you take a look and point out what should I look after?

https://paste.ee/p/fqHCy -> Here is my config.yml
 

WolverinDEV

TeaSpeak Team
Staff member
Administrator
Are you using the web client?
I guess no so you could change io_loops to 1
execute_limit => 8
io_limit => 8

You're currently wasting a lot of performance due to too much threads :)
 

eduardoroeder

Fanatic member
Premium User
Not using the web client.. Should I lower the web one then?
execute_limit was 10, io_limit was 10.
I've followed what you said in the other thread

An example would be:
io_min: 2
io_per_server: 1
io_limit: 6
Result:
For 1 VS: Two IO threads will run
For 3 VS: Three IO threads will run
For 6 VS: Six IO threads will run
For 8 VS: Six IO threads will run
And thought that it was a bottleneck having 10 threads to 20VS (10threads to 15VS should be alright, but when escalating can lead to problems). So increased this to try to bypass this bottleneck.

io_min: 4
io_per_server: 4
io_limit: 30
Result:
For 1 VS: 4 IO threads will run
For 3 VS: 12 IO threads will run
For 6 VS: 24 IO threads will run
For 8 VS: 30 IO threads will run
For 12 VS: 30 IO threads will run
For 20 VS: 30 IO threads will run

The same goes for execute_per_server and events_per server, tryied to escalate a bit up to be more aggressive towards this performance issues.

If I do 'reload config' from the serverquery in 1.4.4 will it be reloaded? This changes in the threading can be updated? What does the other variables in this section means?

Thanks

---- Automatically Merged Double Post ----

Could be a bottleneck of sqlite don't handling multiple transactions?
 
Last edited: