Validators require 128Gb RAM
In a previous update, we implemented a Fast State Serializer, which reduces blockchain state serialization time from 18 hours to ~50 minutes.
This frees up more resources for the validator to work on validating transactions and blocks, which is necessary for network performance in general.
Fast serialization works only if there is enough RAM.
According to requirements validator should have at least 128Gb RAM. If you have less - please upgrade your hardware.
This message is for validators only.
In a previous update, we implemented a Fast State Serializer, which reduces blockchain state serialization time from 18 hours to ~50 minutes.
This frees up more resources for the validator to work on validating transactions and blocks, which is necessary for network performance in general.
Fast serialization works only if there is enough RAM.
According to requirements validator should have at least 128Gb RAM. If you have less - please upgrade your hardware.
This message is for validators only.
Mainnet Validators
Please be prepared to vote on Wednesday August 21 at 8:00 UTC for new transaction executor behavior, dispatch queue activation and increasing minimal split.
Details can be found here. Proposed changes will allow network to more evenly distribute load, improve stability of block generation and serialization process.
All validators MUST be updated to the latest version before voting. Target versions:
mytonctrl
validator
Please be prepared to vote on Wednesday August 21 at 8:00 UTC for new transaction executor behavior, dispatch queue activation and increasing minimal split.
Details can be found here. Proposed changes will allow network to more evenly distribute load, improve stability of block generation and serialization process.
All validators MUST be updated to the latest version before voting. Target versions:
mytonctrl
7e90e26validator
140320bMainnet validators
Please check efficiency of your validators. In case of low efficiency or frequent crashes (including OOM) immediately contact @mytonctrl_help_bot
In particular we ask the following validators to check their nodes
Please check efficiency of your validators. In case of low efficiency or frequent crashes (including OOM) immediately contact @mytonctrl_help_bot
In particular we ask the following validators to check their nodes
2,AE083C661DAD64F734CCBF5A4BEDF398BC4CF5A6BF454E376BB4B4437CBB4C9C
9,D4AB33E3C1F558143BF63ECF82B26B6A1AD635149AFCDA508DFCF53DDEF49EA1
10,20ED0665410992AEC5F476CAC9D452D89B1C34C210C48C1D578D1B46A82A4088
65,481532E012CB8F7E1C1B179F52E31D9B4F20EFE1BB032196E2690975E5989729
88,F82905F3161A1F7108B4A3807FC5C970B2B5B45E58219C033F1614582DDEAAC5
90,95343C09F5D4F1830C8AE4C8577C66EE779FA723D0C8D10ACAFFAC9F346B1ECA
96,F1A0A4153857E5385884E4EFCE50FABBC00662F54139A58CAD6DCE97529F35F5
Mainnet validator
If your validator has index < 100 please be prepared to urgent action at 04:00 UTC (Wed Aug 28 2024 04:00:00 GMT+0000). Please set alarm.
P.S. Due to high recent activity (>20m transactions in recent 2 days), garbage collection overloaded many of validators for enough time for them to lost consensus with each other. To restore consensus back, validators need to be restarted at about the same time with specific flags.
If your validator has index < 100 please be prepared to urgent action at 04:00 UTC (Wed Aug 28 2024 04:00:00 GMT+0000). Please set alarm.
P.S. Due to high recent activity (>20m transactions in recent 2 days), garbage collection overloaded many of validators for enough time for them to lost consensus with each other. To restore consensus back, validators need to be restarted at about the same time with specific flags.
Mainnet validators with index <100
Please restart your nodes with updated flags at 4:00 UTC
1. open
2. add flags
3. restart validator:
Please restart your nodes with updated flags at 4:00 UTC
1. open
/etc/systemd/system/validator.service2. add flags
-F 39987437:600844:7 -F 39987437:600845:7 --state-ttl 86400 to the end of ExecStart3. restart validator:
systemctl daemon-reload
systemctl restart validator
Mainnet validators with index <100
So far not enough validators restarted their nodes with correct flags.
If you didn't restarted your nodes with new flags yet, please do it ASAP.
So far not enough validators restarted their nodes with correct flags.
If you didn't restarted your nodes with new flags yet, please do it ASAP.
Mainnet validators
Block production is restored and stable now. Thank you for cooperation!
For now you can delete
Note: there is NO need to restart validators after removing flags.
In coming days, we will came with update for the node to mitigate recently discovered issues. Please stay tuned.
Block production is restored and stable now. Thank you for cooperation!
For now you can delete
-F 39987437:600844:7 -F 39987437:600845:7 from ExecStart when it will be convenient for you. After that please run:systemctl daemon-reload
Note: there is NO need to restart validators after removing flags.
In coming days, we will came with update for the node to mitigate recently discovered issues. Please stay tuned.
URGENT - Mainnet validators
Please update your nodes and then restart.
In mytonctrl run:
If you don't use mytonctrl manually switch node to stable_testnet branch.
If you have multiple validators you can update them all at once. Please stay tuned for a few hours, in case of subsequent instructions.
Please update your nodes and then restart.
In mytonctrl run:
upgrade stable_testnetIf you don't use mytonctrl manually switch node to stable_testnet branch.
If you have multiple validators you can update them all at once. Please stay tuned for a few hours, in case of subsequent instructions.
URGENT - Mainnet validators
Please make sure you have done the previous update. If you haven't please do.
Then update node flags:
1. open
2. remove
3. add flags
4. restart validator:
Please make sure you have done the previous update. If you haven't please do.
Then update node flags:
1. open
/etc/systemd/system/validator.service2. remove
-F 39987437:600844:7 -F 39987437:600845:7 if they are still present3. add flags
-F 39991868:601006:7 --catchain-max-block-delay 0.5 to the end of ExecStart4. restart validator:
systemctl daemon-reload
systemctl restart validator
Forwarded from TON Community
Dear TON Validators,
We are releasing an important TON Validator Information Form today. This form is crucial for improving our network's performance and stability, especially in light of anticipated increased loads.
- The form collects essential data about your validator setup.
- Your input is vital for network optimization.
- Please complete the form today.
- Your prompt response will help us collaborate effectively on network improvements.
Access the form here
Your cooperation is critical for the TON network's continued success. Thank you for your immediate attention to this matter.
We are releasing an important TON Validator Information Form today. This form is crucial for improving our network's performance and stability, especially in light of anticipated increased loads.
- The form collects essential data about your validator setup.
- Your input is vital for network optimization.
- Please complete the form today.
- Your prompt response will help us collaborate effectively on network improvements.
Access the form here
Your cooperation is critical for the TON network's continued success. Thank you for your immediate attention to this matter.
Paperform
TON Validator Information Form
TON Validator Information Form collects essential data from validators for network management.
Important Instructions for TON Validators
Dear TON Validators, please follow these crucial steps to ensure network stability this week and to avoid new slashing penalties in the future.
Essential Actions:
1) Verify you're on the
2) Ensure your hardware meets or exceeds system requirements: https://t.iss.one/tonstatus/102. Upgrade if necessary, one at a time.
3) We imperatively request you to use mytonctrl.
In your mytonctrl console:
- Update to the latest version:
- Enable telemetry:
4) Set up monitoring dashboards for RAM, Disk, Network, and CPU usage. For technical assistance, contact @mytonctrl_help_bot.
DOGS Project Alert:
This week, the DOGS gaming project (50 million active users) is minting and listing, causing increased network load. We experienced two network outages on August 27 and 28. Please:
1) Stay available this week and respond within 1 hour, around the clock.
2) Closely monitor your hardware. Contact @mytonctrl_help_bot immediately if you need help.
3) Follow @tonstatus and be ready to apply urgent updates if necessary.
‼️ Updates on Slashing Mechanics for TON Validators:
The TON Core team is working on implementing of new slashing penalties for non-performing validators. Executing the actions above is crucial to mitigate the risk of losing rewards for validating the network.
Your cooperation is vital for maintaining network stability and TON's prosperity. Thank you for your prompt attention to these matters.
Dear TON Validators, please follow these crucial steps to ensure network stability this week and to avoid new slashing penalties in the future.
Essential Actions:
1) Verify you're on the
stable_testnet branch (commit 97c57c3). If not, update: https://t.iss.one/tonstatus/130. For multiple validators, update one at a time.2) Ensure your hardware meets or exceeds system requirements: https://t.iss.one/tonstatus/102. Upgrade if necessary, one at a time.
3) We imperatively request you to use mytonctrl.
In your mytonctrl console:
- Update to the latest version:
update master- Enable telemetry:
set sendTelemetry true4) Set up monitoring dashboards for RAM, Disk, Network, and CPU usage. For technical assistance, contact @mytonctrl_help_bot.
DOGS Project Alert:
This week, the DOGS gaming project (50 million active users) is minting and listing, causing increased network load. We experienced two network outages on August 27 and 28. Please:
1) Stay available this week and respond within 1 hour, around the clock.
2) Closely monitor your hardware. Contact @mytonctrl_help_bot immediately if you need help.
3) Follow @tonstatus and be ready to apply urgent updates if necessary.
‼️ Updates on Slashing Mechanics for TON Validators:
The TON Core team is working on implementing of new slashing penalties for non-performing validators. Executing the actions above is crucial to mitigate the risk of losing rewards for validating the network.
Your cooperation is vital for maintaining network stability and TON's prosperity. Thank you for your prompt attention to these matters.
ANNOUNCEMENT: DECENTRALISED SYSTEM OF PENALTIES FOR POORLY PERFORMING VALIDATORS
The current system of penalising poorly performing validators will be fully operational next Monday, September 9.
How do validators determine that another validator has performed poorly?
The TON is supplied with the lite-client utility. In lite-client there is a
This command analyses how many blocks the validator should have processed, and how many it actually processed in a given period of time.
If the validator processed less than 90% of the expected number of blocks during a validation round, it is considered to be performing poorly and should be penalised.
Technical description of the process: https://github.com/ton-blockchain/TIPs/issues/13#issuecomment-786627474
When and by whom a complaint is filed?
After each validation round (~18 hours), the validator stakes of validators that participated in that round are on the Elector smart contract for another ~9 hours.
During this time, anyone can send a complaint against a validator who performed poorly in said round. This happens onchain on the Elector smart contract.
All validators do not need to send a complaint.
How a complaint is validated?
After each validation round, validators receive a list of complaints from the Elector smart contract and double-check them by calling
If a complaint is validated, they onchain vote in favour of that complaint.
These actions are built into mytonctrl and happen automatically.
If the complaint has 66% of the validators' votes (by their weight), a penalty is taken off from the validator's stake.
What is the size of the fine?
The amount of the fine is fixed and equals 101 TON, which is roughly equal to the validator's income per round.
Where does this fine go?
The fine is distributed among the validators minus network costs and a small reward (~8 TON) to the validator who sent the correct complaint.
When was this functionality made?
This functionality was made in mytonctrl back in February 2021 https://github.com/ton-blockchain/TIPs/issues/13.
The complaints and fines functionality in the Elector system smart contract was made initially at the time of network launch.
Why didn't the penalty functionality work well earlier?
The network fee for sending complaints was significant, which as the Toncoin exchange rate increased, made it uneconomical to send them.
This has now been fixed by optimising the complaint message.
In a week from 9 September 2024, an automatic complaint sender will start working on several nodes in the network.
Is the system decentralised?
Yes, anyone can send a complaint, the penalty is only applied by a quorum of validators on the network.
There is no way for anyone to single-handedly fine anyone.
How do I prepare for the start of the penalty system?
Since this functionality is already implemented in node, you don't need to do anything to make the system work.
Please make sure you're complying with the validator guidelines.
During the week, we will publish additional tools and best practices for monitoring and maintaining the effectiveness of your validator.
Will the penalty system get stricter in the future?
Yes, the audience and the number of transactions in TON is growing rapidly and it is vital that the quality of work is at its best.
The system will improve and fines will increase this year. All updates will be announced in advance.
It makes sense to set up hardware, monitoring and validator work properly. If you don't want to do this please consider using staking services https://ton.org/stake.
The current system of penalising poorly performing validators will be fully operational next Monday, September 9.
How do validators determine that another validator has performed poorly?
The TON is supplied with the lite-client utility. In lite-client there is a
checkloadall command.This command analyses how many blocks the validator should have processed, and how many it actually processed in a given period of time.
If the validator processed less than 90% of the expected number of blocks during a validation round, it is considered to be performing poorly and should be penalised.
Technical description of the process: https://github.com/ton-blockchain/TIPs/issues/13#issuecomment-786627474
When and by whom a complaint is filed?
After each validation round (~18 hours), the validator stakes of validators that participated in that round are on the Elector smart contract for another ~9 hours.
During this time, anyone can send a complaint against a validator who performed poorly in said round. This happens onchain on the Elector smart contract.
All validators do not need to send a complaint.
How a complaint is validated?
After each validation round, validators receive a list of complaints from the Elector smart contract and double-check them by calling
checkloadall.If a complaint is validated, they onchain vote in favour of that complaint.
These actions are built into mytonctrl and happen automatically.
If the complaint has 66% of the validators' votes (by their weight), a penalty is taken off from the validator's stake.
What is the size of the fine?
The amount of the fine is fixed and equals 101 TON, which is roughly equal to the validator's income per round.
Where does this fine go?
The fine is distributed among the validators minus network costs and a small reward (~8 TON) to the validator who sent the correct complaint.
When was this functionality made?
This functionality was made in mytonctrl back in February 2021 https://github.com/ton-blockchain/TIPs/issues/13.
The complaints and fines functionality in the Elector system smart contract was made initially at the time of network launch.
Why didn't the penalty functionality work well earlier?
The network fee for sending complaints was significant, which as the Toncoin exchange rate increased, made it uneconomical to send them.
This has now been fixed by optimising the complaint message.
In a week from 9 September 2024, an automatic complaint sender will start working on several nodes in the network.
Is the system decentralised?
Yes, anyone can send a complaint, the penalty is only applied by a quorum of validators on the network.
There is no way for anyone to single-handedly fine anyone.
How do I prepare for the start of the penalty system?
Since this functionality is already implemented in node, you don't need to do anything to make the system work.
Please make sure you're complying with the validator guidelines.
During the week, we will publish additional tools and best practices for monitoring and maintaining the effectiveness of your validator.
Will the penalty system get stricter in the future?
Yes, the audience and the number of transactions in TON is growing rapidly and it is vital that the quality of work is at its best.
The system will improve and fines will increase this year. All updates will be announced in advance.
It makes sense to set up hardware, monitoring and validator work properly. If you don't want to do this please consider using staking services https://ton.org/stake.
Tool for assessing the effectiveness of validators
The new version of mytonctrl has added a new command
Previous efficiency score from the
To update mytonctrl, type
Then type
Note that the previous round may not be displayed immediately after the update.
Note that the current round data becomes more accurate towards the end of the round.
Validators with an index less than 100 (sorted by effective stake) please ensure that your efficiency is greater than 90% (for the full round period).
Validators with an index greater than 100 will not receive penalties next week as they do not participate in masterchain validation, but please follow the validator guidelines, as you will be included in the penalty system in future updates.
If you need tech support please contract @mytonctrl_help_bot (validators only).
The new version of mytonctrl has added a new command
check_ef which outputs your validator efficiency data for the last round and for current round. This command retrieves data by calling checkloadall utility.Previous efficiency score from the
status command outdated and has been removed.To update mytonctrl, type
update in the mytonctrl console.Then type
check_ef in the mytonctrl console.Note that the previous round may not be displayed immediately after the update.
Note that the current round data becomes more accurate towards the end of the round.
Validators with an index less than 100 (sorted by effective stake) please ensure that your efficiency is greater than 90% (for the full round period).
Validators with an index greater than 100 will not receive penalties next week as they do not participate in masterchain validation, but please follow the validator guidelines, as you will be included in the penalty system in future updates.
If you need tech support please contract @mytonctrl_help_bot (validators only).
APIs for validation and effectiveness of validators
1) https://elections.toncenter.com/docs - use this API to get information about current and past validation rounds (cycles) - time of rounds, which validators participated in them, their stakes, etc.
Information on current and past elections (for the validation round) is also available.
2) https://toncenter.com/api/qos/index.html#/ - use this API to get information on the efficiency of validators over time.
This API analyses the information from the catchain and builds an estimate of the validator's efficiency. This API does not use the
Unlike
How to use:
- pass ADNL address of your validator and time inverval (
- get the result. If your
- It is important that your validator participates in validation and has the same ADNL address throughout the specified time period.
For example, if a validator participates in validation every second round - then you need to specify only those intervals when he participated in validation. Otherwise, you will get an incorrect underestimate.
- this works not only for masterchain validators (with index < 100) but also for other validators (with index > 100).
Recommendations
1) Please check the efficiency of your validator and in case of low efficiency - take action to fix the problem. Contact technical support @mytonctrl_help_bot if necessary.
2) Please set up dashboards to monitor your validators using these APIs.
1) https://elections.toncenter.com/docs - use this API to get information about current and past validation rounds (cycles) - time of rounds, which validators participated in them, their stakes, etc.
Information on current and past elections (for the validation round) is also available.
2) https://toncenter.com/api/qos/index.html#/ - use this API to get information on the efficiency of validators over time.
This API analyses the information from the catchain and builds an estimate of the validator's efficiency. This API does not use the
checkloadall utility, but is its alternative.Unlike
checkloadall which works only on validation rounds, in this API you can set any time interval to analyse the validator's efficiency.How to use:
- pass ADNL address of your validator and time inverval (
from_ts, to_ts) to API. For accurate result it makes sense to take a sufficient interval, for example from 18 hours ago the current moment.- get the result. If your
efficiency percentage field is less than 80%, your validator is not working properly. - It is important that your validator participates in validation and has the same ADNL address throughout the specified time period.
For example, if a validator participates in validation every second round - then you need to specify only those intervals when he participated in validation. Otherwise, you will get an incorrect underestimate.
- this works not only for masterchain validators (with index < 100) but also for other validators (with index > 100).
Recommendations
1) Please check the efficiency of your validator and in case of low efficiency - take action to fix the problem. Contact technical support @mytonctrl_help_bot if necessary.
2) Please set up dashboards to monitor your validators using these APIs.
Mainnet validators
Scheduled network update on September 11
We are asking validators to schedule a time on September 11 at 9:00 UTC for validator software update.
This update is mandatory and introduces catchain, serialization, network and collator configuration updates that optimize work of validators.
During preparation of this update we focused on minimal and safest changes which can be reliably tested in short period of time from one hand and substantially improve stability of validation from the other.
Scheduled network update on September 11
We are asking validators to schedule a time on September 11 at 9:00 UTC for validator software update.
This update is mandatory and introduces catchain, serialization, network and collator configuration updates that optimize work of validators.
During preparation of this update we focused on minimal and safest changes which can be reliably tested in short period of time from one hand and substantially improve stability of validation from the other.
Mainnet Validators
Please update your node software (see "Target versions"):
Target versions:
— mytonctrl:74e536b
— node:
If you are not using mytonctrl, check this instruction.
This update is mandatory for validators. Node changelog. Mytonctrl changelog.
After upgrade please remove from
If you have several validator nodes, please update them one by one (update, wait for synchronization, move to the next one).
P.S. We encourage validators to subscribe to @tonstatus_notifications channel with notifications of penalties for poorly performing TON validators.
Please update your node software (see "Target versions"):
update
upgrade master
Target versions:
— mytonctrl:
a467af5 (updated)— node:
1bef6dfIf you are not using mytonctrl, check this instruction.
This update is mandatory for validators. Node changelog. Mytonctrl changelog.
After upgrade please remove from
/etc/systemd/system/validator.service all recently added flags (they are not required anymore):-F *:*:*,--catchain-max-block-delay 0.5--state-ttl * (note, if you intentionally set specific state-ttl, keep it as you need)If you have several validator nodes, please update them one by one (update, wait for synchronization, move to the next one).
P.S. We encourage validators to subscribe to @tonstatus_notifications channel with notifications of penalties for poorly performing TON validators.
Block production in the shard 0x9000000000000000 has stopped. The other shards are working normally. The core team is investigating the problem.
Block production in the shard has recovered. The situation is currently under review.
The network is experiencing performance issues. Transactions may take longer than usual to complete. A fix is being worked on.