We go through several stages before it hits prod. Our test environment -> the dev environment -> staging -> prod and still we are bitten by istio.
A lot of the problems we have seen do not manifest themselves until you get significant traffic in the cluster.
Luckily, it is one of our goals this year to make our setup testable.
Our setup is:
- an ingressgateway running on each node so that our SIEM can get IP addresses
- 1000s of virtual service entries for all the services running (40 per namespace x 25+ namespaces)
- 100s of deploys a day which causes pilot to update all the time
- the clusters are small with like 50 nodes
- we have way too many services and consolidation is slow
- we haven't been able to upgrade to the Istio operator yet because there is an outstanding bug that breaks everything
