Hi,
Are there any benchmarks available for tyk pump ? Or any minimum recommended configurations to build a foolproof, tyk gateway to tyk pump flow of data without any disruptions?
I don’t think we have this, as this depends on your traffic and maybe your pump backend of choice. But this FAQ could give you a good start.
Could you give some example of the kind of disruptions you envision?
The flow of analytics is depicted in our documentation. In short, analytics go into Redis temporarily before pump sends it to your persistent storage of choice. A disruption could happen on the Redis side of things. So configuring your Redis to be optimal would help. Also knowing your potential traffic can assist in setting the proper pump config. A few examples:
- purge_delay
- purge_chunk
- analytics_storage_config.optimisation_max_active
If a single pump is not optimal, then you can always scale horizontally.
If under extreme load then you can enable the use of multiple analytic keys. This is especially useful when your Redis is clustered.
It is worth mentioning that you can separate your analytics storage in Redis for better performance.
So my system configuration is :
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Stepping: 7
CPU MHz: 2593.905
BogoMIPS: 5187.81
Virtualization: VT-x
Hypervisor vendor: Microsoft
We have redis hosted on the same server as tyk gateway. On average we have around 9 GB of free RAM. Per second we recieve around 200 requests. Each would generate around 0.5 MB of data. So I want to know can Tyk Pump pump data fast enough to my database before my redis server fills up.
Is there any way to find out pumping speed of my current configuration?
Please let me know if there is any additional data required.
Your help is really appreciated!
Thanks 
Well the way to know pump isn’t performing well is if the actual purge is taking more time than the value configured for the purge_delay. So in essence pump may be only as fast as the place you store your data.
Pump fetches data every N seconds (purge_delay) + time to process the batch. Considering you are pumping to SQL, you may want to consider setting the batch_size.
I think a single pump can work fine with 200 requests per second. The size of data is what I am unsure of but I think it should be more than enough with your spec. If an issue occurs with your pump backend and somehow analytics are not getting purged, then you may want to ensure your Redis isn’t filled up by setting storage_expiration_time both on the gateway and pump