- Presto doesn't use the (more moderate) YARN resource manager and has its own configuration that is more tailored to ad-hoc queries. So the commonly advised ratio of 10:1 (that is for every 10 cores of YARN workers, 1 core of ODAS is sufficient) won't work well for Presto. What is a better sizing strategy?
- With R4.4xlarge as EMR instance type, it was discovered that a maximum of 64 outbound connections can be made from a single instance (by default, based on 2 x 16 vCPUs for Hyperthreading, and another 2x for active stages). ODAS worker can handle 255 worker connections (by default). Based on this, the Presto cluster should be sized with 3*64 outbound connections, that is, 192.
- This number of connections can be handled by 1 ODAS worker with 255 threads, while keeping a buffer on the ODAS worker side.
- Before scaling up the EMR nodes, you need to look at ODAS worker sizing and must match the number of ODAS workers to the number of EMR nodes in the given ratio.
- Based on this analysis and current testing:
- 3:1, that is, three EMR nodes for every ODAS node of the same instance type is the better sizing strategy for Presto.