Bypass ElasticSearch 7 Max Rows Count limitation

Situation

In ElasticSearch7, when you do a large generic query, it will only select a specific maximum number of elements. This number is set to 10000.

When testing on ElasticSearch 5.6.11

With a simple query, we can have 110K items and more.

Here's an example: See image es-5-6-11.png

When testing on ElasticSearch 7.10.0

With the same query, we can have only 10K items. It does not impact the browsing experience. It just won't display the correct number of information anymore, which can be confusing for some at first.
Here's an example: See image es-7-10-0.png

Please note that the goal behind that is only to display the number of elements returned.

How to fix that?

There are 2 solutions:

1. If the elastic search is not to be updated in the longer term, then we could use the scroll functionality.
   
   Using Postman, you can execute the following: See image es-scroll.png
   
   Simply by adding the parameter 'scroll' with the value '1m' (1 million), it returns what we want in 37ms. We would do it on our side, but this code is handled in ElasticLinq.ElasticConnection and we cannot customize anything in there.
   
   A simple way to do it for you would be to handle it in your code and add a Setting in the Admin Console to let us activate or not that functionality.

2. If the elastic search will be updated in the short term, then we could use the searchAfter functionality.
   
   Since the scroll is deprecated based on the official documentation, we should use what they call the searchAfter functionality. But, that implementation might be a little more complex than the other. You will need to update the body of the request to handle that one.

Thank you