Site Search Response times and Employing a Cached Feed

Currently, the search page routines on Griffmonsters Walks use a script to pull back all posts from the feed as json which then can be interrogated. This has to be performed in a recursive manner as each feed request only contains a certain number of entries, for whatever reason. Therefore, to return all entries, a feed request has to count the entries in the returned data and use this number to make another feed request with the appropriate start-index value, recursing over the feed requests until a feed with no entries is returned.

https://griffmonster-walks.blogspot.com/feeds/posts/summary?alt=json&start-index=38

This works well, albeit the user has to wait a few seconds until all items have been retrieved. As the number of walks on the site increases, this wait time will get progressively worse and it has been something that has been in the back of my mind for some time now. To which, in recent days I have a proposed solution - a cache of feed data. After some research, the direction I am currently testing is a cache generated on GitHub using a simple script and node.js to run a similar routine to what we currently have on the search pages. This, combined with a GitHub workflow that can regenerate the cache file provides a mechanism to provide a cached copy of the full feed available via a uri request to GitHub. To keep the size down, the cache is filtered with only the components required by the search routines.

Initial tests are looking promising, although in some cases there was a 10+ second delay in retrieving the cached data. With some more investigation and tweaks, plus taking advantage of the GitHub CDN, this now seems to be lightning fast compared to the present routines. There is more testing to do, but currently a running test page is looking good, and the promise of more responsive search pages are on the horizon.

To make this complete, there is already a daily cache refresh in the GitHub workflow, plus a routine in my local ant workflow that can be fired when deploying new or updated posts, this will trigger a cache update, and a CDN purge of the data.

I will keep you updated on progress..