pull down to refresh
0 sats \ 4 replies \ @k00b 10 Jan 2022 \ on: Daily discussion thread
Working on social media previews today. Currently when twitter/telegram/mastodon/etc request the little page preview I spin up a headless chome browser and screenshot the page - all inline and on demand. This has problems:
- I do this on the web servers
- headless chrome is not a lightweight process
- these requests can come in large batches in a short time frame
For awhile, this was DOSing SN until I put a rate limiter on it. That's not ideal though because:
- in some cases it prevents the image preview from showing up if the requesting service doesn't intelligently retry
- it's still a strain on the web servers
Instead what I want to do is transactionally queue a job to screenshot the page every time a post/page is updated, store the image in block storage (e.g. S3), and serve that. One problem with this is determining when to screenshot non-post pages like the homepage, recent, user profiles, top, etc. I guess I can daily screenshot those pages but it's relatively inelegant.
Why not just cache the generated images but still generate them on demand?
reply
btw, I am doing the exact same thing on bitcoininflationindex.com. I use netlify for my frontend and use their "On-demand builders" to generate images which are cached on the edge.
Works perfect for this use-case. You could spin up a netlify microservice specifically for this purpose and run it separate from the site.
Here is my code for reference: https://gist.github.com/theshanergy/42fd16010cc968b47f93df1a5413a6f1
reply
That’s an option I’m thinking about. I don’t use netlify but I could do something like this with AWS lambda.
reply
Alternatives ...
- I still do it on demand but cache the result in block storage where it goes stale after some period of time
- I spin up a server dedicated to taking these screenshots on demand
reply