Daily discussion thread \ stacker news

pull down to refresh

11 comments \ @saloon 10 Jan 2022 \ yesterday

This is a thread for random discussions that gets posted everyday at 5am central.

Tell us what you're doing today, ask questions, or vent about your life. Whatever you want, let it rip!

0 sats \ 9 replies \ @k00b 10 Jan 2022

Working on social media previews today. Currently when twitter/telegram/mastodon/etc request the little page preview I spin up a headless chome browser and screenshot the page - all inline and on demand. This has problems:

I do this on the web servers
headless chrome is not a lightweight process
these requests can come in large batches in a short time frame

For awhile, this was DOSing SN until I put a rate limiter on it. That's not ideal though because:

in some cases it prevents the image preview from showing up if the requesting service doesn't intelligently retry
it's still a strain on the web servers

Instead what I want to do is transactionally queue a job to screenshot the page every time a post/page is updated, store the image in block storage (e.g. S3), and serve that. One problem with this is determining when to screenshot non-post pages like the homepage, recent, user profiles, top, etc. I guess I can daily screenshot those pages but it's relatively inelegant.

1 sat \ 5 replies \ @theshanergy 10 Jan 2022

Why not just cache the generated images but still generate them on demand?

1 sat \ 3 replies \ @theshanergy 10 Jan 2022

btw, I am doing the exact same thing on bitcoininflationindex.com. I use netlify for my frontend and use their "On-demand builders" to generate images which are cached on the edge.

https://docs.netlify.com/configure-builds/on-demand-builders/

Works perfect for this use-case. You could spin up a netlify microservice specifically for this purpose and run it separate from the site.

Here is my code for reference: https://gist.github.com/theshanergy/42fd16010cc968b47f93df1a5413a6f1

1 sat \ 1 reply \ @k00b 10 Jan 2022

That’s an option I’m thinking about. I don’t use netlify but I could do something like this with AWS lambda.

0 sats \ 0 replies \ @247bf84fda 25 Aug freebie

deleted by author

0 sats \ 0 replies \ @247bf84fda 25 Aug freebie

deleted by author

0 sats \ 0 replies \ @247bf84fda 25 Aug freebie

deleted by author

0 sats \ 1 reply \ @k00b 10 Jan 2022

Alternatives ...

I still do it on demand but cache the result in block storage where it goes stale after some period of time
I spin up a server dedicated to taking these screenshots on demand