Starting to experiment a little with using data scraped from the destination of link topics

Deimos
Wed, 12 Sep 2018 00:56:48 +0000

This is very minor so far, but I think it's good to have a topic devoted to it so that people have somewhere to discuss it, instead of having it come up randomly in topics that it applies to.

I've recently started scraping some data about the destination of link topics using Embedly's "Extract" API (Embedly was kind enough to give me a reasonable amount of free usage since Tildes is a non-profit). You can put in the url of an article/video/etc. on that page to get an idea of what sort of data I can get from it, if you'd like to see for yourself.

I've only just started tinkering with it, and so far the data is only being used in two small ways:

1. Tweets now display the entire text of the tweet on the topic listing page, similar to the "excerpt" from text topics. You can see an example here.

2. On topic listings, the date that an article was published will be shown (after the domain name) if the publication date was at least 3 days before it was submitted. There are a few examples in the recent posts in ~misc

I'll probably adjust this threshold, but I'd like it to be an amount of time where the age of the content might feel "significant". It would also be possible to just show this info all the time, but I think the topic listings are already fairly cluttered so it's probably best to hide it when it's not interesting/significant.

As I said, these are very tiny changes so far, but there are lots of other possibilities that I hope to start using before long. I've mentioned this before, but something I'd really like to do overall is try to bring in more data about the links where it's possible to be able to show things like the lengths of videos and so on.

Let me know if you have any thoughts about it or notice any issues, thanks.

Comments URL: https://tildes.net/~tildes.official/696/starting_to_experiment_a_little_with_using_data_scraped_from_the_destination_of_link_topics

Votes: 57

Comments: 32