lonk's recent activity

  1. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    I am adding functionality to linklonk.com that would let users share their collections of liked content through a public link. Trying to decide how to structure the public link - should it be...

    I am adding functionality to linklonk.com that would let users share their collections of liked content through a public link. Trying to decide how to structure the public link - should it be randomly generated (like Google Docs) or should it be controlled by the user.

    Trying to get some user feedback here https://linklonk.com/item/3854292744564965376

    One user suggested: let the user control the prefix and make the suffix random (and therefore hard to guess) like: /collection/movies_QRadZdIWL.

    2 votes
  2. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    (edited )
    Link
    It's been 8 months since I "launched" my personal project linklonk.com by announcing it on Tildes. I want to post about it on HackerNews next (planning for this Friday) to get a few more active...

    It's been 8 months since I "launched" my personal project linklonk.com by announcing it on Tildes. I want to post about it on HackerNews next (planning for this Friday) to get a few more active users. Since the amount of traffic the post will get is not predictable, I did some load testing to make sure it won't die. The load test showed that my api server can do about ~10 requests per second. This led to query optimization, code profiling and more caching which improved the performance to ~200 requests per second. I posted the details here.

    I'm now working on adding more caching to the Go server. Right now the popular items are calculated by running a query over ratings and grouping by item id. This is not optimal. It would be much more efficient to maintain a table of [item_id, num_votes] for each of the time periods that we are interested in (last 24 hours, last 7 days, last 30 days, all time).

    This table could be added to the database. But I'm adding it to the Go server as an in-memory structure. The nice thing about using Go is that the server is just one process so you can have state that is shared across all requests.

    I will post the numbers on how much this improves the rate of requests under a load test. I expect >1000 requests per second.

    Edit: with more caching and lazy initialization got it to 1000.

    ab -n 1000 -c 1000 <ip>:<port>/api/recommendations?timePeriod=week
    Document Path:          /api/recommendations?timePeriod=week
    Document Length:        15088 bytes
    
    Concurrency Level:      1000
    Time taken for tests:   0.959 seconds
    Complete requests:      1000
    Failed requests:        0
    Total transferred:      15176000 bytes
    HTML transferred:       15088000 bytes
    Requests per second:    1042.27 [#/sec] (mean)
    Time per request:       959.448 [ms] (mean)
    Time per request:       0.959 [ms] (mean, across all concurrent requests)
    Transfer rate:          15446.71 [Kbytes/sec] received
    
    Percentage of the requests served within a certain time (ms)
      50%    491
      66%    627
      75%    710
      80%    728
      90%    832
      95%    858
      98%    901
      99%    919
     100%    950 (longest request)
    
    3 votes
  3. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    I finished the newsletter functionality for my hobby project https://linklonk.com (announcement post). The email delivery is done through a free SendGrid account that allows up to 100 email per...

    I finished the newsletter functionality for my hobby project https://linklonk.com (announcement post).

    The email delivery is done through a free SendGrid account that allows up to 100 email per day. This is more than enough with the few active users we have. The SendGrid's api library was easy to use from Golang (the language the server is implemented in).

    I needed to render html of the email messages on the server side. This is not something I had to do for the main web ui because there I use JSON to send data to an Angular-based client. To render the emails I used Golang's templating package html/template. It was quite nice.

    One of the interesting challenges was to get the message delivered at the time the user configured. When the user configures that they want their email to be delivered at 10:00am every Tuesday it means that it should be at that time in the user's local time zone (America/New_York). Usually the times on the server side are usually expressed in UTC. But here we need to store the user's selected time ("10:00") and the local time-zone (America/New_York) and calculate when is the next Tuesday 10:00am in America/New_York. The reason we cannot convert it to UTC is because:

    1. America/New_York does not have a constant offset from UTC because of daylight saving.
    2. When it is Tuesday in UTC, it could be still Monday in a different time-zone, and vice versa. So the next day of the week calculation needs to be done in the user's local time zone.

    On the web-version the users can upvote/downvote items so the system can adjust the signal-to-noise ratio of the feeds and other users that voted on that item. I wanted the same functionality to be available in the email. I did it by including this question under each item "Was this worth your time? Yes | No" The links go to https://linklonk.com/api/rateFromEmail?uid=<item id>&collectionId=<collection id>&token=k2nDwff0SQ69OBThn6kr_t1Ozxk:1625280261281&user=<user uid from firebase>
    Where the token parameter is used to authenticate the action. The token is generated using https://github.com/golang/net/blob/master/xsrftoken/xsrf.go - a library that is used to generate tokens that help protect from XSRF attacks. The second part of the token (1625280261281) is the timestamp when the token was generated. The first part (k2nDwff0SQ69OBThn6kr_t1Ozxk) is a hash of the user id, the item id, the collection id (ie, does it go into your upvotes or downvotes), the time and the secret key that is only known to the server. This token helps verify that the token was generated for this specific action by the server (since no one else knows the secret key). And the server can limit how long a token is valid for because the token includes a tamper-proof timestamp.

    I have a few concerns with this implementation:

    • On the one hand the upvote/downvote and the email unsubscribe links (which is authenticated with a similar token) work in one click without the user having to be logged into linklonk.com on the device they are viewing the email on. On the other hand, if you forward this email to someone else then they will be able to vote on your behalf (only on the items included in the email). I wonder which side has more weight. It's probably not a big deal since the recommendations are personalized and are not very suitable for forwarding to other people.
    • Do automatic systems try to fetch links in an email? For example, to scan for malware. If so, my server would get artificial votes.
    1 vote
  4. Comment on What helps keep you off social media? in ~talk

    lonk
    Link
    Let me shamelessly plug my project: https://linklonk.com (I launched it on Tildes last December). It is designed to help you find content that makes you feel like it was worth your time. When you...

    Let me shamelessly plug my project: https://linklonk.com (I launched it on Tildes last December). It is designed to help you find content that makes you feel like it was worth your time. When you like something, you get connected stronger to feeds and people who also liked it - so you see more content from them in the future. When you dislike - the opposite happens - your connections to those who liked it become weaker and you see less content from them.

    This creates a feedback loop where each time you rate content you adjust who you want to get more content in the future. This makes you stop and think if this piece of content was actually worth your time, because it will affect how you will spend your time in the future.

    Plus, LinkLonk remembers the list of recommendations you saw last time and will keep showing it to you unless you press "Mark as read and load new" or "Refresh". It means that the most important content does not dissappear automatically as time passes - less anxiety that you would miss something. I also feel less of an urge to opportunistically check my website for a few seconds, just in case something interesting pops up. I already know what it has - the same content as before I left it. I check Tildes and HackerNews more often to scan for new stuff, but I spend more time reading content on LinkLonk.

    5 votes
  5. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link Parent
    Thank you for the report! The account status was indeed inconsistently handled in the case when the user had created a temporary account that has since expired. On the client side the Firebase...

    Thank you for the report! The account status was indeed inconsistently handled in the case when the user had created a temporary account that has since expired. On the client side the Firebase Authentication library thinks that the user is signed in because the auth cookie is still there, but on the server side I deleted the account from the DB and from Firebase API.

    I now changed the logic on the client to make the state consistent. The client makes a request to the server and if the server response looks as if the user is not signed in (ie, not recognized by the server as an active account), then I will sign out the user on the client side.

    I also made the UI a bit more sensible for the signed out state - now there are no Profile/Your ratings/Create a post links in the nav that make sense only for signed in users.

    The dialog to sign in/create account is now more functional - you can sign in with email or "Continue as guest" (ie., the temporary account). I very much hope that I didn't break anything along the way.

    2 votes
  6. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    (edited )
    Link Parent
    Just to confirm, you see no recommendations above the blue "Mark read & load new" button? LinkLonk saves the state of the recommendations to local storage and restores it on your next visit. Maybe...

    Just to confirm, you see no recommendations above the blue "Mark read & load new" button? LinkLonk saves the state of the recommendations to local storage and restores it on your next visit. Maybe your state was an empty list and now it restores that list?

    I just pushed an update that would ignore previously saved state if the list of saved recommendations is empty. That would force the client to load fresh recommendations. Could you please try refreshing the page and see if it fixed the issue? If it doesn't, please try pressing "Mark read & load new".

    As for spammers, yes, the site is too small to be interesting to them.

    Edit: made a change to forget the locally saved state of the recommendations if it is older than 7 days. I hope it will prevent people from getting stuck in a bad state. After all, the saved state was not meant to be long-term.

  7. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    I made my hobby project https://linklonk.com publicly accessible and creating an account no longer needs an invitation code. The account creation happens when you try to do an action that results...

    I made my hobby project https://linklonk.com publicly accessible and creating an account no longer needs an invitation code. The account creation happens when you try to do an action that results in a non-GET request to the server (POST/PUT/DELETE) to which the server response with an authentication error. On the client side, an HttpInterceptor (I use Angular) handles this error by showing you a dialog to confirm if you want to create an account. It's a bit hacky, but it seems to work.

    Initially I was worried that once the site becomes publicly accessible then web-crawling bots would start pressing upvote/downvote buttons and registering accounts, but it didn't happen in the last two days. I wonder if bots avoid making non-GET requests.

    I'm also adding contextual tips to help new users understand how the site works. It's nothing technical but I think it is more important than any technical improvements I can make at this point. I would appreciate any advice/suggestion regarding the onboarding process.

    3 votes
  8. Comment on Tracing Paper - A brief history of the secret plan to track every printed page in ~tech

    lonk
    Link
    I would also recommend flipping through the Logic website. They have well written articles with authors from different perspectives. You can support them by subscribing to their paper edition....

    I would also recommend flipping through the Logic website. They have well written articles with authors from different perspectives. You can support them by subscribing to their paper edition.

    Looking for inspiration? Try this one on emoji: https://logicmag.io/commons/under-a-blood-red-flag/

  9. Comment on Tracing Paper - A brief history of the secret plan to track every printed page in ~tech

    lonk
    Link
    Found it in my feed on LinkLonk and two more users also liked it. So maybe more Tildes folks will find it useful. If you do, consider upvoting it on LinkLonk to connect to all 3 users that already...

    Found it in my feed on LinkLonk and two more users also liked it. So maybe more Tildes folks will find it useful. If you do, consider upvoting it on LinkLonk to connect to all 3 users that already upvoted it: https://linklonk.com/item/4996019215420096512

    1 vote
  10. Comment on A progress update on LinkLonk - a trust based news aggregator in ~tech

    lonk
    Link Parent
    Reviews definitely seem to fit the LinkLonk's trust model. Many review sites have "Was this review helpful?" ratings but they use it to aggregate general popularity of each review. With the trust...

    Reviews definitely seem to fit the LinkLonk's trust model. Many review sites have "Was this review helpful?" ratings but they use it to aggregate general popularity of each review. With the trust model, you could surface reviews that were rated by people who care about the same aspects of the product or service. For example, for resort reviews you may not care about the amenities for kids if you are not travelling with kids yourself.

    The algorithm is based on Personalize PageRank. To create recommendations for you, we start all possible walks on the bipartite graph of users and item (nodes) that are connected with ratings (edges). The walk is like this:

    • Step 1: Pick an item you liked
    • Step 2: Go to a user that liked the same item before you
    • Step 3: Go to an item that they rated around or after they rated the item from step 1. You have arrived!

    Now, look at how many branches each step had - this goes into the denominator for calculating the probability of taking each path. Aggregate probabilities of taking all paths that end in the same item and you get the ranking score of that item.

    Downvotes work the same way, but paths that start from an item that you downvoted subtract from the ranking score.

    There are more details, but that's the general idea.

    From the user's perspective here is what is happening:

    • When you upvote something - you distribute 1 point of your trust among everyone who upvoted the same item before you. These users become curators of content for you.
    • The users who have your trust distribute it evenly among other items they liked. They, in a way, bet on those other items with your trust that they earned.
    • When you see something that wasted your time and you downvote it - the curators that bet on that item lose your trust.

    Note that you need to know who rated what items. Usually this information is not public. For example, Tildes, Reddit, HackerNews don't make this information available. In a decentralized setting this could be a problem.

    3 votes
  11. A progress update on LinkLonk - a trust based news aggregator

    Hey everyone, I launched my little project LinkLonk here on Tildes back in December and wanted to tell you how it has been going and get your feedback/suggestions. New changes since the launch:...

    Hey everyone,

    I launched my little project LinkLonk here on Tildes back in December and wanted to tell you how it has been going and get your feedback/suggestions.

    New changes since the launch:

    • The temporary accounts now automatically get deleted after 30 days of inactivity. I didn't have the deletion logic at the time of the launch, but had it implemented about 30 days after launch. Automatic account deletion is quite destructive - removes the account from the database (thank goodness for foreign keys and cascade deletes) and from Firebase Authentication. I'm happy that there were nobugs when I ran it the first time.
    • In addition to submitting external links you can now create text posts. The posts are Markdown-formatted (similar to Tildes). One novel thing is that you can post "anonymously". The database has a record of who the author is so the author can delete/edit their post, it's just the name is not show next to the post.
    • Comments - each item has a comment section. The comments are ranked based on how much you trust the people who upvoted each comment (as opposed to being pure popularity). This is the same ranking system that is used to rank the "For you" page, but now applied to comments.
      • Unlike Tildes, the comments have a downvote button. The downvote does not bury the comment for everyone else. Instead, it makes your trust in upvotes of people who upvoted that comment go lower. So the downvote button effects what you see, not what others see. It is much harder to abuse that button that way. For that reason I feel much more comfortable putting it there. However, there is a second order effect. If you downvote a comment that someone else already downvoted - then you will trust the downvotes of that person. When they downvote some other comment - then it will rank lower for you. In a sense they earn your trust to moderate content for you by identifying comments you don't want to see.

    In terms of users, there have been 260 user records created (some from my shameless plug comments on HackerNews). Of those, ~45 rated something - excluding those that were temporary accounts and were deleted. And I think we have 2 regularly active users (excluding myself). In my mind I had 10 as the number of active users that I was hoping to get by the end of 2021. At this rate we may reach it.

    I was pleasantly surprised that there have been no misbehaving users. I didn't need to remove any content even once. This lead me to constantly postpone the implementation of a content reporting system. I hope it stays this way for a long time.

    The whole idea of a trust based recommendation system is based on having someone to trust. Right now it is the RSS feeds that are generating most of the content recommendations for the active users. But ideally it would be mostly users recommending content to users. I have two priorities for the near future:

    • Make the "single-player" experience better so the active users find value already. As an example, I added full-text search through items you liked
    • Find more users to improve the "multi-player" experience. One option is to submit a "Show HN:" post on HackerNews. But you can only do it once and I'm not sure I'm ready to use that shot yet.

    What do you think I should do next on these two fronts?

    If you would like to give LinkLonk a try register with code "tildes" at https://linklonk.com/register. Feel free to comment on this post: https://linklonk.com/item/6347369602224750592

    17 votes
  12. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    Added commenting to https://linklonk.com Done so far: tree-structure, Markdown formatting, sorting by user ratings, post anonymously or with your name. Next step will be to change the sorting to...

    Added commenting to https://linklonk.com

    Done so far: tree-structure, Markdown formatting, sorting by user ratings, post anonymously or with your name.

    Next step will be to change the sorting to be based on how much you trust people who upvoted each comment. This is the trust based system that is used to rank the top level items (and discussed in https://tildes.net/~tech/u7f/linklonk_a_link_aggregator_with_a_trust_system) but now applied to comment. I hope to finish it in a week.

    More details in this post: https://linklonk.com/item/7034545119553585152

    Give it a try. Register with invitation code "tildes" to create a temporary account (no email address necessary).

    Would love to know what Tildes commenting features you like the most so I could prioritize it for LinkLonk :) Personally, I like the "new" comment indicators in the list of posts and in the thread view (the orange highlighting).

    2 votes
  13. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link Parent
    First post is now live: https://linklonk.com/item/3672932074643849216 It is far from the functionality of Tildes, but it's a start.

    First post is now live: https://linklonk.com/item/3672932074643849216

    It is far from the functionality of Tildes, but it's a start.

    1 vote
  14. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    Working on https://linklonk.com/ in spare time (announced on Tildes two months ago, code "tildes" still works, if you'd like to try it). Adding support for text posts (right now you can only...

    Working on https://linklonk.com/ in spare time (announced on Tildes two months ago, code "tildes" still works, if you'd like to try it).

    Adding support for text posts (right now you can only submit links). I don't expect it being used a lot right now since there are few regular users, but I'd like a way to post announcements and respond to the feedback form submissions.

    This brought the need to implement user names, something I didn't need before. Getting some inspirations from the way Tildes validates user names.

    I was originally thinking of implementing comments as a separate kind of thing. That is: store them in a new "comments" table in Postgres, track votes on comments separately from top-level items (ie, links and posts). But now I'm planning to implement them as a text post that just has a parent item id. It would be similar to how:

    I might regret this later, once I find that comments and top-level items differ in some profound way. But for now I see lots of benefits:

    • The code for ranking comments would be largely shared with how LinkLonk ranks top-level items. So the comments will be prioritized if they were upvoted by people you trust (ie, people who previously upvoted comments you liked).
    • We could rank comments across all discussions in a special view.
    • UI and logic for submitting/editing text posts and comments will be shared.

    The reason I'm excited to implement comments on LinkLonk is that I'm curious to see if the personalized ranking of the discussion could help with some of the issues discussed in Tildes is pushing out the minority voice.
    I think part of the issue is that the "volume" of commenters is a fixed thing. No matter how knowledgeable they are. And thanks to the Dunning–Kruger effect we often find clueless people (even if well intentioned) taking up the medium. But what if we can tune the volume of the commenter for ourselves by downvoting useless comments and upvoting the useful ones? Would knowledgeable people be more likely to post their perspective if they knew that they could be heard by people who appreciate their perspective?

    You may say this sounds like a perfect "echo chamber". But there is only one way to find out!

    2 votes
  15. Comment on Many people here believe that social media can't be both large and have good discussion because the human brain isn't made to interact with large numbers of people. What do you think of this? in ~talk

    lonk
    Link
    Instead of phrasing "moderation is impossible to do at scale" I would put it like this: Today's social software is easy to manipulate. Small systems are not worth manipulating. Bigger systems...
    • Exemplary

    Instead of phrasing "moderation is impossible to do at scale" I would put it like this:

    • Today's social software is easy to manipulate.
    • Small systems are not worth manipulating. Bigger systems command more attention and that makes them much more valuable targets. It is like Linux having less malware in big part because it is not as popular as Windows.
    • To solve the root problem we don't need to keep social systems small. We need to fix the vulnerabilities.

    The problem is in the incentive structure.

    Just think: What does the action of voting/retweeting/liking/commenting do on Reddit/Twitter/Facebook/Tildes? Who is this action directed at? Whose experience is changed as a result? In all cases these actions are directed at the reader. The lurker.

    Who is incentivized to vote/retweet/like/comment then? Those who want to influence the experience of the reader.

    It is not a problem at a smaller scale (Tildes) when your user base consists of people with good intentions. They vote/comment to help other users.

    As the system grows, this mechanic attracts bad actors who will game your system to death. And the interesting thing here is that moderation is not going to solve this completely. The goal of a bad actor is often to silence a productive discussion. And to do that they don't need to outright violate any content policy. They can simply derail the discussion, sow "fear uncertainty and doubt". In other words, create enough noise that would bury the signal. Conspiracy theories are an effective tool in that regard.

    How do we solve that?

    My attempt to solve this is through a different incentive structure. Instead of the vote being directed at other users, it is directed at yourself. That is, when you "like" something you say that you want more from other users who liked it. See more details in this post: https://tildes.net/~tech/u7f/linklonk_a_link_aggregator_with_a_trust_system

    9 votes
  16. Comment on What programming/technical projects have you been working on? in ~comp

    lonk
    Link
    I continue working on https://linklonk.com (the invitation code "tildes" is still active) I was very happy with the reception of the Show Tildes post from ~10 days ago. Since then I added: An...

    I continue working on https://linklonk.com (the invitation code "tildes" is still active)

    I was very happy with the reception of the Show Tildes post from ~10 days ago. Since then I added:

    • An invitation system so existing users can invite new users (in /profile). It is modelled very much after the invitation system of Tildes - each user has a number of invitation links they can generate (10) and I will be able to top it up periodically.
    • The profile page now shows you some stats like: "Based on your 5 ratings: you are subscribed to 3 users and 2 feeds; 2 users are subscribed to you" The idea is to help users understand how the system works by being more transparent.

    Next, I want to implement text search functionality for the /ratings page so you can easily look up links that you liked in the past based on the page's title/description/url. The data is stored in PostgreSQL, so I'm hoping a simple SQL query will work well enough. I remember reading http://rachbelaid.com/postgres-full-text-search-is-good-enough/ and I might get some ideas from there.

    3 votes