Is anyone here familiar with crawling the web? I’m interested in broad crawling, rather than focusing on particular sites. I’d appreciate pretty much any information about how this is usually done, and things to watch out for if attempting it.

I'm currently developing a minimal userscript manager who's main goal is to be fully auditable by any user in only ten minutes or so - my prototype uses less than 300 lines of javascript, and I'm trying to keep it that way.

To get the codebase this small, however, I have to be very picky with what features I implement - most notably, the code editor has to be very barebones. Are there any features that I'd be shooting myself in the foot by not including?

For example:

• syntax highlighting
• cloud sync
• regex url matching
• fullscreen editor (currently, it's just a browser popup - the intention is that you write code elsewhere and paste it in)

Any feedback is greatly appreciated!

Those of you who keep hundreds of tabs open: I'm curious how and why you use them. I'd hoard tabs in the past, but in a sad incident a browser (Firefox) restart caused the loss of all my 10s of open tabs that was accumulated over weeks long research about a topic, I decided to never trust tabs again. Now I'm making use of my bookmars toolbar, Org mode and Instapaper for most of the stuff having many tabs open was the method before. So, for me, tabs were for keeping stuff handy during research, read-it-later lists, and temporary bookmarks. What are the use cases for you?

When Edge died, I got worried about loosing competition to the Blink engine and as such, I went exploring other alternatives to realize.. there's not a whole lot, there's blink, gecko and webkit.

So with that, I decided to try epiphany - Gnome's web browser. It uses Webkit which is what Blink was forked from so it's not terribly different in theory but the years apart has made that more apparent. It's fairly elegant in my opinion and it lacks some features, sure.

Anyways, to get to what I wanted to do this week, well, I'd like to challenge you all to use it for a week, mostly for bug hunting purposes and possibly to throw ideas at the project. Worth mentioning, I'm not affiliated with the project, just a user.

So to make sure we're all on the same page, we'll use the development Epiphany flatpak, this way we can be sure that the problem is in the current codebase. So, to install it :

Let's install the gnome-nightly repos as per instructions here :

flatpak remote-add --if-not-exists gnome-nightly https://sdk.gnome.org/gnome-nightly.flatpakrepo
flatpak remote-add --if-not-exists gnome-apps-nightly --from https://sdk.gnome.org/gnome-apps-nightly.flatpakrepo

Then, let's install the development version by doing so :

flatpak install org.gnome.Epiphany.Devel

Then just launch it and have fun with it!

if you run into any bugs, look at the contribution guide here and report the bugs in the repo after checking that the bug is not already present of course!

As someone who is not mainly a web developer, I can barely grasp the immensity of options when it comes to writing a web application.

So far everything I've written has been using PHP and the Slim microframework. PHP because I don't use languages like Python/Ruby/JS that much so I didn't have any prior knowledge of those, and I've found myself to be fairly productive with it. Slim because I didn't want a full-blown framework with 200 files to configure.

I've tried Go because I've used it in the past but I don't see it to be very fit when it comes to websites, I think it's fine for small microservices but doing MVC was a chore, maybe there's a framework out there that solves this.

As for the frontend I've been trying to use as little JavaScript as possible, always vanilla. As of HTML and CSS I'm no designer so I kind of get by copying code and tweaking things here and there.

However I've started a slightly bigger project and I don't fancy myself writing everything from scratch (specially security) besides, ORMs can be useful. Symfony4 is what I've been using for a couple of days, but I've had trouble setting up debugging, and the community/docs don't seem that great since this version is fairly new; so I'm considering trying out something more popular like Django.

So this is why I created the post, I know this will differ greatly depending on the use-case. But I would like to do a quick survey and hear some of your recommendations, both on the backend and frontend. Besides I think it's a good topic for discussion.

Cheers!