-
39 votes
-
Chrome/Firefox Plugin to locally scrape data from multiple URLs
As the title suggests, I am looking for a free chrome or firefox plugin that can locally scrape data from multiple URLs. To be a bit more precise, what I mean by it: A free chrome or firefox...
As the title suggests, I am looking for a free chrome or firefox plugin that can locally scrape data from multiple URLs. To be a bit more precise, what I mean by it:
- A free chrome or firefox plugin
- Local scraping: it runs in the browser itself. No cloud computing or "credits" required to run
- Scrape data: Collects predefined data from certain data fields within a website such as https://www.dastelefonbuch.de/Suche/Test
- Infinite scroll: to load data that only loads once the browser scrolls down (kind of like in the page I linked above)
I am not looking into programming my own scraper using python or anything similar. I have found plugins that "kind of" do what I am describing above, and about two weeks ago I found one that pretty much perfectly does what is described ("DataGrab"), but it starts asking to buy credits after running it a few times.
My own list:
- DataGrab: Excellent, apart from asking to buy credits after a while
- SimpleScraper: Excellent, but asks to buy credits pretty much immediately
- Easy Scraper: Works well for single pages, but no possibility to feed in multiple URLs to crawl
- Instant Data Scraper: Works well for single pages and infinite scroll pages, but no possibility to feed in multiple URLs to crawl
- "Data Scraper - Easy Web Scraping" / dataminer.io: Doesn't work well
- Scrapy.org: Too much programming, but looks quite neat and well documented
Any suggestions are highly welcome!
Edit: A locally run executable or cmd-line based program would be fine too, as long as it just needs to be configured (e.g., creating a list of URLs stored in a .txt or .csv file) instead of coded (e.g., coding an infinite scroll function from scratch).
8 votes -
Midweek Movie Free Talk
Have you watched any movies recently you want to discuss? Any films you want to recommend or are hyped about? Feel free to discuss anything here. Please just try to provide fair warning of...
Have you watched any movies recently you want to discuss? Any films you want to recommend or are hyped about? Feel free to discuss anything here.
Please just try to provide fair warning of spoilers if you can.
10 votes -
What is it like to have both ADHD and autism?
I've been doing a lot of work with my therapist and it has been suggested that I might have a mix of both Autism and ADHD. I have always related to Autism to a certain extent but I would never say...
I've been doing a lot of work with my therapist and it has been suggested that I might have a mix of both Autism and ADHD.
I have always related to Autism to a certain extent but I would never say it because I felt like I didn't really fit because some things I do don't make sense it that context. I was also obsessed with learning to be human since young and have a lot of systems to deal with it and apparently that means I've been masking for a very long time.
But my therapist also brought up ADHD and that has explained a lot about some the things I'm still currently struggling with. But again, when I look it up, I don't feel like I fully fit. But also, I have developed a lot of systems over the years to deal with my issues and function like a normal human being. So a lot of the things I struggle with is hidden because I've "fixed" them.
I'm still reluctant to say I have either of those two things because I feel like an imposter. So I've just settled on Neurodivergent and think that I can always learn the tools that help people with ADHD even if I don't have it. Because if the tools work for you, then why not?
But my therapist has suggested getting diagnosed because it does change what kind of medication you might get. I'm currently on medication for anxiety and depression.
So now I'm wondering how common it is to have a mix of both ADHD and Autism. For those who have both, what is it like for you? What is your daily life like? What are your struggles and how do you deal with them? When did you realise you might have a mix of both? How do you feel about yourself and your brain?
I apologize if this isn't the right place to ask this. I don't post very often and am getting used to it.
38 votes -
FYI: This site claims to have harvested 4B+ Discord chats, today all yours for a price
41 votes -
What makes someone a "decent" person to you?
I'd define decent as the minimum expectation I have for having someone in my life. I use the term a lot, but I had never been clear on what I mean by it. I think for me its that the person broadly...
I'd define decent as the minimum expectation I have for having someone in my life. I use the term a lot, but I had never been clear on what I mean by it. I think for me its that the person broadly reciprocates the work I put into the relationship, both in kind and degree, unless I explicitly say otherwise. I'd say most of my interpersonal problems would fit under this; mainly being expected to put in a lot of emotional effort, and either they don't reciprocate it at all, they unambiguously do not match it or they ignore me when I tell them I don't care about gifts and act like the infrequent gift makes us equal.
But that's me, and I expect there are many more varied expectations from y'all. So, what make some a decent person to you?
26 votes -
Everybody's obsessed with the retro corporate aesthetic
6 votes -
NPR suspends veteran editor as it grapples with his public criticism
40 votes -
'Grand Theft Auto' maker Take-Two to let go 5% of staff, scrap some projects
13 votes -
UK MPs back smoking ban for those born after 2009
13 votes -
The hazy evolution of cannabis
3 votes -
Wikipedia, the free encyclopedia
2 votes -
Strategies for coping with writers block
8 votes -
Cutting class: On the myth of the middle class writer
9 votes -
Hades II - Technical test sign ups now available
45 votes -
A US Food and Drug Administration-approved device offers a new treatment for tinnitus
32 votes -
This is a teenager
36 votes -
TV Tuesdays Free Talk
Have you watched any TV shows recently you want to discuss? Any shows you want to recommend or are hyped about? Feel free to discuss anything here. Please just try to provide fair warning of...
Have you watched any TV shows recently you want to discuss? Any shows you want to recommend or are hyped about? Feel free to discuss anything here.
Please just try to provide fair warning of spoilers if you can.
12 votes -
Aberdeen have agreed a deal for Elfsborg head coach Jimmy Thelin to become their new manager this summer
1 vote -
Tokyo starts ride-hailing service — but it may not be what you expect
19 votes -
Damien Guerot, who fought off Bondi Junction attacker with bollard, to be granted permanent Australia residency, lawyer says
13 votes -
Gold is back — and it has a message for us
6 votes -
The “bad nanny” wars
7 votes -
The search for the ultimate home/apartment spring cleaning guide
I've always considered myself a rather clean and tidy person (and get comments from others to that effect), but realized I was never formally taught on best practices for cleaning. There are...
I've always considered myself a rather clean and tidy person (and get comments from others to that effect), but realized I was never formally taught on best practices for cleaning. There are probably things I tend to overlook, things that may not be visible (best practices for cleaning pipes for example). There is likely a lot of "traditional" knowledge on cleaning I don't know.
I am curious if anyone here has some sort of "ultimate guide for spring cleaning" around the house to get everything to a nice deep clean, and some sort of routine on what someone should do every week/month/whatever in order to upkeep this cleanliness over the course of the year.
Not looking for some extreme clean freaks guide but I am genuinely curious as to what I may have missed and overlook as I was never really "taught" how to clean.
30 votes -
Say Lou Lou – Dust (2024)
3 votes -
To make sure grandmas like his don't get conned, he scams the scammers
25 votes -
React: Some comments from a beginner
New job. I've been wanting to learn something new for a while, so I took a project where a lot of React is done. I'm learning it from scratch while I work with React. I have some comments about...
New job. I've been wanting to learn something new for a while, so I took a project where a lot of React is done. I'm learning it from scratch while I work with React.
I have some comments about it.
- React makes front end work a lot more like programming -- I like that!
- Javascript has changed a lot, and for the better, since I last used it over a decade ago.
- The React-Redux tool kit is the bomb. It should be integrated/absorbed into React. I can't see any reason not to use it, even for small applications as it is less wordy wherever you use it.
- The updating of state values should be more automatic, especially for flag variables not tied to GUI components. It is the major source of hassles with React
- Udemy React videos. My company makes them available free of charge to employees. I've sampled videos from a number of courses. I'm not a fan of the instructors showing you how to do things in older, less efficient ways first in a learning/demo project, the ERASING that code to do it a better way. The should include copies of the project at each stage if they do that. I finally figured out that the best way to take notes I can use later is to comment out the old code and put the new more efficient next stage stuff on top.
- React tests really need to improve. They are often more time consuming than the code itself. The tests have forced me to change my code or do needless testing to get the tests to pass. I had one situation where no matter what I did React test said I didn't cover the code until I broke an else clause off into it's one if clause. Blech.
All in all I've been enjoying learning React. It is neat new ( to me ) thing.
I feel sad that I will likely forget it all when I go back to my specialty language.
16 votes -
What happened to the Nautilus?
9 votes -
Fire breaks out at Copenhagen's historic Stock Exchange
15 votes -
Keanu Reeves joins 'Sonic 3' as Shadow
26 votes -
Song lyrics are getting more repetitive, angrier
18 votes -
Two years to save the planet, says UN climate chief
53 votes -
Ukrainians contemplate the once unthinkable: Losing the war with Russia
37 votes -
The Lonely Island beginnings | The Lonely Island and Seth Meyers Podcast - Episode 1
14 votes -
'I was born intersex and doctors tried to erase me. It's important to speak out' - an interview with Raven van Dorst of the Dutch rock-metal band Dool
40 votes -
From FedExCup glory to PGA Tour struggles – what's gone wrong for Viktor Hovland in 2024?
2 votes -
Human brains and fruit fly brains are built similarly – visualizing how helps researchers better understand how both work
7 votes -
Looking for free or cheap places to learn some SQL and XML
Im looking for places online to explore SQL and XML as a beginner without spending a lot of money and without a lot of performance pressure. Can you help?
17 votes -
What games have you been playing, and what's your opinion on them?
What have you been playing lately? Discussion about video games and board games are both welcome. Please don't just make a list of titles, give some thoughts about the game(s) as well.
20 votes -
What have you been eating, drinking, and cooking?
What food and drinks have you been enjoying (or not enjoying) recently? Have you cooked or created anything interesting? Tell us about it!
6 votes -
Oceans of Slumber - Where Gods Fear to Speak (2024)
2 votes -
Volvo is celebrating its 97th birthday with the opening of what it calls "World of Volvo" in the company's hometown of Gothenburg, Sweden
7 votes -
I was the poster girl for OCD. Then I began to question everything I’d been told about mental illness.
28 votes -
Weekly US politics news and updates thread - week of April 15
This thread is posted weekly - please try to post all relevant US political content in here, such as news, updates, opinion articles, etc. Extremely significant events may warrant a separate...
This thread is posted weekly - please try to post all relevant US political content in here, such as news, updates, opinion articles, etc. Extremely significant events may warrant a separate topic, but almost all should be posted in here.
This is an inherently political thread; please try to avoid antagonistic arguments and bickering matches. Comment threads that devolve into unproductive arguments may be removed so that the overall topic is able to continue.
8 votes -
Weekly Israel-Hamas war megathread - week of April 15
This thread is posted weekly - please try to post all relevant Israel-Hamas war content in here, such as news, updates, opinion articles, etc. Extremely significant events may warrant a separate...
This thread is posted weekly - please try to post all relevant Israel-Hamas war content in here, such as news, updates, opinion articles, etc. Extremely significant events may warrant a separate topic, but almost all should be posted in here.
Please try to avoid antagonistic arguments and bickering matches. Comment threads that devolve into unproductive arguments may be removed so that the overall topic is able to continue.
5 votes -
The Museum of Science and Industry abruptly closed for a day last week to allow it to move “military artifacts from archival storage”
26 votes -
Six badass librarians who changed history
13 votes -
USA, 101 Freeway: Major Los Angeles highway to undergo weeks of closures for wildlife crossing construction
32 votes -
How to make kanelbullar / Swedish cinnamon buns
6 votes -
South Africa recalls cough syrup sold in at least six countries
7 votes