Discussions of progress or collapse often get mired in the question of significant discoveries and inventions. After wrestling with several organisational cencepts for various catalogues, and running into the Ever Growing List dilemma, I hit on what I call BOTI, or Best of the Interval (day, week, month, year, decade, century, etc.). It's similar to the tickler file 43 folder perpetual filing system of GTD. For technical types, a round-robin database or circular buffer.

(As with my bullet journal experiments, the effort is uneven but recoverable, which is its core strength.)

By setting up a cascade of buffers --- day of month, (optionally week or weekdays), month of year, year of decade, decade of century, century of millennium, millennium of 10kyr, a progressively larger scale record (roughly order-of-magnitude based), with a resolution of day but a maximum retention of (here) 10,000 years but only 83 record bins. How much you choose to put in each bin is up to you, but the idea is that only to most significant information is carried forward. Yes, some information is lost but total data storage requirements are known once the bin size and count are established.

Another problem BOTI addresses is finite attention. If you limit yourself to a finite set of items per year, say ten to one hundred (about what a moderately motivated individual could be aware of), BOTI is a form of noise-filtering. Items which seemed urgent or captivating in the moment often fade in significance with time, and often overlooked element rise in significance with time and context. 'Let it settle with time" is a good cure to FOMO.

There's the question of revisiting context. I'd argue that significance might be substantially revised years, decades, possibly centuries after a discovery or inventiion. So an end-of-period purge of all but the top items isn't what we're looking for. Gut a gradual forgetting / pruning seems the general idea.

Back to science and technology: It's hard to assess significance in the moment, and day-to-day reports of science and technology advances are noisy. I've been looking for possible sources to use and am finding little that's satisfactory. I'd like suggestions.

• Many newspapers and magazines run annual "best of" features. These typically include books, but not science (or at least not regularly). Some of the books are science- or technonolgy-related, though.
• There are the Nobel prizes, notably in physics, chemistry, and medicine, with lists at Wikipedia (linked). The Fields Medal in maths. Other fields have their awards, of which lists might prove useful...
• I'm having trouble finding something like a yearbook of science or technology, though some titles match, e.g., McGraw-Hill yearbook of science and technology. On closer look, this might answer my question, at least for yearbooks.
• Wikipedia has some promising but either inconsistent or untidily organised pages or collections, including the List of years in science, Timeline of historic inventions, Timeline of scientific discoveries, Timeline of scientific thought, among numerous other timelines. Compilations are useful but aren't themselves rankings. See also "never ending list" above.

There is a goal here: trends over time. I've a few senses of directions of research and progress, possibly also of biases in awards. Looking at, for example, Nobels in physics, chemistry, and medicine from, say, 1901--1960 vs. 1961--2020, there seems to be a marked shift, though categorising that might be difficult. The breakpoint isn't necessarily 1960 either --- 1950 or 1940 might be argued for.

There is the question of how to measure significance of scientific discoveries or technological inventions. I'm not going to get into that though several standard measures (e.g., counting patents issued) strike me as highly problematic, despite being common in research. Discussion might be interesting.

Mostly, though, I'm looking for data sources.

Stop me if you've heard this one before:

• You get an idea for something you'd like to learn more about. (Maybe you have a question, maybe you want to explore a new hobby, or maybe you want to make a more informed decision.)
• You type something into a search engine.
• You click a result, only to realize that what you're reading is poorly written. It seems rushed, surface-level, and ill-informed. "This doesn't answer my question at all!" you think to yourself.
• You go back, and try another one, and another one, only to give up and put the idea back in your head.

I don't think these webpages are written to be useful in the first place. They seem to be written to attract attention to the website for other reasons (ad revenue, affiliate links, to draw attention to a product or service). Regardless of why it's happening, though, I want to find a better way to search.

The sort of content I'm looking for is written by someone who really cares about the topic. I want to learn from dorks and nerds and passionate people. Once I stumbled across this blog about extra virgin olive oil. The website isn't pretty, and it goes way more in depth than I'll ever need, but I trust the author, and there are some really interesting nuggets of insight on these pages. (e.g. "Another myth debunked: Heating EVOO makes it ‘toxic’")

Do you have any tips or tricks to more reliably find these sorts of sources (whether online or in-person)?

From GiveDirectly's blog:

In 2014, GiveDirectly partnered with academic researchers to launch our largest study ever in Kenya. The ultimate goal: find out how cash transfers affect local economies, including nearby non-recipients, enterprises, and markets. Now, in 2019, the results of this research have been released.

Abstract of the paper:

How large economic stimuli generate individual and aggregate responses is a central question in economics, but has not been studied experimentally. We provided one-time cash transfers of about USD 1000 to over 10,500 poor households across 653 randomized villages in rural Kenya. The implied fiscal shock was 15 percent of local GDP. We find large impacts on consumption and assets for recipients. Importantly, we document large positive spillovers on non-recipient households and firms, and minimal price inflation. We estimate a local fiscal multiplier of 2.6. We interpret welfare implications through the lens of a simple household optimization framework.

Some interesting tidbits from the paper:

Interestingly, sales increased without noticeable changes in firm investment behavior (beyond a modest increase in inventories), and sales do not increase differentially for firms owned by cash recipient households relative to nonrecipients. Both patterns suggest a demand-led rather than an investment-led expansion in economic activity.

[...]

We next examine how these changes affect untreated households. Despite not receiving transfers, they too exhibit large consumption expenditure gains: their annualized consumption expenditure is higher by 13% eighteen months after transfers began, an increase roughly comparable to the gains contemporaneously experienced by the treated households themselves.

(Emphasis added.)

[...]

Average price inflation is 0.1%, and even during periods with the largest transfers, estimated price effects are less than 1% and precisely estimated across all categories of goods.

[...]

Real output increased, and yet there is at most limited evidence of increases in the employment of land (which is in fixed supply), labor, or capital. One plausible, albeit speculative, possibility is that the utilization of these factors was “slack” in at least some enterprises (Lewis 1954). This seems plausible because in the retail and manufacturing sectors, where output responses were concentrated, the typical firm has a single employee (i.e. the proprietor), suggesting that integer constraints may often bind. In addition, many enterprises operate “on demand” in the sense that they produce only when they have customers, and the average non-agricultural enterprise sees just 1.7 customers per hour. In addition to retail, much manufacturing in this setting is “on demand;” for example, a mill owner waits for customers to bring grain and then grinds it for them. The existence of slack may help account for the large multiplier we document, as has also recently been argued in US data, especially in poorer US regions (Michaillat and Saez 2015; Murphy 2017).

Givedirectly blog - Full paper (pdf)