My Miniflux Feed-Polling Choices.

Excuse me, server? Is there new content? … What about now?

In order to read feeds, you first have to fetch them. One of the issues that often comes up is how often you should fetch them. Usually we see this mentioned on fairly popular blogs that draw a lot of readers, and with many readers comes many feed readers, some of which do not do great at limiting how often they look for fresh content. I’m sympathetic to those whose sites get hammered, and I think the main issues are ignorance (which making posts about it can help) and technical difficulties for users (which posts mostly don’t help).

I try to avoid being part of the problem by setting my reader to play nice. Below are the settings I use from my miniflux.conf. (See Miniflux if you want to learn more about the web application.)

# Check feeds more frequently.
# (Default: 60; integer minutes)
POLLING_FREQUENCY=20
# Use the adaptive scheduler that changes refresh based on feed.
# (Default: "round_robin")
POLLING_SCHEDULER=entry_frequency
# Lowest time in minutes to allow check.
# (Default: 5; integer minutes)
SCHEDULER_ENTRY_FREQUENCY_MIN_INTERVAL=20
# Maximum time in minutes to back off to.
# (Default: 1440 = 24h * 60m; integer minutes)
# 2880 = 48h * 60m.
# 4800 = 3 and 1/3 days worth of minutes.
SCHEDULER_ENTRY_FREQUENCY_MAX_INTERVAL=4800

Excerpt from my miniflux.conf.

Being redundant to the configuration’s comments, the settings are:

Try to check for feed updates every 20 minutes. This only looks to see if any feed is ready for a check, and if so it executes for those. (POLLING_FREQUENCY)
This doesn’t directly cause any feeds to be checked, it’s “how often should I check if I should check?”
Feeds will be re-checked based on how frequently they have new entries, with the minimum rest being 20 minutes and the maximum rest being three-ish days. (POLLING_SCHEDULER, SCHEDULER_ENTRY_FREQUENCY_MIN_INTERVAL, and SCHEDULER_ENTRY_FREQUENCY_MAX_INTERVAL)

I don’t want feeds to be checked too often. I’m not trying to hammer anyone’s server, and I have plenty to read, so I try to be judicious. That said, it’s impossible to guess if the infrequently-updated blogs have new posts, so three days seems a good cut-off.

Feeds can implement (for RSS) cloud, ttl, skipHours, and skipDays. Or they can implement Cache-Control, Retry-After, or Expires as part of the HTTP standards. Miniflux doesn’t respect the RSS elements, but does support the HTTP mechanisms, overriding whatever interval it would have used if it runs into them, up to the maximum interval option. (See Issue 1413, from April 2022: “Does Miniflux support the skipHours, skipDays, or ttl tags?”.) It sounds like the cloud route is largely superseded by WebSub, but it’s unknown if Miniflux will ever use it: Issue 639, from April 2020: “WebSub Support (Discussion)”. Given the state of my current reader and of feeds, I think I’m doing alright.

I ended up looking up how it actually works in Miniflux. If you want to see the code: Github: Miniflux: internal/model/feed.go:121. The full path of Miniflux scheduling is:

Default to the minimum interval of Round Robin (60 minutes).
If using the frequency option, set to either:
1. If no entries in the past week, use the maximum interval of frequency. (Default is a day; I use three days.)
2. Otherwise clamp the value between the minimum interval and maximum interval, but attempting to use the fraction of a week divided by the number of entries in the past week. (So if there were three entries, a third of a week, etc.)
  There’s actually a factor option used here, which defaults to 1: SCHEDULER_ENTRY_FREQUENCY_FACTOR. It gets multiplied against the number of updates in the past week. I don’t think it’s useful in most cases.
Take the maximum between that current value and when any relevant HTTP control header specifies.
I’m ignorant but curious what kinds of times are used in these. But I’m also lazy, so if someone ever gathers the data I’d love to see it, but don’t plan to try myself.
Finally, ensure whatever value is not exceeding the maximum interval for the scheduler type.

The biggest sticking point is the maximum frequency. I chose three days because at least some feeds have content relevant to the news cycle. After a half-week that stuff would get fairly stale, making it turn from ham to spam. I’d prefer a chance to read it before that, if I choose.

This does cause over-polling for infrequently-updated feeds. But those feeds might go months without an update, only to receive a burst of them. (This isn’t a rare pattern. Bloggers tend to store up a few good posts and find some spare time to toss them out all at once.)

On the other hand, the frequency scheduler does prevent some polls of slow feeds while allowing timely polls of busy ones. It’s worth the settings, as it allows a better scheduling than plain Round Robin would.

There’s another side to the polling question. For background, when I see new posts in my reader, I go through them and mark anything that looks promising. After that, I chunk through those a bit at a time.

Depending on my reader’s update timing, I’ll often catch a bunch of updates at an odd time. If I have the restraint, I’ll leave them for tomorrow. Other times I’ll be a little peeved digging through them, as I practice “feed reader inbox zero.”

It feels like there could be some system to help organize the feeds so you see items when you want. Maybe a simple priority rating would do, maybe something more complex.

It’s a different topic, but in many ways more essential as it hits most forms of media we deal with: it would be nice if we had more control over what kinds of content we see in a given moment (separate from our general tastes and selections). The analogy I came up with before I cut this section short was: “If you sat down to a meal and you got random spoonfuls of desert, main course, salad, you would find some better way to eat pretty quick. It should be no different for the information streams in our lives.” Something to chew on another time.

Please Deposit Two Cents: Cancel reply