Running makemandb on every boot

[David Brownlee]

Currently the system runs 'makemandb -Q' in the background on every boot.
This updates the apropos database (for 'man -k').

On an system with an existing man database this will stat(2) every manpage
and update /var/db/man.db (an around 18MB sqlite database on a new amd64
system, maybe around 40MB on the same system with a reasonable selection of
packages)

On a modern system rich in disk, memory and CPU this is unnoticeable. On a
slower system (mac68k, vax, embedded 486), this renders the system
effectively unusable for some time.

Given that makemandb is run nightly and weekly by cron anyway, it would
probably be best for those systems to not have makemandb run on boot.

My first thought would be to allow setting it in sysinst, with a per port -
default on for i386, amd64, modern (arm, powerpc, mips), off for others.

What do people think?

[Martin Husemann]

Post by David Brownlee
Given that makemandb is run nightly and weekly by cron anyway, it would
probably be best for those systems to not have makemandb run on boot.

Sounds like a good idea - however, doesn't work well for notebooks that
you do not have running over night.

Post by David Brownlee
My first thought would be to allow setting it in sysinst, with a per port -
default on for i386, amd64, modern (arm, powerpc, mips), off for others.

I don't like that - a generals solution is better.

FWIW: my (very fast) vax says:
49.30 real 32.61 user 14.36 sys

my shark (slow arm, very slow disk) says:
39.78 real 16.13 user 18.94 sys

the mac68k was similar when I last tested.

However, it goes realy bad if there is not enough ram - so maybe we could use
something like this as a generic condition:

if [ $(sysctl hw.physmem64) -lt 25165824 ]; ...

to run it at boot time?

Also we could delay it with a "sleep 120" to have other parts of the
startup finish first)?

Martin

[Greg Troxel]

Post by Martin Husemann

Post by David Brownlee
Given that makemandb is run nightly and weekly by cron anyway, it would
probably be best for those systems to not have makemandb run on boot.

Sounds like a good idea - however, doesn't work well for notebooks that
you do not have running over night.

It sounds like there are two bugs lurking:

doing something at boot every time is being used as a substitute for
scheduling it to be donce once

we have a notion of nightly cron jobs, and there is concern that these
don't happen on notebooks that are not on then

So rather than make running this on boot configurable, it seems better
to perhaps schedule it with some sort of "run this once" faciltiy,
perhaps at(1). Or to chroot and run it at install tiome - why does this
need deferring?

Then, there's the issue of the nightly scripts; perhaps cron should run
them when the machine is first functional if the system has been up. Of
course people will object to that, too, but it seems either we need to
reduce what gets done or figure out a pleasing background way to do it.

[Robert Elz]

Date: Mon, 19 May 2014 11:29:54 -0400
From: Greg Troxel <***@ir.bbn.com>
Message-ID: <***@fnord.ir.bbn.com>

| So rather than make running this on boot configurable, it seems better
| to perhaps schedule it with some sort of "run this once" faciltiy,

In the interests of contributing more than noise, I offer the following
script that fell off the back of a truck somewhere (it is public domain,
unlicenced, unlicenceable as its author will never claim it - I guarantee
that...)

Feel free to do what you like with it, its objective is to run commands
(from cron usually) so they can be specified to both run @boot, and at some
fixed time of day/week/.. whatever is needed. Cron (or similar) takes care
of making sure it gets run (when properly configure) this script arranges to
make sure that the command is run no more than once a day/week/month/year (or
fortnight, though that option is little tested, somewhat weird by design, and
may be buggy).

If this is deemed useful enough to consider, you never know, but that a man
page might fall off another truck sometime...

kre

[Marc Balmer]

Currently the system runs 'makemandb -Q' in the background on every boot. This updates the apropos database (for 'man -k').
On an system with an existing man database this will stat(2) every manpage and update /var/db/man.db (an around 18MB sqlite database on a new amd64 system, maybe around 40MB on the same system with a reasonable selection of packages)
On a modern system rich in disk, memory and CPU this is unnoticeable. On a slower system (mac68k, vax, embedded 486), this renders the system effectively unusable for some time.
Given that makemandb is run nightly and weekly by cron anyway, it would probably be best for those systems to not have makemandb run on boot.
My first thought would be to allow setting it in sysinst, with a per port - default on for i386, amd64, modern (arm, powerpc, mips), off for others.
What do people think?

What say does make sense to me. I run several small machines and I also suffer from this effect. Maybe the behaviour would better be defined in /etc/rc.conf (with sensible per-port default values)?

- mb

[Martin Husemann]

We have a rc.conf setting for this (makemandb=YES/NO), but we could argue
about thedefault (currently: YES).

I am fine with changing that to NO, but I really dislike port-specific
defaults (however they are implemented). They just cause confusion.

I ran a few more tests, and runtimes vary seriously (partly due to different
sets of pkgsrc man pages being installed):

SparcStation LX (50 MHz sparc v8), slow scsi disk, 96 MB ram:
2145.91 real 1856.68 user 92.47 sys

JavaStation Krups (100 MHz sparc v8), diskless, 64MB ram:
1469.48 real 1127.50 user 118.26 sys

HP9000/715/50 (50 MHz hppa), slow scsi disk, 96MB ram:
118.93 real 93.40 user 0.00 sys

GuruPlug (1.2 GHz ARM), sata disk, 512 MB ram:
32.44 real 27.78 user 4.24 sys

GuruPlug (1.2 GHz ARM), diskless, 512 MB ram:
21.00 real 11.33 user 6.52 sys

Apparently something makemandb does is not very performing well on
sparc - this needs closer investigation (maybe we can make it faster
for everything).

Martin

[Martin Husemann]

Post by Martin Husemann
2145.91 real 1856.68 user 92.47 sys

I rebooted that machine (to have clean caches) and ran the below script
on it (which is about as fast as makemandb could get to notice "I have
nothing to do"). The result is:

30.49 real 5.21 user 23.23 sys

So an easy way to speed up makemandb would be: defer all the sqlite
initialization untill we find a file that is newer than the output database.
After sqlite has been initialized, it might still be usefull to check against
the original timestamp before querying, comparing and updating.

This still leaves the sqlite performance on sparc as an open issue - will
check that.

Martin

--8<--
#! /bin/sh

man -p | while read d
do
echo checking $d
find $d -cnewer /var/db/man.db | wc -l
done

[David Brownlee]

I like the idea of the system determining whether to run it or not by
default, but being able to force it on or off.

Maybe a makemandb_force=YES/NO, defaulting to NO.

If makemandb_force=NO then the makemandb will be skipped iff
- Memory is 64M or less
OR
- man.db has been modified in the last N hours (24? 4?)

This does mean on a small old box which is never run overnight makemandb
would never be run, but I think that is a reasonable edge case - and in
fact I'd prefer that behaviour on my small old boxes :)

Post by Martin Husemann
We have a rc.conf setting for this (makemandb=YES/NO), but we could argue
about thedefault (currently: YES).
I am fine with changing that to NO, but I really dislike port-specific
defaults (however they are implemented). They just cause confusion.
I ran a few more tests, and runtimes vary seriously (partly due to different
2145.91 real 1856.68 user 92.47 sys
1469.48 real 1127.50 user 118.26 sys
118.93 real 93.40 user 0.00 sys
32.44 real 27.78 user 4.24 sys
21.00 real 11.33 user 6.52 sys
Apparently something makemandb does is not very performing well on
sparc - this needs closer investigation (maybe we can make it faster
for everything).
Martin