Sun Sep 20 00:17:24 CEST 2020

Welcome, hubert!


And now for something completely different:


The newest addition to the labs is not 7+ years old but spanking new gear: i5-9400, 32GB RAM, 500GB SSD, ... and very, very quiet.

I usually don't put stickers to my machines. The Abteilung-fuer-Redundanz-Abteilung deserves an exception to this rule: their sticker not only matches the colour scheme of the mini-tower/shoebox enclosure perfectly but the six wings also hint at the six cores in the i5-9400.

The machine comes with a big monitor (2560x1440 on 27") and is my new home office workplace.


This replaces the the prior work setup of my Lenovo X220 laptop + 1280x1024 external monitor. (Which is way too noisy when used the whole day long.)

I had to shoot the pictures right away because such a clean desk is very rare with me. It was only possible because I shifted all of the entropy to the neighbouring desk:


Both desks feature a row of 2cm holes in the back. These were originally used as air vents for the heating elements the desks covered in the 70ies. Now they are great for much of the cabling.


The cables will be tied up nicely once things have settled.

I still need to learn how to route the audio via the DisplayPort instead of the analog LineOut/Headphone jacks. Also, having the keyboard USB-hubbed off the monitor means that I have to re-run my xmodmap ~/,Xmodmap every time after switching the monitor off/on. (Probably a job for udev; for now, I let the screensaver do the work.)

I intend to run hubert as a 24/7 server+workstation. I decided on Proxmox-VE/Debian as OS, extended with the desktop services and applications I need at the hypervisor level.

Hubert should inherit many tasks from alexis, the Olinuxino ARM board which provided the 24/7 services for the last years. (XDMCP, bootp/tftp, X11 xfs, DNS, NTP, lpd). Once fz, the NCD X Terminal in the kitchen can boot/work from hubert, I can give alexis its badly needed system upgrade.

Posted by neitzel | Permanent link | File under: learned, done, marshlabs

Sat Sep 19 00:55:52 CEST 2020

Dealing with (not so) dead disks

Harddisks drives fail...

... but in different ways.

Sometimes the integrated electronics board of a disk dies. I cannot cope with these failures -- such a an HDD is truly dead for me.

Sometimes, just "a few" disk blocks become unreadable (and/or unwritable).

And if you are lucky, the disk got just reported as bad but is actually fine for use: I have seen a RAID enclosure for 16 disks which liked to complain about various disks but always in the same two slots -- I rather suspect a problem with the backplane. Then there are those disks which were just subjected to unfair work conditions: too hot. Give them enough cool air to breathe and they cooperate nicely again.

Ditching an entire HDD because a few first blocks become unreadable is justified in a production setting where HDDs are cheaper than the labor to deal with a corrupted disk and the associated data loss.

In my lab, though, this is where the fun begins. Can I put HDDs "with known issues" still to some proper use? Of course:

  • Not all of my data is important. I would not shed a tear about a lost mp3 file ripped from one of my CDs.

  • I can forgo the loss of a filesystem backup -- there will be new ones soon enough.

  • Build RAID systems from risky disks -- that's what RAIDs are for, after all.

This article takes a closer look at a faulty 250 GB disk. It was pulled from a RAID-1 system where it was easily replaced. (By a 500 GB disk because that was the smallest size sold at that time.)

Summary: dd(1), awk(1), jqt(1), smartctl(1).

Test setup

Here is my current backlog of disks to check:

known unknowns

These are mostly 3.5" SATA HDDs, mostly fallout from production servers in the company.

Disks which are checked with some hope for an afterlife go here:

disk reservoir

These are mostly SCSI disks of various vintages. I have no shortage on 68-pin WIDE-SCSI disks. 80-pin SCA disks and 50-pins "FAST" SCSI are the real treasure items, I need these most for the workstation gear I carry. (It has actually become more and more difficult to find the 50pin disks.)

Testing is most easily done with external drive sockets such as these:

eSATA socket

This one takes 3.5" and 2.5" SATA disks (rotating dust or SSDs). It is attached via eSATA to, a Core2Duo box running DragonFlyBSD.

IDE (PATA) disks went mostly out of interest but I would have some IDE-to-USB connector kit for these, too. It's the SCSI drives which cause a bit of pain: these have to be mounted into some computer or enclosure, depending on the interface type.

dd(1)ing through the disk

dd(1) is the swiss army knife to deal with storage devices of all kinds. It works at the block level and isn't dependent on the actual data. If vital blocks on the disk have taken a hit so that your partitioning or filesystem are hosed, dd is not impacted by that.

In the rest of this article, I move from SI-metric Kilo/Mega/Giga units to 2^10 = 1024-based unit factors throughout. Our disk marketed as "250 GB" is really:

ad10: 239429MB <WDC WD2502ABYS-02B7A0 02.03B03> at ata5-master SATA300

This is 251059503104 bytes (justifying the 250 metric marketing GB), or 233.8 (2^30) GiB ("GibiBytes"). The disk is addressable in classic 512-byte blocks, presumably 490350672 of them.

My first command was simply for reading as many blocks from the disk as possible, starting at the beginning (block 0):

% dd if=/dev/ad10 of=/dev/null

This command supposedly runs through the entire disk (/dev/ad10) as "input file", reading all blocks and "copying" them over to Unix' big bit bucket, /dev/null as output (pseudo) file. In case of a problem with reading a disk block, it bails out at that point with an error message. In any case we get informed how many blocks were read/written.

When dd runs, it is silent. On BSD systems, you can issue a SIGINFO signal to the running process by hitting Ctrl-T. Dd will respond with the block position it is currently working on.

Reading just one block at a time is slow. On the first day, I aborted the dd command after having read 200.000.000 blocks (100 GB) successfully.

On the second day, I picked up reading more blocks where I stopped the day before:

% dd if=/dev/ad10 of=/dev/null skip=200000000

This command stopped with this output:

dd: /dev/ad10: Input/output error
135382768+0 records in
135382768+0 records out
69315977216 bytes transferred in 16899.7 secs (4101616 bytes/sec)

Taking the skipped blocks from yesterday's successful but partial run into account, this translate into:

  • The first 200000000 + 135382768 = 335382768 blocks are OK.
  • Blocks are numbered and addressed starting at 0.
  • Block no. 0 though no. 335382767 are good.
  • Block no. 335382768 is unreadable.

I noted that block number down into a simple ascii file and then continued testing, skipping all the 335382768 good and 1 bad block just found:

% dd if=/dev/ad10 of=/dev/null skip=335382769

After just another 1482 good blocks, dd would detect the next error at block no. 335384251. Note that down, skip 335384251 blocks for the next run, rinse and repeat a few times to get an initial feeling how errors appear to be distributed. I ended up with these notes:


335382768 blks OK.

335382768       bad
335384251       bad
335402546       bad
335404029       bad
335408476       bad
335449276       bad
335450759       bad
335452242       bad
335453724       bad
335455208       bad

I finished with these manual stop-and-go tests at this point and had a look at the distances from one bad block to the next:

% awk '/bad/ {if (old) print $1-old; old=$1}' ad10-hd.errs

There's some pattern here: if something bad happened to the magnetic coating in none place, we should see the same sector destroyed along a series of neighbored tracks. Perhaps 1483 is the physical track length at this spot, or the sum of the track length's across all disk surfaces. Perhaps a surface defect in a single spot would look like this?

For further bad block detection, I let dd do the job: the conv=noerror will note every problem but automatically advance past the troubling block (just like we did before) and continue from there, until the end of the disk. I used an initial skip to shortly before the first error already found, block number 335382768:

% dd if=/dev/ad10 of=/dev/null skip=335000000 conv=noerror |& tee ~/ad10-disk.errs

The resulting output starts like this:

dd: /dev/ad10: Input/output error
382768+0 records in
382768+0 records out
195977216 bytes transferred in 26.313414 secs (7447807 bytes/sec)
dd: /dev/ad10: Input/output error
dd: /dev/ad10: Input/output error
384250+0 records in
384250+0 records out
196736000 bytes transferred in 28.703818 secs (6854001 bytes/sec)
dd: /dev/ad10: Input/output error
dd: /dev/ad10: Input/output error
402544+0 records in
402544+0 records out
206102528 bytes transferred in 32.030519 secs (6434567 bytes/sec)
dd: /dev/ad10: Input/output error
dd: /dev/ad10: Input/output error
404026+0 records in
404026+0 records out
206861312 bytes transferred in 34.588290 secs (5980675 bytes/sec)
dd: /dev/ad10: Input/output error

and continues in this style for another 1500 lines. If you look closely, you'll notice the offset-adjusted "records in" numbers increasingly deviate from the bad block numbers established manually:

335382768  335384251  335402546  335404029  ...
335382768  335384250  335402544  335404026  ...

This is because the bad, skipped blocks are not counted as input records. Lesson learned: conv=sync,noerror would replace the unreadable blocks with zeroed blocks for the destination and include these blocks in the the "records in/out" counts.

We will fix this up when extracting the numbers with awk below.

I aborted the run after 13910434 blocks before shutting down the machine and going to bed. The next day, I picked things up at skip=485000000, finding no further errors on the disk.

The results

So these runs located 309 errors on the disk:

% awk '/records in$/  {print $1+bad++}' ~/ad10-disk.errs >
% pr -t5
335382768     335789372     336188550     336561556     337125282
335384251     335790855     336190033     336561768     337126698
335402546     335792337     336191515     336563251     337129770
335404029     335793820     336197446     336566256     337131186
335408476     335795302     336200663     336566468     337132602
335449276     335798520     336205111     336567739     337134018
335450759     335804450     336206593     336567951     337142754
335452242     335805933     336208076     336569221     337199178
335453724     335807415     336214259     336569433     337200594
335455208     335812115     336215741     336570704     337455984
335458424     335813598     336226371     336570916     337458816
335459907     335815080     336227854     336572186     337473216
335461389     335822745     336229337     336572398     337484784
335462872     335831641     336230819     336573669     337487616
335464354     335860568     336232302     336573881     337497768
335465837     335880093     336238484     336575364     337499184
335469054     335887758     336239967     336739950     337503672
335470537     335892207     336241449     336741432     337507920
335472020     335896906     336250597     336742915     337509336
335473502     335898389     336252080     336744397     337510752
335474985     335899871     336253562     336747363     337513824
335476467     335901354     336496544     336753545     337515240
335477950     335902836     336498027     336755028     348457280
335481167     335904320     336504208     336757993     348796478
335482650     335907536     336513105     337054458     348797894
335484132     335909019     336513317     337057290     348800966
335485615     335910502     336514799     337058706     348802382
335487098     335911984     336517805     337061778     348805214
335488580     335913467     336518017     337063194     348806630
335490063     335914949     336522252     337066026     348808046
335494763     335918167     336522464     337068858     348809462
335496245     335919649     336525217     337073346     348812534
335497728     335921132     336525429     337074762     348813950
335499210     335922614     336526912     337076178     348815366
335500693     335924097     336528434     337077594     348816782
335502176     335925580     336529917     337080426     348818198
335505393     335927062     336530129     337081842     348821030
335506876     335930279     336534365     337084914     348829766
335508358     335930280     336534577     337086330     348831182
335509841     335931761     336535848     337087746     348832598
335511323     335931762     336536060     337089162     348837086
335512806     336094401     336537330     337090578     348838502
335514288     336095883     336537542     337091994     348842750
335517506     336133916     336542030     337093410     348844166
335518988     336135399     336542242     337096482     348847238
335520471     336156659     336543513     337097898     348848654
335521954     336158142     336543725     337099314     348850070
335523436     336159625     336544995     337100730     348852902
335532584     336164325     336545207     337102146     348854318
335541732     336165807     336546478     337103562     348857390
335543214     336167290     336546690     337104978     348858806
335553844     336168772     336547961     337108050     348860222
335771077     336170255     336548173     337109466     348861638
335772560     336171737     336552661     337110882     348864470
335775777     336173220     336554143     337112298     348871790
335777259     336176437     336554355     337113714     348873206
335778742     336177920     336555626     337115130     348874622
335780224     336179403     336555838     337118202     348876038
335781707     336180885     336557108     337119618     348877454
335783189     336182368     336557320     337121034     348883358
335784673     336183850     336558591     337122450     348910742
335787889     336185333     336558803     337123866

And here is a plot. The x-axis is along the trouble spots, y-axis the faulty block number. I added the blocks 0 and 490350671 for the first and last block of the disk to put the error positions in relation to the whole disk:

Bad block distribution

So the first 270 bad blocks cluster in one area, the last 40 ones in another.

Entire faulty range: 335382768 to 348910742 = 13527974 blocks = 6.45 GB

Inner good gap: 337515240 to 348457280 = 10942040 blocks = 5.22 GB

Recovering from bad blocks

With 490.350.672 blocks on our 233 GB disk, we should not get mad if one block gets bad. Bad blocks are just a fact of life.

It would sure be nice that this does not go unnotice. Then again: the system should not panic right away due to a bad black. Disk drives are using checksums and error correction internally to prevent wrong bits being returned to the system. Bits can get flipped at many points though, and filesystems such as ZFS or btrfs will do thier checksums to guard against these things.

Even brand new disks will have bad blocks, and more will evolve with time. It wouldn't be economical for a vendor (or buyer) to bet on "perfect" disks.

There are several methods to handle bad blocks:

  1. Sometimes, simply rewriting a bad block can unwedge it again. You have lost the data, of course.

  2. Let the disk designate the bad blocks.

    Digital Equipment Corporation ("DEC") formulated a standard numbered 144 for their system RK06/RK07 disks. The 144 standard describes how bad blocks are marked as bad. Both the drive controller and the OS can then avoid the blocks where the magic pixie dust has fallen off. The standard is designed to cope with up to 126 bad blocks. Tools such as bad144(8) let you investigate the current list of bad blocks and mark more blocks as bad.

    The technology is not dead. Modern drives will do their best to hide such issues from your eyes, too. Disk drives usually come with spare blocks not advertised to the system. The internal drive controller will automatically remap blocks gone bad into spare blocks if possible.

  3. Let the filesystem mask out the bad blocks.

    For a BSD filesystem, badsect(8) can prevent bad blocks from being used for regular files. For linux ext2/3/4 filesystems, there is bladblocks(8).

  4. Partioning around the fault

    Use fdisk/gpt/... to prevent suspicious, large areas from being used.

Repartitioning is what I am going to do here. I will define the 7.15 GB from, say, block 335000000 to 349999999 as a "bad" partition, and the space before and after that is still good for healthy partitions. I won't micro-manage he disk by dealing with the good gap within the two bad streaks differently. I would rather play it safe and include more leading/trailing space with the known to be bad area. 20 GB should be plenty to play it safe and yet have much of the disk for continued use.

Saving time -- using larger blocksizes.

Scanning a disk block by block gives you detailed information where the errors are but is very slow. This is why I spent 3 days on this 233 GB disk.

We can significantly speed up things by reading many blocks at time, i.e. tackle the disk with a much larger blocksize option. Here are the first and second GB read from the disk, once transfered in single blocks, and once with 2024 blocks aggregated into a 1 MB transfer size. We read different GBs from the disk to avoid cache effects:

% dd if=/dev/ad10 of=/dev/null bs=1b count=2097152
2097152+0 records in
2097152+0 records out
1073741824 bytes transferred in 134.774307 secs (7966962 bytes/sec)

% dd if=/dev/ad10 of=/dev/null bs=1m skip=1024 count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 9.100007 secs (117993515 bytes/sec)

Whoa! From 7.6 MB/s to 112.5 MB/s! Warp factor 15! It turns out that even a modest blocksize such as 16 KB gives me 112.2 MB/s throughput.

The maximum throughput is limited by the bottleneck in the chain of

  1. disk drive performance...
  2. speed of the SATA/SCSI/FCAL/... bus connecting the disk to the controller.
  3. speed of the PCI/PCIe/VesaLocal/... bus connecting the controller towards the CPU,
  4. whatever busses any South/North bridges bring with them,
  5. ...your CPU.

Exercise for you:

What dd speed can you achieve with your disk? Is it related to any marketing number you are expecting from your system? (For example: "SATA-II -- 3 Gbit/s!")

Using a sensible aggregated blocksize, we can map out the 233 GB disk in a mere 40 minutes. Any chunk flagged as unreadable can then be subjected to fine-grained, block-wise analysis.

My first attempts turned out to be miserable failures, though: Using bs=1g first and then bs=256m, I caused the machine to

  • have I/O failures with the system disk,
  • have I/O errors with the network interface (just used for ssh),
  • lock up the entire system, requiring the reset button.

I blame the OS for this, DragonFlyBSD-5.8.1 in this case. I finallly got lucky with bs=1m.

dd if=/dev/ad10 of=/dev/null conv=noerror bs=1m | & tee ad10-1m.dd

created a listing with 175 faults, and these were accompanied by 175 messages from the kernel in /var/log/messages:

Sep 18 16:33:06 fred kernel: ad10: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=335382528
Sep 18 16:33:08 fred kernel: ad10: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=335402496
Sep 18 16:33:11 fred kernel: ad10: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=335408384

The plots about the error distributions look very much like the first plot, of course:

Bad 1-MB block distribution

Bad LBA distribution for 1-MB read-attempts

These plots were made by awking the block numbers from the logs and plotting them with with the jqt IDE for the j interactive proramming language. J is very handy to juggle around with data. I used it all the time for this article, mostly as a calculator for dealing with the various values and units, for example turning bytes/sec into MB/s.

If you are stuck with a text terminal, have a look at ministat(1). This is part of the FreeBSD/DragonFlyBSD base system and will give you a simple ascii rendition how values are distributed.

Because our 175 values stack up in just to tall peaks, ending up in some 130 rows with te default ministat output, let me sample down the data to just every 10th value for compactness, and provide the first/last (good) blocks to put the bad block locations into context:

% printf %d\\n 0 0 239429 239429 > bounds
% awk '0==NR%10' bad-1m > bad-samples
% wc -l bad-samples
      17 bad-samples

% ministat -s -w 65 bounds bad-samples 
x bounds
+ bad-samples
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|                                         +                     |
|    x                                    ++               x    |
|    x                                    ++               x    |
|                                         A|                    |
    N           Min           Max        Median           Avg        Stddev
x   4             0        239429      119714.5      119714.5      138234.4
+  17        163796        170278        164278        164922     2035.2878
No difference proven at 95.0% confidence

Beyond dd - be SMART!

The dd command is transferring the data from the disk drive across the busses into memory -- just to /dev/null it there.

Modern disks support the SMART feature. You can ask the drive to do self-tests internally, without impacting the I/O system of the computer.

How this is used shall be part of another article, though.

Posted by neitzel | Permanent link | File under: done, marshlabs

Mon Aug 10 14:08:52 CEST 2020

One-Time Passwords in the BSDs

Some Historic Background

One-Time-Passwords (OTPs) in the BSD systems derive from Bell Communications Research's orignal S/Key system from 1981. Bellcore provided their sources (skey v1.1). Others extended these with patches (such as adding alternative hash algorithms) and devised patches for services such as ftp to use these one-time passwords. Notable was the kit from the US Naval Research Labs (NRL), called OPIE (One-time Passwords In Everthing), avoiding the trademarked S/KEY moniker.

RFC 1760 "The S/KEY One-Time Password System" [1995] describes the system and provides a good bibliography. Followups are RFC 1938, obsoleted by RFC 2289, "A One-Time Password System", which made it to "STD0061".

Back then (in the eighties / early nineties) most TCP/IP traffic was not encrypted and one-time passwords were a significant aid against broken accounts. It was seldomly used in my circles back then (academia), though, and you looked pretty geeky/nerdy when you pulled out a slip of paper to lookup your next password.

These days, ssh has become the defacto standard for remote access. Any paswords are encrypted when using ssh, and more and more people are getting comfortable with using public keys instead of passwords.

So: why still bother with one-time passwords?

I'm travelling around a bit, and it is often convenient to just use the computer of a client customer, friend, hotel lobby, or conference PC connected to the beamer to log into a home machine, without lugging my own laptop around. I generally trust the people I am visiting but can I trust their machines to not have been hacked from wherever and run some keylogger software? Just with my geekier friends.

Do I want to enter my password on these machines? No. Do I want to give access to my ssh private key to these machines? No.

One-time passwords combined with ssh are indeed an attractive option in these situations. So I had a fresh look into the current state of affairs. The last one had been twenty years ago, and I had forgetten everything by now.

Using One-time passwords

You can give one-time passwords a try without having to fear anything bad or inconvenient:

  • Where you used to be prompted for your password, the prompt will also indicate the sequence number for your currently possible OTP.
  • Your standard password will continue to work.
  • Alternatively, you can print a list of numbered OTPs and pick the indicated OTP. (If you ever used a printed TAN sheet with your bank account: that's just the same.)

You can use either password type whenever you like; you are not fixed on just one specific type at any time.

Here is sample session how to use the system, based on NetBSD's commands:

% skeyinit
[Adding neitzel]
Reminder - Only use this method if you are directly connected
       or have an encrypted channel. If you are using telnet
       or rlogin, exit with no password and use skeyinit -s.

Enter secret password: 
Again secret password: 

ID neitzel skey is otp-md4 100 hack37924
Next login password: SOWN FLY MUTE NOR HONK FORE

The first password asked for your is about your standard system password. It is asked to authorize you, after all you are adding another access method to your account right now.

The second, "secret password" is one of the ingredients for generating the list of one-time passwords. You will need it later on to print your OTP list. The "secret password" is supposed to be independent of your system password, but truth be told: I just re-used my standard system password here, it fit the bill for "at least 10 characters".

The last paragraph is kind of "off-by-one" and highly misleading. Yes, the system is now setup to provide a sequence of up to 100 OTPs for you, but you will never encounter the otp-md4 100 hack37924 OTP challenge above, and you can just forget the "Next login password".

All OTP passwords are "fixed width" 64 bit values. They can get rendered as hex strings (16 places) or as a sequence of six pronouncable words. The phrase SOWN FLY MUTE NOR HONK FORE above equate the hex value e3028f039669a309 but are much easier to remember and transcribe.

The following is going to be your real, first OTP challenge. You can enquire it anytime:

% skeyinfo
Your next otp-md4 99 hack37924

Look, Ma, 99 not 100!

At this point, your login prompt also changes:

% ssh -p 222 hackett
Password [ otp-md4 99 hack37924 ]:

You can now use either your usual password or the OTP matching the current otp-md4 99 hack37924 challenge.

How to know that OTP? Create a list when you are in your trusted network and can re-enter the "secret password" from above safely:

% skey -n 5 -t md4 99 hack37924
Reminder - Do not use this program while logged in via telnet or rlogin.
Enter secret password: 

So the OTP matching the the current challenge is the word sequence OLDY PAT CAST GIG VAIL TRIO. You may enter the sequence in lowercase but do not omit the spaces. You could also enter the OTP's hex rendition (c7c649910b7f2fdf, available through skey -x ...) if you enjoy to flagellate yourself.

You could print a longer list with the next, say, twenty OTPs and put this little sheet into your wallet.

Mind you:

(1) This is close to the notorious passwords on yellow post-it stickers.

(2) OTP-ing from remote into your favourite jump-host just to issue long-time passwords from there and have these key-logged may not be the smartest move.

Comparing the current BSDs

  • NetBSD 1.3.3 -- 9.99.x:

    Has the skey suite in the base system:

    skey(3) - one-time password (OTP) library
    pam_skey(8) - S/Key PAM module
    skeyinit(1) - change password or add user to S/Key authentication system
    skeyinfo(1) - obtain the next S/Key challenge for a user
    skeyaudit(1) - warn users if their S/Key will soon expire
    skey(1) - respond to an OTP challenge

    Possible hashes are: MD4 (default), MD5, SHA1

    The heimdahl package comes with its own one-time password tools, otp(1) and otpprint(1).

  • OpenBSD:

    Has the skey suite, too, but different from NetBSD:

    • it provides the login_skey(8) utility to hook S/Key services into other programs. (OpenBSD does not use the "PAM" framework.)
    • It discarded the original MD4 hash and offers MD5, SHA1, and RIPEMD-160 instead.
  • FreeBSD-4.7 and 12.1: provide the "opie" ("one-time passwords in everything") suite of tools in the base system:

    opie(4), OPIE(4) - One-time Passwords In Everything
    opieinfo(1) - Extract sequence number and seed for future OPIE challenges
    opiekey(1), otp-md4(1), otp-md5(1) - Programs for computing responses to OTP challenges
    opiepasswd(1) - Change or set a user's password for the OPIE authentication system
    pam_opie(8) - OPIE PAM module
    pam_opieaccess(8) - OPIEAccess PAM module
    opiekeys(5), /etc/opiekeys(5) - OPIE database of user key information
    opieaccess(5), /etc/opieaccess(5) - OPIE database of trusted networks

    Usable hash algorithms: MD4 (compatible with Bellcore S/Key), MD5 (the default on FreeBSD).

  • DragonFlyBSD-5.8:

    The DragonFly BSD Handbook has a chapter on security. It explains both the OPIE and S/Key suites in detail.

    However, neither is in the base system anymore, or available as a package instead. DragonFlyBSD-5.6 was the last release coming with OPIE support.

The Actual S/Key Algorithm

During skeyinit, the first few letters of the hostname (hackett here) and some random number form the "seed" string: hack37924 here.

The seed string is concatenated with your "secret password".

This is hashed once, using the selected hash algorithm, say md4, into md4("$seed$secret"), 128 bits. These are condensed ("take the first half") into the 64 bits being the OTP matching the challenge md4 1 hack37924.

Take the hash of previous hash, i.e. md4(md4("$seed$secret")), condense those 128 bits that into the 64 bit OTP matching challenge md4 2 hack37924.

Rinse and repeat, say a hundred times.

S/KEY and service integration

On NetBSD, the Pluggable Authentication Modules (PAM) framework is used to to tie the SKEY/OPIE method into various services. As soon as you have gone through the skeyinit procedure, you can use your OTP list all over the place: the system stanza makes it a default authentication method. The following services have it explicitly enabled on top of the default:


It will just work, without turning any knobs.

One small caveat: As already mentioned above, the optional, non-base heimdahl package comes with its own OTP system. Services from this package, such as popper(8), are not tied to the module. You need the setup with otp(1) and thusly a separate OTP list to use these services. Also, popper(8) needs the -a otp option to rely on the OTPs.

Posted by neitzel | Permanent link

Tue Jun 9 17:59:57 CEST 2020

Happy Uptime, ips!

After missing the 2000 days milestone:

Last login: Tue Jun  9 13:16:53 from
ULTRIX V4.4 (Rev. 69) System #17: Tue May 28 10:12:31 MET DST 1996
You have mail.
neitzel 1 > date
Tue Jun  9 13:18:47 MET DST 2020
neitzel 2 > uptime
  1:18pm  up 2222 days, 22:22,  8 users,  load average: 0.21, 0.05, 0.00
neitzel 3 >

Posted by neitzel | Permanent link | File under: marshlabs, bsd

Wed May 27 09:28:47 CEST 2020

lowell on 6.46.6: upgrading RouterOS without enough flash

Upgrading the RouterOS on MikroTik devices is a simple affair:

  • You check for new firmware;
  • if there is any, review the release notes;
  • download, install and reboot;
  • update the bootloader in a separate step,
  • followed by a second reboot.

All this is usually done in five minutes. So far I never had any issues caused by an RouterOS update.

Which makes you buy more MikroTik gear. The youngster in my home is...

[neitzel@lowell] > /system routerboard print 
       routerboard: yes
    board-name: hAP lite
         model: RouterBOARD 941-2nD
     serial-number: 7C2C07C4455B
     firmware-type: qca9531L
  factory-firmware: 3.36
  current-firmware: 6.46.6
  upgrade-firmware: 6.46.6

This is a cheap (22 EUR), small wireless router/switch/access point serving my kitchen. Permanently attached nodes are NCD X terminal terminal fz and the DAB+/FM/Internet/LAN-Media radio gaga.

Lowell is supposed to replace the small cisco switch lab there but, as of now, they all still share the window sill:

kitchen switchen

Updates vs. IPv6-only

I have run out of my public IPv4 addresses at home long ago. Because lowell is currently mostly just operating as an access point it doesn't need any layer-3 address except for management. And so it became my first IPv6-only node, without any IPv4 address at all.

This was all fine. Until I tried the first RouterOS upgrade.

As tcpdump showed the upgrade process will resolve the server name has address has address has IPv6 address 2a02:610:7501:4000::226 has IPv6 address 2a02:610:7501:1000::196

which apparently would serve both the current and the vintage protocol flavours. The hAP though will first try an IPv4 server, notice that that network is unreachable, and... give up. What a shame!

This short-coming is particularly disappointing because the RouterOS can transfer data via IPv6 when asked manually:

[neitzel@lowell] /file> /tool fetch url="" output=user
      status: finished
  downloaded: 0KiB
    data: All my friends and I are crazy. That's the only thing that 
          keeps us sane.

[neitzel@lowell] /file>


Workaround: for RouterOS updates, I temporarily /ip dhcp-client enable 0. Do the upgrade dance, and /ip dhcp-client disable 0 again.

Not nice but there are worse things in life.

Updates vs. Flash Size

Hey, let's just spend fifteen minutes on upgrading all three MikroTik gadgets, I thought around 8pm. When I went into bed, it was around 5am.

The upgrades went without a hitch on the two larger devices, billy and hall but on lowell strange things would happen. The "download" step went fine but the "reboot for install" step would end up in the same old package versions as before (6.46.2), with the download new version (6.46.6) purged from the /file area. Repeated attempts didn't help.

WLKIKIV, as we say here, and a quick /log print shows the problem:

not enough disk space.

The "hdd" flash memory is indeed much more constrained on lowell:

% echo billy hall lowell | \
> xargs -n1 -Ixx ssh xx /system resource print | \
> grep -E 'hdd|board-name'
       free-hdd-space: 107.3MiB
      total-hdd-space: 128.0MiB
           board-name: RB2011L
       free-hdd-space: 109.0MiB
      total-hdd-space: 128.0MiB
           board-name: CRS125-24G-1S-2HnD
       free-hdd-space: 7.1MiB
      total-hdd-space: 16.0MiB
           board-name: hAP lite

The size of the stock "combo" release packages is now approaching half of the 16.0MiB disk size:

% echo 2 6 | xargs -n1 -I X lynx -head -dump \
> |\
> grep Length
Content-Length: 7651050
Content-Length: 7700154

During an upgrade, both the old and new version have to sit side-by-side on the disk, the filesystem structure needs some space, the config needs some space, ... the official documentation is asking for 2 MB spare capacity. After this download though I was down to the last 44 KB(!) on the disk.

A few months ago, in the same situation, I found a surplus support-dump I could delete to gain enough breathing space. No such luck tonight.

With the current RouterOS "Stable" images, things have now simply become too tight for a stock "hAP lite" and similar devices. without much extra config/data on its flash medium to upgrade to newer stock RouterOS versions. To be frank, this is major surprise if not a disgrace.

Me vs. the Web Forum

The official RouterOS documentation doesn't address this problem.

Grudgingly I dived into the "community support". I simply hate sifting through web fora, no matter which ones. It took hours.

Yes, I was not the only one with the problem. There were messages about the problem without any followup at all; there were quite a handful of wrong explanations; there was even a bit of ad-hominem and mud-slinging.

It did pointed me to the proper solution though:

By default, have their software installed from a "combined routeros package" which contains a selection of individual feature packages. It should not happen but the combined package can become to big for smaller platforms. You have then to switch over to deal with the packages individually, selecting you own mix.

The Solution

My first idea was to delete a few of the 6.46.2 packages which I currently don't use in order to create the space for the complete new kit.

Turns out that you cannot /system package uninstall anything when everything comes from the "combined routeros".

The only way forward is this:

  1. Make an extra backup of your configuration beyond of what the automatic reboot/reset backup is providing. The commands are simple and the demands on precious flash space are small:

    [neitzel@lowell] > /system backup save
    [neitzel@lowell] > /export compact file=cfg-mn
    [neitzel@lowell] > /file print where type!=directory
     # NAME                   TYPE             SIZE CREATION-TIME       
     0 cfg-mn.rsc             script         6.4KiB may/27/2020 03:18:50
     1 auto-before-reset.b... backup        19.0KiB jan/02/1970 02:42:11
     2 lowell-20200527-031... backup        30.1KiB may/27/2020 03:14:53
  2. Download the "Extra packages" kit matching your hardware from

    This kit does not contain just "extra" packages for the more obscure features as the title suggests to me. Instead, the filename is much more appropriate: This zip contains the ten packages which comprise the "combined" = "Main" package (=routeros-smips-6.46.6.npk), and only three extra pkgs: multicast, openflow, tr069-client. (A full listing is below.)

  3. Download this .zip archive elsewhere and extract the .npk packages.

  4. Use scp, ftp or RouterOS' /tool fetch to copy a subset of the packages into the /file area for installation. Everybody needs the system package which weighs in with 5.5 MB alone. Another essential package for me is ipv6 (196 KB) to be able to access/manage the hAP-lite at all. dhcp might be that thing for you, and in that case you also need security (155 + 307 = 462 KB). And since security is also required for ssh access, I used that, too. These four pkgs already total at 6+ MB, enough to get nervous.

  5. Reboot to install these packages.

  6. You only get the few selected new packages.

    All packages/features from the old version get removed. The result is not a mix of old and updated packages.

    Your new reduced feature set will load your old configration as much as possible. Settings for now missing features will be lost. For example, without the wireless pkg, I lost my WLAN definition.

    Luckily, you didn't skip the the first step, saving your config, did you?

  7. With the old version's packages gone, you have now plenty of disk space for the other new packages. Install as much as you want by copying them to the /file area and rebooting.

  8. With all wanted new packages in place, you can now reload your configuration:

    /system backup load name=lowell-20200527-0314.backup


    /import cfg-mn.rsc

    As of now I haven't figured out which is better in which case. I suppose that either would do for me.

I believe you can choose between these two strategies:

  • Exercise some restraint and aim at "below 7 MB for everything", so that future upgrades are completely painless. The standard /system package update process should download only those packages you have in use.

    In my case, this would be: system, ipv6, wireless, security, dhcp. As of now, this already totals in 7149168 bytes aka 6.8MiB. Hrrmmm....

  • If you prefer a "all packages" setup, you will have to go through the "update to/with minimal package set / add extras later" on every single update. The only ease is that you can get rid of ballast before doing the upgrade: /system package uninstall will now work. You can then do the (minimal) upgrade and re-add non-minimal packages afterwards. Again, this requires the download of the "Extras" .zip-file. And, of course, the backup of your configuration.

I am wondering how all this will pan out for me. I'll try to automate the "all packages" updates, i.e. the second approach.


For reference, here is my current lowell installation and sizes of the corresponding all_packages:

neitzel 373 > unzip -l
  Length     Date   Time    Name
 --------    ----   ----    ----
    69713  05-14-20 12:14   advanced-tools-6.46.6-smips.npk
   155729  05-14-20 12:14   dhcp-6.46.6-smips.npk
   147537  05-14-20 12:14   hotspot-6.46.6-smips.npk
   196689  05-14-20 12:14   ipv6-6.46.6-smips.npk
    57425  05-14-20 12:14   mpls-6.46.6-smips.npk
    36945  05-14-20 12:14   multicast-6.46.6-smips.npk
    49233  05-14-20 12:14   openflow-6.46.6-smips.npk
   258129  05-14-20 12:14   ppp-6.46.6-smips.npk
    69713  05-14-20 12:14   routing-6.46.6-smips.npk
   307281  05-14-20 12:14   security-6.46.6-smips.npk
  5330220  05-14-20 12:14   system-6.46.6-smips.npk
   114769  05-14-20 12:14   tr069-client-6.46.6-smips.npk
  1159249  05-14-20 12:14   wireless-6.46.6-smips.npk

[neitzel@lowell] > /system package print
Flags: X - disabled
 #   NAME                    VERSION                    SCHEDULED              
 0   security                6.46.6
 1   ipv6                    6.46.6
 2   dhcp                    6.46.6
 3   advanced-tools          6.46.6
 4   system                  6.46.6
 5   wireless                6.46.6
 6   hotspot                 6.46.6
 7   mpls                    6.46.6
 8   multicast               6.46.6
 9   openflow                6.46.6
10   ppp                     6.46.6
11   routing                 6.46.6
12   tr069-client            6.46.6

[neitzel@lowell] > /system reso print
           uptime: 8h49m20s
          version: 6.46.6 (testing)
           build-time: Apr/27/2020 10:32:16
     factory-software: 6.28
          free-memory: 7.7MiB
         total-memory: 32.0MiB
              cpu: MIPS 24Kc V7.4
        cpu-count: 1
        cpu-frequency: 650MHz
         cpu-load: 0%
       free-hdd-space: 7.0MiB
      total-hdd-space: 16.0MiB
  write-sect-since-reboot: 215
     write-sect-total: 194269
           bad-blocks: 0%
    architecture-name: smips
           board-name: hAP lite
         platform: MikroTik

The 22,- EUR are dirt cheap but my time isn't. Automating the the "all packages" updates will certainly be a worthwile learning experience.

How about 44,- EUR for a non-lite hAP? Or a 50,- hAP ac lite? Flash is still sized at 16MiB but you can add a USB stick. Would that help? I couldn't find any statements on this in the manual or product brochures.

If not, then the entire hAP/cAP/wAP range of "16 MB Flash" MikroTik products does not really have a future in the "Stable" RouterOS track for consumers. MikroTik must resolve this issue somehow.

The RB951Ui-2HnD comes at 80,- EUR and with 128MiB NAND storage. This would definitely remove the upgrade pains albeit at a noticeable price increase.

Posted by neitzel | Permanent link | File under: learned, done, marshlabs

Sun May 24 20:42:05 CEST 2020

Another stab at plan9

I used to run plan9 on my old pentium1 laptop and pentium1 PCs in the last millenium. In fact, I had bought specific hardware such as 3com 3C589 or Orinoco Wavelan PCMCIA devices supported by plan9.

I never had any any luck to run plan9 (or variants such as 9atom) on more modern gear. My only successful attempt to run plan9 on a 1000H Eeepc was with Russ Cox' 9vx.

This day started off with some googling after bhyve plan9, took a turn via proxmox plan9, and ended with another read of the 9front pages. Since I'm running now pretty decent hardware capable of running qemu, this is now a viable option:

welcome, 9front!

This is just an initial boot. The actual installation has to wait for another day. I am still a complete klutz with using qemu and had to google how to get my mouse back. (Luckily, another machine was at hand.)

Posted by neitzel | Permanent link | File under: learned, done

Fri May 22 17:00:02 CEST 2020

Posted by neitzel | Permanent link | File under: learned

Sun May 17 22:17:21 CEST 2020

pkgin upgrades working again

A good bug report goes a long way... Matthew Sporleder fixed the netbsd-8.x repositories within just a few hours. Thanks!

Posted by neitzel | Permanent link | File under: bsd

Sat May 16 15:51:19 CEST 2020

pkgin upgrade woes

With the recent upgrade of hackett on the netbsd-8 stable track to "8.2-and-a-bit-later" I also switch the pkgin(1) repository from the 8.1 subdirectory to the 8.2 one. This was on May, 2nd.

For the first nine days after, pkgin update wouldn't find any new catalogue but eventually, on May 11th, there was one.

pkgin upgrade then greeted me with something similar to this:

log 167 # pkgin upgrade
calculating dependencies...done.

30 packages to refresh:
  p5-Socket6-0.29nb1 p5-IO-Socket-INET6-2.72nb5 dehydrated-0.6.5
  p5-IO-CaptureOutput-1.1105 p5-Email-Valid-1.202nb3 Markdown-1.0.1nb7
  p5-Net-Domain-TLD-1.75nb3 p5-Net-SMTP-SSL-1.04nb3 automake-1.16.1nb1
  p5-TimeDate-2.30nb6 p5-Authen-SASL-2.16nb7 p5-Net-IP-1.26nb7
  p5-GSSAPI-0.28nb10 p5-Mozilla-CA-20180117nb2 p5-Net-LibIDN-0.12nb11
  p5-Digest-HMAC-1.03nb9 m4-1.4.18nb2 autoconf-2.69nb9 ytalk-3.3.0nb1
  tig-1.2.1nb3 tcsh-6.22.02 rlwrap-0.43nb3 pcre2-10.34 lzo-2.10 lz4-1.9.2
  libxml2-2.9.10nb1 libuuid-2.32.1 libunistring-0.9.10 iperf-2.0.5nb1

19 packages to upgrade:
  scrollz-2.2.3nb8 screen-4.8.0nb1 python37-3.7.7 perl-5.30.2 p5-Net-SSLeay-1.88
  p5-Net-DNS-1.23 p5-MailTools-2.21 p5-IO-Socket-SSL-2.067 p5-Error-0.17029
  openvpn-2.4.8nb2 nghttp2-1.40.0nb2 libidn-1.35 libffi-3.3nb2 iperf3-3.7nb1
  git-docs-2.25.4 git-base-2.25.4 fossil-2.10nb2 curl-7.69.1 bash-5.0.16nb1

1 package to install:

30 to refresh, 19 to upgrade, 1 to install
14M to download, 15M to install

proceed ? [Y/n]

I welcome the 19 packages to upgrade -- that's why I run the command.

I was very suprised by the refresh list, though. Why do need pkgs to get refreshed even when the pkg is already on board, with the exact same version? Different dependencies? Loss of local cache files? Whatever the reason may be, I was particularly surprised of the ytalk pkg appearing in this list. I had installed that package just five days earlier. And yes, at exactly ytalk-3.3.0nb1 already.

Be that as it may, the real problem showed up when proceeding with this upgrade:

proceed ? [Y/n] y
p5-Authen-SASL-2.16nb7.tgz                    100%   24KB  24.3KB/s   00:00    
download error: p5-Authen-SASL-2.16nb7 size does not match pkg_summary

This error message is not new to me. I have seen it before (last summer) and suspect some inconsistencies within the Fastly CDN hosting the NetBSD repositories.

The relevant commands to debug this:

pkgin 75 > pwd
pkgin 76 > sqlite3 pkgin.db 
SQLite version 3.17.0 2017-02-13 16:02:40
Enter ".help" for usage hints.
sqlite> .mode line
sqlite> select *  from remote_pkg where pkgname like 'p5-Authen-SASL' ;
          PKG_ID = 6173
     FULLPKGNAME = p5-Authen-SASL-2.16nb7
         PKGNAME = p5-Authen-SASL
         PKGVERS = 2.16nb7
      BUILD_DATE = 2020-04-01 03:57:23 +0000
         COMMENT = Perl module to handle SASL authentication
         LICENSE = gnu-gpl-v2 OR artistic
        HOMEPAGE =
      OS_VERSION = 8.0
         PKGPATH = security/p5-Authen-SASL
     PKG_OPTIONS = gssapi
      CATEGORIES = security perl5
        SIZE_PKG = 119267
       FILE_SIZE = 24892
           OPSYS = NetBSD
sqlite> ^D

pkgin 76 > echo 0 1 2 | xargs -n1 -I XX lynx -head -dump \
? |\
? grep Length
Content-Length: 24900
Content-Length: 24892
Content-Length: 24900

So there's the "size does not match". Others on the netbsd-users@ mailing list are seeing this download error, too, and I just joined in with my findings.

Posted by neitzel | Permanent link | File under: done, bsd

Wed May 13 00:03:06 CEST 2020

tweaked blog entry format

If I tweaked things correctly, this entry should show up beneath a header with month/day time (not just time) when viewed on it own "permalink" page.

Posted by neitzel | Permanent link | File under: done