Wed May 27 09:28:47 CEST 2020

lowell on 6.46.6: upgrading RouterOS without enough flash

Upgrading the RouterOS on MikroTik devices is a simple affair:

  • You check for new firmware;
  • if there is any, review the release notes;
  • download, install and reboot;
  • update the bootloader in a separate step,
  • followed by a second reboot.

All this is usually done in five minutes. So far I never had any issues caused by an RouterOS update.

Which makes you buy more MikroTik gear. The youngster in my home is...

lowell.marshlabs.gaertner.de

[neitzel@lowell] > /system routerboard print 
       routerboard: yes
    board-name: hAP lite
         model: RouterBOARD 941-2nD
     serial-number: 7C2C07C4455B
     firmware-type: qca9531L
  factory-firmware: 3.36
  current-firmware: 6.46.6
  upgrade-firmware: 6.46.6

This is a cheap (22 EUR), small wireless router/switch/access point serving my kitchen. Permanently attached nodes are NCD X terminal terminal fz and the DAB+/FM/Internet/LAN-Media radio gaga.

Lowell is supposed to replace the small cisco switch lab there but, as of now, they all still share the window sill:

kitchen switchen

Updates vs. IPv6-only

I have run out of my public IPv4 addresses at home long ago. Because lowell is currently mostly just operating as an access point it doesn't need any layer-3 address except for management. And so it became my first IPv6-only node, without any IPv4 address at all.

This was all fine. Until I tried the first RouterOS upgrade.

As tcpdump showed the upgrade process will resolve the server name

download.mikrotik.com has address 159.148.172.226
download.mikrotik.com has address 159.148.147.204
download.mikrotik.com has IPv6 address 2a02:610:7501:4000::226
download.mikrotik.com has IPv6 address 2a02:610:7501:1000::196

which apparently would serve both the current and the vintage protocol flavours. The hAP though will first try an IPv4 server, notice that that network is unreachable, and... give up. What a shame!

This short-coming is particularly disappointing because the RouterOS can transfer data via IPv6 when asked manually:

[neitzel@lowell] /file> /tool fetch url="http://hackett.6.ml.gaertner.de/index.html" output=user
      status: finished
  downloaded: 0KiB
    data: All my friends and I are crazy. That's the only thing that 
          keeps us sane.

[neitzel@lowell] /file>

D'oh!

Workaround: for RouterOS updates, I temporarily /ip dhcp-client enable 0. Do the upgrade dance, and /ip dhcp-client disable 0 again.

Not nice but there are worse things in life.

Updates vs. Flash Size

Hey, let's just spend fifteen minutes on upgrading all three MikroTik gadgets, I thought around 8pm. When I went into bed, it was around 5am.

The upgrades went without a hitch on the two larger devices, billy and hall but on lowell strange things would happen. The "download" step went fine but the "reboot for install" step would end up in the same old package versions as before (6.46.2), with the download new version (6.46.6) purged from the /file area. Repeated attempts didn't help.

WLKIKIV, as we say here, and a quick /log print shows the problem:

not enough disk space.

The "hdd" flash memory is indeed much more constrained on lowell:

% echo billy hall lowell | \
> xargs -n1 -Ixx ssh xx /system resource print | \
> grep -E 'hdd|board-name'
       free-hdd-space: 107.3MiB
      total-hdd-space: 128.0MiB
           board-name: RB2011L
       free-hdd-space: 109.0MiB
      total-hdd-space: 128.0MiB
           board-name: CRS125-24G-1S-2HnD
       free-hdd-space: 7.1MiB
      total-hdd-space: 16.0MiB
           board-name: hAP lite
%

The size of the stock "combo" release packages is now approaching half of the 16.0MiB disk size:

% echo 2 6 | xargs -n1 -I X lynx -head -dump \
> https://download.mikrotik.com/routeros/6.46.X/routeros-smips-6.46.X.npk |\
> grep Length
Content-Length: 7651050
Content-Length: 7700154
%

During an upgrade, both the old and new version have to sit side-by-side on the disk, the filesystem structure needs some space, the config needs some space, ... the official documentation is asking for 2 MB spare capacity. After this download though I was down to the last 44 KB(!) on the disk.

A few months ago, in the same situation, I found a surplus support-dump I could delete to gain enough breathing space. No such luck tonight.

With the current RouterOS "Stable" images, things have now simply become too tight for a stock "hAP lite" and similar devices. without much extra config/data on its flash medium to upgrade to newer stock RouterOS versions. To be frank, this is major surprise if not a disgrace.

Me vs. the Web Forum

The official RouterOS documentation doesn't address this problem.

Grudgingly I dived into the "community support". I simply hate sifting through web fora, no matter which ones. It took hours.

Yes, I was not the only one with the problem. There were messages about the problem without any followup at all; there were quite a handful of wrong explanations; there was even a bit of ad-hominem and mud-slinging.

It did pointed me to the proper solution though:

By default, have their software installed from a "combined routeros package" which contains a selection of individual feature packages. It should not happen but the combined package can become to big for smaller platforms. You have then to switch over to deal with the packages individually, selecting you own mix.

The Solution

My first idea was to delete a few of the 6.46.2 packages which I currently don't use in order to create the space for the complete new kit.

Turns out that you cannot /system package uninstall anything when everything comes from the "combined routeros".

The only way forward is this:

  1. Make an extra backup of your configuration beyond of what the automatic reboot/reset backup is providing. The commands are simple and the demands on precious flash space are small:

    [neitzel@lowell] > /system backup save
[neitzel@lowell] > /export compact file=cfg-mn
[neitzel@lowell] > /file print where type!=directory
 # NAME                   TYPE             SIZE CREATION-TIME       
 0 cfg-mn.rsc             script         6.4KiB may/27/2020 03:18:50
 1 auto-before-reset.b... backup        19.0KiB jan/02/1970 02:42:11
 2 lowell-20200527-031... backup        30.1KiB may/27/2020 03:14:53

  2. Download the "Extra packages" kit matching your hardware from https://www.mikrotik.com/download.

    This kit does not contain just "extra" packages for the more obscure features as the title suggests to me. Instead, the filename is much more appropriate: all_packages-smips-6.46.6.zip. This zip contains the ten packages which comprise the "combined" = "Main" package (=routeros-smips-6.46.6.npk), and only three extra pkgs: multicast, openflow, tr069-client. (A full listing is below.)

  3. Download this .zip archive elsewhere and extract the .npk packages.

  4. Use scp, ftp or RouterOS' /tool fetch to copy a subset of the packages into the /file area for installation. Everybody needs the system package which weighs in with 5.5 MB alone. Another essential package for me is ipv6 (196 KB) to be able to access/manage the hAP-lite at all. dhcp might be that thing for you, and in that case you also need security (155 + 307 = 462 KB). And since security is also required for ssh access, I used that, too. These four pkgs already total at 6+ MB, enough to get nervous.

  5. Reboot to install these packages.

  6. You only get the few selected new packages.

    All packages/features from the old version get removed. The result is not a mix of old and updated packages.

    Your new reduced feature set will load your old configration as much as possible. Settings for now missing features will be lost. For example, without the wireless pkg, I lost my WLAN definition.

    Luckily, you didn't skip the the first step, saving your config, did you?

  7. With the old version's packages gone, you have now plenty of disk space for the other new packages. Install as much as you want by copying them to the /file area and rebooting.

  8. With all wanted new packages in place, you can now reload your configuration:

    /system backup load name=lowell-20200527-0314.backup

    or

    /import cfg-mn.rsc

    As of now I haven't figured out which is better in which case. I suppose that either would do for me.

I believe you can choose between these two strategies:

  • Exercise some restraint and aim at "below 7 MB for everything", so that future upgrades are completely painless. The standard /system package update process should download only those packages you have in use.

    In my case, this would be: system, ipv6, wireless, security, dhcp. As of now, this already totals in 7149168 bytes aka 6.8MiB. Hrrmmm....

  • If you prefer a "all packages" setup, you will have to go through the "update to/with minimal package set / add extras later" on every single update. The only ease is that you can get rid of ballast before doing the upgrade: /system package uninstall will now work. You can then do the (minimal) upgrade and re-add non-minimal packages afterwards. Again, this requires the download of the "Extras" .zip-file. And, of course, the backup of your configuration.

I am wondering how all this will pan out for me. I'll try to automate the "all packages" updates, i.e. the second approach.

Summary

For reference, here is my current lowell installation and sizes of the corresponding all_packages:

neitzel 373 > unzip -l all_packages-smips-6.46.6.zip
Archive:  all_packages-smips-6.46.6.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    69713  05-14-20 12:14   advanced-tools-6.46.6-smips.npk
   155729  05-14-20 12:14   dhcp-6.46.6-smips.npk
   147537  05-14-20 12:14   hotspot-6.46.6-smips.npk
   196689  05-14-20 12:14   ipv6-6.46.6-smips.npk
    57425  05-14-20 12:14   mpls-6.46.6-smips.npk
    36945  05-14-20 12:14   multicast-6.46.6-smips.npk
    49233  05-14-20 12:14   openflow-6.46.6-smips.npk
   258129  05-14-20 12:14   ppp-6.46.6-smips.npk
    69713  05-14-20 12:14   routing-6.46.6-smips.npk
   307281  05-14-20 12:14   security-6.46.6-smips.npk
  5330220  05-14-20 12:14   system-6.46.6-smips.npk
   114769  05-14-20 12:14   tr069-client-6.46.6-smips.npk
  1159249  05-14-20 12:14   wireless-6.46.6-smips.npk

[neitzel@lowell] > /system package print
Flags: X - disabled
 #   NAME                    VERSION                    SCHEDULED              
 0   security                6.46.6
 1   ipv6                    6.46.6
 2   dhcp                    6.46.6
 3   advanced-tools          6.46.6
 4   system                  6.46.6
 5   wireless                6.46.6
 6   hotspot                 6.46.6
 7   mpls                    6.46.6
 8   multicast               6.46.6
 9   openflow                6.46.6
10   ppp                     6.46.6
11   routing                 6.46.6
12   tr069-client            6.46.6

[neitzel@lowell] > /system reso print
           uptime: 8h49m20s
          version: 6.46.6 (testing)
           build-time: Apr/27/2020 10:32:16
     factory-software: 6.28
          free-memory: 7.7MiB
         total-memory: 32.0MiB
              cpu: MIPS 24Kc V7.4
        cpu-count: 1
        cpu-frequency: 650MHz
         cpu-load: 0%
       free-hdd-space: 7.0MiB
      total-hdd-space: 16.0MiB
  write-sect-since-reboot: 215
     write-sect-total: 194269
           bad-blocks: 0%
    architecture-name: smips
           board-name: hAP lite
         platform: MikroTik

The 22,- EUR are dirt cheap but my time isn't. Automating the the "all packages" updates will certainly be a worthwile learning experience.

How about 44,- EUR for a non-lite hAP? Or a 50,- hAP ac lite? Flash is still sized at 16MiB but you can add a USB stick. Would that help? I couldn't find any statements on this in the manual or product brochures.

If not, then the entire hAP/cAP/wAP range of "16 MB Flash" MikroTik products does not really have a future in the "Stable" RouterOS track for consumers. MikroTik must resolve this issue somehow.

The RB951Ui-2HnD comes at 80,- EUR and with 128MiB NAND storage. This would definitely remove the upgrade pains albeit at a noticeable price increase.


Posted by neitzel | Permanent link | File under: learned, done, marshlabs
/\