Performance using a microSD passthru to host OS MSC CIRCUITPY_SDCARD_USB

[mikeysklar]

Opened to review performance issues related to exposing microSD cards through the mass storage controller. Related to a PR #10967 that was added to enable macOS support and marked all LUNs as removable.

@bablokb @dhalbert @tannewt

[mikeysklar]

@bablokb -

During earlier testing I saw Linux showing the best host OS behavior. macOS had a long cold plug delay and windows just hammered away on a microSD disk with read(10) for over 40 minutes.

I'd like to run the tests again with todays main branch. One build in default mode and the other compiled with CIRCUITPY_SDCARD_USB=0 (off).

• Metro RP2040
• 16GB SanDisk class 10 (FAT32)
• 64MB (FAT16)

Running this benchmark code.py. 30 second usb sniffs.

That way we can put some numbers to the behavior and see the difference between FAT16 and different host OSs. Also probably come up with some host OS settings to help with the user experience. eg. Turn off Windows File Explorer searching new media on plug-in.

Does this make sense or would you like to see something different?

[bablokb]

I'd like to run the tests again with todays main branch. One build in default mode and the other compiled with CIRCUITPY_SDCARD_USB=0 (off).

That is what I basically did. As a baseline, I also used 9.2.9.

Note that we already have a similar issue #10733. The title of that issue is now misleading.

[mikeysklar]

I am trying to match your testing. You have shown that PR #10967 does not have a significant effect on performance aside from first boot penalty.

I'd like to run a round of benchmarks across FAT16 vs FAT32, and with SDCARD_USB on vs off, to put concrete numbers to it. Will post a summary when done.

[mikeysklar]

Here are the results.

Board: Metro RP2040, firmware 2026-05-07 main @ 13a8fb9f7b

Firmware	SD format	Host OS	Slow-window (s)	Peak cpu (ms)
usb-on	FAT32	macOS	~17	492
usb-on	FAT32	Linux	~8.4	492
usb-on	FAT32	Windows	never settled	465
usb-on	FAT16	macOS	~0.6	335
usb-on	FAT16	Linux	~3.4	458
usb-on	FAT16	Windows	~2	419
usb-off	FAT32	macOS	~0	247
usb-off	FAT32	Linux	~1.5	306
usb-off	FAT32	Windows	0	229
usb-off	FAT16	macOS	~0	253
usb-off	FAT16	Linux	~1.5	306
usb-off	FAT16	Windows	0	229

Slow-window = seconds after power-cycle before CPU timing returns to steady-state (229 ms).

Peak cpu = worst single iteration during that window.

The main variables are card format and host OS:

• FAT16 helps a lot. Windows with FAT32+usb-on never settled within the 30s window. FAT16 drops it to ~2s — fewer sectors for the host to scan.

• usb-off — essentially zero slow-window on macOS and Windows. The ~1.5s Linux blip is CP's own internal SD mount, not host activity.

Full captures, SCSI decoded logs, and bench output: https://github.com/mikeysklar/bench-microsd-msc

Host OS	Version
macOS Tahoe	26.3.1 (build 25D2128)
Ubuntu	24.04.3 LTS, kernel 6.17.0-22-generic
Windows 11 Pro	10.0.26200 Build 26200

Format	Card
FAT32	16 GB SanDisk Ultra Class 10 microSD
FAT16	64 MB microSD (manufacturer unknown)

[bablokb]

I can't confirm the "slow-window", at least not in this size. With my 8GB card (which leads to a crash), there is nothing like that up to the crash. The crash is 122s after POR and the print statements just before crashing take about 1.5s each. But my test does much more than "cpu-only": it dynamically loads code, uses I2C, connects to WLAN, does a request and so on. Especially the networking should add load to background processing and will probably compete with tiny-usb.

BTW: the card itself currently only has 8 small files and is working fine with CP since I guess something like 8.0.5. I also checked the filesystem on the card, but nothing special to note (clean, FAT32 with 4K cluster-size).

Maybe the crash is a hint of something really going wild within tiny-usb: e.g. too many requests from the host not serviced fast enougth might lead to long queues with trashing and memory corruption.

Another thought:

Instead of making the compile-time decision to turn usb-presentation on/off, I would (again) suggest to give control back to the user: an additional mount option with sensible defaults (i.e. transparently backward compatible) could turn this on/off as needed. The rationale is simple: I have a few PCB-designs with XTSD chips. During normal operation, the program writes data to this "SD". Getting the data out is a bit more complicated, since you cannot take the chip off of the PCB and put it in a card reader. So mounting it with host-visibility would really be helpful in this case. As it would be to all who have the SD-card within a case.

[bablokb]

@mikeysklar
I ran your cpu-only benchmark with minor changes (actually mounting the SD-card and printing each iteration). It crashes again after about two minutes. The first iteration seems ok, but already the second one is really slow.

[10:51:49.343] mounting SD...
[10:51:53.178] SD-card mounted on /sd
[10:51:53.178] done
[10:51:53.420] iter001: t+  0.00s  cpu= 239ms
[10:53:43.654] iter002: t+  0.24s  cpu=109731ms
[10:54:03.469] Wi-Fi: aus | Fertig | 10.3.0-main-10967
Automatisches Neuladen ist deaktiviert.
[10:54:03.469] Sicherheitsmodus aktiv! Gespeicherter Code wird nicht ausgeführt

[10:54:03.470] Sie befinden sich im abgesicherten Modus, weil:
[10:54:03.470] Der CircuitPython-Kerncode ist hart abgestürzt. Hoppla!
[10:54:03.470] Harter Fehler: Speicherzugriff- oder Anweisungsfehler.
[10:54:03.470] Reiche bitte ein Problem mit deinem Programm bei github.com/adafruit/circuitpython/issues ein.
[10:54:03.470] Drücke Reset, um den Sicherheitsmodus zu beenden.

[10:54:03.470] Drücke eine beliebige Taste um REPL zu betreten. Drücke STRG-D zum neuladen.
[10:54:03.814] Wi-Fi: aus | REPL | 10.3.0-main-10967
[10:54:03.815] Adafruit CircuitPython 10.3.0-main-10967 on 2026-05-08; Raspberry Pi Pico W with rp2040

[bablokb]

Related logs from journalctl:

May 08 10:51:47 canopus kernel: usb 1-2: new full-speed USB device number 61 using xhci_hcd
May 08 10:51:48 canopus kernel: usb 1-2: New USB device found, idVendor=239a, idProduct=8120, bcdDevice= 1.00
May 08 10:51:48 canopus kernel: usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
May 08 10:51:48 canopus kernel: usb 1-2: Product: Pico W
May 08 10:51:48 canopus kernel: usb 1-2: Manufacturer: Raspberry Pi
May 08 10:51:48 canopus kernel: usb 1-2: SerialNumber: E6614104033E9E32
May 08 10:51:48 canopus kernel: cdc_acm 1-2:1.0: ttyACM0: USB ACM device
May 08 10:51:48 canopus kernel: usb-storage 1-2:1.2: USB Mass Storage device detected
May 08 10:51:48 canopus kernel: scsi host0: usb-storage 1-2:1.2
...
May 08 10:51:49 canopus kernel: scsi 0:0:0:0: Direct-Access     Raspberr Pico W           1.0  PQ: 0 ANSI: 2
May 08 10:51:49 canopus kernel: scsi 0:0:0:1: Direct-Access     Raspberr Pico W           1.0  PQ: 0 ANSI: 2
May 08 10:51:49 canopus kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May 08 10:51:49 canopus kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] 1017 512-byte logical blocks: (521 kB/509 KiB)
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Write Protect is off
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Mode Sense: 03 00 00 00
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] No Caching mode page found
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May 08 10:51:49 canopus kernel: sd 0:0:0:1: [sdb] Media removed, stopped polling
May 08 10:51:49 canopus kernel: sd 0:0:0:1: [sdb] Attached SCSI removable disk
May 08 10:51:49 canopus kernel:  sda: sda1
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
...
May 08 10:51:48 canopus kernel: cdc_acm 1-2:1.0: ttyACM0: USB ACM device
May 08 10:51:48 canopus kernel: usb-storage 1-2:1.2: USB Mass Storage device detected
May 08 10:51:48 canopus kernel: scsi host0: usb-storage 1-2:1.2
...
May 08 10:51:49 canopus kernel: scsi 0:0:0:0: Direct-Access     Raspberr Pico W           1.0  PQ: 0 ANSI: 2
May 08 10:51:49 canopus kernel: scsi 0:0:0:1: Direct-Access     Raspberr Pico W           1.0  PQ: 0 ANSI: 2
May 08 10:51:49 canopus kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May 08 10:51:49 canopus kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] 1017 512-byte logical blocks: (521 kB/509 KiB)
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Write Protect is off
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Mode Sense: 03 00 00 00
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] No Caching mode page found
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May 08 10:51:49 canopus kernel: sd 0:0:0:1: [sdb] Media removed, stopped polling
May 08 10:51:49 canopus kernel: sd 0:0:0:1: [sdb] Attached SCSI removable disk
May 08 10:51:49 canopus kernel:  sda: sda1
May 08 10:51:49 canopus kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
...

====> At this point the Pico mounts the SD, the host detects it
May 08 10:51:53 canopus kernel: sd 0:0:0:1: [sdb] 15661056 512-byte logical blocks: (8.02 GB/7.47 GiB)
May 08 10:51:53 canopus kernel: sd 0:0:0:1: [sdb] Write Protect is on
May 08 10:51:53 canopus kernel: sd 0:0:0:1: [sdb] Mode Sense: 03 00 80 00
May 08 10:51:53 canopus kernel: sdb: detected capacity change from 0 to 15661056
May 08 10:51:53 canopus kernel:  sdb: sdb1
...

====> The host notices a very slow response from the device
May 08 10:52:54 canopus systemd-udevd[13107]: sdb: Worker [21342] processing SEQNUM=572782 is taking a long time

====> At this point the Pico crashes
May 08 10:53:59 canopus kernel: usb 1-2: reset full-speed USB device number 61 using xhci_hcd
May 08 10:54:00 canopus kernel: usb 1-2: USB disconnect, device number 61

[bablokb]

Here is the test with the XTSD (1GBit, i.e. 128MB). Slowdown is for about 15s. So this is clearly hardware (and host) dependent.

[16:21:27.995] mounting SD...
[16:21:33.209] SD-card mounted on /sd
[16:21:33.209] done
[16:21:33.443] iter001: t+  0.00s  cpu= 229ms
[16:21:33.727] iter002: t+  0.23s  cpu= 278ms
[16:21:34.365] iter003: t+  0.52s  cpu= 632ms
[16:21:35.042] iter004: t+  1.16s  cpu= 667ms
[16:21:35.678] iter005: t+  1.83s  cpu= 629ms
[16:21:36.312] iter006: t+  2.47s  cpu= 626ms
[16:21:36.898] iter007: t+  3.10s  cpu= 578ms
[16:21:37.574] iter008: t+  3.69s  cpu= 668ms
[16:21:38.235] iter009: t+  4.37s  cpu= 652ms
[16:21:38.867] iter010: t+  5.03s  cpu= 622ms
[16:21:39.471] iter011: t+  5.66s  cpu= 596ms
[16:21:40.105] iter012: t+  6.26s  cpu= 629ms
[16:21:40.780] iter013: t+  6.89s  cpu= 665ms
[16:21:41.420] iter014: t+  7.57s  cpu= 630ms
[16:21:42.051] iter015: t+  8.21s  cpu= 625ms
[16:21:42.640] iter016: t+  8.84s  cpu= 581ms
[16:21:43.306] iter017: t+  9.43s  cpu= 657ms
[16:21:43.973] iter018: t+ 10.10s  cpu= 660ms
[16:21:44.605] iter019: t+ 10.76s  cpu= 625ms
[16:21:45.237] iter020: t+ 11.40s  cpu= 626ms
[16:21:45.879] iter021: t+ 12.03s  cpu= 636ms
[16:21:46.540] iter022: t+ 12.67s  cpu= 651ms
[16:21:47.171] iter023: t+ 13.33s  cpu= 624ms
[16:21:47.802] iter024: t+ 13.96s  cpu= 623ms
[16:21:48.274] iter025: t+ 14.59s  cpu= 467ms
[16:21:48.507] iter026: t+ 15.06s  cpu= 229ms
[16:21:48.740] iter027: t+ 15.30s  cpu= 229ms
[16:21:48.973] iter028: t+ 15.53s  cpu= 229ms
[16:21:49.205] iter029: t+ 15.76s  cpu= 229ms
[16:21:49.438] iter030: t+ 16.00s  cpu= 229ms
[16:21:49.671] iter031: t+ 16.23s  cpu= 229ms
[16:21:49.903] iter032: t+ 16.46s  cpu= 229ms
[16:21:50.135] iter033: t+ 16.69s  cpu= 229ms
[16:21:50.369] iter034: t+ 16.92s  cpu= 230ms

[mikeysklar]

Yes, that is the slowdown I would expect to see with FAT32. Would you mind running the same test to confirm the 1.5s or less delay with FAT16?

[bablokb]

Hm, how would I do that? The filesystem is not visible to the host until I mount it and then it is readonly.

[mikeysklar]

I've used the SD Formatter script which is an Arduino example sketch from the Adafruit SdFat Fork.

Now, I think this can all be done in CircuitPython with a XTSD using mkfs. I can wire it up and give it a try on a feather RP2040. I have an XTSD somewhere around here. Should be about 1.5s slow window on Linux and 0 on macOS / Windows with XTSD+FAT16.

format storage card with mkfs as FAT16:

import storage, sdcardio, busio, board

spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
sd = sdcardio.SDCard(spi, board.D5)
storage.VfsFat.mkfs(sd)

[bablokb]

I did think about doing this but the documentation is not clear about which FAT version is used by the implementation. Looking at the code, it first tries FAT16 and if this fails it does a second try with FAT32.

[mikeysklar]

I can verify that the format code worked for me to get a FAT16 XTSD.

Feather RP2040 + 512MB XTSD

code.py:

import storage, sdcardio, busio, board

spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
sd = sdcardio.SDCard(spi, board.D5)
storage.VfsFat.mkfs(sd)

macos fs verify:

sklarm@ciao ada % diskutil info /Volumes/NO\ NAME | grep -E "File System|Type \(Bundle\)|Name \(User"
   File System Personality:   MS-DOS FAT16
   Type (Bundle):             msdos
   Name (User Visible):       MS-DOS (FAT16)

[dhalbert]

Whether FAT12, FAT16, or FAT32 is used depends only on the number of clusters (cluster size may vary):

From https://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc, page 15:

This is the one and only way that FAT type is determined. There is no such thing as a FAT12 volume that has more than 4084 clusters. There is no such thing as a FAT16 volume that has less than 4085 clusters or more than 65,524 clusters. There is no such thing as a FAT32 volume that has less than 65,525 clusters. If you try to make a FAT volume that violates this rule, Microsoft operating systems will not handle them correctly because they will think the volume has a different type of FAT than what you think it does.

NOTE: As is noted numerous times earlier, the world is full of FAT code that is wrong. There is a lot of FAT type code that is off by 1 or 2 or 8 or 10 or 16. For this reason, it is highly recommended that if you are formatting a FAT volume which has maximum compatibility with all existing FAT code, then you should you avoid making volumes of any type that have close to 4,085 or 65,525 clusters. Stay at least 16 clusters on each side away from these cut-over cluster counts.