Home
JAQForum Ver 24.01
Log In or Join  
Active Topics
Local Time 16:22 05 Sep 2026 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Bytes lost from the serial port to a file

Author Message
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 681
Posted: 09:51pm 21 Aug 2026
Copy link to clipboard 
Print this post

Hi Peter,
I tried to write a data stream from the serial port to a file, but I always lose a few bytes in the process—even with PRINT MEMORY. Then I had the idea to write it not to the flash disk, but to the SD card, where it now works perfectly.

Is this because of this explanation I tried to find using AI?
I hope the translation is good enough...
I would really appreciate some confirmation or an explanation from you. Thanks in advance!

Matthias



The UART hardware (typically a PL011-type UART on the RP2350 with a very small hardware FIFO, often only 16–32 bytes) must be read. This hardware FIFO represents a bottleneck: 32 bytes × 320 µs = 10,240 µs ≈ 10.2 ms.

This means that if more than ~10 ms elapses between two reads of the hardware FIFO (performed by the underlying MMBasic firmware, not directly by your Midiin subroutine), the tiny hardware FIFO overflows before your 4,096-byte software buffer even notices it. This happens silently, without an error message.

That would explain why:

a 4096-byte software buffer (with a 1.3-second reserve) is insufficient, since the bytes are already lost at the hardware level beforehand.
The losses occur irregularly and are consistent with normal SD write latencies (a range from a few milliseconds to the low double-digit millisecond range is sufficient).
Some SD/SPI drivers temporarily disable global interrupts during critical transfer phases (which is common during bit-banging or DMA setup phases). If MIDI bytes arrive precisely during this window, even a “real” hardware interrupt cannot respond in time, and the 10-ms hardware FIFO buffer is quickly exhausted.

That’s a very good question—and yes, that could indeed be the crucial point. I’ve looked at the source code for the PicoMite/MMBasic firmware (the basis for RP2350 MMBasic devices like PicoCalc) as well as the underlying Raspberry Pi Pico SDK documentation on flash access.


The fundamental difference: A: (Flash) vs. B: (SD card)

A: Internal flash memory

The RP2350’s internal flash chip operates in XIP (Execute-In-Place) mode. This means that the processor executes its own program code directly from the flash while it is running. This has a serious hardware consequence, which is explicitly mentioned in the official Pico SDK documentation:

“Flash cannot be erased or written to when in XIP mode. […] It is therefore critical that no code or data is being read from flash while flash is being written or erased.”

And further, specifically regarding interrupts:

“They are unsafe if you have interrupt handlers or an interrupt vector table in flash, so you must disable interrupts before calling [flash_range_program/erase]”

This means: Every write or erase operation on the internal flash forces the SDK to completely disable all interrupts on the writing core. This is not optional but physically necessary, since the flash chip cannot be read simultaneously while it is being written to or erased (not even for interrupt vectors or handlers that are themselves stored in flash!). A flash sector erase typically takes 10–100+ ms, while a pure program (without erase) tends to take 1–3 ms. However, as soon as an erase cycle is due (which happens regularly with sequential data appending), the interrupts are completely disabled for the entire duration.

Interestingly, there is even a note to this effect in the README of the PicoMite source code: It describes that the GPIO interrupt handler had to be deliberately placed in RAM rather than in flash—an indication that interrupt/flash conflicts are a known and actively addressed issue in this codebase.

B: (SD Card)

The SD card, on the other hand, is connected to an external SPI/SDIO bus. It is a standard peripheral device through which data transfer occurs not via the internal flash address space, but via a separate hardware component. No global interrupts need to be disabled for this access. Your Midiin interrupt can continue to fire during an SD write operation and retrieve bytes from the UART hardware.

Why this matches your symptoms exactly: A 10 ms hardware FIFO buffer (from my previous calculation) is more than enough to handle normal SD card latencies (which occur even with SPI without an interrupt disable), but not a flash erase cycle of, say, 50 ms, during which literally nothing responds.
It is precisely in those moments when a flash write operation coincides with a dense burst of notes that bytes are lost.
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 10:06pm 21 Aug 2026
Copy link to clipboard 
Print this post

what s/w? what version? Options if applicable? Example code?
Not a mind reader
Edited 2026-08-22 08:07 by matherp
 
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 681
Posted: 06:57pm 22 Aug 2026
Copy link to clipboard 
Print this post

  matherp said  ... Not a mind reader


Why not? That would be impressive!
I’ve been thinking about this problem and how to optimise it for the last two days, so I simply forgot all about it.

> option list
PicoMite MMBasic RP2350A V6.03.01
OPTION LCD SPI GP10,GP11,GP12
OPTION SYSTEM I2C GP6,GP7, SLOW
OPTION FLASH SIZE 4194304
OPTION LIBRARY_FLASH_SIZE  4B000
OPTION COLOURCODE ON
OPTION DEFAULT COLOURS GREEN, BLACK
OPTION KEYBOARD PICOCALC
OPTION HEARTBEAT OFF
OPTION CPUSPEED (KHz) 252000
OPTION LCDPANEL CONSOLE ,, FF00
OPTION DISPLAY 26, 40
OPTION LCDPANEL ST7365P, PORTRAIT,GP14,GP15,GP13
OPTION BACKLIGHT KBD 32
OPTION BACKLIGHT LCD 64
OPTION SDCARD GP17, GP18, GP19, GP16
OPTION AUDIO GP26,GP27', ON PWM CHANNEL 5
OPTION MODBUFF ENABLE  192
OPTION PLATFORM PicoCalc
>


' midiRec3v02.bas

' Ring buffer setup (4096 entries for bytes and timestamps)
Dim rb_byte%(4095)
Dim rb_time%(4095)
Dim rb_head% = 0
Dim rb_tail% = 0

' Open the serial interface using a 1-byte interrupt trigger!
SetPin GP4, GP5, COM2
Open "COM2:31250, 4096, midiin, 1" As #2
Open "temp.bin" For RANDOM As #3  

' Flash-Init dummy start
Dim dummy%(7)
For i% = 0 To 7: dummy%(i%) = 0: Next i%
Memory PRINT #3, 8, dummy%()

' OPTIMISATION: Micro-burst buffer (only 32 events = 256 bytes)
Dim buffer%(31)
Dim idx% = 0

Dim st% = 0, rs% = 0
Dim dat1% = 0, exp% = 0
Dim t_evt% = 0
Dim run_flag% = 0

' Variables for loop reception
Dim b% = 0
Dim t_incoming% = 0

Print Chr$(27)+"[2J"+Chr$(27)+"[H"
CLS
Print "midiRec3v02 Ready (Micro-Burst Mode)..."
Timer = 0

Do
 If LCASE$(Inkey$) = "s" Then Exit Do

' Check whether the ring buffer contains any new data'
 If rb_head% = rb_tail% Then Continue Do

' Retrieve a byte and a precise timestamp from the ring buffer
 b% = rb_byte%(rb_tail%)
 t_incoming% = rb_time%(rb_tail%)
 rb_tail% = (rb_tail% + 1) Mod 4096

 If b% >= &HF8 Then Continue Do

 If b% >= &HF0 And b% <= &HF7 Then
   rs% = 0
   If b% = &HF0 Then st% = 3
   If b% = &HF7 Then st% = 0
   Continue Do
 EndIf
 If st% = 3 And b% < &H80 Then Continue Do

 ' Status-Byte
 If b% >= &H80 Then
   rs% = b%
   st% = 1
   run_flag% = 0
   t_evt% = t_incoming%
   
   Select Case (b% And &HF0)
     Case &HC0, &HD0: exp% = 1
     Case Else :      exp% = 2
   End Select
   Continue Do
 EndIf

 If rs% = 0 Then Continue Do

 If st% = 1 Then
   If run_flag% = 1 Then
     t_evt% = t_incoming%
     run_flag% = 0
   EndIf
   
   dat1% = b%
   
   If exp% = 1 Then
     buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40)
     idx% = idx% + 1
     st% = 1
     run_flag% = 1
   Else
     st% = 2
   EndIf

 ElseIf st% = 2 Then
   buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) Or (b% << 48)
   idx% = idx% + 1
   st% = 1
   run_flag% = 1
 EndIf

' MICRO-BURST WRITE: Write as few as 32 events (256 bytes)!
 If idx% = 32 Then
   Memory PRINT #3, 256, buffer%()
   idx% = 0
 EndIf
Loop

Close #2

' Run the remaining data in the ring buffer through the state machine in its entirety'
Do While rb_head% <> rb_tail%
 b% = rb_byte%(rb_tail%)
 t_incoming% = rb_time%(rb_tail%)
 rb_tail% = (rb_tail% + 1) Mod 4096
 
 If b% >= &HF8 Then Continue Do
 If b% >= &HF0 And b% <= &HF7 Then
   rs% = 0
   If b% = &HF0 Then st% = 3
   If b% = &HF7 Then st% = 0
   Continue Do
 EndIf
 If st% = 3 And b% < &H80 Then Continue Do

 If b% >= &H80 Then
   rs% = b%
   st% = 1
   run_flag% = 0
   t_evt% = t_incoming%
   Select Case (b% And &HF0)
     Case &HC0, &HD0: exp% = 1
     Case Else :      exp% = 2
   End Select
   Continue Do
 EndIf

 If rs% = 0 Then Continue Do

 If st% = 1 Then
   If run_flag% = 1 Then
     t_evt% = t_incoming%
     run_flag% = 0
   EndIf
   dat1% = b%
   If exp% = 1 Then
     buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40)
     idx% = idx% + 1
     st% = 1
     run_flag% = 1
   Else
     st% = 2
   EndIf
 ElseIf st% = 2 Then
   buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) Or (b% << 48)
   idx% = idx% + 1
   st% = 1
   run_flag% = 1
 EndIf
 
 If idx% = 32 Then
   Memory PRINT #3, 256, buffer%()
   idx% = 0
 EndIf
Loop

' Write the remaining events to the event buffer
If idx% > 0 Then
 Memory PRINT #3, idx% * 8, buffer%()
EndIf

Close #3
Print "Saved temp.bin"
End

SUB midiin
 Local t%
 Local b_in%
 Local next_head%
 
 Do While Loc(#2) > 0
   t% = Fix(Timer) And &HFFFFFFFF
   b_in% = Asc(Input$(1, #2))
   next_head% = (rb_head% + 1) Mod 4096
   
   If next_head% <> rb_tail% Then
     rb_byte%(rb_head%) = b_in%
     rb_time%(rb_head%) = t%
     rb_head% = next_head%
   EndIf
 Loop
END SUB


Comparison between the transmitted data stream and the recording:




As soon as I switch the working directory from the flash drive (drive a:) to the SD card (b:), I no longer have any missing data!

Hence my question to the AI and my attempt to verify this explanation with you.

Matthias
 
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 681
Posted: 07:48am 28 Aug 2026
Copy link to clipboard 
Print this post

@Peter (matherp): Did you notice the points I added to the question? I’d appreciate your feedback.
Matthias
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 05:01pm 28 Aug 2026
Copy link to clipboard 
Print this post

Not sure what you are asking. The diagnosis is correct. Writing to flash requires disabling interrupts and if serial interrupts occur while writing they will be lost. Nothing I can do about that. If the flash write requires a block erase then the time missed could be significant. The only way round it is f.or you to control the serial data such that it is paused while writes take place
 
lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3872
Posted: 05:28pm 28 Aug 2026
Copy link to clipboard 
Print this post

  matherp said  The only way round it is for you to control the serial data such that it is paused while writes take place


Here's an XOFF/XON solution (if your sender will respect XOFF and your data doesn't include binary): XON and XOFF

It's also possible to code RTS/CTS (again if the sender respects that).
PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on FOTS
 
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 681
Posted: 08:04pm 28 Aug 2026
Copy link to clipboard 
Print this post

  matherp said  Not sure what you are asking. The diagnosis is correct. Writing to flash requires disabling interrupts and if serial interrupts occur while writing they will be lost. Nothing I can do about that. If the flash write requires a block erase then the time missed could be significant. The only way round it is f.or you to control the serial data such that it is paused while writes take place


Since this involves MIDI data, it unfortunately can't be paused.
My question was: Does stopping the interrupt apply only to the internal flash (drive A:) [as I understand it] or also to the external SD card (drive B:) [surprisingly, it worked there—was that a coincidence, or is it guaranteed to work because no interrupt is stopped?!?].

I’d appreciate some clarification on that, thanks!

Matthias
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 08:32pm 28 Aug 2026
Copy link to clipboard 
Print this post

Interrupts are only stopped for flash write and some time critical commands like WS2812
 
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 681
Posted: 08:43pm 28 Aug 2026
Copy link to clipboard 
Print this post

thx
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 6600
Posted: 07:52am 29 Aug 2026
Copy link to clipboard 
Print this post

Are you able to save the raw data stream to a file for later processing?
That way time is not critical.

Jim
VK7JH
MMedit
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 4377
Posted: 09:01am 29 Aug 2026
Copy link to clipboard 
Print this post

If it arrives quickly, sounds like it would have to fit in RAM.

John
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 9105
Posted: 10:05am 29 Aug 2026
Copy link to clipboard 
Print this post

Why? It works ok saving to SD card. MIDI isn't incredibly fast, but the presence of interrupts will screw it up because it's time-sensitive.

Ah... memories of hanging an old Casio CZ-101 off an Atari STFM. I was useless with music then - and I haven't improved one iota. :) I still have the CZ-101 though.
Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 4377
Posted: 11:36am 29 Aug 2026
Copy link to clipboard 
Print this post

Might work with SD but depends how long is the worst case time spent inside the SD card code.

John
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 9105
Posted: 11:41am 29 Aug 2026
Copy link to clipboard 
Print this post

A RAMDISK would be nice for this sort of thing. There are always potential flaws in any approach. :)
.
Edited 2026-08-29 21:42 by Mixtel90
Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 11:42am 29 Aug 2026
Copy link to clipboard 
Print this post

RAM drive doesn't solve this issue. Interrupts are lost because interrupts are disabled and have to be disabled to write to flash.
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 9105
Posted: 12:13pm 29 Aug 2026
Copy link to clipboard 
Print this post

Ah, good point! I'd missed that.
I've just gone back and re-read the AI diagnosis. It makes a lot more sense to me now. :) It all hinges on the tiny UART buffer. There's far more chance of successfully writing to a SPI SD card than to flash.
Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 4377
Posted: 12:46pm 29 Aug 2026
Copy link to clipboard 
Print this post

How much data is this?  Why won't it fit in RAM and then can be processed after grabbing it?

Don't do SD card or flash writes unless no alternative.

John
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 01:01pm 29 Aug 2026
Copy link to clipboard 
Print this post

sd writes are no issue. This is a simple limitation of flash usage
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 9105
Posted: 01:10pm 29 Aug 2026
Copy link to clipboard 
Print this post

The only user-accessible volatile RAM, I think, is the variables area unless you can write it into a screen layer. IIRC the program area is flash (flash slot 0), using XIP to run it. There isn't really all that much volatile RAM in the system.

You can't successfully do flash writes in this case as doing so disables the serial input interrupts, some bytes are guaranteed to be lost during the write.

Writing to the SD card, although it does have its own delays, can apparently be managed by the little RX buffer of the UART as the interrupts don't need to be disabled.
Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11815
Posted: 01:30pm 29 Aug 2026
Copy link to clipboard 
Print this post

  Quote  can apparently be managed by the little RX buffer of the UART


Com ports have a circular buffer with the size able to be defined in the open statement (standard MMbasic). As long as interrupts are serviced then huge numbers of bytes can be buffered while the sdcard writes
 
Print this page


To reply to this topic, you need to log in.

The Back Shed's forum code is written, and hosted, in Australia.
© JAQ Software 2026