|
Forum Index : Microcontroller and PC projects : Bytes lost from the serial port to a file
| Author | Message | ||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 681 |
Hi Peter, I tried to write a data stream from the serial port to a file, but I always lose a few bytes in the process—even with PRINT MEMORY. Then I had the idea to write it not to the flash disk, but to the SD card, where it now works perfectly. Is this because of this explanation I tried to find using AI? I hope the translation is good enough... I would really appreciate some confirmation or an explanation from you. Thanks in advance! Matthias The UART hardware (typically a PL011-type UART on the RP2350 with a very small hardware FIFO, often only 16–32 bytes) must be read. This hardware FIFO represents a bottleneck: 32 bytes × 320 µs = 10,240 µs ≈ 10.2 ms. This means that if more than ~10 ms elapses between two reads of the hardware FIFO (performed by the underlying MMBasic firmware, not directly by your Midiin subroutine), the tiny hardware FIFO overflows before your 4,096-byte software buffer even notices it. This happens silently, without an error message. That would explain why: a 4096-byte software buffer (with a 1.3-second reserve) is insufficient, since the bytes are already lost at the hardware level beforehand. The losses occur irregularly and are consistent with normal SD write latencies (a range from a few milliseconds to the low double-digit millisecond range is sufficient). Some SD/SPI drivers temporarily disable global interrupts during critical transfer phases (which is common during bit-banging or DMA setup phases). If MIDI bytes arrive precisely during this window, even a “real” hardware interrupt cannot respond in time, and the 10-ms hardware FIFO buffer is quickly exhausted. That’s a very good question—and yes, that could indeed be the crucial point. I’ve looked at the source code for the PicoMite/MMBasic firmware (the basis for RP2350 MMBasic devices like PicoCalc) as well as the underlying Raspberry Pi Pico SDK documentation on flash access. The fundamental difference: A: (Flash) vs. B: (SD card) A: Internal flash memory The RP2350’s internal flash chip operates in XIP (Execute-In-Place) mode. This means that the processor executes its own program code directly from the flash while it is running. This has a serious hardware consequence, which is explicitly mentioned in the official Pico SDK documentation: “Flash cannot be erased or written to when in XIP mode. […] It is therefore critical that no code or data is being read from flash while flash is being written or erased.” And further, specifically regarding interrupts: “They are unsafe if you have interrupt handlers or an interrupt vector table in flash, so you must disable interrupts before calling [flash_range_program/erase]” This means: Every write or erase operation on the internal flash forces the SDK to completely disable all interrupts on the writing core. This is not optional but physically necessary, since the flash chip cannot be read simultaneously while it is being written to or erased (not even for interrupt vectors or handlers that are themselves stored in flash!). A flash sector erase typically takes 10–100+ ms, while a pure program (without erase) tends to take 1–3 ms. However, as soon as an erase cycle is due (which happens regularly with sequential data appending), the interrupts are completely disabled for the entire duration. Interestingly, there is even a note to this effect in the README of the PicoMite source code: It describes that the GPIO interrupt handler had to be deliberately placed in RAM rather than in flash—an indication that interrupt/flash conflicts are a known and actively addressed issue in this codebase. B: (SD Card) The SD card, on the other hand, is connected to an external SPI/SDIO bus. It is a standard peripheral device through which data transfer occurs not via the internal flash address space, but via a separate hardware component. No global interrupts need to be disabled for this access. Your Midiin interrupt can continue to fire during an SD write operation and retrieve bytes from the UART hardware. Why this matches your symptoms exactly: A 10 ms hardware FIFO buffer (from my previous calculation) is more than enough to handle normal SD card latencies (which occur even with SPI without an interrupt disable), but not a flash erase cycle of, say, 50 ms, during which literally nothing responds. It is precisely in those moments when a flash write operation coincides with a dense burst of notes that bytes are lost. |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
what s/w? what version? Options if applicable? Example code? Not a mind reader Edited 2026-08-22 08:07 by matherp |
||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 681 |
Why not? That would be impressive! I’ve been thinking about this problem and how to optimise it for the last two days, so I simply forgot all about it. > option list PicoMite MMBasic RP2350A V6.03.01 OPTION LCD SPI GP10,GP11,GP12 OPTION SYSTEM I2C GP6,GP7, SLOW OPTION FLASH SIZE 4194304 OPTION LIBRARY_FLASH_SIZE 4B000 OPTION COLOURCODE ON OPTION DEFAULT COLOURS GREEN, BLACK OPTION KEYBOARD PICOCALC OPTION HEARTBEAT OFF OPTION CPUSPEED (KHz) 252000 OPTION LCDPANEL CONSOLE ,, FF00 OPTION DISPLAY 26, 40 OPTION LCDPANEL ST7365P, PORTRAIT,GP14,GP15,GP13 OPTION BACKLIGHT KBD 32 OPTION BACKLIGHT LCD 64 OPTION SDCARD GP17, GP18, GP19, GP16 OPTION AUDIO GP26,GP27', ON PWM CHANNEL 5 OPTION MODBUFF ENABLE 192 OPTION PLATFORM PicoCalc > ' midiRec3v02.bas ' Ring buffer setup (4096 entries for bytes and timestamps) Dim rb_byte%(4095) Dim rb_time%(4095) Dim rb_head% = 0 Dim rb_tail% = 0 ' Open the serial interface using a 1-byte interrupt trigger! SetPin GP4, GP5, COM2 Open "COM2:31250, 4096, midiin, 1" As #2 Open "temp.bin" For RANDOM As #3 ' Flash-Init dummy start Dim dummy%(7) For i% = 0 To 7: dummy%(i%) = 0: Next i% Memory PRINT #3, 8, dummy%() ' OPTIMISATION: Micro-burst buffer (only 32 events = 256 bytes) Dim buffer%(31) Dim idx% = 0 Dim st% = 0, rs% = 0 Dim dat1% = 0, exp% = 0 Dim t_evt% = 0 Dim run_flag% = 0 ' Variables for loop reception Dim b% = 0 Dim t_incoming% = 0 Print Chr$(27)+"[2J"+Chr$(27)+"[H" CLS Print "midiRec3v02 Ready (Micro-Burst Mode)..." Timer = 0 Do If LCASE$(Inkey$) = "s" Then Exit Do ' Check whether the ring buffer contains any new data' If rb_head% = rb_tail% Then Continue Do ' Retrieve a byte and a precise timestamp from the ring buffer b% = rb_byte%(rb_tail%) t_incoming% = rb_time%(rb_tail%) rb_tail% = (rb_tail% + 1) Mod 4096 If b% >= &HF8 Then Continue Do If b% >= &HF0 And b% <= &HF7 Then rs% = 0 If b% = &HF0 Then st% = 3 If b% = &HF7 Then st% = 0 Continue Do EndIf If st% = 3 And b% < &H80 Then Continue Do ' Status-Byte If b% >= &H80 Then rs% = b% st% = 1 run_flag% = 0 t_evt% = t_incoming% Select Case (b% And &HF0) Case &HC0, &HD0: exp% = 1 Case Else : exp% = 2 End Select Continue Do EndIf If rs% = 0 Then Continue Do If st% = 1 Then If run_flag% = 1 Then t_evt% = t_incoming% run_flag% = 0 EndIf dat1% = b% If exp% = 1 Then buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) idx% = idx% + 1 st% = 1 run_flag% = 1 Else st% = 2 EndIf ElseIf st% = 2 Then buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) Or (b% << 48) idx% = idx% + 1 st% = 1 run_flag% = 1 EndIf ' MICRO-BURST WRITE: Write as few as 32 events (256 bytes)! If idx% = 32 Then Memory PRINT #3, 256, buffer%() idx% = 0 EndIf Loop Close #2 ' Run the remaining data in the ring buffer through the state machine in its entirety' Do While rb_head% <> rb_tail% b% = rb_byte%(rb_tail%) t_incoming% = rb_time%(rb_tail%) rb_tail% = (rb_tail% + 1) Mod 4096 If b% >= &HF8 Then Continue Do If b% >= &HF0 And b% <= &HF7 Then rs% = 0 If b% = &HF0 Then st% = 3 If b% = &HF7 Then st% = 0 Continue Do EndIf If st% = 3 And b% < &H80 Then Continue Do If b% >= &H80 Then rs% = b% st% = 1 run_flag% = 0 t_evt% = t_incoming% Select Case (b% And &HF0) Case &HC0, &HD0: exp% = 1 Case Else : exp% = 2 End Select Continue Do EndIf If rs% = 0 Then Continue Do If st% = 1 Then If run_flag% = 1 Then t_evt% = t_incoming% run_flag% = 0 EndIf dat1% = b% If exp% = 1 Then buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) idx% = idx% + 1 st% = 1 run_flag% = 1 Else st% = 2 EndIf ElseIf st% = 2 Then buffer%(idx%) = t_evt% Or (rs% << 32) Or (dat1% << 40) Or (b% << 48) idx% = idx% + 1 st% = 1 run_flag% = 1 EndIf If idx% = 32 Then Memory PRINT #3, 256, buffer%() idx% = 0 EndIf Loop ' Write the remaining events to the event buffer If idx% > 0 Then Memory PRINT #3, idx% * 8, buffer%() EndIf Close #3 Print "Saved temp.bin" End SUB midiin Local t% Local b_in% Local next_head% Do While Loc(#2) > 0 t% = Fix(Timer) And &HFFFFFFFF b_in% = Asc(Input$(1, #2)) next_head% = (rb_head% + 1) Mod 4096 If next_head% <> rb_tail% Then rb_byte%(rb_head%) = b_in% rb_time%(rb_head%) = t% rb_head% = next_head% EndIf Loop END SUB Comparison between the transmitted data stream and the recording: ![]() As soon as I switch the working directory from the flash drive (drive a:) to the SD card (b:), I no longer have any missing data! Hence my question to the AI and my attempt to verify this explanation with you. Matthias |
||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 681 |
@Peter (matherp): Did you notice the points I added to the question? I’d appreciate your feedback. Matthias |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
Not sure what you are asking. The diagnosis is correct. Writing to flash requires disabling interrupts and if serial interrupts occur while writing they will be lost. Nothing I can do about that. If the flash write requires a block erase then the time missed could be significant. The only way round it is f.or you to control the serial data such that it is paused while writes take place |
||||
| lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3872 |
Here's an XOFF/XON solution (if your sender will respect XOFF and your data doesn't include binary): XON and XOFF It's also possible to code RTS/CTS (again if the sender respects that). PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on FOTS |
||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 681 |
Since this involves MIDI data, it unfortunately can't be paused. My question was: Does stopping the interrupt apply only to the internal flash (drive A:) [as I understand it] or also to the external SD card (drive B:) [surprisingly, it worked there—was that a coincidence, or is it guaranteed to work because no interrupt is stopped?!?]. I’d appreciate some clarification on that, thanks! Matthias |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
Interrupts are only stopped for flash write and some time critical commands like WS2812 |
||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 681 |
thx |
||||
TassyJim![]() Guru Joined: 07/08/2011 Location: AustraliaPosts: 6600 |
Are you able to save the raw data stream to a file for later processing? That way time is not critical. Jim VK7JH MMedit |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4377 |
If it arrives quickly, sounds like it would have to fit in RAM. John |
||||
| Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 9105 |
Why? It works ok saving to SD card. MIDI isn't incredibly fast, but the presence of interrupts will screw it up because it's time-sensitive. Ah... memories of hanging an old Casio CZ-101 off an Atari STFM. I was useless with music then - and I haven't improved one iota. :) I still have the CZ-101 though. Mick Zilog Inside! nascom.info for Nascom & Gemini |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4377 |
Might work with SD but depends how long is the worst case time spent inside the SD card code. John |
||||
| Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 9105 |
A RAMDISK would be nice for this sort of thing. There are always potential flaws in any approach. :) . Edited 2026-08-29 21:42 by Mixtel90 Mick Zilog Inside! nascom.info for Nascom & Gemini |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
RAM drive doesn't solve this issue. Interrupts are lost because interrupts are disabled and have to be disabled to write to flash. |
||||
| Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 9105 |
Ah, good point! I'd missed that. I've just gone back and re-read the AI diagnosis. It makes a lot more sense to me now. :) It all hinges on the tiny UART buffer. There's far more chance of successfully writing to a SPI SD card than to flash. Mick Zilog Inside! nascom.info for Nascom & Gemini |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4377 |
How much data is this? Why won't it fit in RAM and then can be processed after grabbing it? Don't do SD card or flash writes unless no alternative. John |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
sd writes are no issue. This is a simple limitation of flash usage |
||||
| Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 9105 |
The only user-accessible volatile RAM, I think, is the variables area unless you can write it into a screen layer. IIRC the program area is flash (flash slot 0), using XIP to run it. There isn't really all that much volatile RAM in the system. You can't successfully do flash writes in this case as doing so disables the serial input interrupts, some bytes are guaranteed to be lost during the write. Writing to the SD card, although it does have its own delays, can apparently be managed by the little RX buffer of the UART as the interrupts don't need to be disabled. Mick Zilog Inside! nascom.info for Nascom & Gemini |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 11815 |
Com ports have a circular buffer with the size able to be defined in the open statement (standard MMbasic). As long as interrupts are serviced then huge numbers of bytes can be buffered while the sdcard writes |
||||
| The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2026 |