16.0.0-RC4 not starting

[A380Coding]

Update from 15.7.0 to 16.0.0-RC4 using
AI-on-the-edge-device__update__16.0.0-RC4.zip

The device does not start up again.
I checked the console and get this:

rst:0x1 (POWERON_RESET),boot:0x3b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7664
load:0x40078000,len:15360
load:0x40080400,len:3744
entry 0x4008067c
I (27) boot: ESP-IDF 4.3.2 2nd stage bootloader
I (27) boot: compile time 20:02:47
I (27) boot: chip revision: 1
I (29) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (37) boot.esp32: SPI Speed : 40MHz
I (41) boot.esp32: SPI Mode : DIO
I (46) boot.esp32: SPI Flash Size : 4MB
I (50) boot: Enabling RNG early entropy source...
I (56) boot: Partition Table:
I (59) boot: ## Label Usage Type ST Offset Length
I (67) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (74) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (81) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (89) boot: 3 ota_0 OTA app 00 10 00010000 001db000
I (96) boot: 4 ota_1 OTA app 00 11 001f0000 001db000
I (104) boot: End of partition table
I (108) boot_comm: chip revision: 1, min. application chip revision: 0
I (115) esp_image: segment 0: paddr=001f0020 vaddr=3f400020 size=32714h (206612) map
I (203) esp_image: segment 1: paddr=0022273c vaddr=3ffb0000 size=049ech ( 18924) load
I (211) esp_image: segment 2: paddr=00227130 vaddr=40080000 size=08ee8h ( 36584) load
I (227) esp_image: segment 3: paddr=00230020 vaddr=400d0020 size=11528ch (1135244) map
I (662) esp_image: segment 4: paddr=003452b4 vaddr=40088ee8 size=11d78h ( 73080) load
I (707) boot: Loaded app from partition at offset 0x1f0000
I (707) boot: Disabling RNG early entropy source...
I (719) quad_psram: This chip is ESP32-D0WD
I (719) esp_psram: Found 8MB PSRAM device
I (719) esp_psram: Speed: 40MHz
I (720) esp_psram: PSRAM initialized, cache is in low/high (2-core) mode.
W (727) esp_psram: Virtual address not enough for PSRAM, map as much as we can. 4MB is mapped
I (735) cpu_start: Multicore app
I (1616) esp_psram: SPI SRAM memory test OK
I (1625) cpu_start: Pro cpu start user code
I (1625) cpu_start: cpu freq: 160000000 Hz
I (1625) app_init: Application information:
I (1626) app_init: Project name: AI-on-the-edge
I (1630) app_init: App version: 16.0.0-RC4
I (1634) app_init: Compile time: Oct 6 2024 22:26:08
I (1640) app_init: ELF file SHA256: d8e5a6d60...
I (1644) app_init: ESP-IDF: 5.3.1
I (1648) efuse_init: Min chip rev: v0.0
I (1652) efuse_init: Max chip rev: v3.99
I (1656) efuse_init: Chip rev: v1.0
I (1660) heap_init: Initializing. RAM available for dynamic allocation:
I (1666) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (1671) heap_init: At 3FFBB4C8 len 00024B38 (146 KiB): DRAM
I (1677) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1682) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1688) heap_init: At 4009AC60 len 000053A0 (20 KiB): IRAM
I (1694) esp_psram: Adding pool of 4081K of PSRAM memory to heap allocator
I (1701) spi_flash: detected chip: generic
I (1703) spi_flash: flash io: dio
W (1708) i2c: This driver is an old driver, please migrate your application code to adapt driver/i2c_master.h
I (1728) main_task: Started on CPU0
I (1738) esp_psram: Reserving pool of 40K of internal memory for DMA/internal allocations
I (1738) main_task: Calling app_main()
I (1738) MAIN:

================ Start app_main =================
I (1848) MAIN: =================================================
I (1868) MAIN: ==================== Start ======================
Guru Meditation Error: Core 0 panic'ed (InstrFetchProhibited). Exception was unhandled.

Core 0 register dump:
PC : 0xd4c00b0b PS : 0x00060530 A0 : 0x8008ac74 A1 : 0x3ffbcfe0
A2 : 0x3ffafe6c A3 : 0x00000006 A4 : 0x00000000 A5 : 0x00000002
A6 : 0x3ffafe10 A7 : 0x3f811f14 A8 : 0x80137b78 A9 : 0x00002000
A10 : 0x00000000 A11 : 0x00000000 A12 : 0x00000002 A13 : 0x00000000
A14 : 0x3ffc90a0 A15 : 0x3ffc9054 SAR : 0x0000001d EXCCAUSE: 0x00000014
EXCVADDR: 0xd4c00b08 LBEG : 0x4008bcc5 LEND : 0x4008bcd5 LCOUNT : 0xffffffe7

Backtrace: 0x54c00b08:0x3ffbcfe0 0x4008ac71:0x3ffbd000 0x400895e5:0x3ffbd020 0x40089427:0x3ffbd040 0x400894b7:0x3ffbd060 0x40117123:0x3ffbd080 0x40117918:0x3ffbd0a0 0x40117941:0x3ffbd240 0x400d56cf:0x3ffbd290 0x401e3314:0x3ffbd3e0

ELF file SHA256: d8e5a6d60

Rebooting...
ets Jun 8 2016 00:22:57

Any ideas?

[SybexX]

@A380Coding At this point the PSRAM is initializing and apparently there is something wrong with it (the initialization apparently causes a timeout). Did any version even work on it or is the board new?

[SybexX]

Why don't you use the OTA site to update the firmware?

[A380Coding]

That is what I tried initially and got stuck with the above console output. I then erased and tried flashing manually, but no luck. Just to see if I can get it running without OTA. That is when I started to test other boards to exclude a faulty one. But not luck as well. Running out of ideas why :( Flashing back to 15.7 worked. I do not recall now, but I think I tried the OTA way two times.

I want to test again on the spare boards I have, but I probably need to do something different to get it running.

[SybexX]

Please also note that with a manual update, you must manually copy the appropriate SD files to the SD.
Between version 15.7.0 and 16 RC4, many SD files have changed and if something is missing or different, this can lead to bootlopp.

[A380Coding]

I'm aware of that. I removed all files and copied the once from the sd_card.zip included in the manual installation set. So basically doing a fresh install.
In case of the above error message, I think it doesn't even read the SD card as it is stuck before that?

[SybexX]

The SD is initialized directly beforehand and the following folders are created if they do not exist:
"/sdcard/log", "/sdcard/log/data", "/sdcard/log/analog", "/sdcard/log/digit", "/sdcard/log/message", "/sdcard/log/source"

[Slider0007]

@A380Coding: Your trace log is not pointing to one direction but at least it shows some indication, e.g. migrateConfiguration(). The MQTTPublish entries are confusing me, though.

Decoding stack results
0x4008ac71: __swrite at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp-elf/src/newlib/newlib/libc/stdio/stdio.c line 86
0x400895e5: __sflush_r at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp-elf/src/newlib/newlib/libc/stdio/fflush.c line 224
0x40089427: _fclose_r at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp-elf/src/newlib/newlib/libc/stdio/fclose.c line 94
0x400894b7: fclose at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp-elf/src/newlib/newlib/libc/stdio/fclose.c line 125
0x40117123: std::less   , std::allocator  > >::operator()(std::__cxx11::basic_string  , std::allocator  > const&, std::__cxx11::basic_string  , std::allocator  > const&) const at C:/Users/Admin/.platformio/packages/toolchain-xtensa-esp-elf@13.2.0+20240530/xtensa-esp-elf/include/c++/13.2.0/bits/char_traits.h line 399
0x40117918: MQTTPublish(std::__cxx11::basic_string  , std::allocator  >, std::__cxx11::basic_string  , std::allocator  >, int, bool) at components/jomjol_mqtt/interface_mqtt.cpp line 46
0x40117941: MQTTPublish(std::__cxx11::basic_string  , std::allocator  >, std::__cxx11::basic_string  , std::allocator  >, int, bool) at components/jomjol_mqtt/interface_mqtt.cpp line 60
0x400d56cf: migrateConfiguration() at C:/Users/Admin/.platformio/packages/toolchain-xtensa-esp-elf@13.2.0+20240530/xtensa-esp-elf/include/c++/13.2.0/bits/basic_string.h line 631
0x401e3314: ram_spur_coef_cfg at phy_chip_v7.c line 1916

You could test if 16.0.0RC1 is booting on your device, because in 16.0.0RC2 and newer migration routine is modified. Be aware, this would be only for testing to get idea where it's stuck, because 16.0.0RC1 does have other issues in place.

[A380Coding]

I will try again on the spare boards as soon as I find some time. The other board is back in use with 15.7. I'm not using MQTT, strange.

[A380Coding]

I was able to flash RC4 on one of the spare boards. Initial setup via the wizard is extremely slow. After rebooting I got an "0x00000004 Cam bad" error. Camera was working fine during initial setup.

[Slider0007]

I was able to flash RC4 on one of the spare boards. Initial setup via the wizard is extremely slow. After rebooting I got an "0x00000004 Cam bad" error. Camera was working fine during initial setup.

Have you tested RC4 again or RC1?

[A380Coding]

I just tried RC4 and at least the flashing part worked the first time and it started up. Not sure why it was successful this time.
I disconnected power several times and get it running now and then with working cam, but it does not seem to be stable. Also reconnected the cam, just to be sure. If I manage to start up with working cam, it feels like it is running at 100% cpu load.