Bootloader uploading at 0x2000 and certificates not generating

[Szybet]

Hello Jason
First, I appreciate your work, I see you everywhere in platformio issues, you also helped me a lot, I use this repo as a fork for my small project inkWatchy. I have 2 small issues, one after recently updating everything to the newer esp idf v5.1 branch

Also, don't bother with this if you don't have time, I found worked around those issues anyway

First, for the target esp32 (I checked if the mcu variable was esp32, it was) it gives me the bootloader offset for the p4, so 0x2000

platform-espressif32/builder/frameworks/espidf.py

Line 1666 in 1b989b0

"0x0" if mcu in ("esp32c2", "esp32c3", "esp32c6", "esp32s3", "esp32h2") else ("0x2000" if mcu in ("esp32p4") else "0x1000"),

I have no explanation for that, to my python knowledge everything seems fine

secondly, Compilation for me stops on certificates not generating, the work around is this:
https://github.com/Szybet/InkWatchy/blob/984dbd73a9166b8da3f0ebc2fbd120b1d0a90779/resources/tools/buildTime/preBuild.sh#L12-L28

How to do it properly? I see the function generate_mbedtls_bundle which uses MBEDTLS_CERTIFICATE_BUNDLE, should I set it to false? I want https to work.

Thanks!

[Jason2866]

@Szybet No idea why the bootloader for an esp32 is placed at 0x2000. Have to investigate.
For the certificates, check your settings -> https://github.com/Szybet/InkWatchy/blob/master/resources/tools/buildTime/sdkconfig-watchy-2.defaults
My assumption is that it is not as it should be. See Arduino orig settings https://github.com/espressif/esp32-arduino-lib-builder/blob/master/configs/defconfig.common

EDIT: Probably missing this, when using default IDF and Arduino

[env:esp32dev]
board = esp32dev
board_build.embed_txtfiles =
    managed_components/espressif__esp_insights/server_certs/https_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_mqtt_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_claim_service_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_ota_server.crt

[Jason2866]

Interesting platformio.ini setup part

board_build.cmake_extra_args = 
    -DSDKCONFIG_DEFAULTS="resources/tools/buildTime/sdkconfig-watchy-2.defaults"

conflicts with the Platformio IDF cmake/Ninja build script. May end in unexpected results/issues.
Why is it there in general?

[Szybet]

Interesting platformio.ini setup part

board_build.cmake_extra_args = 
    -DSDKCONFIG_DEFAULTS="resources/tools/buildTime/sdkconfig-watchy-2.defaults"

conflicts with the Platformio IDF cmake/Ninja build script. May end in unexpected results/issues. Why is it there in general?

I needed different sdkconfigs for different envs, this was the best solution? I think the main reason was I didn't wanted those sdkconfigs to be at default in the main directory, but placed somewhere else. Can't remember really

How to do it properly?

As for the certificates, I'm gonna test it, thanks

[Jason2866]

I think you can safe work of doing forks. Use this variant (it is from me too ;-))
It uses official not modified or removed components / libraries.
https://github.com/pioarduino/platform-espressif32

For sdkconfig.defaults files it is best to leave it where the are. The are correctly fetched. No settings needed. See here https://github.com/pioarduino/platform-espressif32/blob/develop/examples/espidf-arduino-littlefs/sdkconfig.defaults
you can even specify defaults for the MCU variants. Look for details in Platformio documentation

[Jason2866]

Bug with flash offset verified. Will revert, since IDF 5.1 does not support p4....
Don't see the error in the logic?!

[Szybet]

When trying the board_build.embed_txtfiles option
I tried making the NVS partition larger and I erased the flash then tried again

  #0  0x401ade51 in esp_cpu_clear_watchpoint at /root/.platformio/packages/framework-espidf/components/esp_hw_support/cpu.c:245
  #1  0x3ffc0270 in _xt_exception_table at ??:?
  #2  0x4008dcf9 in vPortEnterCritical at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:575
      (inlined by) xTaskIncrementTick at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/tasks.c:3315
  #3  0x3ffc02a0 in _xt_exception_table at ??:?
  #4  0x4008fa33 in xPortSysTickHandler at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/portable/port_systick.c:192
  #5  0x3ffc02d0 in _xt_exception_table at ??:?
  #6  0x4008f87d in _frxt_timer_int at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/portable/xtensa/portasm.S:343
  #7  0x3ffc02f0 in _xt_exception_table at ??:?
  #8  0x4008466a in _xt_lowint1 at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/portable/xtensa/xtensa_vectors.S:1240
  #9  0x3ffc0300 in _xt_exception_table at ??:?
  #10 0x40090bba in spi_flash_hal_poll_cmd_done at /root/.platformio/packages/framework-espidf/components/hal/spi_flash_hal_common.inc:37
  #11 0x40090966 in spi_flash_hal_read at /root/.platformio/packages/framework-espidf/components/hal/spi_flash_hal_common.inc:190
  #12 0x400987b9 in spi_flash_chip_winbond_read at /root/.platformio/packages/framework-espidf/components/spi_flash/spi_flash_chip_winbond.c:69
  #13 0x40085a93 in esp_flash_read at /root/.platformio/packages/framework-espidf/components/spi_flash/esp_flash_api.c:898
  #14 0x4010ad19 in esp_partition_read_raw at /root/.platformio/packages/framework-espidf/components/esp_partition/partition_target.c:99
  #15 0x4012bd76 in nvs::NVSPartition::read_raw(unsigned int, void*, unsigned int) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_partition.cpp:36
  #16 0x4012d1fc in nvs::Page::load(nvs::Partition*, unsigned long) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_page.cpp:51
  #17 0x4012d617 in nvs::PageManager::load(nvs::Partition*, unsigned long, unsigned long) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_pagemanager.cpp:25
  #18 0x4012aeaf in nvs::Storage::init(unsigned long, unsigned long) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_storage.cpp:172
  #19 0x4012bef5 in nvs::NVSPartitionManager::init_custom(nvs::Partition*, unsigned long, unsigned long) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_partition_manager.cpp:92
  #20 0x4012bfcf in nvs::NVSPartitionManager::init_partition(char const*) at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_partition_manager.cpp:54
  #21 0x4012a594 in nvs_flash_init at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_api.cpp:125
      (inlined by) nvs_flash_init_partition at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_api.cpp:117
      (inlined by) nvs_flash_init at /root/.platformio/packages/framework-espidf/components/nvs_flash/src/nvs_api.cpp:161
  #22 0x400eed13 in initArduino at components/arduino/cores/esp32/esp32-hal-misc.c:275
  #23 0x400eff91 in app_main at components/arduino/cores/esp32/main.cpp:104
  #24 0x401afdc6 in main_task at /root/.platformio/packages/framework-espidf/components/freertos/app_startup.c:208 (discriminator 15)
  #25 0x4008f432 in vPortTaskWrapper at /root/.platformio/packages/framework-espidf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:162

What is the difference between this https://github.com/pioarduino/platform-espressif32 and the current repo we are in? In practical words?

Well yes, but how do I specify many sdkconfig.defaults? Would be best if they could be at a custom path too

[Jason2866]

Change of NVS partition size is not supported with Platformio. You can add a second NVS partition. But do not touch the existing.

[Szybet]

It was just my ques that's what I need to do, well board_build.embed_txtfiles doesn't work for this issue

[Jason2866]

Idk what's wrong in your setup. The other fork I mentioned does include Rainmaker, Insights, mbedtls with all cryptos. This fork has removed this and changed a lot of sdkconfig settings!
The pioarduino fork is using unchanged Arduino and IDF repos. So it is fully compatible.
With the settings for the certificates I posted, IDF and Arduino/IDF projects compile successfully. See CI run https://github.com/pioarduino/platform-espressif32/actions/runs/10285459768