04-13 App Storage Issues

Today I tried to connect the ESP32 to Wifi. I seem to have reached a new sort of roadblock which I have never encountered before. There isnt sufficient storage space on the ESP32 for the nice high level ESP32 BLE libraries. I never paid too close attention to it, but before including Wifi libs, the program already used 74% of storage just to scan for BLE devices.

I only have 1.280 Mb or 1310720 bytes of storage. Now with the Wifi library I am using 103% of storage space.

Possible solutions

Get rid of tft_espi library

The Library provides an interface for the program to function with the TFT_eSPI library file. I do not consider logging things to the screens absolutely essential, they are just nice to have. Perhaps I can save a few precious bytes here. I really would prefer not to do this though since I like seeing the output directly on screen.

Use the ESP32 API from espressif systems directly

The BLE library of the Arduino ESP32 community provides very high level classes to interface with the esp32 ble api. It has been fairly easy to create servers and clients with it. This convenience comes with one large drawback.

Considering this Github Post, the inclusion of memory seems very expensive. BLEDevice::init(" "); consumes 696 Kb of storage, which is a whole lot when one only has 1.2 Mb of storage available total. Perhaps it woud make more sense to move away from the Arduino ESP32 library and directly invoke the BLE libraries from Espressif Systems.

It is questionable though how helpful a switch to custom programmed BLE functions would be. The github user Nkolban, who is the creator of the high level classes said himself that the cost of Bluetooth LE "is what it is" regardless of using his Library or not. Bluetooth is not cheap in general. The cost of using bluetooth on a memory and storage strapped device such as the ESP32 is expensive(500kB Storage, which takes away from my 1.2 Mb). Therefore rewriting everything might not make a lot of sense.

Adjust ESP32 Partition tables

According to various blog posts, the Espressif IDF has a direct way to allocate more storage to a ESP32 application. I can then give the ESP32 more storage to use my existing code and dont have to change anything. This angle of attack allows me to vertically scale system resources to account for an increasingly expensive program. I see some inherent problems for the future with this angle of attack.

This implies that people using my Client will have to go through an additional step in the setup process. Furthermore, a very resource intensive application requires adequate hardware (4 Mb Flash Memory at least).

Vertical scaling might make life really terrible when my APP surpasses the max. partition table size, since I am less flexible to introduce the above solutions.

Increasing the System requirements may make my library less accessible to potential users. Not having to be careful about system resource usage might reduce code quality as well.

This is probably the easiest solution to my problem

Chosen Solution and Conclusion

I chose the third option and adjusted the partition tables, there is even a UI for it in the Boards Manager. I had used the factory defaults when setting up the project. With the previous configuration (shown above) I only allocated 1.2 Mb for application storage (which are the 1280 Kb I mentioned above) and 1.5 Mb for SPIFFS, (the filesystem of the ESP32).

Now the app partition has 2 Mb of space, which comfortably accomodates the big BLE library as well. Storage usage went from 102% to 64%.

Despite possibly potentially being a huge antipattern, I can adjust the partition table size even further with the option "Huge APP" which allocates a whopping 3 Mb. I was reading about OTA for the ESP32, but this is no longer possible with both this configuration and the one that I have just newly adopted. There is still an incentive to keep the program size under 2Mb, since the option "Minimal SPIFFS" with 1.9mb APP and 190kb SPIFFS includes OTA support.