Regression in decode unicode

Created by: fvanroie

Problem Description

I am using the ARDUINOJSON_DECODE_UNICODE=1 compiler flag to enable the decode unicode feature of ArduinoJson.

After upgrading to version 6.14.0 of the library, unicode characters read from a json lines file are not decoded properly anymore. There seems to be a shift of 0x1000 in the decoded character codepoint.

Switching back to version 6.13.0 solves the problem.

Target Platform

ESP8266 & ESP32

Compiler model and version

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 2.3.2 > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: toolchain-xtensa 2.40802.191122 (4.8.2),
framework-arduinoespressif8266 2.20603.191216 (2.6.3),
tool-esptool 1.413.0 (4.13), tool-esptoolpy 1.20800.0 (2.8.0),
tool-mkspiffs 1.200.0 (2.0)

platformio.ini configuration file:

[env:d1_mini]
platform = espressif8266
board = d1_mini
framework = arduino
upload_port = COM4       ; Change to the correct port
monitor_port = COM4      ; Change to the correct port
monitor_speed = 115200
upload_speed = 921600
lib_deps =
    ArduinoJson@6.13.0   ; 6.14.0 breaks unicode decoding!
build_flags =
    -D SPIFFS_TEMPORAL_FD_CACHE     ; speed up opening recent files
    -D ARDUINOJSON_DECODE_UNICODE=1 ; for utf-8 symbols
    -D ARDUINOJSON_ENABLE_PROGMEM=1 ; for PROGMEM arguments
board_build.f_flash = 40000000L
board_build.flash_mode = dout

Minimal, Complete and Verifiable Example

#include <Arduino.h>
#include "ArduinoJson.h"
#include "FS.h"

void setup()
{
  Serial.begin(115200); /* prepare for possible serial debug */
  Serial.flush();
  Serial.println();

  SPIFFS.begin();
  if (!SPIFFS.begin())
    Serial.println("FILE: %sSPI flash init failed. Unable to mount FS.");
  else
    Serial.println("FILE: [SUCCESS] SPI flash FS mounted");

  File file = SPIFFS.open("/pages.jsonl", "r");
  if (!file)
    Serial.println("FILE: File not opened");
  else
    Serial.println("FILE: [SUCCESS] file opened");

  DynamicJsonDocument config(256);
  while (deserializeJson(config, file) == DeserializationError::Ok)
  {
    serializeJson(config, Serial);
    Serial.println();

    String message = config["txt"].as<String>();

    byte bytes[message.length() + 1];
    message.getBytes(bytes, message.length() + 1);

    for (int i = 0; i < sizeof(bytes); i++)
    {
      Serial.print(bytes[i], HEX);
      Serial.print(" ");
    }
    Serial.println("\n");
  }
  file.close();
}

void loop()
{
  // put your main code here, to run repeatedly:
}

Contents of input file /pages.jsonl uploaded to SPIFFS:

{"txt":"\uF053 Prev"}
{"txt":"\uF015 Home"}
{"txt":"Next \uF054"}

Result

Test 1 Using version 6.13.0

lib_deps =
    ArduinoJson@6.13.0

Result:

{"txt":" Prev"}
EF 81 93 20 50 72 65 76 0

{"txt":" Home"}
EF 80 95 20 48 6F 6D 65 0

{"txt":"Next "}
4E 65 78 74 20 EF 81 94 0

Test 2 Using version 6.14.0

lib_deps =
    ArduinoJson@6.14.0

Result:

{"txt":"𐁓 Prev"}
F0 90 81 93 20 50 72 65 76 0

{"txt":"𐀕 Home"}
F0 90 80 95 20 48 6F 6D 65 0

{"txt":"Next 𐁔"}
4E 65 78 74 20 F0 90 81 94 0

Conclusion

The decoded unicode characters \uF053 \uF015 \uF054 are shifted by 0x1000:

	Relevant Output Received	Unicode Character
Test 1	EF 81 93EF 80 95 EF 81 94	\uF053 \uF015 \uF054
Test 2	F0 90 81 93F0 90 80 95F0 90 81 94	‭\u10053‬ \u10015 \u‭10054‬