feat(fs): #3243 add encoding option to readTextFile functions

[SeijiOkuda]

Summary

Added support for reading text files encoded in character sets other than UTF-8 (e.g. Shift_JIS).

Previously, the application assumed UTF-8 encoding, which caused garbled text in some cases. With this change, multiple character encodings are now supported and such files can be read correctly.

Closes #3243

Changes

• Added support for reading files saved in encodings other than UTF-8
• Added logic to specify and/or detect the character encoding and decode accordingly
• Confirmed that existing behavior for UTF-8 files is not affected

Background / Motivation

• To support existing memo/text files that are saved in encodings such as Shift_JIS
• To allow users to handle text files without character corruption

Verification

• Confirmed that UTF-8 files can still be read as before
• Confirmed that files encoded in Shift_JIS and other encodings are displayed correctly

Usage

You can specify the encoding when reading a text file:

await readTextFile(path, { encoding: "shift_jis" });

[github-actions]

Package Changes Through 5ad3d80

There are 2 changes which include fs with minor, fs-js with minor

Planned Package Versions

The following package releases are the planned based on the context of changes in this pull request.

package	current	next
api-example	2.0.41	2.0.42
api-example-js	2.0.37	2.0.38
fs	2.4.5	2.5.0
fs-js	2.4.5	2.5.0
dialog	2.6.0	2.6.1
dialog-js	2.6.0	2.6.1
http	2.5.7	2.5.8
http-js	2.5.7	2.5.8
persisted-scope	2.3.5	2.3.6

Add another change file through the GitHub UI by following this link.

---

Read about change files or the docs at github.com/jbolda/covector

[amrbashir]

Can you please add a change file and also update readTextFileLines to do the same decoding

[amrbashir]

Missing documentation

[SeijiOkuda]

@amrbashir
Thanks for the review!
Added documentation comments.

[SeijiOkuda]

Can you please add a change file and also update readTextFileLines to do the same decoding

@amrbashir
Thanks for the review!
Done! I added the change file and updated readTextFileLines to apply the same decoding as well.

[aiueo13]

readTextFileLines detects line breaks on the Rust side using a byte-level check of \n and \r\n for 0x0A and 0x0D 0x0A. Therefore, for encodings that are not compatible with this approach. it may be necessary to either return an error or change the implementation.😱

plugins-workspace/plugins/fs/src/commands.rs

Lines 1208 to 1227 in 05045f9

	impl<B: BufRead> Iterator for LinesBytes<B> {
	type Item = std::io::Result<Vec<u8>>;
	fn next(&mut self) -> Option<std::io::Result<Vec<u8>>> {
	let mut buf = Vec::new();
	match self.0.read_until(b'\n', &mut buf) {
	Ok(0) => None,
	Ok(_n) => {
	if buf.last() == Some(&b'\n') {
	buf.pop();
	if buf.last() == Some(&b'\r') {
	buf.pop();
	}
	}
	Some(Ok(buf))
	}
	Err(e) => Some(Err(e)),
	}
	}
	}

[amrbashir]

I think that could be fixed in another PR