Ensure non-unicode strings aren't normalised (#3556)

In rare scenarios it's possible to end up calling `normalise_whitespace`
on a non-unicode string. In this case, we shouldn't be removing
whitespace characters that are specific to unicode as a
`Encoding::CompatibilityError` error ends up being raised.

This has only happened in practice once, and only in the test
environment: https://good-machine.sentry.io/issues/6604598285/

In this case, a CSV file was uploaded where the encoding couldn't be
detected, which lead to a unhandled exception being raised, rather than
a validation error on the file.

Author: thomasleese

lib/core_ext/string/normalise_whitespace.rb

@@ -4,11 +4,13 @@ class String
   def normalise_whitespace
     result = self
 
-    # \u200D is a zero-width joiner (ZWJ) which is used in the frontend to display the NHS number
-    result = result.tr("\u200D", "")
+    if result.encoding == Encoding::UTF_8
+      # \u200D is a zero-width joiner (ZWJ) which is used in the frontend to display the NHS number
+      result = result.tr("\u200D", "")
 
-    # \u00A0 is a non-breaking space
-    result = result.tr("\u00A0", " ")
+      # \u00A0 is a non-breaking space
+      result = result.tr("\u00A0", " ")
+    end
 
     result.strip.gsub(/\s+/, " ").presence
   end

spec/lib/core_ext/string_normalise_whitespace_spec.rb

@@ -40,9 +40,14 @@
     end
 
     context "with non-UTF-8 encoded strings" do
+      let(:windows_1252_encoded_string) do
+        "hello’s world".encode(Encoding::WINDOWS_1252)
+      end
+
       it "does not apply Unicode-specific transformations" do
-        ascii_string = "hello world".encode(Encoding::ASCII)
-        expect(ascii_string.normalise_whitespace).to eq("hello world")
+        expect(windows_1252_encoded_string.normalise_whitespace).to eq(
+          windows_1252_encoded_string
+        )
       end
     end
   end

spec/lib/csv_parser_spec.rb

@@ -42,4 +42,14 @@
       expect(row[:header].value).to eq("value with ’ character")
     end
   end
+
+  context "when the encoding cannot be detected" do
+    # This is not a good example of a CSV file, but it was the only
+    # string I could find where the encoding wasn't detected.
+    let(:data) { "\x92".b }
+
+    it "doesn't raise an error" do
+      expect { table }.not_to raise_error
+    end
+  end
 end