Affected version
3.5.5
Bug description
This is a dup of #2961. It seems to be closed because they could not provide repro file. I am able to reproduce this in an open source test. https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala#L266 The test is expected to fail and produce lots of errors from the exception. Offending portion:
CsvFormat:
Comment character=#
Field delimiter=,
Line separator (normalized)=�
Line separator sequence=�
Quote character="
Quote escape character=\
Quote escape escape character=null
Notice the 0xFFFF in in separator. That print out comes from here. https://github.com/uniVocity/univocity-parsers/blob/7e7d1b3c0a3dceaed4a8413875eb1500f2a028ec/src/main/java/com/univocity/parsers/common/Format.java#L220 Every time there's an exception, it tries to print out the current config. Printing non-printable binary strings on screen is a questionable practice that could really mess up someone's terminal. But I guess we shouldn't choke on it neither.
This is what Xmllint has to say:
❯ xmllint TEST-org.apache.spark.sql.CsvFunctionsSuite.xml
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:227: parser error : Char 0xFFFF out of allowed range
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:227: parser error : CData section not finished
17:33:13.268 [Executor task launch worker for task
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:227: parser error : Char 0xFFFF out of allowed range
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:227: parser error : PCDATA invalid Char value 65535
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:228: parser error : Char 0xFFFF out of allowed range
Line separator sequence=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:228: parser error : PCDATA invalid Char value 65535
Line separator sequence=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:304: parser error : Char 0xFFFF out of allowed range
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:304: parser error : PCDATA invalid Char value 65535
Line separator (normalized)=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:305: parser error : Char 0xFFFF out of allowed range
Line separator sequence=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:305: parser error : PCDATA invalid Char value 65535
Line separator sequence=�
^
TEST-org.apache.spark.sql.CsvFunctionsSuite.xml:344: parser error : Sequence ']]>' not allowed in content
park.util.Utils$.tryWithSafeFinally(Utils.scala:94) ~[spark-core_2.12-3.5.1.jar:
Affected version
3.5.5
Bug description
This is a dup of #2961. It seems to be closed because they could not provide repro file. I am able to reproduce this in an open source test. https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala#L266 The test is expected to fail and produce lots of errors from the exception. Offending portion:
Notice the 0xFFFF in in separator. That print out comes from here. https://github.com/uniVocity/univocity-parsers/blob/7e7d1b3c0a3dceaed4a8413875eb1500f2a028ec/src/main/java/com/univocity/parsers/common/Format.java#L220 Every time there's an exception, it tries to print out the current config. Printing non-printable binary strings on screen is a questionable practice that could really mess up someone's terminal. But I guess we shouldn't choke on it neither.
This is what Xmllint has to say: