Fix character encoding problem for non-ASCII/Chinese data (#795) - #1693
Fix character encoding problem for non-ASCII/Chinese data (#795)#1693sanjaykumarmtt wants to merge 6 commits into
Conversation
tsegismont
left a comment
There was a problem hiding this comment.
Thank you @sanjaykumarmtt
Could you please add a test?
|
@tsegismont thank you for the review I have added the unit test and formatted the code accordingly |
tsegismont
left a comment
There was a problem hiding this comment.
Thanks for adding a test, but I was thinking about actually testing a DB2 query rather than a unit test. Is it possible?
|
thanks for the feedback @tsegismont yes that makes sense I will add a DB2 query integration test for this and update the PR soon |
|
hi @tsegismont apologies for the delay I have replaced the previous unit test with a DB2 query integration test in DB2QueriesTest as suggested please review it and let me know if any further changes are needed |
Signed-off-by: Thomas Segismont <tsegismont@gmail.com>
|
I have updated the PR with the suggested fix: casting the untyped parameter marker with CAST(? AS VARCHAR(100)) in DB2QueriesTest, and reverted the unrelated changes. |
tsegismont
left a comment
There was a problem hiding this comment.
Thanks. For me the test passes without the code changes. Can you check if it's actually testing what you want?
…ip in DB2QueriesTest
|
sorry for the delay I found why the test passed earlier the target chinese string was not present in the database table so the query returned 0 rows and did not properly test the encoding. |
|
I have updated the test added the Chinese string to init.sql updated the test assertion to verify that 1 row is returned and the decoded message matches the expected value. |
|
Could you please approve and trigger the CI workflow when you have a moment? |
|
The test still passes without the changes. I think this happens because it's all UTF-8. Please make sure to create a test that does not pass without the changes. |
|
hi @tsegismont the original test was passing without changes because the database environment and connection default to UTF-8 CCSID 1208 which bypasses the specific Chinese CCSID mapping logic to trigger the mapping I tried casting the column using CAST('你好世界' AS VARCHAR(100) CCSID 1386) but DB2 threw SQLCODE=-622 (SQLSTATE=56031) because this database configuration does not support that clause |
|
I tried writing a dedicated unit test (CCSIDConstantsTest) locally to verify the mapping and it passes with the changes and fails without them |
|
I'm not a DB2 expert unfortunately. Perhaps is it possible to create a specific database with different encoding? |
### Motivation
Fixes #795
When reading Chinese or other multi byte characters from DB2 they show up as
???because the driver only checked OS type instead of the server CCSID.This change fixes the issue:
CCSIDConstants.sda.ccsid_) if present; otherwise, uses connection level encoding (getCcsidMbcEncoding()).Conformance