Skip to content

UTF-8 Decoding Error #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Parent: Update README.md
merged 3 commits into from
Mar 15, 2021
Merged

UTF-8 Decoding Error #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c2f914a
Add a test to reproduce the invalid utf-8 error.
amjith Mar 14, 2021
5b86afa
Fix the utf-8 decoding error.
amjith Mar 14, 2021
b71fb3d
Update changelog.
amjith Mar 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
## Unreleased - TBD

<!-- add upcoming change notes here -->

### Features

- Add verbose feature to `favorite_query` command. (Thanks: [Zhaolong Zhu])
Expand All @@ -12,6 +10,7 @@
### Bug Fixes

- Fix compatibility with sqlparse >= 0.4.0. (Thanks: [chocolateboy])
- Fix invalid utf-8 exception. (Thanks: [Amjith])

## 1.4.1 - 2020-07-27

Expand Down
8 changes: 8 additions & 0 deletions litecli/sqlexecute.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@
# })


def utf8_resilient_decoder(s):
try:
return s.decode("utf-8")
except UnicodeDecodeError:
return s.decode("latin-1")


class SQLExecute(object):

databases_query = """
Expand Down Expand Up @@ -61,6 +68,7 @@ def connect(self, database=None):
raise Exception("Path does not exist: {}".format(db_dir_name))

conn = sqlite3.connect(database=db_name, isolation_level=None)
conn.text_factory = utf8_resilient_decoder
if self.conn:
self.conn.close()

Expand Down
17 changes: 12 additions & 5 deletions tests/test_sqlexecute.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,17 @@ def test_unicode_support_in_output(executor):
assert_result_equal(results, headers=["t"], rows=[(u"é",)])


@dbtest
def test_invalid_unicode_values_dont_choke(executor):
run(executor, "create table unicodechars(t text)")
# \xc3 is not a valid utf-8 char. But we can insert it into the database
# which can break querying if not handled correctly.
run(executor, u"insert into unicodechars (t) values (cast(x'c3' as text))")

results = run(executor, u"select * from unicodechars")
assert_result_equal(results, headers=["t"], rows=[(u"Ã",)])


@dbtest
def test_multiple_queries_same_line(executor):
results = run(executor, "select 'foo'; select 'bar'")
Expand Down Expand Up @@ -199,11 +210,7 @@ def test_verbose_feature_of_favorite_query(executor):

results = run(executor, "\\f sh_param 1")
assert_result_equal(
results,
title=None,
headers=["a", "id"],
rows=[("abc", 1)],
auto_status=False,
results, title=None, headers=["a", "id"], rows=[("abc", 1)], auto_status=False,
)

results = run(executor, "\\f+ sh_param 1")
Expand Down