Skip to content

Commit 2f608e7

Browse files
JamieTaylor-TUOSJamieTaylor-TUOS
committed
Support Pandas DataFrames
added support for converting DBConnector.query() results to pandas dataframe by setting `df=True` as input arg
1 parent 6f25d3a commit 2f608e7

22 files changed

Lines changed: 434 additions & 358 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ A wrapper for the MySQL Python connector that provides additional resilience an
66

77
* A wrapper for the MySQL Python connector that provides additional resilience and functionality.
88
* Implements connection pooling that's more reliable.
9-
* Version 1.1.4
9+
* Version 1.2.0
1010
* Works with Python 2.7+ or 3.5+
1111

1212
## How do I get set up? ##

‎dbconnector/dbconnector.py‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -353,14 +353,17 @@ def _safe_query(self, query_type, **kwargs):
353353
self.sleep_interval = 1
354354
return result
355355

356-
def query(self, sqlquery):
356+
def query(self, sqlquery, df=False):
357357
"""
358358
Query the database using select (with resilience).
359359
360360
Parameters
361361
----------
362362
`sqlquery` : string
363363
SQL `select` statement to be executed.
364+
`df` : boolean
365+
Set to True to return query results as Pandas DataFrame. Column names will be extracted
366+
from the *sqlquery* and converted to lowercase (do not use "select * from").
364367
Returns
365368
-------
366369
list
@@ -375,7 +378,7 @@ def query(self, sqlquery):
375378
MySQL Connector Python is prone to dropped connections - this wrapper adds resilience by
376379
retrying.
377380
"""
378-
return self._safe_query(self._select_query, sqlquery=sqlquery)
381+
return self._safe_query(self._select_query, sqlquery=sqlquery, df=df)
379382

380383
def proc(self, proc, args):
381384
"""Execute a MySQL procedure with resilience."""
@@ -407,10 +410,21 @@ def _safe_close(cnx):
407410
def _select_query(cnx, **kwargs):
408411
"""Execute a select query."""
409412
sqlquery = kwargs.get('sqlquery', None)
413+
df = kwargs.get('df', False)
410414
cursor = cnx.cursor()
411415
cursor.execute(sqlquery)
412416
result = cursor.fetchall()
413417
cursor.close()
418+
if df:
419+
from pandas import DataFrame
420+
import re
421+
col_regex = "(?<=^select)[a-zA-Z0-9_\s*(),`]+(?=from)"
422+
cols = [c.strip().split(" as ")[-1].strip().strip("`") for c in
423+
re.findall(col_regex, sqlquery.lower())[0].strip().split(",")]
424+
if result[0]:
425+
return DataFrame(result, columns=cols)
426+
else:
427+
return DataFrame({c: [] for c in cols})
414428
return result
415429

416430
@staticmethod

‎docs/build/doctrees/environment.pickle‎

9.88 KB
Binary file not shown.

‎docs/build/doctrees/index.doctree‎

6 Bytes
Binary file not shown.

‎docs/build/doctrees/modules.doctree‎

3.21 KB
Binary file not shown.

‎docs/build/html/.buildinfo‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
# Sphinx build info version 1
22
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
3-
config: 47bae999860315bcc2c3f55a918b274b
3+
config: afdfe3eaf6b0f76469cadb0c04fadb56
44
tags: 645f666f9bcd5a90fca523b33c5a78b7

‎docs/build/html/_modules/dbconnector/dbconnector.html‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@
99
<title>dbconnector.dbconnector &#8212; MySQL-DBConnector 1.1.2 documentation</title>
1010
<link rel="stylesheet" href="../../_static/classic.css" type="text/css" />
1111
<link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
12+
1213
<script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
1314
<script type="text/javascript" src="../../_static/jquery.js"></script>
1415
<script type="text/javascript" src="../../_static/underscore.js"></script>
1516
<script type="text/javascript" src="../../_static/doctools.js"></script>
16-
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
17+
<script type="text/javascript" src="../../_static/language_data.js"></script>
18+
1719
<link rel="index" title="Index" href="../../genindex.html" />
1820
<link rel="search" title="Search" href="../../search.html" />
1921
</head><body>
@@ -387,14 +389,17 @@ <h1>Source code for dbconnector.dbconnector</h1><div class="highlight"><pre>
387389
<span class="bp">self</span><span class="o">.</span><span class="n">sleep_interval</span> <span class="o">=</span> <span class="mi">1</span>
388390
<span class="k">return</span> <span class="n">result</span>
389391

390-
<div class="viewcode-block" id="DBConnector.query"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.query">[docs]</a> <span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sqlquery</span><span class="p">):</span>
392+
<div class="viewcode-block" id="DBConnector.query"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.query">[docs]</a> <span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sqlquery</span><span class="p">,</span> <span class="n">df</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
391393
<span class="sd">&quot;&quot;&quot;</span>
392394
<span class="sd"> Query the database using select (with resilience).</span>
393395

394396
<span class="sd"> Parameters</span>
395397
<span class="sd"> ----------</span>
396398
<span class="sd"> `sqlquery` : string</span>
397399
<span class="sd"> SQL `select` statement to be executed.</span>
400+
<span class="sd"> `df` : boolean</span>
401+
<span class="sd"> Set to True to return query results as Pandas DataFrame. Column names will be extracted</span>
402+
<span class="sd"> from the *sqlquery* and converted to lowercase (do not use &quot;select * from&quot;).</span>
398403
<span class="sd"> Returns</span>
399404
<span class="sd"> -------</span>
400405
<span class="sd"> list</span>
@@ -409,15 +414,15 @@ <h1>Source code for dbconnector.dbconnector</h1><div class="highlight"><pre>
409414
<span class="sd"> MySQL Connector Python is prone to dropped connections - this wrapper adds resilience by</span>
410415
<span class="sd"> retrying.</span>
411416
<span class="sd"> &quot;&quot;&quot;</span>
412-
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_safe_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_select_query</span><span class="p">,</span> <span class="n">sqlquery</span><span class="o">=</span><span class="n">sqlquery</span><span class="p">)</span></div>
417+
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_safe_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_select_query</span><span class="p">,</span> <span class="n">sqlquery</span><span class="o">=</span><span class="n">sqlquery</span><span class="p">,</span> <span class="n">df</span><span class="o">=</span><span class="n">df</span><span class="p">)</span></div>
413418

414419
<div class="viewcode-block" id="DBConnector.proc"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.proc">[docs]</a> <span class="k">def</span> <span class="nf">proc</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">proc</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
415420
<span class="sd">&quot;&quot;&quot;Execute a MySQL procedure with resilience.&quot;&quot;&quot;</span>
416421
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_safe_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_proc_query</span><span class="p">,</span> <span class="n">proc</span><span class="o">=</span><span class="n">proc</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="n">args</span><span class="p">)</span></div>
417422

418-
<div class="viewcode-block" id="DBConnector.iud_query"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.iud_query">[docs]</a> <span class="k">def</span> <span class="nf">iud_query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sqlquery</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
423+
<div class="viewcode-block" id="DBConnector.iud_query"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.iud_query">[docs]</a> <span class="k">def</span> <span class="nf">iud_query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sqlquery</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
419424
<span class="sd">&quot;&quot;&quot;Execute an insert/update/delete SQL statement with resilience.&quot;&quot;&quot;</span>
420-
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_safe_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_iud_query</span><span class="p">,</span> <span class="n">sqlquery</span><span class="o">=</span><span class="n">sqlquery</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">)</span></div>
425+
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_safe_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_iud_query</span><span class="p">,</span> <span class="n">sqlquery</span><span class="o">=</span><span class="n">sqlquery</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">size</span><span class="p">)</span></div>
421426

422427
<div class="viewcode-block" id="DBConnector.close_connections"><a class="viewcode-back" href="../../modules.html#dbconnector.dbconnector.DBConnector.close_connections">[docs]</a> <span class="k">def</span> <span class="nf">close_connections</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
423428
<span class="sd">&quot;&quot;&quot;Close all connections when done for optimal DB efficiency.&quot;&quot;&quot;</span>
@@ -441,10 +446,21 @@ <h1>Source code for dbconnector.dbconnector</h1><div class="highlight"><pre>
441446
<span class="k">def</span> <span class="nf">_select_query</span><span class="p">(</span><span class="n">cnx</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
442447
<span class="sd">&quot;&quot;&quot;Execute a select query.&quot;&quot;&quot;</span>
443448
<span class="n">sqlquery</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;sqlquery&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
449+
<span class="n">df</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;df&#39;</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
444450
<span class="n">cursor</span> <span class="o">=</span> <span class="n">cnx</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
445451
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sqlquery</span><span class="p">)</span>
446452
<span class="n">result</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
447453
<span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
454+
<span class="k">if</span> <span class="n">df</span><span class="p">:</span>
455+
<span class="kn">from</span> <span class="nn">pandas</span> <span class="k">import</span> <span class="n">DataFrame</span>
456+
<span class="kn">import</span> <span class="nn">re</span>
457+
<span class="n">col_regex</span> <span class="o">=</span> <span class="s2">&quot;(?&lt;=^select)[a-zA-Z0-9_\s*(),`]+(?=from)&quot;</span>
458+
<span class="n">cols</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; as &quot;</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s2">&quot;`&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span>
459+
<span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">col_regex</span><span class="p">,</span> <span class="n">sqlquery</span><span class="o">.</span><span class="n">lower</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
460+
<span class="k">if</span> <span class="n">result</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span>
461+
<span class="k">return</span> <span class="n">DataFrame</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">cols</span><span class="p">)</span>
462+
<span class="k">else</span><span class="p">:</span>
463+
<span class="k">return</span> <span class="n">DataFrame</span><span class="p">({</span><span class="n">c</span><span class="p">:</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">cols</span><span class="p">})</span>
448464
<span class="k">return</span> <span class="n">result</span>
449465

450466
<span class="nd">@staticmethod</span>
@@ -463,7 +479,7 @@ <h1>Source code for dbconnector.dbconnector</h1><div class="highlight"><pre>
463479
<span class="nd">@staticmethod</span>
464480
<span class="k">def</span> <span class="nf">_iud_query</span><span class="p">(</span><span class="n">cnx</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
465481
<span class="sd">&quot;&quot;&quot;Execute an insert/update/delete SQL statement.&quot;&quot;&quot;</span>
466-
<span class="n">size</span> <span class="o">=</span> <span class="mi">100</span>
482+
<span class="n">size</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;size&#39;</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span>
467483
<span class="n">data</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;data&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
468484
<span class="n">sqlquery</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;sqlquery&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
469485
<span class="n">cursor</span> <span class="o">=</span> <span class="n">cnx</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
@@ -510,7 +526,7 @@ <h3>Navigation</h3>
510526
</div>
511527
<div class="footer" role="contentinfo">
512528
&#169; Copyright 2018, Jamie Taylor.
513-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.8.
529+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.5.
514530
</div>
515531
</body>
516532
</html>

‎docs/build/html/_modules/index.html‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@
99
<title>Overview: module code &#8212; MySQL-DBConnector 1.1.2 documentation</title>
1010
<link rel="stylesheet" href="../_static/classic.css" type="text/css" />
1111
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
12+
1213
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
1314
<script type="text/javascript" src="../_static/jquery.js"></script>
1415
<script type="text/javascript" src="../_static/underscore.js"></script>
1516
<script type="text/javascript" src="../_static/doctools.js"></script>
16-
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
17+
<script type="text/javascript" src="../_static/language_data.js"></script>
18+
1719
<link rel="index" title="Index" href="../genindex.html" />
1820
<link rel="search" title="Search" href="../search.html" />
1921
</head><body>
@@ -68,7 +70,7 @@ <h3>Navigation</h3>
6870
</div>
6971
<div class="footer" role="contentinfo">
7072
&#169; Copyright 2018, Jamie Taylor.
71-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.8.
73+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.5.
7274
</div>
7375
</body>
7476
</html>

‎docs/build/html/_static/basic.css‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
*
55
* Sphinx stylesheet -- basic theme.
66
*
7-
* :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
7+
* :copyright: Copyright 2007-2019 by the Sphinx team, see AUTHORS.
88
* :license: BSD, see LICENSE for details.
99
*
1010
*/
@@ -81,6 +81,10 @@ div.sphinxsidebar input {
8181
font-size: 1em;
8282
}
8383

84+
div.sphinxsidebar #searchbox form.search {
85+
overflow: hidden;
86+
}
87+
8488
div.sphinxsidebar #searchbox input[type="text"] {
8589
float: left;
8690
width: 80%;
@@ -427,6 +431,13 @@ table.field-list td, table.field-list th {
427431
hyphens: manual;
428432
}
429433

434+
/* -- hlist styles ---------------------------------------------------------- */
435+
436+
table.hlist td {
437+
vertical-align: top;
438+
}
439+
440+
430441
/* -- other body styles ----------------------------------------------------- */
431442

432443
ol.arabic {

‎docs/build/html/_static/classic.css‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
*
55
* Sphinx stylesheet -- classic theme.
66
*
7-
* :copyright: Copyright 2007-2018 by the Sphinx team, see AUTHORS.
7+
* :copyright: Copyright 2007-2019 by the Sphinx team, see AUTHORS.
88
* :license: BSD, see LICENSE for details.
99
*
1010
*/

0 commit comments

Comments
 (0)