Skip to content

Commit fe6d3d1

Browse files
nsheffvreuterdonaldcampbelljr
authored
Sqlmodel (#36)
* improve error message when namespace can't be found; apply formatter; close #3 * only call helper when it's needed * tweak main object docstring for better CLI help message * restructure parser creation logic * better isolation of branches of conditional expression * cleaner setup of PSM config * be more explicit that a test expectation is simply to not crash * better test names and description * better validation sequence of PSM ctor args; #3 * add exception subtype and test to solve #3 * export PipestatError at package namespace level * simplify / condense sytnax * collect more errors, avoid repetition * tweak message * add log info for reading the schema * collect all reserved word violations * first-pass at handling looper-like structure for schema; #20 * simplify syntax by using default default arg * update test case for more specific error type; #3 * use name directly * tweak implementation and fix syntax * introduce sqlmodel * simplify implementation * simplify implementation * temp, amid refactor to use SQLModel * update schemas in test/sample files * start testing ParsedSchema * wave of parsed schema tests * fix import path * add more parsed schema testing * more parsed schema testing * finish happy path ParsedSchema tests * repairing old tests * using pydantic model to avoid SQLModel collection / list type restriction It appears that SQLModel doesn't yet support collection types. More generally, there appears to be deliberate, or at least known, lack of support for types for which there's no matching SQLAlchemy type. See 3rd feature from 0.0.7 release: https://sqlmodel.tiangolo.com/release-notes/#features See related PR: fastapi/sqlmodel#18 * pausing on sample-level handling; splitting off files from the scalars * collecting ORMs * add record identifier * WORKING EXAMPLE with totally broken logic, but functional machinery * include record identifier, and extend table models use __table_args__={'extend_existing': True} to extend table models, such that subsequent calls to create_model aren't problematic. * update expected test failure reason syntax, and constructor signature * better test names * remove ABC tag from PipestatError since we instantiate it in fact * fix missing schema test * update results file / namespace test * remove connection tests * fix results file 1:1 logic with namespace * fix test expectation * more test updates * more namespace removal * remove debugging; temp * better parametrization scheme * update what to ignore based on what can be left by interrupted tests * update schema format based on merge of status and results * start updating access to status table * apply formatter * relax constraint for data type for status subschema * add primary key for status table * add record identifier to status table * remove unnecessary test nesting * remove now-outdated reference to 'self' in what were member functions * better organization and syntax in status test suite * restructuring model builds and tests * require project or sample data in schema * add formatter to test reqs * nulls rather than empty mappings; update schema structures * reorganizing and cleaning up constants * test for requirement of pipeline identifier * add test for extra schema keys * condense validation of schema, add tests stubs * finishing overhaul of constants and minimization, finishing parsed schema tests and stubs for integration with PSM * ironing out more read_yaml_data bugs, testing * amid updating signatures to account for project / samples disambiguation * Remove namespace and status_schema from PipestatManager creation to align with Class changes. * Import constants from argparser for use in cli.py. * Change psm.schema -> psm.schema.results_data due to changes in parsing schemas. * Apply formatting. * Add const import for ENV_VARS * Add ability to determine project_level flag from config.yaml.Allow project_level to default to False and modified logic during table name acquisition to get either table_name__sample or table_name__project based on project_level flag. Apply formatting. * Simplified 2 tests within pytest. Apply formatting. * Resolve pytest failure: TestPipestatManagerInstantiation.test_missing_cfg_data * Add force_overwrite=True for reporting test. * Add "array" type to CLASSES_BY_TYPE to prevent failure during complex model creation. #40 * Apply _recursively_replace_custom_types to sample data to parse more complex objects. #40 * Add pytest for complex objects. #40 * Clean up pytest for complex/nested objects. Convert nested objects (dicts) to strings before pushing to db columns. #40 * Capture 'namespace' from config file and add it as a separate column in the sample table. #41 * Add database context manager such that the db tables are cleared between tests. #38 * Add comment to context manager class. #38 * Fix removal test. Check if db mode is false AND if results_file.yaml is supplied. #39 * Formatting. #39 * Allow for storing dict as JSONB. Implemented filtering of data within JSONB column using sqlmodel. #40 * Fix bug with context manager in pytests for DB connection. #38 * Added ability to parse collection of images as array[dict] and store as JSONB. #40 * Import contextmanager class into test_status.py to prevent pytest failures when testing db backend. #38 * Fix failures in TestNoRecordID. #38 * Fix runtime warning due to __table_args__. #49 * Apply formatting. #49 * Fixed issue with setting self.project_level and added pytest to report both sample and project level data. #37 * Refactor pytest constants. * Refactor test_report_samples_and_project. * Create deepcopy of values before reporting. #50 * Only import deepcopy function. #50 * Re-implement record_count. #44 * Re-implement highlighted_results and associated pytest. #45 * Move helper function into helpers file * lint * move select_value to helpers * lint * move constants to const See #46 * refactor and simplify PSM constructor * Implement project and sample level disambiguation and fix associated PyTests #51 * move select_value method to yacman * Reconcile refactoring changes with project_level disambiguation. #51 * Refactor project_level check into function. #51 * Apply formatting. * updates on pipeline type * Fix results_file.yaml case issue. * increase black line length to 99 * black formatting * Re-implement test_str_representation. #38 * simplify constructor further * Decouple tests and ensure results_file.yaml is not persistent between tests. #38 * Implement xFail tests from test_init.py. #38 * Implement test_custom_status_schema #38 * Split backend classes (#55) * Incomplete work toward splitting backend into classes * go back... * revert more * revert more * redo changes * backend work * file reporting works * file backend work * Fix typing import for python 3.10 #52 * Partial implementation of remove for file backend. #52 * Add check_which_results_exist and check_results_exist to file backend. #52 * Move check_results_exist to abstract class. #52 * Implement check_record_exists in file backend. #52 * Implement get_status and set_status in file backend. #52 * Implement clear_status in file backend. #52 * partial DBBackend report implementation. * continue DBBackend report implementation. * Implement retrieve for DBBackend. #52 * Implement removal for DBBackend. #52 * refactor check_record_exists and add to interface * refactor check_which_results_exist to list_existing_results * finish implementing remove for DB backend #52 * Add option to return all results for list_existing_results DB Backend * Add option to return all results for list_existing_results File Backend * consolidate report and report_db in DB Backend * create remove_record functions for file and DB backends * Add basic implementation for Pytesting new split classes. * remove unused decorator * simplify list results func * clarify docstring * rename to list_results * rename some variables * fix priority vars for check-result_exists * lint * simplify list_results for DB Backend by using list comprehension * extend basic testing for split_backend, fix remove_record to update local file * clean up report in pipestat manager, fix related pytests * clean up remove and retrieve * add pytest condition for DB backend * Implement select function #52 * Implement select_txt function #52 * ReImplement record_count function * Implement set_status * Implement get_status * Remove redundant code from pipestat.py * move select_distinct to DB backend. * remove redundant functions, clean up docstrings * add raising exception if no backend specified * Correct doc strings. * remove ".new" usage. * add PROJECT_NAME as constant * Clean up pipestat context managers. * lint * add some type hints. * implement require backend decorator * no need for intermediate variable * Change self[DATA_KEY] to self.data and align both parent and child classes to use attribute instead of attribute or item. * fix pytest assertion * replace status" with constant * Change backend data attribute to _data * Fix pipeline_type/table_name priority * Remove select functions from pipestatmanager parent class * Move initializing and loading results file to FileBackend. * Refactor namespace and project name to pipeline_name. * Refactor pipeline_id to pipeline_name, remove namespace property. * Organize functions * unify some properties retrievals * standardize retrieve function. Fix #58. * simplify field definitions, refactor namespace to project_name * use const in field definitions * Remove commented, unused code from backend * Add and edit docstrings * add some tests * move orm and engine creation to DB backend. * Split backend files for readability. * Clean up redundant importing, re-organize functions and properties --------- Co-authored-by: nsheff <nsheff@users.noreply.github.com> --------- Co-authored-by: Vince Reuter <vince.reuter@gmail.com> Co-authored-by: Donald C <125581724+donaldcampbelljr@users.noreply.github.com>
1 parent 67051cd commit fe6d3d1

40 files changed

Lines changed: 3569 additions & 2130 deletions

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@ coverage.xml
5757
*.py,cover
5858
.hypothesis/
5959
.pytest_cache/
60+
tests/data/*.flag
61+
*.db
6062

6163
# Translations
6264
*.mo

docs/autodoc_build/pipestat.md

Lines changed: 31 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -29,25 +29,26 @@ h4 .content {
2929

3030
# Package `pipestat` Documentation
3131

32+
## <a name="PipestatError"></a> Class `PipestatError`
33+
Base exception type for this package
34+
35+
3236
## <a name="PipestatManager"></a> Class `PipestatManager`
33-
Pipestat standardizes reporting of pipeline results and pipeline status management. It formalizes a way for pipeline developers and downstream tools developers to communicate -- results produced by a pipeline can easily and reliably become an input for downstream analyses. The object exposes API for interacting with the results and pipeline status and can be backed by either a YAML-formatted file or a database.
37+
Pipestat standardizes reporting of pipeline results and pipeline status management. It formalizes a way for pipeline developers and downstream tools developers to communicate -- results produced by a pipeline can easily and reliably become an input for downstream analyses. A PipestatManager object exposes an API for interacting with the results and pipeline status and can be backed by either a YAML-formatted file or a database.
3438

3539

3640
```python
37-
def __init__(self, namespace: Union[str, NoneType]=None, record_identifier: Union[str, NoneType]=None, schema_path: Union[str, NoneType]=None, results_file_path: Union[str, NoneType]=None, database_only: Union[bool, NoneType]=True, config: Union[str, dict, NoneType]=None, status_schema_path: Union[str, NoneType]=None, flag_file_dir: Union[str, NoneType]=None, custom_declarative_base: Union[sqlalchemy.orm.decl_api.DeclarativeMeta, NoneType]=None, show_db_logs: bool=False)
41+
def __init__(self, record_identifier: Optional[str] = None, schema_path: Optional[str] = None, results_file_path: Optional[str] = None, database_only: Optional[bool] = True, config: Union[str, dict, NoneType] = None, flag_file_dir: Optional[str] = None, show_db_logs: bool = False, project_level: Optional[bool] = False)
3842
```
3943

4044
Initialize the object
4145
#### Parameters:
4246

43-
- `namespace` (`str`): namespace to report into. This will be the DBtable name if using DB as the object back-end
44-
- `record_identifier` (`str`): record identifier to report for. Thiscreates a weak bound to the record, which can be overriden in this object method calls
47+
- `record_identifier` (`str`): record identifier to report for. Thiscreates a weak bound to the record, which can be overridden in this object method calls
4548
- `schema_path` (`str`): path to the output schema that formalizesthe results structure
4649
- `results_file_path` (`str`): YAML file to report into, if file isused as the object back-end
4750
- `database_only` (`bool`): whether the reported data should not bestored in the memory, but only in the database
4851
- `config` (`str | dict`): path to the configuration file or a mappingwith the config file content
49-
- `status_schema_path` (`str`): path to the status schema that formalizesthe status flags structure
50-
- `custom_declarative_base` (`sqlalchemy.orm.DeclarativeMeta`): a declarative base touse for ORMs creation a new instance will be created if not provided
5152

5253

5354

@@ -70,7 +71,7 @@ Assert provided list of results is defined in the schema
7071

7172

7273
```python
73-
def check_record_exists(self, record_identifier: str, table_name: str=None) -> bool
74+
def check_record_exists(self, record_identifier: str, table_name: str) -> bool
7475
```
7576

7677
Check if the specified record exists in the table
@@ -88,7 +89,7 @@ Check if the specified record exists in the table
8889

8990

9091
```python
91-
def check_result_exists(self, result_identifier: str, record_identifier: str=None) -> bool
92+
def check_result_exists(self, result_identifier: str, record_identifier: str = None) -> bool
9293
```
9394

9495
Check if the result has been reported
@@ -106,25 +107,26 @@ Check if the result has been reported
106107

107108

108109
```python
109-
def check_which_results_exist(self, results: List[str], rid: Union[str, NoneType]=None, table_name: Union[str, NoneType]=None) -> List[str]
110+
def check_which_results_exist(self, results: List[str], rid: Optional[str] = None, table_name: Optional[str] = None) -> List[str]
110111
```
111112

112113
Check which results have been reported
113114
#### Parameters:
114115

115-
- `rid` (`str`): unique identifier of the record
116116
- `results` (`List[str]`): names of the results to check
117+
- `rid` (`str`): unique identifier of the record
118+
- `table_name` (`str`): name of the table for which to check results
117119

118120

119121
#### Returns:
120122

121-
- `List[str]`: whether the specified result has been reported for theindicated record in current namespace
123+
- `List[str]`: names of results which exist
122124

123125

124126

125127

126128
```python
127-
def clear_status(self, record_identifier: str=None, flag_names: List[str]=None) -> List[Union[str, NoneType]]
129+
def clear_status(self, record_identifier: str = None, flag_names: List[str] = None) -> List[Optional[str]]
128130
```
129131

130132
Remove status flags
@@ -160,7 +162,7 @@ def data(self)
160162
Data object
161163
#### Returns:
162164

163-
- `yacman.YacAttMap`: the object that stores the reported data
165+
- `yacman.YAMLConfigManager`: the object that stores the reported data
164166

165167

166168

@@ -177,20 +179,6 @@ Database column key word arguments for every result, sourced from the results sc
177179

178180

179181

180-
```python
181-
def db_column_relationships_by_result(self)
182-
```
183-
184-
Database column relationships for every result, sourced from the results schema in the `relationship` section
185-
186-
*Note: this is an experimental feature*
187-
#### Returns:
188-
189-
- `Dict[str, Dict[str, str]]`: relationships for every result
190-
191-
192-
193-
194182
```python
195183
def db_url(self)
196184
```
@@ -209,31 +197,25 @@ Database URL, generated based on config credentials
209197

210198

211199
```python
212-
def establish_db_connection(self) -> bool
200+
def file(self)
213201
```
214202

215-
Establish DB connection using the config data
203+
File path that the object is reporting the results into
216204
#### Returns:
217205

218-
- `bool`: whether the connection has been established successfully
206+
- `str`: file path that the object is reporting the results into
219207

220208

221209

222210

223211
```python
224-
def file(self)
212+
def get_one_record(self, table_name: str, rid: Optional[str] = None)
225213
```
226214

227-
File path that the object is reporting the results into
228-
#### Returns:
229-
230-
- `str`: file path that the object is reporting the results into
231-
232-
233215

234216

235217
```python
236-
def get_orm(self, table_name: str=None) -> Any
218+
def get_orm(self, table_name: str) -> Any
237219
```
238220

239221
Get an object relational mapper class
@@ -250,7 +232,7 @@ Get an object relational mapper class
250232

251233

252234
```python
253-
def get_status(self, record_identifier: str=None) -> Union[str, NoneType]
235+
def get_status(self, record_identifier: str = None) -> Optional[str]
254236
```
255237

256238
Get the current pipeline status
@@ -291,18 +273,6 @@ Highlighted results
291273

292274

293275

294-
```python
295-
def is_db_connected(self) -> bool
296-
```
297-
298-
Check whether a DB connection has been established
299-
#### Returns:
300-
301-
- `bool`: whether the connection has been established
302-
303-
304-
305-
306276
```python
307277
def namespace(self)
308278
```
@@ -340,7 +310,7 @@ Unique identifier of the record
340310

341311

342312
```python
343-
def remove(self, record_identifier: str=None, result_identifier: str=None) -> bool
313+
def remove(self, record_identifier: str = None, result_identifier: str = None) -> bool
344314
```
345315

346316
Remove a result.
@@ -361,7 +331,7 @@ will be removed.
361331

362332

363333
```python
364-
def report(self, values: Dict[str, Any], record_identifier: str=None, force_overwrite: bool=False, strict_type: bool=True, return_id: bool=False) -> Union[bool, int]
334+
def report(self, values: Dict[str, Any], record_identifier: str = None, force_overwrite: bool = False, strict_type: bool = True, return_id: bool = False, project_level: Optional[bool] = None) -> Union[bool, int]
365335
```
366336

367337
Report a result.
@@ -372,6 +342,7 @@ Report a result.
372342
- `force_overwrite` (`bool`): whether to overwrite the existing record
373343
- `strict_type` (`bool`): whether the type of the reported values shouldremain as is. Pipestat would attempt to convert to the schema-defined one otherwise
374344
- `return_id` (`bool`): PostgreSQL IDs of the records that have beenupdated. Not available with results file as backend
345+
- `project_level` (``): whether what's being reported pertains to project-level,rather than sample-level, attribute(s)
375346

376347

377348
#### Returns:
@@ -394,7 +365,7 @@ Result schema mappings
394365

395366

396367
```python
397-
def retrieve(self, record_identifier: Union[str, NoneType]=None, result_identifier: Union[str, NoneType]=None) -> Union[Any, Dict[str, Any]]
368+
def retrieve(self, record_identifier: Optional[str] = None, result_identifier: Optional[str] = None) -> Union[Any, Dict[str, Any]]
398369
```
399370

400371
Retrieve a result for a record.
@@ -439,7 +410,7 @@ Schema path
439410

440411

441412
```python
442-
def select(self, table_name: Union[str, NoneType]=None, columns: Union[List[str], NoneType]=None, filter_conditions: Union[List[Tuple[str, str, Union[str, List[str]]]], NoneType]=None, json_filter_conditions: Union[List[Tuple[str, str, str]], NoneType]=None, offset: Union[int, NoneType]=None, limit: Union[int, NoneType]=None) -> List[Any]
413+
def select(self, table_name: Optional[str] = None, columns: Optional[List[str]] = None, filter_conditions: Optional[List[Tuple[str, str, Union[str, List[str]]]]] = None, json_filter_conditions: Optional[List[Tuple[str, str, str]]] = None, offset: Optional[int] = None, limit: Optional[int] = None, project_level: Optional[bool] = None) -> List[Any]
443414
```
444415

445416
Perform a `SELECT` on the table
@@ -469,7 +440,7 @@ Perform a `SELECT DISTINCT` on given table and column
469440

470441

471442
```python
472-
def select_txt(self, columns: Union[List[str], NoneType]=None, filter_templ: Union[str, NoneType]='', filter_params: Union[Dict[str, Any], NoneType]={}, table_name: Union[str, NoneType]=None, offset: Union[int, NoneType]=None, limit: Union[int, NoneType]=None) -> List[Any]
443+
def select_txt(self, columns: Optional[List[str]] = None, filter_templ: Optional[str] = '', filter_params: Optional[Dict[str, Any]] = {}, table_name: Optional[str] = None, offset: Optional[int] = None, limit: Optional[int] = None) -> List[Any]
473444
```
474445

475446
Execute a query with a textual filter. Returns all results.
@@ -501,18 +472,19 @@ Provide a transactional scope around a series of query operations.
501472

502473

503474
```python
504-
def set_status(self, status_identifier: str, record_identifier: str=None) -> None
475+
def set_status(self, status_identifier: str, record_identifier: str = None, project_level: Optional[bool] = None) -> None
505476
```
506477

507478
Set pipeline run status.
508479

509480
The status identifier needs to match one of identifiers specified in
510481
the status schema. A basic, ready to use, status schema is shipped with
511-
this package.
482+
this package.
512483
#### Parameters:
513484

514485
- `status_identifier` (`str`): status to set, one of statuses definedin the status schema
515486
- `record_identifier` (`str`): record identifier to set thepipeline status for
487+
- `project_level` (`bool`): whether status is being set for a project-level pipeline, or sample-level
516488

517489

518490

@@ -556,4 +528,4 @@ Check schema for any possible issues
556528

557529

558530

559-
*Version Information: `pipestat` v0.3.0, generated by `lucidoc` v0.4.3*
531+
*Version Information: `pipestat` v0.3.1, generated by `lucidoc` v0.4.4*

pipestat/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,10 @@
44

55
from ._version import __version__
66
from .helpers import *
7+
from .exceptions import PipestatError
78
from .pipestat import *
89

9-
__classes__ = ["PipestatManager"]
10+
__classes__ = ["PipestatError", "PipestatManager"]
1011
__all__ = __classes__
1112

1213
logmuse.init_logger(PKG_NAME)

0 commit comments

Comments
 (0)