Fix core bugs and make the app actually work#5
Open
rulingAnts wants to merge 12 commits into
Open
Conversation
- Rewrite XML import for the real Dekereke format: UTF-16 LE decoding (BOM detection for UTF-16 LE/BE and UTF-8) and the phon_data/data_form record structure used by the QWOM data file. The previous parser expected a Wordlist/Entry structure real files don't use and read UTF-16 files as UTF-8, so imports failed entirely. - Fix duplicate/failed inserts: entries were inserted with explicit id:0, breaking AUTOINCREMENT. Ids are now auto-assigned; references have a UNIQUE index (schema v2 + migration); imports run in one transaction; in-file duplicate references are skipped and reported. - Fix compile error: dart:convert has no utf16 codec; export now writes real UTF-16 LE bytes with BOM, preserving all original XML fields and filling Phonetic/SoundFile from collected data (round-trip safe). - Fix export ZIP layout (no more export_temp/ nesting), clean up stale archives, always include consent_log.json. - Elicitation: session resume at first incomplete word, saved data shown when revisiting entries, recording lifecycle handled on save/navigate, wordlist-assigned SoundFile names used for recordings, Indonesian and Tok Pisin glosses displayed, optional picture display. - Add the missing consent screen (MVP ethics requirement) with written + optional verbal consent, persisted consent log, and elicitation gating. - Merge-on-import option preserves collected data on wordlist updates. - Regenerate the Android scaffold (was missing gradle wrapper, resources, launch themes — could not build an APK); minSdk 23 for the recorder. - Add comprehensive test suite (43 tests) incl. UTF-16 handling, schema migration, duplicate prevention, export round-trip, and an integration test against the real QWOM file when present. - Add GitHub Actions CI: analyze, test, and APK artifact builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR upgrades the Flutter app from a non-compiling prototype to a working MVP by implementing the real QWOM/Dekereke XML import/export pipeline (UTF‑16, <phon_data>/<data_form>), fixing SQLite import/ID/duplication problems via schema v2 + migration, and adding consent gating + CI + a comprehensive test suite.
Changes:
- Rewrite XML import/export to support real Dekereke structure, UTF‑16 BOM/sniffing, full-field round-trip, and export ZIP layout fixes.
- Add/upgrade DB schema (unique reference index, migration + dedupe), provider/session-resume logic, recording filename rules, and consent gating UI + persistence.
- Add extensive tests and GitHub Actions CI (analyze/test + APK artifact), plus Android Gradle scaffold regeneration.
Reviewed changes
Copilot reviewed 43 out of 50 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/xml_service_test.dart | Validates real-format Dekereke parsing, field preservation, UTF‑16 decoding/encoding, and export round-trip behavior. |
| test/wordlist_provider_test.dart | Tests provider ordering, resume behavior, navigation bounds, persistence semantics, and clear behavior. |
| test/widget_test.dart | Adds smoke tests for HomeScreen rendering and progress display with seeded DB state. |
| test/real_qwom_data_test.dart | Optional integration test against external real QWOM XML when available via env/path. |
| test/models_test.dart | Extends model tests for null id mapping, XML field round-trip, and recording filename sanitization. |
| test/export_service_test.dart | Verifies ZIP export layout, UTF‑16 XML bytes, consent log inclusion, and stale archive cleanup. |
| test/database_service_test.dart | Covers AUTOINCREMENT/id bug, unique reference enforcement, batch import, merge behavior, and schema migration. |
| test_data/sample_wordlist.xml | Updates sample XML to match real <phon_data>/<data_form> structure. |
| README.md | Updates repo status and points to Flutter docs, changelog, and CI APK artifact workflow. |
| pubspec.yaml | Updates dependency versions and adds sqflite_common_ffi for test support. |
| pubspec.lock | Adds a lockfile for reproducible dependency resolution. |
| lib/services/xml_service.dart | Implements UTF‑16 BOM/sniff decode, real-format parsing with field preservation, import result reporting, and Dekereke export. |
| lib/services/export_service.dart | Fixes export encoding, ZIP layout (flat), consent log always included, and stale export cleanup. |
| lib/services/database_service.dart | Introduces schema v2, unique reference index, migration/dedupe, transactional replace/import, merge-on-import, and app settings/device id. |
| lib/services/audio_service.dart | Adds wordlist-driven recording filenames, consent recording support, safer stop/cancel semantics, and utility existence check. |
| lib/screens/import_screen.dart | Adds replace-vs-merge prompt when prior collected data exists; reports skipped invalid/duplicate counts. |
| lib/screens/home_screen.dart | Gates elicitation route behind consent status. |
| lib/screens/elicitation_screen.dart | Reworks elicitation UI to persist/display per-entry transcription/audio, resume logic, and picture display. |
| lib/screens/consent_screen.dart | Adds consent UI with optional verbal recording and persistence; routes to elicitation on assent. |
| lib/providers/wordlist_provider.dart | Adds resume-at-first-incomplete, safe notify after async, error tracking, and navigation helpers. |
| lib/models/wordlist_entry.dart | Makes id nullable, adds Dekereke field support + XML field preservation JSON, and recording filename sanitization. |
| lib/models/consent_record.dart | Makes id nullable for SQLite auto-assignment and updates mapping accordingly. |
| lib/main.dart | Adjusts supported locales list (adds Indonesian, simplifies Locale constructors). |
| FLUTTER_README.md | Updates XML format docs to reflect real Dekereke structure and current app capabilities. |
| CHANGELOG.md | Documents the major fixes/additions delivered by this PR. |
| android/settings.gradle.kts | Migrates Android settings to Kotlin DSL and Flutter plugin loader. |
| android/settings.gradle | Removes legacy Groovy settings.gradle. |
| android/gradle/wrapper/gradle-wrapper.properties | Adds Gradle wrapper configuration. |
| android/gradle.properties | Updates Gradle JVM args. |
| android/build.gradle.kts | Adds Kotlin DSL top-level Gradle configuration and build dir routing. |
| android/build.gradle | Removes legacy Groovy build.gradle. |
| android/app/src/profile/AndroidManifest.xml | Adds profile manifest with INTERNET permission for Flutter tooling. |
| android/app/src/main/res/values/styles.xml | Adds launch/normal themes required for Flutter embedding v2 scaffolding. |
| android/app/src/main/res/values-night/styles.xml | Adds night-mode equivalents of launch/normal themes. |
| android/app/src/main/res/drawable/launch_background.xml | Adds splash background resource. |
| android/app/src/main/res/drawable-v21/launch_background.xml | Adds v21+ splash background resource. |
| android/app/src/main/kotlin/com/example/wordlist_elicitation/MainActivity.kt | Simplifies MainActivity to expression body. |
| android/app/src/main/AndroidManifest.xml | Adds RECORD_AUDIO permission and updates embedding metadata/queries per modern template. |
| android/app/src/debug/AndroidManifest.xml | Adds debug manifest with INTERNET permission for Flutter tooling. |
| android/app/build.gradle.kts | Adds Kotlin DSL app build config (minSdk 23, Flutter plugin, Java 11). |
| android/app/build.gradle | Removes legacy Groovy app build.gradle. |
| android/.gitignore | Adds Android-specific ignore rules. |
| .metadata | Updates Flutter tool metadata revisions and unmanaged_files list. |
| .gitignore | Stops ignoring pubspec.lock, un-ignores wrapper jar, and adjusts Android Studio/Gradle ignore rules. |
| .github/workflows/ci.yml | Adds CI workflow running analyze/test and building/uploading an APK artifact. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+6
| gradle-wrapper.jar | ||
| /.gradle | ||
| /captures/ | ||
| /gradlew | ||
| /gradlew.bat | ||
| /local.properties |
| @@ -1,3 +1,3 @@ | |||
| org.gradle.jvmargs=-Xmx4G | |||
| org.gradle.jvmargs=-Xmx8G -XX:MaxMetaspaceSize=4G -XX:ReservedCodeCacheSize=512m -XX:+HeapDumpOnOutOfMemoryError | |||
| void dispose() { | ||
| if (_isRecording) { | ||
| // Leaving mid-recording discards the unfinished take. | ||
| widget.audioService.cancelRecording(); |
| @override | ||
| void dispose() { | ||
| if (_isRecording) { | ||
| _audioService.cancelRecording(); |
| 1. wordlist_data.xml - Dekereke XML format with collected transcriptions | ||
| 2. audio/ - Directory with all WAV audio recordings (16-bit) | ||
| 1. wordlist_data.xml - Dekereke XML (UTF-16) with collected transcriptions | ||
| 2. audio/ - WAV audio recordings (16-bit), named <Reference><gloss>.wav |
…rovement - record 5.x ships record_linux 0.7.2, which no longer implements record_platform_interface 1.6.0; the dart plugin registrant compiles it during Android kernel snapshots, breaking `flutter build apk` (caught by CI, not reproducible via analyze/test). - mergeEntries now accepts incoming collected data (transcription, completion) when this device has none, so restoring an exported backup via merge-import works; preserved-data behavior unchanged otherwise. - Doc corrections: minSdk 23 in DEVELOPMENT.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…I build Review-confirmed fixes: - Recording safety (critical): new takes now record into audio/tmp/ and are moved over the archived WAV only on save. Previously re-recording overwrote the saved recording the moment the mic was tapped, and cancel/navigation deleted it permanently. Cancel now only ever discards the temp take; leftover takes are cleaned at screen open; consent recordings are finalized on response. - Double-tap race (major): Save & Next sets _isSaving synchronously and only advances after the DB write succeeds, so a double tap can no longer skip an entry or write one word's data onto the next. - Whitespace corruption (confirmed by new test): the XML pretty-printer collapsed whitespace runs inside text values (IPA transcriptions); export now preserves text content verbatim. - Export robustness: new archives are fully built before old ones are deleted (a failed export can't destroy the last backup) and only WAVs referenced by current entries/consent records are included, keeping recordings from replaced wordlists out of the archive. - Save failures (DB or file move) now surface a snackbar and keep the user on the word instead of silently advancing. Build fix: - file_picker upgraded to 8.x: 6.x uses the removed Flutter v1 embedding API and no longer compiles in Android release builds (caught by CI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ft v1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eous scope note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…can't keep columns hidden/read-only) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/HANDOFF.md carries everything a repo-only cloud session needs: verified Dekereke ground truth, FlexText patterns to reuse, decision log, P0 checklist (VM-only items), and a prioritized backlog starting with packages/dekereke_core. Fixtures are synthetic but format-faithful UTF-16 LE Dekereke files (duplicate/missing References, nested fragments, boolean tags, multi-file cells, irregular suffixes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eth answers §7 in-session Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.claude/settings.json pins model=claude-fable-5, sets an allowlist of only that model (enforced), declares an empty fallback list, and disables switch-on-flag — so sessions stop on limits rather than downgrading. HANDOFF.md adds the agent-side backstop instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR takes the initial Flutter implementation from non-compiling to a working, tested app that handles the real QWOM/Dekereke data format.
Critical fixes
export_service.dartreferenced a nonexistentutf16codec, andpubspec.yamlcould not resolve against current Flutter (intl pin conflict).<Wordlist>/<Entry>structure, but the real QWOM file uses<phon_data>/<data_form>records in UTF-16 LE — which the old code read as UTF-8. The importer now decodes UTF-16 LE/BE/UTF-8 (BOM + sniffing) and parses the real record structure, preserving all fields for round-trip export.id: 0, breaking SQLite AUTOINCREMENT. Ids are now auto-assigned, references are UNIQUE at the schema level (v2 migration dedupes existing data), imports run in a single transaction, and in-file duplicates are skipped and reported.Correctness & UX fixes
<Phonetic>from the collected transcription and<SoundFile>from the recording, and round-trips every original field — verified against the full 987-entry QWOM file.export_temp/; stale archives are cleaned up;consent_log.jsonis always included.<SoundFile>filename (e.g.0002skin.wavfor "skin (human)").<documents>/pictures/.Added
Test plan
flutter analyzecleanflutter test— 43/43 passing locally (including integration test against the real QWOM2025-08.xml)🤖 Generated with Claude Code