Add Tier 4 export improvements and Tier 5 metadata/annotations

Tier 4 (export/roundtrip):
- Export segments as source-level where: clauses
- Export renames as rename: (detect simple identifier dimensions)
- Export full join on conditions from relationship metadata
- join_cross: already handled in prior commit

Tier 5 (metadata/annotations):
- Non-description tags (# line_chart, # percent, etc.) stored in
  dimension/measure/model metadata["tags"]
- #@ persist annotations stored in Model.metadata["persist"]
- timezone: statements stored in Model.metadata["timezone"]
- Standalone # annotations in extend blocks stored as model tags
- declare: field declarations processed as dimensions in + syntax
- _parse_annotations_full returns both description and tag list

Author: nicosuave

docs/compatibility/malloy.md

@@ -115,10 +115,11 @@ Not mapped: `access` modifiers (`public`, `private`, `internal`), `order_by:` wi
 | `# description: value` tag annotation | Supported (extracted as `description`) |
 | Multiple `##` lines on one entity | Supported (joined with spaces) |
 | Statement-level `#` tags (before `source:`) | Supported (applied as source description if the source itself has none) |
-| `# tag_name` (non-description tags) | Partial support: parsed without error but only `desc:` and `description:` prefixed tags are extracted. Other tags are discarded. |
-| `#@ persist` and `#@ persist name=...` | Unsupported (parsed by the grammar but not recognized by the visitor). |
+| `# tag_name` (non-description tags) | Supported (stored in `metadata["tags"]` on dimensions, measures, and models; includes `line_chart`, `bar_chart`, `percent`, `currency`, etc.) |
+| `#@ persist` and `#@ persist name=...` | Supported (stored in `Model.metadata["persist"]` and `metadata["persist_name"]`) |
+| Standalone `#` annotations in extend blocks | Supported (stored in `Model.metadata["tags"]` via `DefExploreAnnotationContext`) |
 
-Not mapped: visualization hint tags (`# line_chart`, `# bar_chart`, `# list_detail`, `# shape_map`, `# percent`, `# currency`, `# number`), `--! styles` directives, `##! experimental` pragmas.
+Not mapped: `--! styles` directives, `##! experimental` pragmas.
 
 ---
 
@@ -163,8 +164,8 @@ Not mapped: join `type` (`left`, `right`, `full`, `inner`).
 | `where: condition` in source extend block | Supported (mapped to `Segment`) |
 | Multiple filter conditions (comma-separated) | Supported (each becomes a separate segment) |
 | Filter expressions with comparisons, `and`, `or` | Supported (expression preserved as-is) |
-| Malloy partial application (`field ? pick ... when ...`) | Partial support: expression text is captured but the `?` operator is not evaluated. |
-| Malloy value matching (`field ? 'a' \| 'b'`) | Partial support: expression preserved as-is, not converted to SQL `IN`. |
+| Malloy partial application (`field ? pick ... when ...`) | Supported in dimension context (expanded to CASE); partial in filter context |
+| Malloy value matching (`field ? 'a' \| 'b'`) | Supported (transformed to `field IN ('a', 'b')`) |
 
 Segment naming: first filter is named `default_filter`, subsequent filters are named `default_filter_1`, `default_filter_2`, etc.
 
@@ -300,16 +301,18 @@ Sidemantic can export its semantic model back to Malloy format.
 | Ratio metrics | Supported (exported as `numerator / denominator`) |
 | `primary_key:` | Supported (exported when not the default `id`) |
 | `join_one:` / `join_many:` with `with` clause | Supported |
+| `join_one:` / `join_many:` with `on` condition | Supported (full `on` condition exported from `metadata["on_condition"]` when available) |
+| `where:` (segments) | Supported (source-level where clauses exported) |
 | Roundtrip fidelity (parse -> export -> re-parse) | Supported (semantically equivalent graphs; passthrough dimensions intentionally dropped) |
 | `join_cross:` export | Supported (one_to_one relationships exported as `join_cross:`) |
-| `rename:` export | Partial support: renames are captured as dimensions during parsing; exported as `dimension:` not `rename:` |
+| `rename:` export | Supported (simple identifier dimensions detected and exported as `rename: new is old`) |
 | `view:` export | Unsupported (views are not captured during parsing) |
 
 ---
 
 ## Experimental and Advanced Features
 
-Unsupported. `##! experimental{...}` pragma annotations, `compose()` for composite sources, `timezone:` statements, `sample:` specifications, and `declare:` field declarations are all parsed by the grammar without error but not processed by the visitor.
+Partially supported. `timezone:` statements are stored in `Model.metadata["timezone"]`. `declare:` field declarations are processed as dimensions in old `+` syntax blocks. `compose()` sources process the first composed source. `##! experimental{...}` pragma annotations and `sample:` specifications are parsed by the grammar without error but not processed by the visitor.
 
 ---
 

sidemantic/adapters/malloy.py

@@ -60,42 +60,46 @@ def _reset_current(self):
         self.current_metrics = []
         self.current_relationships = []
         self.current_segments = []
+        self._timezone = None
+        self._model_tags = []
+        self._accept_fields = []
+        self._except_fields = []
 
     def _parse_annotations(self, tags_ctx) -> str | None:
-        """Parse annotations from tags context.
+        """Parse annotations from tags context, returning description text.
 
-        Malloy annotations:
-        - ## Description text -> description (DOC_ANNOTATION)
-        - # key: value -> metadata (ANNOTATION)
+        Also stores non-description tags via _parse_annotations_full.
+        """
+        if tags_ctx is None:
+            return None
+        desc, _ = self._parse_annotations_full(tags_ctx)
+        return desc
 
-        We extract:
-        - Any ## text as description
-        - # desc: value as description
+    def _parse_annotations_full(self, tags_ctx) -> tuple[str | None, list[str]]:
+        """Parse annotations from tags context.
 
-        Returns the description text if found.
+        Returns (description, tags) where tags is a list of non-description
+        tag strings like "line_chart", "percent", "currency", etc.
         """
         if tags_ctx is None:
-            return None
+            return None, []
 
         descriptions = []
+        tags = []
 
-        # Iterate through ANNOTATION tokens
         for i in range(tags_ctx.getChildCount()):
             child = tags_ctx.getChild(i)
             if child is not None:
                 text = child.getText()
 
                 # ## is a doc annotation (description)
                 if text.startswith("##"):
-                    # Strip ## and whitespace
                     desc = text[2:].strip()
                     if desc:
                         descriptions.append(desc)
                 # # is a tag annotation
                 elif text.startswith("#"):
-                    # Strip # and check for desc: or description:
                     tag_text = text[1:].strip()
-                    # Common patterns: desc: value, description: value
                     if tag_text.lower().startswith("desc:"):
                         desc = tag_text[5:].strip()
                         if desc:
@@ -104,10 +108,11 @@ def _parse_annotations(self, tags_ctx) -> str | None:
                         desc = tag_text[12:].strip()
                         if desc:
                             descriptions.append(desc)
+                    elif tag_text:
+                        tags.append(tag_text)
 
-        if descriptions:
-            return " ".join(descriptions)
-        return None
+        desc = " ".join(descriptions) if descriptions else None
+        return desc, tags
 
     def visitImportStatement(self, ctx: MalloyParser.ImportStatementContext):  # noqa: N802
         """Visit import statement and extract dependencies.
@@ -499,7 +504,20 @@ def visitDefineSourceStatement(self, ctx: MalloyParser.DefineSourceStatementCont
         # These apply to all sources in the statement if there's only one,
         # or can be overridden by source-specific tags
         stmt_tags = ctx.tags()
-        stmt_description = self._parse_annotations(stmt_tags) if stmt_tags else None
+        stmt_description = None
+        stmt_persist = None
+        if stmt_tags:
+            stmt_description, stmt_tag_list = self._parse_annotations_full(stmt_tags)
+            # Check for #@ persist annotations
+            for tag in stmt_tag_list:
+                if tag.startswith("@ persist") or tag.startswith("@persist"):
+                    persist_text = tag[len("@ persist") :] if tag.startswith("@ persist") else tag[len("@persist") :]
+                    persist_text = persist_text.strip()
+                    stmt_persist = {"persist": True}
+                    # Parse name=value
+                    name_match = re.match(r"name\s*=\s*(\S+)", persist_text)
+                    if name_match:
+                        stmt_persist["persist_name"] = name_match.group(1)
 
         # Get source definitions
         source_list = ctx.sourcePropertyList()
@@ -510,14 +528,19 @@ def visitDefineSourceStatement(self, ctx: MalloyParser.DefineSourceStatementCont
                 self._process_source_definition(source_def)
 
                 # If source has no description but statement does, use statement description
-                # (only for single-source statements, or as fallback)
                 if self.current_description is None and stmt_description is not None:
                     self.current_description = stmt_description
 
                 if self.current_model_name:
                     metadata = {}
                     if self.current_connection:
                         metadata["connection"] = self.current_connection
+                    if stmt_persist:
+                        metadata.update(stmt_persist)
+                    if self._timezone:
+                        metadata["timezone"] = self._timezone
+                    if self._model_tags:
+                        metadata["tags"] = self._model_tags
                     model = Model(
                         name=self.current_model_name,
                         table=self.current_table,
@@ -715,6 +738,12 @@ def _process_query_properties_as_explore(self, ctx):
                 where_stmt = stmt.whereStatement()
                 if where_stmt:
                     self._process_where_as_segment(where_stmt)
+            elif isinstance(stmt, MalloyParser.DeclareStatementContext):
+                # declare: creates fields accessible within the source
+                def_list = stmt.defList()
+                if def_list:
+                    for field_def in def_list.fieldDef():
+                        self._process_dimension_def(field_def)
 
     def _process_explore_statement(self, ctx: MalloyParser.ExploreStatementContext):
         """Process a single statement in explore properties."""
@@ -775,6 +804,34 @@ def _process_explore_statement(self, ctx: MalloyParser.ExploreStatementContext):
                     self._accept_fields.extend(field_names)
             return
 
+        # Timezone statement: timezone: 'US/Pacific'
+        if isinstance(ctx, MalloyParser.DefExploreTimezoneContext):
+            tz_stmt = ctx.timezoneStatement()
+            if tz_stmt:
+                tz_string = tz_stmt.string()
+                if tz_string:
+                    tz_value = self._extract_string(self._get_text(tz_string))
+                    if not hasattr(self, "_timezone"):
+                        self._timezone = None
+                    self._timezone = tz_value
+            return
+
+        # Standalone annotations in extend blocks
+        if isinstance(ctx, MalloyParser.DefExploreAnnotationContext):
+            # These are # tag annotations not attached to a field
+            # Store as model-level tags
+            for i in range(ctx.getChildCount()):
+                child = ctx.getChild(i)
+                if child is not None:
+                    text = child.getText()
+                    if text.startswith("#"):
+                        tag_text = text[1:].strip()
+                        if tag_text:
+                            if not hasattr(self, "_model_tags"):
+                                self._model_tags = []
+                            self._model_tags.append(tag_text)
+            return
+
         # Rename statements: rename: new_name is old_name
         if isinstance(ctx, MalloyParser.DefExploreRenameContext):
             rename_list = ctx.renameList()
@@ -812,8 +869,13 @@ def _process_dimension_def(self, ctx: MalloyParser.FieldDefContext):
         name = self._get_text(name_def)
 
         # Get annotations from tags
-        tags = ctx.tags()
-        description = self._parse_annotations(tags) if tags else None
+        tags_ctx = ctx.tags()
+        description = None
+        dim_metadata = None
+        if tags_ctx:
+            description, tag_list = self._parse_annotations_full(tags_ctx)
+            if tag_list:
+                dim_metadata = {"tags": tag_list}
 
         # Get the expression
         field_expr = ctx.fieldExpr()
@@ -848,6 +910,7 @@ def _process_dimension_def(self, ctx: MalloyParser.FieldDefContext):
                 sql=sql,
                 granularity=granularity,
                 description=description,
+                metadata=dim_metadata,
             )
         )
 
@@ -869,8 +932,13 @@ def _process_measure_def(self, ctx: MalloyParser.FieldDefContext):
         name = self._get_text(name_def)
 
         # Get annotations from tags
-        tags = ctx.tags()
-        description = self._parse_annotations(tags) if tags else None
+        tags_ctx = ctx.tags()
+        description = None
+        measure_tags = None
+        if tags_ctx:
+            description, tag_list = self._parse_annotations_full(tags_ctx)
+            if tag_list:
+                measure_tags = tag_list
 
         # Get the expression
         field_expr = ctx.fieldExpr()
@@ -942,9 +1010,11 @@ def _process_measure_def(self, ctx: MalloyParser.FieldDefContext):
             else:
                 metric_type = "derived"
 
-        metric_metadata = None
+        metric_metadata = {}
         if measure_granularity:
-            metric_metadata = {"granularity": measure_granularity}
+            metric_metadata["granularity"] = measure_granularity
+        if measure_tags:
+            metric_metadata["tags"] = measure_tags
 
         self.current_metrics.append(
             Metric(
@@ -954,7 +1024,7 @@ def _process_measure_def(self, ctx: MalloyParser.FieldDefContext):
                 sql=sql,
                 filters=filters,
                 description=description,
-                metadata=metric_metadata,
+                metadata=metric_metadata if metric_metadata else None,
             )
         )
 
@@ -1314,7 +1384,7 @@ def _export_source(self, model: Model) -> list[str]:
         """
         lines = []
 
-        # Model description as annotation
+        # Model description as tag annotation
         if model.description:
             lines.append(f"# desc: {model.description}")
 
@@ -1331,10 +1401,13 @@ def _export_source(self, model: Model) -> list[str]:
         if model.primary_key and model.primary_key != "id":
             lines.append(f"  primary_key: {model.primary_key}")
 
-        # Dimensions
-        # Skip dimensions that match the primary key (Malloy auto-exposes the PK column).
-        # Skip passthrough dimensions (sql == name) since Malloy auto-exposes underlying
-        # table columns. Only export dimensions with actual transformations.
+        # Segments (source-level where clauses) - Tier 4.1
+        if model.segments:
+            for segment in model.segments:
+                lines.append(f"  where: {self._strip_model_prefix(segment.sql)}")
+
+        # Separate renames from computed dimensions for proper export
+        renames_to_export: list[tuple[str, str]] = []
         dims_to_export: list[tuple[Dimension, str]] = []
         for dim in model.dimensions:
             if dim.name == model.primary_key:
@@ -1343,23 +1416,32 @@ def _export_source(self, model: Model) -> list[str]:
             # Skip passthrough dimensions - Malloy auto-exposes table columns
             if sql == dim.name:
                 continue
-            dims_to_export.append((dim, sql))
+            # Tier 4.5: detect renames (simple identifier, no operators/functions)
+            if re.match(r"^[`\w]+$", sql) and sql != dim.name:
+                renames_to_export.append((dim.name, sql))
+            else:
+                dims_to_export.append((dim, sql))
+
+        # Export renames
+        if renames_to_export:
+            lines.append("")
+            lines.append("  rename:")
+            for new_name, old_name in renames_to_export:
+                lines.append(f"    {new_name} is {old_name}")
 
+        # Export computed dimensions
         if dims_to_export:
             lines.append("")
             lines.append("  dimension:")
             for dim, sql in dims_to_export:
                 if dim.description:
                     lines.append(f"    # desc: {dim.description}")
-                # For time dimensions with granularity, use Malloy's time truncation syntax
-                # But only if the SQL doesn't already contain a truncation function
                 if dim.type == "time" and dim.granularity:
                     sql_lower = sql.lower()
                     already_has_truncation = (
                         "date_trunc" in sql_lower or "::date" in sql_lower or sql_lower.endswith(f".{dim.granularity}")
                     )
                     if not already_has_truncation:
-                        # Append Malloy time accessor: .second, .minute, .hour, .day, .week, .month, .quarter, .year
                         lines.append(f"    {dim.name} is {sql}.{dim.granularity}")
                     else:
                         lines.append(f"    {dim.name} is {sql}")
@@ -1376,7 +1458,7 @@ def _export_source(self, model: Model) -> list[str]:
                 measure_expr = self._format_measure(metric)
                 lines.append(f"    {metric.name} is {measure_expr}")
 
-        # Joins
+        # Joins - Tier 4.4: use on condition from metadata when available
         for rel in model.relationships:
             lines.append("")
             if rel.type == "many_to_one":
@@ -1385,7 +1467,10 @@ def _export_source(self, model: Model) -> list[str]:
                 join_type = "join_cross"
             else:
                 join_type = "join_many"
-            if rel.foreign_key:
+            on_condition = (rel.metadata or {}).get("on_condition")
+            if on_condition:
+                lines.append(f"  {join_type}: {rel.name} on {on_condition}")
+            elif rel.foreign_key:
                 lines.append(f"  {join_type}: {rel.name} with {rel.foreign_key}")
             else:
                 lines.append(f"  {join_type}: {rel.name}")