[Detail Bug] Domain Data Products tab silently truncates results to first 50 items (no pagination)

[detail-app]

Detail Bug Report

https://app.detail.dev/org_3377c26d-da48-4ccd-b83a-22c542f4fe83/bugs/bug_86531c62-0ce5-4f1d-862b-7c503bcd4899

Introduced in #12839 by @karanh37 on Sep 7, 2023

Summary

• Context: DataProductsTab renders the list of Data Products that belong to a Domain (and its sub-domains) on the Domain detail page's "Data Products" tab, by issuing a searchQuery against SearchIndex.DATA_PRODUCT and mapping the hits into clickable cards.
• Bug: The fetchDataProducts call hardcodes pageNumber: 1, pageSize: PAGE_SIZE_LARGE (50) and then renders only dataProducts.data.map(...) with no pagination UI, so any Data Products beyond the first 50 are silently dropped from the tab.
• Actual vs. expected: Expected: the user can page through every Data Product of the Domain from this tab, as the sibling listing tabs allow (Evidence Create teams page similar to tags #1). Actual: only the first 50 Data Products are ever rendered here; to reach the rest the user must leave the Domain page and use the global Explore search. The overflow is not reachable from this tab — there is no pager, no "show more", and sub-domain navigation does not recover the parent's directly-attached Data Products.
• Impact: On any single Domain with more than 50 Data Products directly attached to it, the overflow (>50) is not reachable from this tab. The badge displays the true total (see Evidence Apply apache 2.0 license to all the files in the project #3), so the user can see that cards are missing, but the only in-product route to the rest is the global Explore search (a separate page, not per-Domain scoped by default). The defect is therefore one of tab completeness, not of data loss.

Code with Bug

openmetadata-ui/src/main/resources/ui/src/components/Domain/DomainTabs/DataProductsTab/DataProductsTab.component.tsx:

const res = await searchQuery({
  query: '',
  pageNumber: 1,
  pageSize: PAGE_SIZE_LARGE,            // <-- BUG 🔴 hardcoded first page of 50, no later pages fetched
  queryFilter: getQueryFilterForDataProducts(
    urlDomainFqn || domainFqn || ''
  ),
  searchIndex: SearchIndex.DATA_PRODUCT,
});

const data = formatDataProductResponse(res.hits.hits);
setDataProducts({
  data: data,
  paging: { total: res.hits.total.value ?? 0 }, // total is known but no pager consumes it
});
...
// firstPanel.children:
<>
  {dataProducts.data.map((dataProduct) => (   // <-- BUG 🔴 only renders ≤50 cards; no pager to view the rest
    <ExploreSearchCard ... />
  ))}
</>

Explanation

• searchQuery is invoked for only the first page (pageNumber: 1) and limited to PAGE_SIZE_LARGE (50). The component never requests subsequent pages and provides no pagination / navigation UI.
• Even though the total hit count is available (res.hits.total.value) and stored into component state, nothing renders or consumes that total to enable reaching overflow results.
• On domains with more than 50 directly-attached Data Products, the tab will always show only 50 cards, regardless of the true total.

Codebase Inconsistency

• The Domain page tab badge count is computed via a separate count-only search (pageSize: 0) and therefore shows the true total, while the tab content remains capped at 50—an observable UI mismatch.

openmetadata-ui/src/main/resources/ui/src/components/Domain/DomainDetails/DomainDetails.component.tsx:

const res = await searchQuery({
  query: '',
  pageNumber: 1,
  pageSize: 0,                                       // <-- count-only
  queryFilter: getQueryFilterForDataProducts(domainFqn),
  searchIndex: SearchIndex.DATA_PRODUCT,
});
setDataProductsCount(res.hits.total.value ?? 0);

Failing Test

it('renders a paging control when total Data Products exceed the visible page', async () => {
  const hits = Array.from({ length: 50 }, (_, i) => ({
    _source: {
      id: `dp-${i}`, name: `dp${i}`,
      fullyQualifiedName: `Commerce.dp${i}`,
      domains: [{ fullyQualifiedName: 'Commerce', type: 'domain' }],
    },
  }));
  mockSearchQuery.mockResolvedValue({ hits: { hits, total: { value: 73 } } });

  render(<DataProductsTab permissions={MOCK_PERMISSIONS} onAddDataProduct={jest.fn()} />);

  // The full set (73) exceeds what renders (50), so the user must be given a
  // way to reach the overflow. Asserting a paging/navigation control exists
  // covers NextPrevious, Pagination, and any "load more"/cursor scheme.
  expect(await screen.findByRole('navigation')).toBeInTheDocument();
});

Recommended Fix

Reuse the same pagination pattern as AssetsTabs: drive pageNumber/pageSize via usePaging(), update total via handlePagingChange({ total: res.hits.total.value ?? 0 }), and render a paging control (e.g., <NextPrevious ... />) so users can reach all Data Products.

History

This bug was introduced in commit b6bab6c7dd1 (PR #12839, 2023-09-07), which created DataProductsTab.component.tsx with a hardcoded limit of 50 and no pager. A later refactor e0e246772d (PR #13178, 2023-09-16) switched to searchQuery and began storing res.hits.total.value into state, but still did not add pagination, preserving the truncation behavior.