Skip to content

Commit 3608425

Browse files
Improve dpnp.partition implementation (#2766)
The PR propose to improve implementation and to use `dpnp.sort` call when - input array has number of dimensions > 1 - input array has previously not supported integer dtype - `axis` keyword is passed (previously not supported) - sequence of `kth` is passed (previously not supported) In case of `ndim > 1` previously the implementation from legacy backend was used, which is significantly slow (see performance comparation below). It used a copy of input data into the shared USM memory and included computations on the host. This PR proposes to reuse `dpnp.sort` for all the above cases. While in case when the legacy implementation is stable and fast (for 1D input array), it will remain, because it relays on `std::nth_element` from OneDPL. The benchmark results were collected on PVC with help of the below code: ```python import dpnp, numpy as np from dpnp.tests.helper import generate_random_numpy_array a = generate_random_numpy_array(10**7, dtype=np.float64, seed_value=117) ia = dpnp.array(a) %timeit x = dpnp.partition(ia, 513); x.sycl_queue.wait() ``` Below tables contains data in case of 1D input array (shape=(10**7,)), where the implementation path was kept the same, plus adding support of missing integer dtypes using fallback on the sort function: | Implementation | int32 | uint32 | int64 | uint64 | float32 | float64 | complex64 | complex128 | |--------|--------|--------|--------|--------|--------|--------|--------|--------| | old (legacy backend) | 7.46 ms | not supported | 9.46 ms | not supported | 7.39 ms | 8.92 ms | 10.9 ms | 21.2 ms | | new (backend + sort) | 7.34 ms | 10.8 ms | 9.48 ms | 12.5 ms | 7.37 ms | 8.89 ms | 11 ms | 21.2 ms | The following code was used for 2D input array with shape=(10**4, 10**4): ```python import dpnp, numpy as np from dpnp.tests.helper import generate_random_numpy_array a = generate_random_numpy_array((10**4, 10**4), dtype=np.float64, seed_value=117) ia = dpnp.array(a) %timeit x = dpnp.partition(ia, 1513); x.sycl_queue.wait() ``` In that case the new implementation is fully based on the sort call: | Implementation | int32 | int64 | float32 | float64 | complex64 | complex128 | |--------|--------|--------|--------|--------|--------|--------| | old (legacy backend) | 6.4 s | 6.89 s | 7.36 s | 7.66 s | 8.61 s | 10 s | | new (sort) | 57.4 ms | 64.7 ms | 62.2 ms | 68 ms | 77 ms | 151 ms | 9c4aed2
1 parent 060c771 commit 3608425

759 files changed

Lines changed: 1781 additions & 1575 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.buildinfo

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
# Sphinx build info version 1
22
# This file records the configuration used when building these files. When it is not found, a full rebuild will be done.
3-
config: dabfa318cc06c6ce826220cdde96bd82
3+
config: bf51692a0db2c4243b085a8d82c6bc8b
44
tags: 645f666f9bcd5a90fca523b33c5a78b7

_modules/dpnp/dpnp_array.html

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_array &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_array &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>
@@ -1623,35 +1623,54 @@ <h1>Source code for dpnp.dpnp_array</h1><div class="highlight"><pre>
16231623
<a class="viewcode-back" href="../../reference/generated/dpnp.dpnp_array.dpnp_array.partition.html#dpnp.dpnp_array.dpnp_array.partition">[docs]</a>
16241624
<span class="k">def</span><span class="w"> </span><span class="nf">partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">/</span><span class="p">,</span> <span class="n">kth</span><span class="p">,</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="s2">&quot;introselect&quot;</span><span class="p">,</span> <span class="n">order</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
16251625
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
1626-
<span class="sd"> Return a partitioned copy of an array.</span>
1626+
<span class="sd"> Partially sorts the elements in the array in such a way that the value</span>
1627+
<span class="sd"> of the element in k-th position is in the position it would be in a</span>
1628+
<span class="sd"> sorted array. In the output array, all elements smaller than the k-th</span>
1629+
<span class="sd"> element are located to the left of this element and all equal or</span>
1630+
<span class="sd"> greater are located to its right. The ordering of the elements in the</span>
1631+
<span class="sd"> two partitions on the either side of the k-th element in the output</span>
1632+
<span class="sd"> array is undefined.</span>
16271633

1628-
<span class="sd"> Rearranges the elements in the array in such a way that the value of</span>
1629-
<span class="sd"> the element in `kth` position is in the position it would be in</span>
1630-
<span class="sd"> a sorted array.</span>
1634+
<span class="sd"> Refer to `dpnp.partition` for full documentation.</span>
16311635

1632-
<span class="sd"> All elements smaller than the `kth` element are moved before this</span>
1633-
<span class="sd"> element and all equal or greater are moved behind it. The ordering</span>
1634-
<span class="sd"> of the elements in the two partitions is undefined.</span>
1636+
<span class="sd"> kth : {int, sequence of ints}</span>
1637+
<span class="sd"> Element index to partition by. The kth element value will be in its</span>
1638+
<span class="sd"> final sorted position and all smaller elements will be moved before</span>
1639+
<span class="sd"> it and all equal or greater elements behind it.</span>
1640+
<span class="sd"> The order of all elements in the partitions is undefined. If</span>
1641+
<span class="sd"> provided with a sequence of kth it will partition all elements</span>
1642+
<span class="sd"> indexed by kth of them into their sorted position at once.</span>
1643+
<span class="sd"> axis : int, optional</span>
1644+
<span class="sd"> Axis along which to sort. The default is ``-1``, which means sort</span>
1645+
<span class="sd"> along the the last axis.</span>
16351646

1636-
<span class="sd"> Refer to `dpnp.partition` for full documentation.</span>
1647+
<span class="sd"> Default: ``-1``.</span>
16371648

16381649
<span class="sd"> See Also</span>
16391650
<span class="sd"> --------</span>
16401651
<span class="sd"> :obj:`dpnp.partition` : Return a partitioned copy of an array.</span>
1652+
<span class="sd"> :obj:`dpnp.argpartition` : Indirect partition.</span>
1653+
<span class="sd"> :obj:`dpnp.sort` : Full sort.</span>
16411654

16421655
<span class="sd"> Examples</span>
16431656
<span class="sd"> --------</span>
16441657
<span class="sd"> &gt;&gt;&gt; import dpnp as np</span>
16451658
<span class="sd"> &gt;&gt;&gt; a = np.array([3, 4, 2, 1])</span>
16461659
<span class="sd"> &gt;&gt;&gt; a.partition(3)</span>
16471660
<span class="sd"> &gt;&gt;&gt; a</span>
1661+
<span class="sd"> array([1, 2, 3, 4]) # may vary</span>
1662+
1663+
<span class="sd"> &gt;&gt;&gt; a.partition((1, 3))</span>
1664+
<span class="sd"> &gt;&gt;&gt; a</span>
16481665
<span class="sd"> array([1, 2, 3, 4])</span>
16491666

16501667
<span class="sd"> &quot;&quot;&quot;</span>
16511668

1652-
<span class="bp">self</span><span class="o">.</span><span class="n">_array_obj</span> <span class="o">=</span> <span class="n">dpnp</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span>
1653-
<span class="bp">self</span><span class="p">,</span> <span class="n">kth</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="n">axis</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">kind</span><span class="p">,</span> <span class="n">order</span><span class="o">=</span><span class="n">order</span>
1654-
<span class="p">)</span><span class="o">.</span><span class="n">get_array</span><span class="p">()</span></div>
1669+
<span class="k">if</span> <span class="n">axis</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
1670+
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span>
1671+
<span class="s2">&quot;&#39;NoneType&#39; object cannot be interpreted as an integer&quot;</span>
1672+
<span class="p">)</span>
1673+
<span class="bp">self</span><span class="p">[</span><span class="o">...</span><span class="p">]</span> <span class="o">=</span> <span class="n">dpnp</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">kth</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="n">axis</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="n">kind</span><span class="p">,</span> <span class="n">order</span><span class="o">=</span><span class="n">order</span><span class="p">)</span></div>
16551674

16561675

16571676
<div class="viewcode-block" id="dpnp_array.prod">

_modules/dpnp/dpnp_array_api_info.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_array_api_info &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_array_api_info &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_flatiter.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_flatiter &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_flatiter &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface_arraycreation.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface_arraycreation &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface_arraycreation &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface_bitwise.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface_bitwise &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface_bitwise &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface_counting.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface_counting &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface_counting &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface_functional.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface_functional &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface_functional &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

_modules/dpnp/dpnp_iface_histograms.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@
1414
</script>
1515

1616
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
17-
<title>dpnp.dpnp_iface_histograms &mdash; Data Parallel Extension for NumPy 0.20.0dev2+20.g9d6d5a50e6f documentation</title>
17+
<title>dpnp.dpnp_iface_histograms &mdash; Data Parallel Extension for NumPy 0.20.0dev2+21.g9c4aed22eb8 documentation</title>
1818
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=03e43079" />
1919
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=9edc463e" />
2020

2121

2222
<script src="../../_static/jquery.js?v=5d32c60e"></script>
2323
<script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
24-
<script src="../../_static/documentation_options.js?v=3ba98b21"></script>
24+
<script src="../../_static/documentation_options.js?v=20ce9460"></script>
2525
<script src="../../_static/doctools.js?v=9bcbadda"></script>
2626
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
2727
<script src="../../_static/js/theme.js"></script>

0 commit comments

Comments
 (0)