iCVTEAM.github.io/DanceIt.html at master · iCVTEAM/iCVTEAM.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>DanceIt</title>
    <link rel="stylesheet" type="text/css" href="assets/scripts/bulma.min.css">
    <link rel="stylesheet" type="text/css" href="assets/scripts/theme.css">
    <link rel="stylesheet" type="text/css" href="https://cdn.bootcdn.net/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  </head>
  <body>
    <section class="hero is-light" style="">
      <div class="hero-body" style="padding-top: 50px;">
        <div class="container" style="text-align: center;margin-bottom:5px;">
          <h1 class="title">
            DanceIt: Music-inspired Dancing Video Synthesis
          </h1>

          <div class="author">Xin Guo</div>
          <div class="author">Jia Li</div>
          <div class="author">Yifan Zhao</div>
          <div class="group">
            <a href="http://cvteam.net/">CVTEAM</a>
          </div>
          <div class="aff">
            <p><sup>1</sup>State Key Laboratory of Virtual Reality Technology and Systems, SCSE, Beihang University, Beijing, China</p>
          </div>
          <div class="con">
            <p  style="font-size: 24px; margin-top:5px; margin-bottom: 15px;">
            IEEE Transactions on Image Processing 2021
            </p>
          </div>
          <div class="columns">
            <div class="column"></div>
            <div class="column"></div>
            <div class="column">
              <a href="https://arxiv.org/abs/2009.08027" target="_blank">
                <p class="link">Paper</p>
              </a>
            </div>
            <div class="column">
              <a href="https://github.com/iCVTEAM/MDVS/" target="_blank">
                <p class="link">Code</p>
              </a>
            </div>
            <div class="column"></div>
            <div class="column"></div>
          </div>
        </div>
      </div>
    </section>
    <div style="text-align: center;">
      <div class="container" style="max-width:850px">
        <div style="text-align: center;">
          <img src="assets/MDVS/head.png" class="centerImage">
        </div>
      </div>
      <div class="head_cap">
        <p style="color:gray;">
          Pipeline of the proposed framework
        </p>
      </div>
    </div>
    <section class="hero">
      <div class="hero-body">
        <div class="container" style="max-width: 800px" >
          <h1 style="">Abstract</h1>
          <p  style="text-align: justify; font-size: 17px;">
            Close your eyes and listen to music, one can easily
            imagine an actor dancing rhythmically along with the music.
            These dance movements are usually made up of dance movements
            you have seen before. In this paper, we propose to reproduce such
            an inherent capability of the human-being within a computer
            vision system. The proposed system consists of three modules. To
            explore the relationship between music and dance movements,
            we propose a cross-modal alignment module that focuses on
            dancing video clips, accompanied on pre-designed music, to learn
            a system that can judge the consistency between the visual
            features of pose sequences and the acoustic features of music.
            The learnt model is then used in the imagination module to
            select a pose sequence for the given music. Such pose sequence
            selected from the music, however, is usually discontinuous. To
            solve this problem, in the spatial-temporal alignment module we
            develop a spatial alignment algorithm based on the tendency
            and periodicity of dance movements to predict dance movements
            between discontinuous fragments. In addition, the selected pose
            sequence is often misaligned with the music beat. To solve this
            problem, we further develop a temporal alignment algorithm
            to align the rhythm of music and dance. Finally, the processed
            pose sequence is used to synthesize realistic dancing videos in
            the imagination module. The generated dancing videos match
            the content and rhythm of the music. Experimental results and
            subjective evaluations show that the proposed approach can
            perform the function of generating promising dancing videos by
            inputting music.
          </p>
        </div>
      </div>
    </section>
    <section class="hero is-light" style="background-color:#FFFFFF;">
      <div class="hero-body">
        <div class="container" style="max-width:800px;margin-bottom:20px;">
          <h1>
             Synthesized results of multiple targets
          </h1>
        </div>
        <div class="container" style="max-width:800px">
          <div style="text-align: center;">
            <img src="assets/MDVS/result.png" class="centerImage">
          </div>
        </div>
      </div>
    </section>
  </body>
</html>