Use gym and atari wrappers instead of chainerrl.envs.ale

[muupan]

Resolves #251

• Compare performance of agents
• Compare speed
• Apply to all examples under examples/ale

[muupan]

atari_wrappers.py in openai/baselines use FireResetEnv to send FIRE after every reset, but there is some doubt that DeepMind uses the same trick. openai/baselines#240 Thus I disabled it by default.

[muupan]

Below are elapsed time (sec) on 10000 steps of chainerrl.envs.ale.ALE and new atari_wrappers (https://github.com/muupan/chainerrl/blob/atari-wrappers-speed/examples/ale/check_atari_wrappers_speed.py).
Measured on MacBook Pro (Retina, 15-inch, Mid 2014).
New atari_wrappers is significantly faster.

$ python examples/ale/check_atari_wrappers_speed.py
rom name: pong
env name: PongNoFrameskip-v4
old env
.............
15.631451486144215
.............
15.614390858914703
.............
15.652993701864034
new env
.............
10.40902072796598
.............
10.401655629044399
.............
10.374119991902262
old env (test)
.............
15.4871492870152
.............
15.508433992043138
.............
15.478705200133845
new env (test)
.............
10.310282153077424
.............
10.431455522077158
.............
10.324188519036397

[muupan]

A3C (16 processes, 80M steps)

[muupan]

CategoricalDQN

[muupan]

PPO, DQN, ACER (only with atari_wrappers)

[muupan]

Since DeepMind papers use 5 min (= 5 * 60 * 60 // 4 steps) to evaluation episodes (see https://arxiv.org/abs/1507.04296 etc), I applied it to all the examples. It seems it affect performance, especially on BeamRider.

[toslunar]

It seems to me that the wrappers assume NoFrameskip envs. I'd like to have

chainerrl.envs.ale_unwrapped('pong')  # => gym.make('PongNoFrameSkip-v4')

[muupan]

chainerrl.envs.ale_unwrapped('pong') # => gym.make('PongNoFrameSkip-v4')

Don't you think stating that only `*NoFrameskip-v4' is supported is sufficient? The rule to convert 'pong' to 'PongNoFrameskip-v4' may not be simple, e.g., underscore and version upgrade.

It would be helpful to list up possible env names in the error message if the user specifies invalid one.

[toslunar]

LGTM except that hacking 1.1 warns gym.undo_logger_setup()