Center for Analysis and Design of Intelligent Agents

Aim: This assignment is meant to give you a better insight into cumulative learning and life-long learning on the example of human learning.

Summary: In the last assignment you have learned about the learning of deep reinforcement learners, like the actor-critic, and the role task environments play in current AI technologies. In this assignment you are supposed to get a better idea on cumulative learning and its difference to the learning of current AI systems, by example of a human learner - i.e. yourself. Furthermore, this assignment will highlight the importance of perception for learning and task solving. You are given Python code with an implementation of the cart-pole task designed as a game for a human to play. For this, 4 different conditions have been implemented, giving you the chance to experience the importance of the presentation of data for yourself.
The files for this assignment can be found here: assignment 2 files
The same installation as for assignment 1 can be used.

Condition 1: SYMBOLIC
You are presented with an alphanumeric display of a continuously updated state of observable variables relevant to the task (e.g. x, v, theta, and omega). With the arrow keys you can apply, as previously the reinforcement learner, a force of -10 (left) or +10 (right) Newton to the cart. Your task is to keep the pole upright for as long as possible. In the top-right you can see your current score (the total reward you achieved in this epoch (+1 for each successful iteration)), in the center are the values of the observables. You can set the environment to run synchronous or async, meaning that, if sync is not set the environment updates automatically after 100 ms. Further, you can invert the forces by pressing the i key on your keyboard.

Condition 1 - symbolic

Condition 2: COLOR CODING
You are presented with colors for each of the variables as well as an arrow indicating if the value is negative (pointing to the left) or positive (pointing to the right). Green means a value around 0, the redder it becomes the closer to either edge you are. For v and omega (both not in the solution space restriction) red implies a high value, green a low value. Other things are the same as in Condition 1.

Condition 2 - color coding

Condition 3: COLORS & LINES
In the setting colors_lines the same colors are used as in Level 2; however, additionally a line is drawn on the bar giving you additional information of the current state of the observables. Otherwise things are the same as in Conditions 1 and 2.

Condition 3 - colors and lines

Condition 4: ANIMATION
In the animation setting an animation of the cart-pole is presented to you, including the cart (rectangle), the pole (line to the top of the image), the velocity (line inside the cart to the left or right), and the angular velocity (line at the top of the pole) indicating the current position and velocity of the cart. Otherwise this is the same as in Conditions 1.

Condition 4 - animation

The code for the cart-pole task-environment used for this is exactly the same as in Assignment 1 and can/ should therefore be reused completely.

Read the full instructions before beginning.

1. Your task is to get good at performing the task in all 4 Levels. Record the data for your training in each condition.
2. Apply the following settings to the environment (If you downloaded and use the provided environment this should be the case already):
   1. All variables (x, v, theta, omega) are observables.
   2. Apply noise (all of them at the same time) to the environment with a mean of 0 and a standard deviation of
      1. x: 0.2 m
      2. v: 0.2 m/s
      3. theta: 1.0 deg
      4. omega: 0.2 rad/s
   3. Set the environment to run asynchronous
3. Play the game on Conditions 1, 2, 3, and 4 in that order for at least 10 epochs (or better phrased: until you are confident in playing in the condition, but for at least 10 epochs) each and note for each condition your
   1. highest score,
   2. average score and standard deviation,
   3. median score.
4. Invert the forces by pressing the “i” key on your keyboard during a run (after 5-10 restarts/ fails) and continue for another 5-10 episodes. Do this in all four conditions in the same order as previously. (Redo instruction 3 with this inversion). What can you say about your learning speed with force inversion?
5. Apply the following settings to the environment (all of them at the same time):
   1. Only the variables x, v, omega are observables.
   2. Do not apply noise.
   3. Set the environment to run asynchronous.
6. And replay the game as stated under instruction 3.
7. Change the environment in any way you like to make it harder (or easier) to play note your changes and your scores accordingly.
8. Reset the settings back to the ones from the beginning and replay the game (as described in the third instruction).
9. Compare your results from the first tries from number 3 to the other, and especially the last tries from number 7. What can you conclude about the possibilities of cumulative, life-long learning?
10. Report. Write a report on your results including the different scores and comparisons between the different tries. Compare your results with the results from the last assignment and discuss. Discuss the advantages, and disadvantages of human learning (and human nature) this might include (but is not restricted to):
    1. Previously acquired knowledge used in this game.
    2. Cumulative learning.
    3. Transfer learning.
    4. Boredom.

Try to not only do what you are asked, but rather investigate learning further by yourself (if you have the time to do so, of course). Some of the things you could to:

1. Change dt, make it faster or slower.
2. Try out the synchronous environment - this is actually, how the RL plays the game
3. Try the things you changed in assignment 1 for the RL now on yourself, for example changing the task in the middle of a run.
4. Try even more (things the RL could not do, for example change the observation state during a run (e.g. after 100 iterations)).
5. What else can you think of?

Try to make a point on the advantages and disadvantages of cumulative/ human learning.


Set up the conditions:
For most of the tasks given to you all you need is to change parts in the main.py or cart_pole_env.py, I hope it is self explaining, if any questions arise please ask!