EXPLICIT LEARNING
- By Sam Jack
- Published 05/27/2008
Sam Jack
Try your lucky Although formally endorsed by everyone, the tips below are held as 'common knowledge' by many players and represent solid general strategy for the long-run.
In strategic games like chess, the performance loss by
ignoring opponent modeling is small, and hence it is
usually ignored (although it has been studied [5,11,12]).
In contrast, not only does opponent modeling have
tremendous value in poker, it can be the distinguishing feature between players at different skill levels. If a set of
players all have a comparable knowledge of poker
fundamentals, the ability to alter decisions based on an
accurate model of the opponent may have a greater impact
on success than any other strategic principle.
To assess a hand, the Hand Evaluator compares those
cards against all possible opponent holdings. Naively, one
could treat all opponent hands as equally likely, however
this skews the hand evaluations compared to more
realistic assumptions. Many weak hands are likely to have
been folded before the flop, making them less likely to be
held later in the hand. Similarly, a hand made strong by
the turn and river cards may have been folded on the flop.
Therefore, for each starting hand, we need to define a
probability that our opponent would have played that hand
in the observed manner. We call the probabilities for each
of these (52 choose 2) = 1,326 subcases weights since they
act as multipliers in the enumeration computations.3
The use of these weights is the first step toward opponent
modeling since we are changing our computations based
on the relative probabilities our opponent’s possible hole
cards. The simplest approach to determining these weights
is to treat all opponents the same, calculating a single set
of weights to reflect “reasonable” behavior, and use them
for all opponents. An initial set of weights was determined
by ranking the starting hands (as determined by off-line
simulations) and assigning a probability commensurate
with the average return on investment of each hand. These
results closely approximate the ranking of hands by strong
players [18].
In Loki, the Opponent Modeler uses probability triples to
update the Weight Table after each opponent action. To
accomplish this, the Triple Generator is called for each
possible two-card hand. It then multiplies each weight in
the Weight Table by the entry in the probability triple that
corresponds to the opponent’s action. For example,
suppose the previous weight for Ace-Ace is 0.7 (meaning
that if it has been dealt, there is a 70% chance the
opponent would have played it in exactly the manner
observed so far), and the opponent now calls. If the
probability triple for the current context is [0.0, 0.2, 0.8],
then the updated weight for this case would be 0.7 x 0.2 =
0.14. The relative likelihood of the opponent holding Ace-
Ace has decreased to 14% because they did not raise.
The call value of 0.2 reflects the possibility that this
particular opponent might deliberately try to mislead us by
calling instead of raising. Using a probability distribution
allows us to account for uncertainty in our beliefs. This
process of updating the weight table is repeated for each
entry.
The above corresponds to what we call Generic Opponent
Modeling (GOM). Each hand is viewed in isolation and all
opponents are treated as the same player. Each player’s
Weight Table is initially identical, and gets modified
based on their betting action. Although rather simplistic, this model is quite powerful in that it does a good job of
skewing the hand evaluations to take into account the
most likely opponent holdings.
Obviously, treating all opponents the same is clearly
wrong. Each player has a different style, ranging from
loose (plays most hands beyond the flop) to tight (usually
plays the few hands that have a very high probability of
winning), and from aggressive to conservative. Knowing
the style of the opponents allows a player to adjust their
betting decisions. For example, if a perceived tight player
is betting aggressively, there is a good chance that they
have a strong hand. A loose player will play many
marginal hands or may bluff a lot. This is useful
information and may allow you to fold a strong hand or
call with a weak one when it is correct to do so. In
general, a bet made by a loose or aggressive player should
not be taken as seriously as one made by a tight or
conservative player.
Specific Opponent Modeling (SOM) customizes the
probability triple function to represent the playing style of
each opponent. For a given game, the reweighting factor
applied to the entries of the Weight table is adjusted by
betting frequency statistics gathered on that opponent
from previous hands. This results in a shift of the assumed
call and raise thresholds for each player. In the case of a
tight player, the call and raise thresholds will increase,
indicating fewer hands that are likely to be played.
Conversely, a loose player’s thresholds will be lowered.
During each round of a game, the history of previous
actions by the opponent is used to influence the
probability triple generated for that opponent.
In competitive poker, opponent modeling is much more
complex than portrayed here. For example, players can act
to mislead their opponents into constructing an erroneous
model. Early in a session a strong poker player may try to
create the impression of being very conservative, only to
exploit that image later in that session when the opponents
are using an incorrect opponent model. A strong player
has to adapt their model to the opponents varying their
playing style.
ignoring opponent modeling is small, and hence it is
usually ignored (although it has been studied [5,11,12]).
In contrast, not only does opponent modeling have
tremendous value in poker, it can be the distinguishing feature between players at different skill levels. If a set of
players all have a comparable knowledge of poker
fundamentals, the ability to alter decisions based on an
accurate model of the opponent may have a greater impact
on success than any other strategic principle.
To assess a hand, the Hand Evaluator compares those
cards against all possible opponent holdings. Naively, one
could treat all opponent hands as equally likely, however
this skews the hand evaluations compared to more
realistic assumptions. Many weak hands are likely to have
been folded before the flop, making them less likely to be
held later in the hand. Similarly, a hand made strong by
the turn and river cards may have been folded on the flop.
Therefore, for each starting hand, we need to define a
probability that our opponent would have played that hand
in the observed manner. We call the probabilities for each
of these (52 choose 2) = 1,326 subcases weights since they
act as multipliers in the enumeration computations.3
The use of these weights is the first step toward opponent
modeling since we are changing our computations based
on the relative probabilities our opponent’s possible hole
cards. The simplest approach to determining these weights
is to treat all opponents the same, calculating a single set
of weights to reflect “reasonable” behavior, and use them
for all opponents. An initial set of weights was determined
by ranking the starting hands (as determined by off-line
simulations) and assigning a probability commensurate
with the average return on investment of each hand. These
results closely approximate the ranking of hands by strong
players [18].
In Loki, the Opponent Modeler uses probability triples to
update the Weight Table after each opponent action. To
accomplish this, the Triple Generator is called for each
possible two-card hand. It then multiplies each weight in
the Weight Table by the entry in the probability triple that
corresponds to the opponent’s action. For example,
suppose the previous weight for Ace-Ace is 0.7 (meaning
that if it has been dealt, there is a 70% chance the
opponent would have played it in exactly the manner
observed so far), and the opponent now calls. If the
probability triple for the current context is [0.0, 0.2, 0.8],
then the updated weight for this case would be 0.7 x 0.2 =
0.14. The relative likelihood of the opponent holding Ace-
Ace has decreased to 14% because they did not raise.
The call value of 0.2 reflects the possibility that this
particular opponent might deliberately try to mislead us by
calling instead of raising. Using a probability distribution
allows us to account for uncertainty in our beliefs. This
process of updating the weight table is repeated for each
entry.
The above corresponds to what we call Generic Opponent
Modeling (GOM). Each hand is viewed in isolation and all
opponents are treated as the same player. Each player’s
Weight Table is initially identical, and gets modified
based on their betting action. Although rather simplistic, this model is quite powerful in that it does a good job of
skewing the hand evaluations to take into account the
most likely opponent holdings.
Obviously, treating all opponents the same is clearly
wrong. Each player has a different style, ranging from
loose (plays most hands beyond the flop) to tight (usually
plays the few hands that have a very high probability of
winning), and from aggressive to conservative. Knowing
the style of the opponents allows a player to adjust their
betting decisions. For example, if a perceived tight player
is betting aggressively, there is a good chance that they
have a strong hand. A loose player will play many
marginal hands or may bluff a lot. This is useful
information and may allow you to fold a strong hand or
call with a weak one when it is correct to do so. In
general, a bet made by a loose or aggressive player should
not be taken as seriously as one made by a tight or
conservative player.
Specific Opponent Modeling (SOM) customizes the
probability triple function to represent the playing style of
each opponent. For a given game, the reweighting factor
applied to the entries of the Weight table is adjusted by
betting frequency statistics gathered on that opponent
from previous hands. This results in a shift of the assumed
call and raise thresholds for each player. In the case of a
tight player, the call and raise thresholds will increase,
indicating fewer hands that are likely to be played.
Conversely, a loose player’s thresholds will be lowered.
During each round of a game, the history of previous
actions by the opponent is used to influence the
probability triple generated for that opponent.
In competitive poker, opponent modeling is much more
complex than portrayed here. For example, players can act
to mislead their opponents into constructing an erroneous
model. Early in a session a strong poker player may try to
create the impression of being very conservative, only to
exploit that image later in that session when the opponents
are using an incorrect opponent model. A strong player
has to adapt their model to the opponents varying their
playing style.
