A stateful packet firewall has an understanding of the current connection, it records the communications state. This allows the firewall to know if an incoming message is a new message, or a reply to a message that it was expecting.
For example, a stateful packet firewall would allow an ICMP packet to leave the network, and when the ping reply comes back to the firewall it would allow this packet through as it was expecting it.
A stateful packet firewall offers more security than say over a stateless firewall, because it allows the firewall to use some logic on what traffic to allow or drop. An incoming TCP packet with a SYN-ACK flag set, but no record of an outgoing TCP SYN packet, would result in a drop of the packet as it may be something illegitamate.
A strength therefore for a stateful packet firewall is knowing the context of a message and whether it forms part of an expected communication protocol. A possible weakness is it does not though have any ability to 'see' within a packets payload, it's is only operating at layer 3/4 of the OSI model. Another weakness, very common in low end firewalls, is that the messages must 'tracked' so that the firewall can understand the context of the messages. This means the state table in memory will fill up quickly on a busy network, potentially leading to a denial of service should the state table fill up completely. A busy firewall may well have a state table expiry set a lower time, but the risk of this is a delayed packet may be seen as an illegitimate packet if the state is lost.
Expanding on the other firewalls is the application proxy firewall, this has a full network stack for sending and receiving messages. This is because an application firewall, unlike a stateful/stateless firewall, terminates the message at the firewall and builds a new packet with the original contents (changes in the senders and destination IP though). This allows therefore the firewall to 'see' the whole OSI stack up to and including layer 7. This allows the firewall to make application level decisions on whether to allow or drop the message. An example would be a HTTP message that has a POST request, the firewall may be configured to only allow GET requests and therefore drop the message based on that alone, and not an IP/Port restriction which a stateless/stateful firewall would do.