Understanding BGP Port Numbers

Your reasoning is correct; it is because BGP uses different source and destination ports other than 179 depending on who originates the session. BGP is essentially a standard TCP based protocol, which means that it is client and server based.
When a TCP client attempts to establish a connection to a TCP server it first sends a TCP SYN packet to the server with the destination port as the well known port. This first SYN essentially is a request to open a session. If the server permits the session it will respond with a TCP SYN ACK saying that it acknowledges the request to open the session, and that it also wants to open the session. In this SYN ACK response the server uses the well known port as the source port, and a randomly negotiated destination port. The last step of the three way handshake is the client responding to the server with a TCP ACK, which acknowledges the server’s response and completes the connection establishment.
Now from the perspective of BGP specifically the TCP clients and servers are routers. When the “client” router initiates the BGP session is sends a request to the server with a destination port of 179 and a random source port X. The server then responds with a source port of 179 and a destination port of X. Therefore all client to server traffic uses destination 179, while all server to client traffic uses source 179. We can also verify this from the debug output in IOS.
In the below topology R1 and R2 are directly connected BGP peers on an Ethernet segment with configurations as follows:

R1:
interface Ethernet0/0
 ip address 10.0.0.1 255.255.255.0
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
router bgp 1
 network 1.1.1.1 mask 255.255.255.255
 neighbor 10.0.0.2 remote-as 1

R2:
interface Ethernet0/0
 ip address 10.0.0.2 255.255.255.0
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
router bgp 1
 network 2.2.2.2 mask 255.255.255.255
 neighbor 10.0.0.1 remote-as 1

From the “debug ip packet detail” output we can see the initial TCP session establishment for BGP between these neighbors as follows:

R1:
IP: tableid=0, s=10.0.0.2 (Ethernet0/0), d=10.0.0.1 (Ethernet0/0), routed via RIB
IP: s=10.0.0.2 (Ethernet0/0), d=10.0.0.1 (Ethernet0/0), len 44, rcvd 3
    TCP src=11000, dst=179, seq=3564860120, ack=0, win=16384 SYN

R2:
IP: tableid=0, s=10.0.0.1 (Ethernet0/0), d=10.0.0.2 (Ethernet0/0), routed via RIB
IP: s=10.0.0.1 (Ethernet0/0), d=10.0.0.2 (Ethernet0/0), len 44, rcvd 3
    TCP src=179, dst=11000, seq=1977460611, ack=3564860121, win=16384 ACK SYN

R1:
IP: tableid=0, s=10.0.0.2 (Ethernet0/0), d=10.0.0.1 (Ethernet0/0), routed via RIB
IP: s=10.0.0.2 (Ethernet0/0), d=10.0.0.1 (Ethernet0/0), len 40, rcvd 3
    TCP src=11000, dst=179, seq=3564860121, ack=1977460612, win=16384 ACK

In the first code snippet we can see that R1 has received a TCP packet from R2. This packet was sourced from 10.0.0.2 (R2), destined to 10.0.0.1 (R1), has a TCP source port of 11000 (the random port), a destination TCP port of 179 (the well known port), and is a SYN packet (1st step of three way handshake).
Next R2 receives the response from R1 that is a TCP SYN ACK, but in this case we can see that the port numbers are swapped. R1, the TCP server, now uses TCP port 179 as the source port, and the randomly negotiated port 11000 as the destination.
Further debugging of the BGP flow between these neighbors will show R1 using destination 179 (as the TCP client) and R2 using source 179 (as the TCP server).
The implication of this operation is that if BGP needs to be matched for some reason, i.e. to filter it out in an access-list, to match it in a QoS policy, we must account for traffic that is both going to TCP port 179 and coming from TCP port 179.