Which seems to let me PGP encrypt files using a passphrase. Is a public key just the same as a passphrase that is super super long? Like could I use Get-Content and point it to the key file or just a normal text file that has the key all on one line?
Is a public key just the same as a passphrase that is super super long?
Nope. The link you posted uses symmetric encryption. Symmetric encryption uses the same key (passphrase) to encrypt and decrypt the data.
Public/Private key pairs use asymmetric encryption. For security you encrypt with the public key and only the holder of the private key can decrypt the data.
You can also encrypt (or sign) data with the private key; however anyone with the public key (that's everybody, it's public) can decrypt it so this is isn't used for keeping data secret. However, as only the person with the private key could have encrypted it you can trust the data came from the person it claims to have come from and that it hasn't been changed during transmission.