Power Sharing

At a men's breakfast I attend on Friday mornings, we are currently studying the Holy Spirit. One of the passages used last week was John 20:21-22. Jesus had just appeared in a room to His disciples shortly after His resurrection. Of course, since He was supposed to be dead, He had to do the man-thing where He got to show off His cool scars to His friends. After validating His identity, He gave them instruction which included the verse we were focusing on where Jesus breathed on them while saying "Receive the Holy Spirit".
While everyone else moved on, I couldn't help but notice that the words highlighted in red continued for another verse. It said, "Receive the Holy Spirit. If you forgive the sins of any, they are forgiven them, if you retain the sins of any, they are retained."
That's a bold statement. Last time I checked, God was the only one wielding that power. So what did Jesus mean by that when He was speaking to those ten guys? I've gotten various opinions from different commentaries, and most of them link this with our responsibility to share the Gospel with the rest of the world. If people receive Christ, their sins are forgiven. If they don't, they continue to live in condemnation. That may be the intent of the verse. I assume that the Catholic church has linked it to the ability of their priests to host confessionals and mandate terms of forgiveness on God's behalf.
Whatever the proper interpretation, I was struck with a completely different take on the whole thing. It probably has absolutely nothing to do with that upper room encounter, but I still think God wanted to bring something to my attention.
Even though asking God for forgiveness is paramount, we are also called upon to confess our sins to one another so we can be prayed for and healed. I'm okay with that if it relates to somebody I feel that I know pretty well and can judge their character. However, let's take the example of a pastor caught in adultery. There is no denying that what he did was disgusting and carries dire consequences, but let's also assume that in this example, he succumbed to temptation ~ he sinned ~ his heart was truly broken after the reality of what he had done to his wife set in ~ he begged God and his wife for forgiveness ~ he also confessed publicly what had taken place, asking for prayer and forgiveness. God knows the heart, and He forgave him. His wife had him agree in writing to be castrated and then forgave him.
When he repented to God, that sin should have been gone forever. Instead, he made the mistake of public confession. Instead of praying for and also forgiving the man, I chose to retain his sin. I decided that he wasn't sorry he did it; he was sorry he got caught. I talked to my friends about what a scumbag this guy was. They also agreed with my opinion and they chose to retain his sin.
Funny thing is, that while that guy was clean before God and his sin was forgotten, I've now not only taken on sin through my lack of forgiveness for something that was never my business to begin with, but I've caused others to do the same. Now who is standing before God in judgment? "Receive the Holy Spirit. If you forgive the sins of any, they are forgiven, if you retain the sins of any, they are retained."
I am a Christian, and the Holy Spirit dwells within me. I have access to a lot of power. Do I exercise my power to forgive, or do I exercise my power to retain?