What kind of authority do pastors have? One helpful text for considering this question is 1 Corinthians 4:1:
This is how one should regard us, as servants of Christ and stewards of the mysteries of God. (1 Cor. 4:1)
The "us" in this verse refers to Paul and to Apollos (1 Cor. 4:6). Notably, Apollos was not an apostle, but simply a pastor who ministered at the Church in Corinth for a time (1 Cor. 3:5–6). Even so, both the Apostle Paul and the Pastor Apollos are servants and stewards.
Servants of Christ: Under Authority
The word servant here literally means "under-rower" (a low-class, common sailor), and metaphorically, to any kind of "lower class of servants" (Hodge, A Commentary on 1 & 2 Corinthians, 64.)
As servants, Paul explains that pastors have no independent authority. In fact, they are under the absolute authority of their Master. They are on the bottom rung of society, as Paul will state more graphically a few verses later: "We have become, and are still, like the scum of the world, the refuse of all things" (1 Cor. 4:13).
Pastors, therefore, have no authority of their own at all.
Stewards of the Mysteries of God: Dispensers of Another's Authority
The word stewards refers still to a servant. The steward, however, is entrusted to dispense the Master's provisions to the other servants in the household. The steward is still a man under strict authority; however, as the steward, he has responsibility for and authority over the other servants in the household.
Pastors, therefore, are dispensers of the mysteries of God. Paul defined the mysteries of God earlier in this letter to the Corinthians. He explains that these mysteries are God's plan in eternity past to send the Son into the world to be crucified for the salvation of sinners:
...but we speak God’s wisdom in a mystery, the hidden wisdom which God predestined before the ages to our glory; 8 the wisdom which none of the rulers of this age has understood; for if they had understood it they would not have crucified the Lord of glory; (1 Cor. 2:7; NASB)
Pastors have no authority of their own. Instead, God entrusts pastors to dispense the gospel of Christ and him crucified to the church. Thus, pastors are dispensers of another's authority: that is, of the authority of King Jesus.
Laboring in Preaching and Teaching: Authority by the Word
Paul makes the same point in a different way elsewhere:
 Let the elders who rule well be considered worthy of double honor, especially those who labor in preaching and teaching. (1 Tim. 5:17)
All elders rule—that is, all elders exercise ministerial authority within the Church of Jesus Christ. Among the keys of the kingdom that Jesus entrusted to the officers of his Church, this is the Key of Discipline.
Some elders not only rule, but they "labor in preaching and teaching." These elders exercise declarative authority within the Church of Jesus Christ by dispensing the mysteries of God. This is the Key of Doctrine.
Ruling Elders vs. Teaching Elders: Administering Christ's Authority Together
Based on this verse, Presbyterians have historically distinguished between Ruling Elders and Teaching Elders. Ruling Elders in Christ's Church primarily engage in ministerial authority by the Key of Discipline.
Teaching Elders (i.e., pastors) exercise the Key of Discipline in ministerial authority along with the Ruling Elders. Teaching Elders also, however, exercise the Key of Doctrine in declarative authority by laboring in preaching and teaching—that is, by dispensing the mysteries of God.
The Authority of Christ's Word and Christ's Name
In both these areas of discipline and doctrine, elders never rule by their own, independent authority. Rather, they operate under the strict authority of King Jesus—by Christ's Word, and in Christ's Name.
So, Pastors and Ruling Elders in Christ's Church do have authority, but all church power is only ministerial and declarative. We are only servants of Christ with no independent authority of our own. Furthermore, we are only stewards charged with laboring in preaching and teaching in order to declare and dispense the gospel of King Jesus.
Conclusion: Under Authority to Administer Christ's Authority
Pastors are first and foremost men under Christ's authority. Secondarily, Christ entrusts pastors with authority for building up the body of Christ by declaring and ministering the word of Christ.