I agree with putting them in front of a bunch of people. I make sure my peer Infrastructure managers talk to them, dev managers talk to them, and business customer managers. For the main technical interview I just bring in my entire team, usually, for both tech and fit. If everyone thinks the guy’s an a-hole, then regardless of how his skills are, it won’t be good for a collaborative environment.

You totally have to be thorough and not compromise. And that hurts – I’ve had several times where I went through an entire recruiting process – post jobs, get in 80 resumes to sort through, do a dozen phone interviews and four in-person ones… And then have a “best candidate” that I think is kinda B-. When you throw them back and start the process again, it’s tough (and can cost you months). But similar to you, every time I’ve said “Well… I’m really not feeling this guy’s level of initiative, but I really need some headcount…” it’s come back to burn me. Insisting on A players is good but hard.

Especially when we speak about SMBs, it is important to get the lingo right. There are too many so called “operations people” who are merely glorified network admins and would choke at the first sight of a forecast, employee problem, or strategy session with the board.

High-value member of operations team (especially a leader) not only will know his/her way around, but will have a track record with IT systems, HR, project management, service delivery, customer service management, accounting, and even marketing. Operations person is supposed to be ultra-generalist.

Here are two posts on the subject:
“What to Look for in a Chief Right Hand Person (COO, VP of Operations)” http://theoperationsguy.com/what-to-look-for-in-coo-head-of-operations
“Defining the Difference Between (Business) Operations and Technology Operations”
http://theoperationsguy.com/difference-between-business-and-technology-operations

On the subject of hiring technology operations people:
1. Ability to learn is king! Technologies change now on weekly basis. What was relevant yesterday is no longer relevant today. Just because you puked up every bit of tech jargon you know, that gives me absolutely no guarantee you can do the job. Show me how you think, show me how you learn, and show me you are not stuck in your dinosaur era with related IBM-era entitlement.
2. I no longer pay attention to certifications. With all the mind dumps available out there and rampant cheating, I simply no longer have trust in the certs. To add on top of that, even if you did not cheat, all it says to me is that you take multiple choice tests well.

Hopefully, new hiring managers will benefit from your experience and advice, but I also suspect that much of this needs to be learned firsthand, unfortunately.