Fixing UNIX Filenames – An interesting discussion of the non-standard handling of the “standards” in UNIX filesystem naming, such as newlines in names.

An ‘Accordion’ of Wood and Glass – A look at where all the money you spent on your calculus textbook went.

Strace – the Sysadmin’s Microscope – An excellent article on using strace(1), the best way to find out what your process is really doing in Linux.

Print

Garner’s Modern American Usage - Okay, I’ve not read the whole thing, but I keep it on my desk at work to figure out if my word choice is correct, or if it makes me sound like an asshole (and, all of the usage examples are from recent media, including great references to Superbad when needed).

Web Operations – Allspaw is listed as the author but this book is written by many people in the DevOps community (and hey, I reviewed Patrick DeBois’ chapter on monitoring). This book offers excellent practical advice to people doing web operations.

Rework – The 37signals guys wrote this book about the lessons they’ve learned running a successful startup, and while the book is pretentious, it does have solid advice (for example: don’t hire fast; don’t worry about being “professional”, build software you want to use).

The format was a shortened pub trivia, with songs played after reading each question:

• 1 round of 4 questions with each question being worth either 1, 3, 5 or 7 points (you decide, you can only use each value once)
• 1 special picture round (2 points for each item correctly identified)
• 1 round of 4 questions with each question being worth 2, 4, 6 or 8 points (akin to the first round)
• 2 final questions on which you wager up to 10 points each (win or lose what you wager)

Without further ado, here are the questions:

Famous Nerds: Who wrote in a famous RFC, “be conservative in what you do, be liberal in what you accept from others”?

Databases: What does ACID stand for?

Networking: How many bytes are there in an Ethernet address?

Systems Programming: What is the only UNIX syscall that returns twice?

Data Structures: This colorful binary tree is self-balancing and contains no data in the tree’s leaf nodes – what is it?

Operating Systems: Which process typically has the PID of 1 on a UNIX system?

Computing History: Which of these technologies was not invented at XEROX PARC: Ethernet, The Mouse, The Windowing GUI, laser printer?

SciFi/Fantasy: Who is the Kwistaz Haderach?

Video Games: What item must you retrieve from the Dungeons Of Doom in Nethack?

Role Playing Games: What does THAC0 stand for?

And of course, we had a picture round: ID The Programming Language & ID Carrier/GDS By IATA Code (hey, we’re an airline software company, I have to have some airline trivia).

I won’t post the answers here but feel free to post your answers in the comments (and be aware there’s a few correct answers to some of these questions).

Also, if you’re a nerd looking for a job at a cool place to work, check out ITA’s current job offerings. Not to spam my readers but it really is fun to work at a place where you can host a nerd pub trivia and 10 teams join the fun.

Vladimir has setup a registration link here.

This promises to be a great show; Vlad and Jeff Buchbinder will be giving a presentation on lessons learned while reengineering deployments at their companies. You can read more about what is on the agenda on this post on Vlad’s blog.

See you there – Tuesday, August 3rd, 2010, from 6pm until 8pm.

No Agenda, No Meeting

Meetings without agendas are likely to go nowhere and do nothing. Even recurring meetings need agendas. The agenda should be a few lines covering the specific reason(s) for the meeting, and this agenda should be emailed to the people who will be at the meeting at least 24 hours before the meeting. The reason for emailing the agenda, rather than only putting the meeting agenda in the invitation, is that people will often reply to the emailed agenda addressing the very questions you wanted to answer in the meeting. Sometimes, you’ll address all the reasons for having that meeting via email and you don’t need to have the meeting – which is always a good outcome. Emailed agendas are like the pre-interview phone screen – if the agenda can’t get past this step, no need to waste everyone’s time in person.

An agenda also serves to focus the meeting as it progresses, helping people keep the meeting on track.

Always Take Notes

You, as the organizer, should take detailed notes (except for the times when you are at the whiteboard, or other such situation). You set up the meeting, you have to take the notes. It was important enough for you to break up every attendee’s day, and to cost the company the combined salary of the people you invited, so you should make sure you record what happened in the meeting. Good notes contain:

• A list of attendees
• A copy of the agenda
• Short summaries of discussions/disagreements
• A list of what was decided, including who will do what

You don’t need to worry about grammar and spelling as you take the notes – you can clean them up later. What you do need to do is email all the attendees the notes (some of my friends check the notes into revision control as well; this is a good practice that I need to adopt). When you email out the notes you’re giving people time to correct your notes and clarify what everyone meant.

If you whiteboarded something during the meeting, take a picture of it and include the picture in the email.

Also, if you have trouble paying attention in meetings (like I do), volunteering to take notes is a great way to stay focused on the meeting.

Let People Leave The Meeting

If someone doesn’t need to be there for the whole meeting, don’t make them stay. Let them go after they’ve contributed their part.

Meetings Happen In Person….

Or do they? Scrums and other short status meetings are good candidates for meeting over IRC. A friend of mine who runs a lot of distributed groups runs many of his shorter (and longer) meetings this way, with the help of a few tricks like an IRC bot that can be told when the meeting has started and stopped: after the stop command is received the bot will make a web page from the meeting notes. Scrums are particularly amenable to this format because people can post their prepared list of updates into the chat window for all to see and comment on. Another advantage of scrums or status reporting meetings over IRC is that people don’t have to leave their desks (this seems obvious, but making meetings like scrums fit into people’s workflows is important). Meetings over IRC also let people observe a meeting by lurking in the chat room, which is something that is harder to do in person.

IRC can be a useful tool for in-person meetings as well, providing a place to paste texts, share links, record notes, and so on. I think that Google Wave (or something like SubEthaEdit) has potential for this purpose as well, but haven’t tried it.

What Works For You?

What techniques do you use to maximize the value of your meetings? I’d love to hear about your experiences in the comments.

When I’m not being a computer nerd I’m often studying music (I’ve even created website which helps me study), and in an effort to keep me and my friends making music I’ve started Iron Musician, where my friends and I must create at least a minute of music once week or owe $5 to the beer pool.

Take a listen to some of the entries and feel to free to provide some constructive criticism.

Give me back my list of key performance indicators!

Last week I brought up the topic of measuring managers. I had just gone through review season at work and (as is typically for my group at work), we’d spent a lot of time revising our review process for individual contributors and for team leads. We spend a lot of time on reviews because reviews are important–reviews are when you not only tell someone how they are performing at work, but reward or punish them with raises, bonuses and promotions–and I think at work the group I’m in has come up with a system that sucks less than most systems I’ve been part of. The problem is that my group has now grown to the point where our system needs to measure and review people managers.

I brought up the topic of measuring managers during the last Breakfast Club call so I could hear other my friends’ thoughts on the matter. Below are some of the ideas from this phone call.

I need to define what skills makes a good manager.

At work, we track eleven different skills that we think are required for individual contributors and team leads, and we give each of these skills a weighting depending on the role. My friends had similar systems but with manager-specific skills and I think that’s what I need to do as well. Which skills are important for a manager is different for every company and for every level of management.

I haven’t defined the skills that are important for management at my work in my group yet (I’ve set aside time for this in the next few weeks), and I’m going to this with input from many of the people in my group. Any suggestions for skills that make up a good manager are appreciated.

Manager reviews, just like individual contributor reviews, need evidence.

I try to include evidence for my comments in any review, and I try to have that evidence show that some behavior was observed over time. For example, I’ll include ticket, bug, email or commit information as evidence, and I’ll include at least two data points, one from the first half of the review period and the other from the second half of the review period. You’d be surprised at how much you’ve forgotten someone has done (or not done). You’ll quickly realize that your intuition about how a review for someone should go is biased towards that person’s performance in the last month or so.

For managers, this evidence is harder to extract, as the artifacts of management are often decisions that aren’t in a formal tracking system. You’ll need to dig deeper, often in emails or meeting notes, to find the evidence that shows a manager making a decision, and you’ll need to keep on digging to find the reasons behind that decision and the outcome of that decision. Furthermore, by reviewing email threads and notes, you get an (incomplete) picture of the manager’s treatment of other people.

360 degree reviews are a good way to gather evidence, but be aware that the responses in a 360 have the same time bias I warned about before, and you should consider the motivations behind any response given in a 360.

I would go far as to say that a review that doesn’t have evidence for a majority of the comments is a poorly done review and the reviewer has failed to do his or her job.

Managers need to be evaluated in the context of their team, but that does not mean a manager’s review is based only the team’s deliverables.

Team-based deliverables should not be a manager’s only deliverables. I’ve worked in companies where everyone but the manager’s manager knows that the team is full of superstars doing great work in spite of their manager. The same team could be managed by my cat and that team would still do amazing work (there’s a point in here about knowing what kind of management is needed and when, but that’s for another day). On the other hand, a manager can make an entire team superstars by removing the barriers to the team’s effectiveness, or getting the team the resources it needs, or by clearly setting direction, or acting as a “shit umbrella” to help the team avoid distractions. On the third hand, a manager could do all those wonderful things for his team and still miss a team deliverable for a hundred other reasons.

I think the important thing about team-based deliverables is that you still need to balance the delivery of those objectives with the evaluation of the manager’s skills, and then you need to evaluate all of the evidence found during the review process.

Final Thoughts.

I’m going to spend some time reading up on manager evaluation on the Internet, and also reading many books with bad titles from the business section of the book store. What do you think about reviewing managers? How is it done where you work? If you’re an individual contributor, do you think your manager is being well-reviewed by his or her boss?

DevOps Days is the day after Velocity 2010 (which I sadly won’t be able to make), so if you are already coming out for Velocity you should stay an extra day and come to DevOps Days!

I’ve mostly hired “DevOps” people

Almost all of my hires have been for teams that are either too small to have defined boundaries or for teams that have charters that involve problems that don’t fit into traditional operations roles (“System Engineer II”). I’m much more interested in building a team of people that will own problems and find solutions regardless of their job description and have therefore tried to avoid creating jobs that rely on a specific skill. I want people who will go outside their role boundaries and marshal the technical resources needed to solve a problem correctly.

I want to hire people to do people work and let computers do computer work. People are good at creative problem solving; they’re good at understanding subtlety and context and intention. Computers are very good at doing the same thing over and over again very quickly. In other words, if it can be automated, I want to hire the people who will automate it and move on to harder, more human problems. I want to hire smart people who can learn a particular tool if necessary, not hire people just because they’ve used that tool before.

That is not to say I like to hire purely creative people with no thought to their technical ability, but rather that I don’t hire based on knowledge of a specific technology. For example, I may run postfix on my mail servers but I’d still hire someone with sendmail experience if that candidate understood internet mail routing really well.

On the other hand, there’s a baseline level of technical ability you need to do your job well. I’m not hiring people to teach them Linux. You need to be experienced with the basics so that you can get to work building better tools and solving hard problems for the company that hired you. We’ve got work to do, after all.

I’ve made a lot of mistakes in hiring

Here are just a few of the many mistakes I’ve made, and some of the lessons I’ve learned from those mistakes:

Rushing a hire because of a current need (a specific project, a needed skill on a team, a work overload).

Your best admin is quitting work to become a piano tuner and he’s taking his storage management expertise with him. You need a replacement stat. What you should not do is rush to hire the first admin with storage experience on his or her resume, because it is inevitable that the hire will work on a diverse set of technologies in their career at your company. Spend the time to hire an admin who you’ll later think of as your best admin.

Some tips for speeding up hiring without sacrificing quality:

• Fast track your internal references, especially from your most valued employees. Smart people are friends with other smart people.
• Use phone screens to test candidates for the requisite technical ability; phone screens should have questions with specific answers on topics the candidate simply must know. You’ll weed out a lot of unqualified people in a 30 minute phone call.
• If they pass the first phone, do a 45 minute phone interview; discuss an interesting topic from their resume.
• Don’t be afraid to walk someone out early; it’ll save you time and they probably already know the interview isn’t going well.
• ITA is well known for our pre-interview programming puzzles, and there’s legitimate debate about this process, but if you don’t require a puzzle at least ask for a code sample or equivalent. Like the technical phone screen, this process can help identify candidates for in-person interviews.

Not interviewing thoroughly enough.

This is especially important in a small company where the success or failure of a hire can have a drastic effect on the health of the company. Have the candidate talk to at least 5 people, and make sure most of them are technical people. It’s also important to discuss who will talk about what in each interview before the interview. Otherwise, you’ll all talk to the candidate about the same interesting bit on his or her resume. Don’t hesitate to bring the candidate back for another round of interviews if you think it is warranted.

Compromising on ability.

I made this mistake at a previous job after being unable to find the correct person for the job for months. I eventually hired a person simply to get a body in, and I knew the person wasn’t good enough for the job. Sure enough we had to part ways after a few months. Firing someone is expensive and time consuming, and you’re left back where you started. Part of the problem with this hire was that my budget wasn’t correct for the skill set I wanted, but in the end it was me compromising that led to the failure.

Some lessons from this mistake:

• Fight for the correct budget for the skills you need.
• Make sure everyone agrees the person is correct for the job.
• If you haven’t hired for a position you’ve had open for a long time, perhaps you need to re-evaluate what you are looking for and if you really need the hire.

Thinking I’ll have the time to train someone in skills fundamental to the job.

I feel worst about this mistake because I should have known better; I was already overloaded at work, so what made me think I could train someone in my zero free time? Even if you think you have the free time it is inevitable that work will take back that free time.

What didn’t work for me may work for you…

… but it also may not, so be careful. Hopefully you’ll learn from my mistakes in any future hiring you do.