return to OCLUG Web Site
A Django site.
May 7, 2010

» Liveblogging: Senior Skills: Sysadmin Patterns

The Beacon Pattern:
- This is a “Get out of the business” pattern
- Identify an oft-occurring and annoying task
- Automate and document it to the point of being able to hand it off to someone far less technical

- System admins were being put in charge of scheduling rooms in the building
- They wrote a PHP web application to help them automate the task
- They refined the app, documented how to use it, and handed it off to a secretary
- They have to maintain the app, but it’s far less work.

The Community Pattern:

- Prior to launch of a new service, create user documentation for it.
- Point a few early adopters at the documentation and see if they can use the service with minimal support
- Use feedback to improve documentation, and the service
- Upon launch, create a mailing list, forum, IRC channel, or Jabber chat room and ask early adopters to help test it out.
- Upon launch, your early adopters are the community, and they’ll tell new users to use the tools you’ve provided instead of calling you.

- A beowulf cluster for an academic department
- Documented like crazy, early adopters were given early access to the cluster (demand was high)
- Crated a mailing list, early adopters were added to it with their consent, functionality was tested with them.
- Email announcing launch mentioned the early adopters in a ‘thank you’ secion, and linked them to their mailing list.

The DRY pattern
DRY = Don’t repeat yourself
Identify duplicate code in your automation scripts
Put subroutines that exist in an include file, and include them in your scripts.

- “sysadmin library”
- /var/lib/adm/.*pl
- Elapsed time and # of lines to script a task for which the library was useful plunged dramatically
– new tasks were thought up that were not considered before but were obvious now (ie, users that want to change their username)
– migrating to new services became much easier

The Chameleon Pattern
- Identify commonalities among your services
- Leverage those to create “Chameleon” servers that can be re-purposed on the fly
- Abstract as much of this away from the physical hardware
- Doesn’t need to involve virtualization, though it’s awfully handy if you can do it that way.
[this one is a bit harder to do with MySQL config files]

[puppet/cfengine were mentioned...] – more than a script: a methodology

- But isn’t installing packages you don’t need bad? Depends on the package….ie, gcc is bad for enterprise

“Junior annoynances”

Terminal issues

open terminal, login to machine1
think issue is with machine2, talks to machine1.
log out of machine1
log into machine2

opens 2 terminals each of machine1 and machine2 to start

networking issue ticket arrives
logs into server
runs tcpdump

networking issue ticket arrives
logs into server
looks at logs

“Fix” vs. “Solution” ie “taking orders”
Junior will try fix a problem, senior will try to figure out what the problem is. ie, “I need a samba directory mounted under an NFS mount” a junior admin will try to do exactly that, a senior admin will ask “what are you trying to do with that?” because maybe all they need is a symlink.

Signs you might be a fanboy:
- Disparaging users of latest stable release of $THING for not using the nightly (unstable) build which fixes more issues
- Creating false/invalid comparisons based on popular opinion instead of experience/facts
- Going against internal standards, breaking environmental consistency, to use $THING instead of $STANDARD (but this is also how disruptive technology works)
- Being in complete denial that most technology at some point or another stinks.
- Evaluating solutions based on “I like” instead of “we need” and “this does”.

» Liveblogging: Seeking Senior and Beyond

I am attending the Professional IT Community Conference – it is put on by the League of Professional System Administrators (LOPSA), and is a 2-day community conference. There are technical and “soft” topics — the audience is system administrators. While technical topics such as Essential IPv6 for Linux Administrators are not essential for my job, many of the “soft” topics are directly applicable and relevant to DBAs too. (I am speaking on How to Stop Hating MySQL tomorrow.)

So I am in Seeking Senior and Beyond: The Tech Skills That Get You Promoted. The first part talks about the definition of what it means to be senior, and it completely relates to DBA work:
works and plays well with other
understands “ability”
leads by example
lives to share knowledge
understands “Service”
thoughtful of the consequences of their actions
understands projects
cool under pressure

Good Qualities:

Bad Qualities:
[my own addition - no follow through, lack of attention to detail]

The Dice/Monster Factor – what do job sites see as important for a senior position?

They back up the SAGE 5-year experience requirement
Ability to code in newer languages (Ruby/Python) is more prevalent (perhaps cloud-induced?)

The cloud allows sysadmin tasks to be done by anyone… developers can do sysadmin work, and you end up seeing schizophrenic job descriptions such as

About the 5-year requirement:
- Senior after 5 years? What happens after 10 years?
- Most electricians, by comparison, haven’t even completed an *apprenticeship* in 5 years.

Senior Administrators Code
- not just 20-line shell scripts
- coding skills are part of a sysadmin skill
- ability to code competently *is* a factor that separates juniors from seniors
- hiring managers expect senior admins to be competent coders.

If you are not a coder
- pick a language, any language
- do not listen to fans, find one that fits how you think, they all work…..
- …that being said, some languages are more practical than others (ie, .NET probably is not the best language to learn if you are a Unix sysadmin).

Popular admin languages:
- Perl: classic admin scripting language. Learn at least the basics, because you will see it in any environment that has been around for more than 5 years.

- Ruby: object-oriented language for people who mostly like Perl (except for its OO implementation)

- Python: object-oriented language for people who mostly hate Perl, objects or no objects. For example, you don’t have to create a String object to send an output.

But what if you do not have time to learn how to program?

- senior admins are better at managing their time than junior admins, so perhaps managing time
- time management means you’ll have more time to do things, it doesn’t mean all work work work.
- Read Time Management for System Administrators – there is Google Video of a presentation by the author, Tom Limoncelli.

Consider “The Cloud”
- starting to use developer APIs to perform sysadmin tasks, so learning programming is good.
- still growing, could supplant large portions of datacenter real estate
- a coder with sysadmin knowledge: Good
- a sysadmin with coding knowledge: Good
- a coder without sysadmin knowledge: OK
- a sysadmin with no coding interest/experience: Tough place to be in

Senior Admins Have Problems Too
Many don’t document or share knowledge
Maany don’t do a good job keeping up with their craft
Cannot always be highlighted as an example of how to deal with clients
Often reinvent the wheel – also usually there is no repository
Often don’t progress beyond the “senior admin” role

….on the other hand…..
cynicism can be good…..

learn from the good traits
observe how others respond to their bad traits
think about how you might improve upon that
strive to work and play well with others, even if you don’t have a mentor for good/bad examples.

Now he’s going into talking about Patterns in System Administration….