We'll see | Matt Zimmerman

For the first time in several years, I had the opportunity to attend a software conference in the city where I lived at the time. I’ve benefited from many InfoQ articles in the past couple of years, and watched recordings of some excellent talks from previous QCon events, so I jumped at the opportunity to attend QCon London 2010. It is being held in the Queen Elizabeth II Conference Center, conveniently located a short walk away from Canonical’s London office.

Whenever I attend conferences, I can’t help taking note of which operating systems are in use, and this tells me something about the audience. I was surprised to notice that in addition to the expected Mac and Windows presence, there was a substantial Ubuntu contingent and some Fedora as well.

Today’s tracks included two of particular interest to me at the moment: Dev and Ops: A single team and the unfortunately gendered Software Craftsmanship.

Jason Gorman: Beyond Masters and Apprentices

A Scalable, Peer-led Model For Building Good Habits In Large & Diverse Development Teams

Jason explained the method he uses to coach software developers.
I got a bad seat on the left side of the auditorium, where it was hard to see the slides because they were blocked by the lectern, so I may have missed a few points.

He began by outlining some of the primary factors which make software more difficult to change over time:

• Readability: developers spend a lot of their time trying to understand code that they (or someone else) have written
• Complexity: as well as making code more difficult to understand, complexity increases the chance of errors. More complex code can fail in more ways.
• Duplication: when code is duplicated, it’s more difficult to change because we need to keep track of the copies and often change them all
• Dependencies and the “ripple effect”: highly interdependent code is more difficult to change, because a change in one place requires corresponding changes elsewhere
• Regression Test Assurance: I didn’t quite follow how this fit into the list, to be honest. Regression tests are supposed to make it easier to change the code, because errors can be caught more easily.

He then outlined the fundamental principles of his method:

• Focus on Learning over Teaching – a motivated learner will find their own way, so focus on enabling them to pull the lesson rather than pushing it to them (“there is a big difference between knowing how to do something and being able to do it”)
• Focus on Ability over Knowledge – learn by doing, and evaluate progress through practice as well (“how do you know when a juggler can juggle?”)

…and went on to outline the process from start to finish:

1. Orientation, where peers agree on good habits related to the subject being learned. The goal seemed to be to draw out knowledge from the group, allowing them to define their own school of thought with regard to how the work should be done. In other words, learn to do what they know, rather than trying to inject knowledge.
2. Practice programming, trying to exercise these habits and learn “the right way to do it”
3. Evaluation through peer review, where team members pair up and observe each other. Over the course of 40-60 hours, they watch each other program and check off where they are observed practicing the habits.
4. Assessment, where learners practice a time-boxed programming exercise, which is recorded. The focus is on methodical correctness, not speed of progress. Observers watch the recording (which only displays the code), and note instances where the habit was not practiced. The assessment is passed only if less than three errors are noticed.
5. Recognition, which comes through a certificate issued by the coach, but also through admission to a networking group on LinkedIn, promoting peer recognition

Jason noted that this method of assessing was good practice in itself, helping learners to practice pairing and observation in a rigorous way.

After the principal coach coaches a pilot group, the pilot group then goes on to coach others while they study the next stage of material.

To conclude, Jason gave us a live demo of the assessment technique, by launching Eclipse and writing a simple class using TDD live on the projector. The audience were provided with worksheets containing a list of the habits to observe, and instructed to note instances where he did not practice them.

Julian Simpson: Siloes are for farmers

Production deployments using all your team

After a brief introduction to the problems targeted by the devops approach, Julian offered some advice on how to do it right.

He began with the people issues, reminding us of Weinberg’s second law, which is “no matter what they tell you, it’s always a people problem”.

His people tips:

• In keeping with a recent trend, he criticized email as a severely flawed communication medium, best avoided.
• respect everyone
• have lunch with people on the other side of the wall
• discuss your problems with other groups (don’t just ask for a specific solution)
• invite everyone to stand-ups and retrospectives
• co-locate the sysadmins and developers (thomas allen)

Next, a few process suggestions:

• Avoid code ownership generally (or rather, promote joint/collective ownership)
• Pair developers with sysadmins
• It’s done when the code is in production (I would rephrase as: it’s not done until the code is in production)

and then tools:

• Teach your sysadmins to use version control
• Help your developers write performant code
• Help developers with managing their dev environment
• Run your deploy scripts via continuous integration (leading toward continuous deployment)
• Use Puppet or Chef (useful as a form of documentation as well as deployment tools, and on developer workstations as well as servers)
• Integrate monitoring and continuous integration (test monitoring in the development environment)
• Deliver code as OS packages (e.g. RPM, DEB)
• Separate binaries and configuration
• Harden systems immediately and enable logging for tuning security configuration (i.e. configure developer workstations with real security, making the development environment closer to production)
• Give developers access to production logs and data
• Re-create the developer environment often (to clear out accumulated cruft)

I agreed with a lot of what was said, objected to some, and lacked clarity on a few points. I think this kind of material is well suited to a multi-way BOF style discussion rather than a presentation format, and would have liked more opportunity for discussion.

Lars George and Fabrizio Schmidt: Social networks and the Richness of Data

Getting distributed webservices done with Nosql

Lars and Fabrizio described the general “social network problem”, and how they went about solving it. This problem space involves the processing, aggregation and dissemination of notifications for a very high volume of events, as commonly manifest in social networking websites such as Facebook and Twitter which connect people to each other to share updates. Apparently simple functionality, such as displaying the most recent updates from one’s “friends”, quickly become complex at scale.

As an example of the magnitude of the problem, he explained that they process 18 million events per day, and how in the course of storing and sharing these across the social graph, some operations peak as high as 150,000 per second. Such large and rapidly changing data sets represent a serious scaling challenge.

They originally built a monolithic, synchronous system called Phoenix, built on:

• LAMP frontends: Apache+PHP+APC (500 of them)
• Sharded MySQL multi-master databases (150 of them)
• memcache nodes with 1TB+ (60 of them)

They then added on asynchronous services alongside this, to handle things like Twitter and mobile devices, using Java (Tomcat) and RabbitMQ. The web frontend would send out AMQP messages, which would then be picked up by the asynchronous services, which would (where applicable) communicate back to Phoenix through an HTTP API call.

When the time came to re-architect their activity , they identified the following requirements:

• endless scalability
• storage- and cloud-independent
• fast
• flexible and extensible data model

This led them to an architecture based on:

• Nginx + Janitor
• Embedded Jetty + RESTeasy
• NoSQL storage backends (no fewer than three: Redis, Voldemort and Hazelcast)

They described this architecture in depth. The things which stood out for me were:

• They used different update strategies (push vs. pull) depending on the level of fan-out for the node (i.e. number of “friends”)
• They implemented a time-based activity filter which recorded a global timeline, from minutes out to days. Rather than traversing all of the user’s “friends” looking for events, they just scan the most recent events to see if their friends appear there.
• They created a distributed, scalable concurrent ID generator based on Hazelcast, which uses distributed locking to assign ranges to nodes, so that nodes can then quickly (locally) assign individual IDs
• It’s interesting how many of the off-the-shelf components had native scaling, replication, and sharding features. This sort of thing is effectively standard equipment now.

Their list of lessons learned:

• Start benchmarking and profiling your app early
• A fast and easy deployment keeps motivation high
• Configure Voldemort carefully (especially on large heap machines)
• Read the mailing lists of the NoSQL system you use
• No solution in docs? – read the sources
• At some point stop discussing and just do it

Andres Kitt: Building Skype

Learnings from almost five years as a Skype Architect

Andres began with an overview of Skype, which serves 800,000 registered users per employee (650 vs. 521 million). Their core team is based in Estonia. Their main functionality is peer-to-peer, but they do need substantial server infrastructure (PHP, C, C++, PostgreSQL) for things like peer-to-peer supporting glue, e-commerce and SIP integration. Skype uses PostgreSQL heavily in some interesting ways, in a complex multi-tiered architecture of databases and proxies.

His first lesson was that technical rules of thumb can lead us astray. It is always tempting to use patterns that have worked for us previously, in a different project, team or company, but they may not be right for another context. They can and should be used as a starting point for discussion, but not presumed to be the solution.

Second, he emphasized the importance of paying attention to functional architecture, not only technical architecture. As an example, he showed how the Skype web store, which sells only 4 products (skype in, skype out, voicemail, and subscription bundles of the previous three) became incredibly complex, because no one was responsible for this. Complex functional architecture leads to complex technical architecture, which is undesirable as he noted in his next point.

Keep it simple: minimize functionality, and minimize complexity. He gave an example of how their queuing system’s performance and scalability were greatly enhanced by removing functionality (the guarantee to deliver messages exactly once), which enabled the simplification of the system.

He also shared some organizational learnings, which I appreciated. Maybe my filters are playing tricks on me, but it seems as if more and more discussion of software engineering is focusing on organizing people. I interpret this as a sign of growing maturity in the industry, which (as Andres noted) has its roots in a somewhat asocial culture.

He noted that architecture needs to fit your organization. Design needs to be measured primarily by how well they solve business problems, rather than beauty or elegance.

He stressed the importance of communication, a term which I think is becoming so overused and diluted in organizations that it is not very useful. It’s used to refer to everything from roles and responsibilities, to personal relationships, to cultural norming, and more. In the case of Skype, what Andres learned was the importance of organizing and empowering people to facilitate alignment, information flow and understanding between different parts of the business. Skype evolved an architecture team which interfaces between (multiple) business units and (multiple) engineering teams, helping each to understand the other and taking responsibility for the overall system design.

Conclusion

Overall, I thought the day’s talks gave me new insight into how Internet applications are being developed and deployed in the real world today. They affirmed some of what I’ve been wondering about, and gave me some new things to think about as well. I’m looking forward to tomorrow.

Seeing Process: Making Process Visible (Steve Smith)

This session allowed me to practice capturing and visualizing the processes used by a team. In order to create a process to work with, we first conducted a simulation, where we (about 15 participants) organized ourselves to produce a product (sorted decks of cards) from raw materials (many cards from different decks). Volunteers took on the roles of the customer, the customer’s QA manager, the factory owner and the factory production manager, while the rest of us (myself included) acted as factory workers. As a manager in my day job, I wanted to get a different perspective in this exercise, and practice different kinds of leadership.

While the customer and management were sequestered for further instruction, the workers waited outside and talked about what would happen. There was quite a bit of talk of disrupting the activity, making things difficult for management, which I hoped wouldn’t amount to anything once we got started. The point of the exercise was to learn something, and being too disruptive would hinder that.

More interestingly, several of the workers had participated in other, similar simulations, and so were keen to avoid the obvious pitfalls. We collectively resolved to work cooperatively and treat each other well during the simulation: to act as a team. This turned out to be a defining moment in the simulation.

The appointed manager started out by briefing us on the parameters of the simulation, insofar as he was aware of them, and setting some ground rules. He explained that there was a mess inside, preparing us for what to expect. Realizing the group might be too big for the task at hand, he explained that some people might not be needed, and they should find something to do on their own without interfering with the rest of the group. This was also a key action.

The manager also made a point of wielding explicit power, which I found a bit excessive. He said there would be layoffs, and threatened that as punishment for the workers. Particularly given the team dynamic, this threatened to unite the team against him, which I felt was an unnecessary risk.

When we were ready to start work, the manager gave a lot of explicit instruction, providing an algorithm for the first phase of sorting. It wasn’t a very good algorithm, and this was evident to some members of the group, but the workers didn’t seem to feel it was worth the time which would be wasted arguing. That is, all but one, who caused the group to block for a couple of minutes by refusing to cooperate or suggest an alternative.

What was missing at this point was an understanding of the goal. The team was prepared to work on the task, but without knowing what the result was expected to be, they hesitated and were confused. The manager offered a vote on a small change to the algorithm, and it was agreed. Gradually, we established that the result should be piles of cards sorted according to the pattern on the back, and the team quickly self-organized around this goal, largely disregarding the proposed algorithms. This beautifully illustrated one of the lessons of the exercise, by showing how quickly and naturally processes diverge from what is prescribed.

I took quite an active role in learning the parameters of the simulation and helping to facilitate the group’s self-organization. During the initial card-sorting phase, I established a clearinghouse for cards of all types, and invited my colleagues to bring me any type of card to add to my stacks. This simple act made it obvious how to contribute to the goal, and organized the physical materials for later use. Someone else was doing the same thing, and when all of the cards were stacked, we just combined our stacks.

The manager explained that we needed to sort the decks. I asked what ordering should be used, and he pointed to a flipchart where instructions were written. The workers took the stacks of cards off to various parts of the room to sort them. Once I had one complete, I took it to the QA manager, who said it was incorrect. I checked it against the instructions, and it was clear that they were not effective in communicating the correct ordering. I rewrote them on another flipchart, checked them with the QA manager, and took the old instructions down.

Before I went back to sorting cards, I talked to other people sorting, and made sure that they knew about the new instructions. I checked if they had any complete decks, helped them do the final (quick) sorting by suit, and carried their completed decks to QA. This simple action which both boosted our immediate output (by pushing work in progress through the process) and accelerated production (by spreading information about the requirements).

The next time I handed in a batch of cards to QA, I was recruited to work at the QA table, because there was a shortage of capacity there. After checking a couple of decks, I had had experience with every stage of the process in the “factory” and thus a strong working knowledge of the overall process. For the rest of the time, I sought out work in progress on the factory floor and helped to finish it off. Wherever I found stacks of cards, I just sorted them and took them to QA.

Throughout this process, I remember hearing the disconcerted voice of the owner, who seemed to be worried about everything. She was frightened when I fixed the instructions on the flipchart, confused when I was changing roles or anything else unexpected. She noticed when people were not working, and wanted the manager to do something about it. I’m not sure how that role was structured, and what pressures she was responding to. I had no contact at all with the customer.

Once the simulation was over, the customer checked several of the delivered products, rejecting one which had an error, but still ending up with enough to meet the requirements. All told, the simulation ran very smoothly, and I was keen to analyze why.

Next, each of us drew a diagram of the process from our point of view. I often struggle with creating diagrams by hand, because I tend to work out the structure as I go. This time, I stopped for a moment and sketched the major phases of the process in my notebook before I started drawing. This resulted in a much clearer and neater diagram than I normally produce.

We discussed the diagrams as a group, which revealed a lot about how things had really worked, more than anyone had been able to see individually. It was surprising how much information was disseminated in such a short amount of time through drawing and discussing the diagrams. It helped that the entire exercise was very fresh in our minds, but I got the impression that this would be a useful way to gather and share information from a group under normal circumstances as well.

I had to leave a bit early in order to catch my flight, so I missed the end of the discussion.

Today is my last day at the conference, and I’ll be leaving before the welcome dinner in order to catch my flight to Dallas. It has been a pleasure meeting everyone, and an invaluable opportunity to get perspective on my work and other parts of my life.

Morning session: Retrospectives (Esther Derby)

The full title of this session was Looking Back, Moving Forward: Continuous Improvement With Effective Retrospectives. I find retrospectives to be a useful tool for process improvement, and am interested in gaining more experience with them and learning new techniques.

To start, the group (about 12 people) conducted a project together. We were provided with a specification for a 3-dimensional model of an urban development project, and some supplies with which to build it. Over the course of 30 minutes, we organized ourselves, performed the task, and successfully delivered the result to a satisfied customer.

Overall, the project went very well, so I was looking forward to the retrospective. In a retrospective, I often find myself focusing on the problems, so this would be good practice in identifying and repeating successes.

To start, we “set the stage” for the retrospective by reviewing the agenda and “working agreements” (pre-established goals and guidelines for behavior). I always try to do this in retrospectives, but the idea of agreeing the guidelines in advance was new to me. This would both help to build support in the team, and save time in the meeting.

Then, we conducted a quick poll of the group to assess the outcome of the project, using a weather metaphor: was it “partly sunny”, “cloudy”, “rain”, etc. I liked this metaphor because I found it more neutral than a rating: weather is something which just happens and is observable, while numbers, for example, seem to carry more judgement.

Next, we broke into smaller groups to collect data (our observations) on the project we had just completed. I liked the idea of doing this in smaller groups, as it promoted more points of view, and also produced some redundant data (which was useful in the analysis to follow). We wrote these observations on sticky notes.

To analyze the data, everyone came back together and plotted it on a “FRIM” chart (frequency vs. impact), where one axis indicated frequency (how often it occurred during the project?) and the other impact (how much impact did it have, and was it positive or negative?). We looked at this data from near and far, looking for patterns and considering individual notes.

The upper left quadrant (low frequency and positive impact) was dominated things which happened near the beginning of the project: establishing roles, establishing the workspace, etc. The lower left quadrant (low frequency and negative impact) showed mostly items to do with ambiguity: lack of information, confusion and so on. The upper right quadrant (high frequency and positive impact) included ongoing teamwork, the ongoing progress which helped coalesce and motivate the team. There was nothing in the lower right quadrant (high frequency and negative impact), which I took as a sign of a healthy project. The fact that some of the observations were redundant helped make the patterns clearer, by showing what had been noticed more.

With these observations in mind, we broke into smaller groups again, this time with the objective of generating ideas for improvement, or for repeating our successes. I thought that splitting up the group would bring more ideas out, but in fact we came up with many of the same suggestions. Still, this helped to reinforce the suggestions because they were independently proposed by different groups.

The team was reformed, and suggestions were posted on a flipchart page with a table for evaluation. The team rated the suggestions according to the amount of effort required, the anticipated impact, and their individual willingness to work on it. Finally, based on this data, Esther asked for a volunteer to implement the most promising suggestion, choosing just one to keep the amount of change manageable. She asked for a second volunteer to back up and support the first. I liked the idea of rating the suggestions together, and also having someone in a supporting role for the followup action.

We then polled the team again, reusing the weather metaphor, to predict the “weather” for the next project, based on what we learned. Most of the group was optimistic, though a couple of people predicted storms. When asked about their concerns, one of them pointed out that our decision to focus on improving one aspect of our process could cause us to waste too much time up front working on that. To mitigate this, we resolved to limit the amount of time we would spend on it, and he seemed satisfied. The other doomsayer was unrepentant, and said he suspected we would fall prey to the “second system” effect. I got the impression that he wanted to be proven right, and wasn’t interested in avoiding such a problem.

We concluded by reviewing the overall structure of the retrospective, and how the stages fit together: setting the stage, gathering data, analyzing it, selecting actions and wrapping up. The wrap-up included a retrospective on the retrospective itself, to promote improvement of that process. I was a bit concerned we would end up in infinite regress, but Esther stopped there, and didn’t do a retrospective of the retrospective of the retrospective. I don’t normally even do one meta-retrospective, but am considering trying that out now.

Morning Session: How Do I Communicate With You? (Don Gray)

This session explored interpersonal communication, in particular how to identify communication styles and preferences and apply this knowledge to communicate more effectively.

In one of the exercises, we broke into pairs and exchanged stories from recent memory. While one person told their story, the other observed their language, noting the types of predicates used (visual, auditory, kinesthetic, olfactory/gustatory and “unspecified”) as indicators of the speaker’s representational system. We used this data as the basis for a discussion about cues which indicate a person’s preferences, ranging from word choice to eye movement.

Another exercise divided the group into two based on personality type (Myers-Briggs “N” and “S”). Both groups were given identical objects (Starbucks paper coffee cups) and asked to write down descriptive phrases. The “N” group’s descriptions were wide-ranging, exploring the possible uses of the cup, the memories and meanings it evoked in them. The “S” group focused more on the physical properties of the cup, and where there were some comments on the ideas associated with it, these were carefully separated on a separate list.

A third exercise had us folding a piece of paper according to instructions read aloud. The instructions were ambiguous, and would be interpreted differently depending on how the person happened to be holding the paper, how they turned it as they folded it, or how they interpreted the sentences. Unsurprisingly, everyone ended up with a different pattern.

There was then some free-flowing discussion about communication styles where people in the group shared experiences of communication challenges. I learned more from these scenarios than from the exercises, which were covering familiar ground. The challenge, for me, is applying what I know about communication theory, by raising my awareness of patterns in real life situations. Practicing this through discussing examples was more helpful to me than exploring more theory.

Finally, the group was presented with a choice of whether to explore a canned scenario (a new executive looking for feedback from others in the organization) or a real-world scenario put forward by someone in the group. The latter option prevailed, but the scenario was a price negotiation with few details, which I didn’t think would be valuable for me. I excused myself and left early, purchasing a copy of Tom DeMarco’s Slack from the book table.

Afternoon session: Beyond the Org Chart Illusions (Jerry Weinberg)

This session alone would have made this conference worthwhile. In it, we explored the structure and dynamics of organizations from a first person point of view. The idea was to discover the tacit structures which make up organizational culture. Setting aside the “objective” description provided by an organizational chart or other such tool, we instead created individual representations of the organization as we experienced it.

To do this, we used an exercise based on Virginia Satir’s technique of family mapping. We each selected an organization we were familiar with, and drew a picture with symbols: first ourselves, then other people (both with no names or words), then physical objects and structures, and labels. We then overlaid the points of pain, pleasure, problems, plans, performance (high and low) and power in the organization.

Unsurprisingly, the diagrams were all vastly different in symbolism, structure, order, level and technique. Jerry focused the group on a pair of diagrams created by two different people in the same organization. The creators explained the meanings of the symbols they used, illustrating the problems faced by the organization. This spawned a discussion about how to address those problems, which occupied the rest of our time.

This discussion really got me thinking, and drawing parallels with my own experience. In particular, i identified with some of the fears and other roadblocks which prevented the people from taking action. One memorable point from Jerry was that you will never convince someone with logic to change something they did not arrive at through logic. I make that mistake a lot.

I also gained some insight on how companies can be successful despite poor management, at least for a time. For example, circumstances may favor the company, such as having little or no competition. Advantages like that don’t last, though, and when things change, the management gap can cripple the company.

I can’t describe here much of what I learned in this session, but I found it extremely valuable.

Morning Session: Project Patterns

I chose to attend a session entitled: Is this the Way We Really Want to do Things? Seeing Project Patterns and Changing What You Don’t Like (Johanna Rothman). My goal was to explore the causes of the troublesome patterns I see in projects at work. In particular, I see:

• Too many projects starting up at once
• Projects being instantiated without enough consideration for the probability of success (“is this a good idea?” rather than “can we realistically achieve this?”)
• Key people finding out too late about projects which affect them

All of these patterns lead to increased project risk, communication bottlenecks, low motivation, and high stress.

In the session, we conducted a simulation of a team, with engineers, a project manager, a senior manager and a customer. I took on the role of the senior manager.

In the course of the simulation, we received requirements from the customer, implemented them, and delivered products. While the team was working on implementation, I talked with the customer about what was coming next: what would happen when we delivered, what the next project would be, and so on. Part of the simulation was that I had to be separated from the group while they were working.

When we delivered the first batch of products, and the customer was happy with them, it was time to decide what to do next. We gave the customer a choice of two projects we had discussed, one of which was similar to the previous one (but larger scale and more involved), while the other was different. Despite repeated attempts, we could not persuade the customer to prioritize one over the other.

So, I decided that we should change gears and start work on the “different” project. It seemed to be of greater economic value to the customer, and simpler to execute. One of the engineers disagreed with this decision, but didn’t explain why. The project manager seemed to agree, and I left the team to work. They produced a prototype, which the customer liked, and with a few small changes it was accepted as a finished product.

To my surprise, though, I found out later that the team was in fact working on both projects at once, delivering two different types of products. The decision hadn’t actually been made. These unexpected products were delivered to the customer, but didn’t meet the expanded requirements, and that work was wasted.

The debrief which followed was unfortunately too short, and I didn’t feel that we were able to fully explore what the simulation had revealed. The project manager indicated that he hadn’t understood the decision to have been made, pointing to a communication failure.

This reminded me that while we often think of a decision as an event which happens at a point in time, it is more commonly a process of change, which takes time and must be revisited in order to check progress and evaluate. A decision is really just an idea for a change, and there is more work to be done in order to implement that idea. This can be true even when there is a very explicit “decision point”: it still takes time for that message to be received, interpreted and accepted.

One of the tangents we followed during the debrief had to do with how humans think about numbers. Jerry asked each member of the group to write down a random number, and then we wrote them all on a flipchart. They were: 8, 75, 47, 72, 45, 44, 32, 6, 13 and 47. This reminded me of the analyses of election results which indicate fraud.

Afternoon Session: Saying No

After lunch, I decided to attend Jerry Weinberg’s session, Saying No That Really Means No. This was much larger than the morning session, with over 40 people sitting in a large circle.

The subject of discussion was the variety of difficulties that people face in saying “no” to things which don’t seem right for them. For example, saying “yes” to a project which is doomed to failure. This seemed like a good follow-on to the morning’s exercise.

Jerry began by asking the audience to name some of their difficulties, and tell stories of times when they had trouble saying “no”. One of these stories was role-played and analyzed as an example. Most of the time, though, was filled with storytelling and discussion.

This is a deeply complex topic, because this problem is rooted in self-image, social norms, egocentrism, misperception, and other cognitive phenomena. There was no key insight for me, just a reinforcement of the necessity of self-awareness. The only way to avoid patterns like this is to notice when they are happening, and that can be challenging, especially in a stressful situation.

Once you realize what’s happening, there are all sorts of tools which can be applied to the problem: negotiation, requests for help, problem-solving, even simple inaction.

The AYE conference is an annual conference “designed to increase your effectiveness—in leadership, coaching, managing, influencing, and working in teams, whether you work in systems development, testing, product development, quality assurance, customer service, or consulting.” It starts tomorrow, and I’ve arrived today eager to meet the other attendees and see what the conference is like.

My work involves all of those things, but the main reason I decided to attend was that I learned a lot at the Problem Solving Leadership workshop, which I attended in January and wrote about on this blog. When I heard about this event, organized in part by the same people, and realized I would be in the US that week anyway, I seized the opportunity to attend.

The 2009 program is overflowing with sessions on topics I’m interested in, some of which I have written about here previously.

The speakers are all people whose insight I’ve appreciated through their writing, speaking and teaching, so I expect I’ll have a tough time deciding where to spend my time these next few days. The sessions are said to be experiential ones rather than lectures, so I don’t think that I’ll be able to hop between them, but I’ll see tomorrow when the main program starts.

I know that I have too much on my mind when:

1. I have a brilliant idea
2. I realize that I can’t do anything with it right now
3. I realize that if I don’t record it, I will lose it, because I have a lot to think about
4. I consider creating a list of ideas to come back to later
5. I notice that I have already done this (and forgotten about it)
6. I open the list, and notice that the idea I just had is already on it
7. It has been there for a year
8. This is not the first time this has happened

I’ve been thinking lately that I need to put some energy into organizing my life better, and this is a good example of why.  I am flooded with information, creative ideas, desires, and responsibilities through my work, study, home life and reflection.  I have no illusions about being able to fully honor all of these: that is clearly impossible.  However, I instinctively feel that I could do a much better job of sorting and prioritizing them to maximize my personal effectiveness and satisfaction.

I am a great fan of keeping lists: to-do lists, agendas, my inbox, journals, and other tools all serve to help me capture my thoughts and consider them in a larger context.  Rather than reacting to them one by one, I can look at all of them together and make a conscious choice about what to do right now.  List-keeping is one of the most basic strategies of personal organization, and practicing it has made a dramatic difference in my life.

However, I can see that it is no longer sufficient, and that in order to continue to improve, I will need something more powerful.  I’m not looking for a new list management tool: Remember the Milk, Futz, Tomboy, Jott, and many others provide highly optimized list-keeping facilities.  I don’t use any of them myself (being a text file junkie), but they look great, and offer the right tradeoffs for different people.  My method list-keeping is good enough for me, for now.  Rather, list-keeping is not enough.

Similarly, scheduling has been a successful strategy for me, helping me to decide how I spend my time.  I am not as proficient in scheduling as I am in list-keeping, but I understand the basics and apply them.

What I need is a new paradigm, a new way of thinking about this problem which incorporates and transcends list-keeping and scheduling, and addresses their shortcomings.  I’ve only just begun to research this area, and so far, the most relevant material I’ve found has come from Stephen Covey’s classic text The 7 Habits of Highly Effective People.  “Habit 3″ describes a historical progression of time management tools and approaches which resonates with my personal experience, and prescribes next steps to improve upon it.  I’m not sure yet whether it’s the right direction for me, but here is what I think so far.

Things I like about it:

• Explicit recognition of the various roles I occupy in my life
• Helping to balance priorities across different roles
• Promoting preventative and growth activities, in balance with day-to-day progress

Things which I feel are missing:

• Simplicity: It seems like a lot of bookkeeping, compared to how I’m used to doing things.  I want a system which is as lightweight as possible, because organizational tools which create friction are self-defeating.
• Feedback: I want a mechanism which helps me to regularly evaluate what I’ve done and improve upon it.  This should be as easy and automatic as possible, without requiring too much time tracking and data entry
• Technology: As a technologist, I’m always looking for ways to bring the latest technology to bear on my problems, to make me more efficient.  Covey’s approach was designed without the benefit of the past 20 years of Internet revolution, and the software which is based on it seems a bit dry and monolithic.

Dear readers, do you find yourselves in a similar position?  What are you using to manage your life?  What else did you try?  What was good or bad about it?

We all witness bad behavior at some point or other.  For many of us, the most common examples are provided by men misbehaving toward women.  Whether it’s in public at a conference, on an IRC channel, in an errant wiki page or two, or in a private conversation, how we respond to it is critically important.  This is particularly true where the behavior undermines the security or agency of another person.  Perhaps most of all, it applies where someone is speaking up about it.

If I’m standing with a group of people, and one of them behaves badly, I think that they’re a jerk.  If no one else seems to notice or object, then I start to wonder if they’re all jerks.  If someone speaks up, and is attacked, ignored or discredited, then I’m certain that I’m in a den of wolves.  Feelings like these are toxic to communities, and I don’t want anyone to have to feel this way in one of mine.

Managing one’s own behavior, although it is an essential first step, is not enough.  We must also critique the behavior of others, and signal to our peers that we object to bad behavior.  Furthermore, we must support those who speak up, particularly when they are doing so on their own behalf, or as a member of an underprivileged or under-represented group.  It may be difficult to speak up when you are an observer, but it is much more difficult when you are a target.  This isn’t about coming to anyone’s rescue, but openly accepting their objection and their right to voice it—even if it’s directed at you.

I will not trivialize the effort required to do this.  It is not easy to “break ranks” and stand as (or with) an objector. It is, however, often the right thing to do, and justifies the application of will and the taking of risks for the sake of integrity.  I will also not profess that I have always made the right choice myself.  Indeed, too many times, I have stood by, and I am ashamed for it.  I have made excuses for myself and rationalized my choices, explaining to myself why I couldn’t do what was right in a particular situation.

That is why, in the title of this article, I am addressing myself above all.  I am calling myself out, and calling on my peers in the Ubuntu community to do the same.  Don’t accept bad behavior.  Stand behind those who object to it.  Hold yourself and others accountable for the well-being of your community, and let others know that you are doing so.

It’s a phrase I hear every day: “I don’t have enough time to do that.”

Recently, I’ve been thinking a lot about the trivial things we say, particularly when a poor choice of words can obscure our meaning.  Conventional wisdom supports “calling it like it is,” but taking an objective view is rarely as easy as that.  Our use of language is inescapably tied to our personal experience of the world, and therefore pure objectivity in speech is virtually absurd.  Perhaps instead we should “call it by its true name,” which is sufficiently mystical and subjective to get to the heart of the matter.

What do we mean when we say that we don’t have time?  If a task would require two hours, but is due in an hour, then we don’t have time to do it.  Who could argue with that?  However, more frequently, we mean “I can’t do that in addition to everything else I’m doing.”  If I have a full week’s work to do, and something new comes in which would require a whole day to complete, I might say that I “can’t” do it this week because “I don’t have the time.”  I clearly have enough time, though, because a day is less than a week.  If the task were important enough, I would do it instead of something else.

This brings us a step closer to what we really mean, which is “that is less important than my other obligations.”  What a difference it makes to consider the situation in this way!  I have shifted from a position of powerlessness (“I can’t”) to one of decisiveness (“I choose not to”).  Nothing about my circumstances has changed, but I’m now communicating a different view of the world, one in which I’m taking responsibility for my choice.  This also opens the way for discussing what your priorities are, and making adjustments if circumstances have changed.

In other news, I’m told that London youth have taken to saying “I have a lot of time for that” to mean that they like something.

We are all called upon to act as leaders in some stage or capacity of our lives. A healthy, developing person is constantly growing and changing, and exercises personal leadership to continually assess and adjust their direction. Leaders in organizations apply the same pattern to the teams and other systems around them. They recognize inconsistencies between the way things work today in the organization, and the present and future challenges of its environment.

Having identified such an opportunity for improvement, we are faced with a question: where to begin? How can the system evolve from this form to that? How can I facilitate that change? The most effective leaders I’ve seen don’t look very far for the answer. They start by changing themselves, and thereby their own behavior toward others. They do this for a number of good reasons.

First, we have greater influence over ourselves than our environment, because we can understand our internal world more readily than anyone else’s. We therefore have a better chance of success in our efforts. Building on that foundation, we will begin to affect others in small ways, often without even knowing.

Furthermore, meaningful and lasting change is usually inspired rather than directed. Consider the traits you value in yourself, and the people who have helped you to develop them. Do you more strongly identify with the paternalistic figure who says “you must be…” or the role model who showed you a different way by silent example?

We are highly adapted to recognizing problems in our environment, and socially conditioned to try to correct them through action. This emotional impulse is a strong one: I see something I don’t like, and I want to stop it or change it. This approach works well for inanimate objects, but not so well with people. We resent others for trying to change us, because we value ourselves as we are. Who are they to tell us how to be, when they too are imperfect?

The catch, of course, is that changing ourselves is hard. It is nearly hopeless without understanding what we are, and perhaps some of how we got that way. It is easy to become caught up in other people’s apparent problems, thinking that you can fix them, or even know what they are. If I stop and think, though, there is generally always something I can change within myself which contribute to solving the problem. Often, this means working at a deeper level, setting aside my superficial view of the situation and understanding what is behind it. What might they be feeling which inspires their behavior, and how am I affecting them? How can I change in a way which will give them the confidence and perspective to see the problem more clearly and act for themselves?

I probably heard this principle, in some form, a thousand times before I managed to internalize it, and I still forget, sometimes when I need it most. I hope that writing this will help to remind me of it in my life. It is simple enough, but requires great discipline and practice to apply consistently.