In 2010, I managed to attend a business trip to Tokyo. Over five weeks, I worked alongside the best people from one department at NEC. In this task, there were only two of us: Alexey M. and I.
Alexey, thank you for your support and communication during the trip. You thoughtfully brought a spare 3G phone with a tourist SIM card. Arigatou gozaimashita! (Thank you very much!)
My task involved assembling two server configurations, customized by NEC, with Oracle database installed:
- Four blades operating in parallel in a RAC (Real Application Cluster) configuration.
- Four blades connected via a special bus as a single unit (Single Oracle Instance).
The tests were optimized for parallel queries, and we expected impressive results from the first configuration on the most modern hardware.
At the Fuchū data center (Tokyo), the department head — a Japanese gentleman with gray hair — personally plugged cables into the servers, not allowing anyone from the team to assist. Both sides were anxious about the work, so we poured our best efforts into doing everything to the highest standard.
We began with the second configuration. During testing, we obtained respectable results (I don’t remember the exact numbers). We repeated the tests several times and then switched the servers to RAC cluster mode.
Days passed, but the tests showed RAC lagging significantly behind the single-instance configuration. Colleagues from Moscow repeatedly rewrote the tests, but we could never match the performance of the second configuration. We did not share our results.
Our Japanese colleagues noticed our discomfort. On day 32 of the assignment, taking responsibility for the tests, I decided to go to management and admit that, according to our measurements, the RAC configuration significantly underperformed the single-instance setup. No matter how we optimized or tuned the kernel or ext3 filesystem on RHEL, we couldn’t improve the results.
The elderly Japanese man politely listened to my report right next to the server. He called the rest of the team over and asked them to repeat what he had just heard. Everyone smiled along with him. And he said:
- These results are exactly what we expected. The company modified standard servers so that their processors and memory work together. We were eagerly awaiting your tests under heavy load — we didn’t have the opportunity to test this ourselves. Therefore, we are very pleased with the results — all our expectations for the new servers have been met. Thank you!
What relief came at that moment! The large company I represented didn’t care what was going on inside me, nor did they care that I had chosen to present their results in such a way — by admitting my own helplessness. But it turns out, such things can happen. A bad result can be an expected result.
The director and his assistant — also a friend — invited Alexey and me to lunch at a small family-run restaurant opposite the data center. I believe offering us to try their dishes was a sign of the highest respect and hospitality.
The Japanese don’t use their high-ranking positions. Although the gate at the exit from the premises was open, we waited three minutes until 2:00 p.m. to leave together.
This was the most impressive trip I’ve ever taken. Interacting with people in Japan at that time opened my eyes to many things. At first, I expected to see flying saucers at Narita Airport, then I searched for them on the streets of the city. But I saw Toyota Crowns in taxis and boxy cars. Ordinary tractors and trucks. And the Narita Express train, in the rain, barely moved at 40 km/h. Meanwhile, the metro runs with precision down to the second, and each line’s trains emit their own unique sound. The Japanese are railroad enthusiasts. The word “JR” carries far more weight than “RZD.”> In terms of work, I learned about the approach that brought the industry stability. Things may have changed now, and people say Toyota is no longer the same. But it was precisely there that the expression emerged: “First, you must cultivate good employees, and then you can make good cars.” Japanese companies initiated the Lean approach—lean production. Many of its principles are applicable to software development. Here too, there are the same conveyors and people. I want to incorporate the Lean and Gemba approaches into my work, about which I will soon begin a series of articles and share my experience of how it works in practice.
And finally:
