MemoryOutOfBound Error

Today when I reached office, my manager called me to look into a production issue. We were getting some MemoryOutOfBound error. He was sure that application was not huge and does not use much memory, so the error should be something else than memory not being available.

We tweaked the settings to get the dump of the heap. And systems were failing regularly. Once we bounce the server, server starts working and another server fails in 15-20 minutes. I checked memory status using jmap, and heap size was no where close to 2 GB. Development tech lead told that the system was working fine when may heap size was 512 mb in stead of 2 GB, so we reverted back to 512 mb. And suddenly it started working. And that was right, reducing heap size helped. The reason was,

We were using a 32 bit system with 4 GB memory. OS was using 2 GB for its internal usage. When we provide 2 GB for JVM heap, there were little memory left for any native calls such as creating OS thread, opening a socket etc. And we used to get MemoryOutOfBound error. Reducing it to 512 MB helped.

Paid to make mistake

In one of the leadership training, the trainer said we are paid to make mistake! I did not agree with him. How can one be paid to make mistake? I am paid to write good software and he is paid to provide appropriate training. And neither he nor I worth the salary if we don't do our job well. How are we paid to make mistake? He clarified, we are paid to make mistake and not for not doing our tasks. He explained his view:

Man is to err and will make mistake. We do mistake, even Vivek Paul makes mistake. But we should not repeat our mistakes. We should learn from mistakes. Anybody who makes and repeats mistake needs to be fired (too harsh, isn't it?). So one has the option of working in the comfort zone and not making any mistake or stretching her ability and do something new and may make mistake. And we are all paid to grow and stretch ourselves, do new things which is out of our comfort zone and make mistakes.

I was reading Michael Nygard's Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers). He also agreed with the trainer. He claims he does make mistake, but all the mistakes are unique. Once he learns about a mistake, he ensures that it does not happen twice. And as a cynic he expects something will go wrong in the production.

I got a different prospective from my manager. He at certain cases could have put the blame on certain mistakes on some of the team members including me. In stead he took the responsibility and ensured that the mistake did not happen again. It is not about blaming others but owning the mistake, fixing it and moving forward ensuring it does not happen again.

Doing it right at right time

I am following cricket like most Indians. Currently I am following CLT20. Victoria played 4 matches won 2 and lost 2 matches, so does Delhi and Bangalore. But Victoria went to semifinal where as Bangalore and Delhi were knocked out. Though Victoria have better run rate than either Delhi or Bangalore, it went to semifinal based on points without considering the run rate. I was looking into the All Star 11 It has 3 players from Bangalore, 2 from Delhi and only one from Victoria (so does 1 from Cape Cobra and New South Wales). So despite showing better individual brilliance and 2 teams have been knocked out of the tournament.

Both Delhi and Bangalore knocked out because they lost the group match to a team which went to the next round. So they went to the next round but with 0 points. Victoria lost a match but to a team which was eliminated from the group stage. That was the differentiator between Victoria and other teams.

So it is not only a mistake (loosing a match) but the mistake which got added (carried forward points) costed Bangalore and Delhi.