{"id":1373,"date":"2013-08-12T10:12:52","date_gmt":"2013-08-12T08:12:52","guid":{"rendered":"http:\/\/www.oradba.ch\/?p=1373"},"modified":"2013-09-26T14:02:33","modified_gmt":"2013-09-26T12:02:33","slug":"avdf-linux-kernel-could-not-recognize-whole-ram","status":"publish","type":"post","link":"https:\/\/www.oradba.ch\/wordpress\/2013\/08\/avdf-linux-kernel-could-not-recognize-whole-ram\/","title":{"rendered":"AVDF Linux kernel could not recognize whole RAM"},"content":{"rendered":"

After initial setup of an Audit Vault and Database Firewall engineering system, I’ve started to add several audit vault agents and secure targets. In the beginning it went quite smoothly. But after a certain number of secured targets, there were continuously ORA-04031 errors. Most of the errors were related to large pool and PX Msg buffers issues. The analysis of the trace files has shown interesting stuff. \ud83d\ude09 But more on that in a later blog post. The real problem is the available memory. <\/p>\n

Symptoms<\/h3>\n

The Audit Vault and Database Firewall engineering system is running on a HP ProLiant BL465c Gen 8<\/em>. It comes with 32GB Memory. Should actually be sufficient for a system engineering. It turned out that the 32GB are not recognized by operating system. As you can see below the system has just 3GB memory in total.<\/p>\n

[root@melete2 ~]# free\n                     total    used   free shared buffers  cached\nMem:               3048108 2385888 662220      0   10720 1525036\n-\/+ buffers\/cache:  850132 2197976\nSwap:              4194296  453564 3740732<\/pre>\n
Reviewing dmesg<\/code> shows that we lose 29 GB of memory.<\/p>\n
Initializing cgroup subsys cpuset\nInitializing cgroup subsys cpu\nLinux version 2.6.32-300.39.5.el5uek (mockbuild@ca-build56.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Wed Mar 13 11:26:53 PDT 2013\nCommand line: ro root=\/dev\/vg_root\/lv_root console=tty9 udevtimeout=10\nKERNEL supported cpus:\n  Intel GenuineIntel\n  AMD AuthenticAMD\n  Centaur CentaurHauls\nBIOS-provided physical RAM map:\n BIOS-e820: 0000000000000000 - 000000000009f000 (usable)\n BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)\n BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)\n BIOS-e820: 0000000000100000 - 00000000bddde000 (usable)\n BIOS-e820: 00000000bddde000 - 00000000bde0e000 (ACPI data)\n BIOS-e820: 00000000bde0e000 - 00000000d0000000 (reserved)\n BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)\n BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)\n BIOS-e820: 0000000100000000 - 000000083efff000 (usable)\nDMI 2.7 present.\nlast_pfn = 0x83efff max_arch_pfn = 0x400000000\nMTRR default type: uncachable\nMTRR fixed ranges enabled:\n  00000-9FFFF write-back\n  A0000-BFFFF uncachable\n  C0000-FFFFF write-back\nMTRR variable ranges enabled:\n  0 base 000000000000 mask FFFF80000000 write-back\n  1 base 000080000000 mask FFFFC0000000 write-back\n  2 disabled\n  3 disabled\n  4 disabled\n  5 disabled\n  6 disabled\n  7 disabled\nx86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106\ne820 update range: 00000000c0000000 - 000000083efff000 (usable) ==> (reserved)\nWARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing 29679MB of RAM.\n------------[ cut here ]------------\n<\/pre>\n
Cause<\/h3>\n
According to an Oracle Metalink Note 1448147.1<\/a><\/em> this problem is related to a BIOS issue. <\/p>\n
Solutions and Workaround<\/h3>\n
The solution described in Oracle Metalink Note 1448147.1<\/a><\/em> is to upgrade the BIOS or disable MTRR in kernel. Since BIOS upgrade is not an option for this environment I’ll try to workaround by disable MTRR.<\/p>\n
Disable MTRR<\/h4>\n
Changing the grub.conf<\/code> is basically quite easy if you find the boot files. When I first try it, I’d realized that there is no grub configuration available. It seems that Oracle decided to not mount \/boot<\/code> at startup. So it is mandatory to first mount the boot partition. Afterward you just can add disable_mtrr_trim<\/code> as  additional kernel option. <\/p>\n
\n[root@melete2 ~]# mount \/boot\n\n[root@melete2 ~]# df -kh \/boot\nFilesystem            Size  Used Avail Use% Mounted on\n\/dev\/sda1             145M   26M  112M  19% \/boot\n\n[root@melete2 ~]# vi \/boot\/grub\/grub.conf \n# grub.conf generated by anaconda\n#\n# Note that you do not have to rerun grub after making changes to this file\n# NOTICE:  You have a \/boot partition.  This means that\n#          all kernel and initrd paths are relative to \/boot\/, eg.\n#          root (hd0,0)\n#          kernel \/vmlinuz-version ro root=\/dev\/vg_root\/lv_root\n#          initrd \/initrd-version.img\n#boot=\/dev\/sda\ndefault=0\ntimeout=5\nsplashimage=(hd0,0)\/grub\/splash.xpm.gz\nhiddenmenu\ntitle Audit Vault Server 12.1.1.0.0\n        root (hd0,0)\n        kernel \/vmlinuz-2.6.32-300.39.5.el5uek ro root=\/dev\/vg_root\/lv_root console=tty9 \nudevtimeout=10 disable_mtrr_trim\n        initrd \/initrd-2.6.32-300.39.5.el5uek.img\ntitle Audit Vault Server 12.1.1.0.0\n        root (hd0,0)\n        kernel \/vmlinuz-2.6.32-300.38.1.el5uek ro root=\/dev\/vg_root\/lv_root console=tty9 \nudevtimeout=10 disable_mtrr_trim\n        initrd \/initrd-2.6.32-300.38.1.el5uek.img\n\n[root@melete2 ~]# reboot\n\nBroadcast message from root (pts\/0) (Thu Jul 11 20:17:56 2013):\n\nThe system is going down for reboot NOW!\n[root@melete2 ~]# Connection to melete2 closed by remote host.\nConnection to melete2 closed.\n<\/pre>\n
After reboot we now have 32GB memory available.<\/p>\n
[root@melete2 ~]# free\n                      total     used     free shared buffers  cached\nMem:               33024372  3930724 29093648      0   17868 2640744\n-\/+ buffers\/cache:  1272112 31752260\nSwap:              14680056        0 14680056<\/pre>\n
Unfortunately, the configuration of the AVDF appliance is not automatically updated to use the extra memory. We have to do some manual changes.<\/p>\n
Update Kernel Parameters<\/h4>\n
The kernel setting have to be changed to allow a bigger SGA. See Metalink Note 1529433.1<\/a><\/em> for more detailed information on how calculate and set the kernel parameters. For the engineering system we will define a SGA with 20GB therefor we set the shmmax and shmall as follows:<\/p>\n
[root@melete2 ~]# vi \/etc\/sysctl.conf\n\u2026\nkernel.shmmax=23622320128\nkernel.shmall=5368709120\n...\n[root@melete2 ~]# sysctl -p\n<\/pre>\n
Increase SWAP<\/h4>\n
With 32GB memory, it is also advisable to enlarge the swap space. I’ve discussed this already in the blog post Resize swap space on linux<\/a>. Since the AVDF appliance does use logical volumes it’s even a bit easier.<\/p>\n
[root@melete2 ~]# swapoff -v \/dev\/vg_root\/lv_swap\n\n[root@melete2 ~]# lvresize \/dev\/vg_root\/lv_swap -L +8G\n\n[root@melete2 ~]# mkswap \/dev\/vg_root\/lv_swap\n\n[root@melete2 ~]# swapon -v \/dev\/vg_root\/lv_swap\n<\/pre>\n
Increase SGA<\/h4>\n
Finally we can increase the SGA.<\/p>\n
\n\nSQL> alter system set sga_max_size=20G scope=spfile;\nSystem altered.\n\nSQL> alter system set sga_target=20G scope=spfile;\nSystem altered.\n\nSQL> startup force\n<\/pre>\n
Conclusion<\/h3>\n
Although AVDF is an appliance, it is mandatory to examine the system after installation. Eg. are there errors in the log files in \/var\/log<\/code>, memory, storage etc. available. The solution described here makes it possible to use all the memory. Nevertheless, the appliance has been adjusted to an extent where is necessary to consider whether the support is still archive. If you run into a similar issue on your production AVDF setup I would recommend opening an Oracle SR. Looking forward to the next AVDF patchset. I hope this system stays patchable.<\/p>\n
References<\/h3>\n
Some links related to this post.<\/p>\n