Customizing Redhat/Oracle Linux CDROMs, ISOs and USBs - Part 1

Redhat/Oracle Linux custom CDROMs, ISOs and USBs can be very useful... They speed up server installation with predefined kickstart files; they reduce distributions to the few hundreds of MB you really need; they provide easy customization for configurations that are not supported by anaconda's initial setup program or your distribution (e.g. here) and they might be enhanced with additional RPMs and custom scripts. This few posts demonstrate how to modify Oracle Linux ISOs and how to use them.

Obviously if you are dealing with a large number of servers and VMs, using RPM repositories with PXE, xenpvnetboot or templates is the way to go. But if you are dealing with only a few VMs on your laptop or with manual server installation, custom ISOs are simple and very fast. You can literally create one in less than 10 minutes. This first part will be presenting how to customize an ISO with a simple kickstart file. It will be using the Oracle Linux 7 Early Adopter ISO but other Redhat Based distributions and other releases are not much different...

Installing Oracle Database 12.1 in Command Line and "Silent Mode"

Installing and configuring Oracle databases is probably the first task you will want to automate when you are a DBA. Well, it is not the most expensive, neither the most difficult and probably not even the worthiest. Be also careful, it might come at a very bad time if you are not in charge or did not already have automated your Operating Systems installation and configuration. Anyway, we can assume this operation has to be done frequently enough to start with it as an automation.

Every Oracle release and, sometimes Patch Set, come with a few changes. Oracle Database 12.1 is no exception to that rule! Before you get your installation fully automated and rely on your preferred tool to deploy them, this blog post will present the command lines you can use to perform a silent mode installation of Oracle on most operating systems

Alter Blog Open Resetlogs... to the Cloud

People have gone one step further to use Oracle with the cloud this year. For 6 months now, in addition to people using AWS and subscribing to SaaS, I've started to meet people looking at new public clouds including Microsoft Azure or even Oracle Backup Database Cloud -for now. I've also met people providing production private Platform as a Service with Oracle Databases. So that, I'm now thinking 2014 is a game changer for Oracle technology with the Cloud.

Don't get me wrong, there is no new magic cloud wands for Oracle. I don't believe there is anything like, you buy a few Exa-servers with some extra-packs or options and... woohoo: you run an extreme, elastic, secured, never stopping, self-service, pay-per-use Oracle Cloud. I don't think it is the case at all... But still, I'm very excited by what the Oracle cloud journey promises!

Querying an Oracle VM Server Xenstore from a Linux Guest

Accessing the Oracle VM Xenstore from a Linux Guest is the most efficient and sometimes most reliable way to collect data about your Oracle virtualized environment. Among the various things you can get from Xen internal database are (1) the DomU name, (2) the Dom0 VNC access point or (3) the mapping between the VM disk devices and the underlying storage.

For instance, if you want to trigger a storage snapshot from your guest, you can directly and automatically discover the LUN serial number with the xenstore-ls command and request a snapshot without connecting to the Oracle VM Manager.

This article explains how to install and use the various xenstore tools/utilities to get those values.

Building Xenstore Tools

Oracle doesn’t provide the xenstore tools for your guest as part of its RPM repository. Building your own is not a big deal. Download the latest Oracle VM Server Source ISO from My Oracle Support; Oracle keeps it in the 16410428 placeholder. The file you are looking for is named xen-4.x.x-yy.el5.zz.src.rpm and is available from the src.iso file.

Then install xen source file on one of your Oracle Linux guest. Below is an example of how to install the source code from Oracle VM 3.2.6 :

rpm -ivh xen-4.1.3-25.el5.53.src.rpm
cd rpmbuild/SOURCES
tar -zxvf xen-4.1.3-ovs.tar.gz
cd xen-4.1.3-ovs

Once you’ve installed the source, make sure you have all the necessary RPMs to build it. below is an example that works with Oracle Linux 6:

yum install oracle-rdbms-server-11gR2-preinstall \
     libuuid-devel openssl-devel ncurses-devel   \
     dev86 iasl python-devel SDL-devel

And build the tools with a simple make command:

cd ~/rpmbuild/SOURCES/xen-4.1.3-ovs/tools
make

Installing xen-detect and the Xenstore Tools

Building a RPM to install the tools you need for your guest is straightforward. All you need are the xen-detect, libxenstore.so.3.0, xenstore and xenstore-control files. Assuming you’ve installed and built them from the source on your guest, the script below performs a typical installation:

cd ~/rpmbuild/SOURCES/xen-4.1.3-ovs/tools/misc
install xen-detect /usr/local/bin
cd ~/rpmbuild/SOURCES/xen-4.1.3-ovs/tools/xenstore
install libxenstore.so.3.0 /usr/local/lib
install xenstore xenstore-control /usr/local/bin
cd /usr/local/bin
ln -f xenstore xenstore-chmod
ln -f xenstore xenstore-exists
ln -f xenstore xenstore-list
ln -f xenstore xenstore-ls
ln -f xenstore xenstore-read
ln -f xenstore xenstore-rm
ln -f xenstore xenstore-write

Using xen-detect and Xenstore Tools

xen-detect allows to quickly figure out, from your guest, the type of virtualization that is currently in use. This is a sample output taken from a paravirtualized DomU:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
xen-detect
Running in PV context on Xen v4.1.

To access the server xenstore from the guest, you must first allow access to the xenbus to the xenstore tools running in userspace. Mount the xenfs systems to /proc/xen to allow the access:

mount -t xenfs none /proc/xen

You can then use the various xenstore commands to get the DomU Name or Id:

xenstore-read name
0004fb000006000042165a8746757cac

xenstore-read domid
3

You can list backend devices from the store:

xenstore-ls /local/domain/`xenstore-read domid` |grep "backend ="

backend = "/local/domain/0/backend/vkbd/3/0"
backend = "/local/domain/0/backend/vfb/3/0"
backend = "/local/domain/0/backend/vbd/3/51712"
backend = "/local/domain/0/backend/vif/3/0"
backend = "/local/domain/0/backend/console/3/0"

Check out the storage details; below is a command that allows to match the storage device to its virtualized storage source:

xenstore-ls -f /local/domain/0/backend/vbd/3/51712

/local/domain/0/backend/vbd/3/51712/domain = "0004fb000006000042165a8746757cac"
/local/domain/0/backend/vbd/3/51712/frontend = "/local/domain/3/device/vbd/51712"
/local/domain/0/backend/vbd/3/51712/uuid = "383157e9-4204-3c43-325f-df6d38f2f1ff"
/local/domain/0/backend/vbd/3/51712/bootable = "1"
/local/domain/0/backend/vbd/3/51712/dev = "xvda"
/local/domain/0/backend/vbd/3/51712/state = "4"
/local/domain/0/backend/vbd/3/51712/params = "/OVS/Repositories/0004fb0000030000b85220e0f581c9d4/VirtualDisks/0004fb0000120000237098d4453a0ae7.img"
/local/domain/0/backend/vbd/3/51712/mode = "w"
/local/domain/0/backend/vbd/3/51712/online = "1"
/local/domain/0/backend/vbd/3/51712/frontend-id = "3"
/local/domain/0/backend/vbd/3/51712/type = "file"
/local/domain/0/backend/vbd/3/51712/node = "/dev/loop0"
/local/domain/0/backend/vbd/3/51712/physical-device = "7:0"
/local/domain/0/backend/vbd/3/51712/hotplug-status = "connected"
/local/domain/0/backend/vbd/3/51712/feature-flush-cache = "1"
/local/domain/0/backend/vbd/3/51712/feature-discard = "0"
/local/domain/0/backend/vbd/3/51712/feature-barrier = "1"
/local/domain/0/backend/vbd/3/51712/sectors = "25165824"
/local/domain/0/backend/vbd/3/51712/info = "0"
/local/domain/0/backend/vbd/3/51712/sector-size = "512"

You can discover how a network device relates to a Dom0 network bridge:

xenstore-ls -f /local/domain/0/backend/vif/3/0

/local/domain/0/backend/vif/3/0/bridge = "c0a83900"
/local/domain/0/backend/vif/3/0/domain = "0004fb000006000042165a8746757cac"
/local/domain/0/backend/vif/3/0/handle = "0"
/local/domain/0/backend/vif/3/0/uuid = "874c2912-38a6-a27c-3728-7858ca82f0eb"
/local/domain/0/backend/vif/3/0/script = "/etc/xen/scripts/vif-bridge"
/local/domain/0/backend/vif/3/0/state = "4"
/local/domain/0/backend/vif/3/0/frontend = "/local/domain/3/device/vif/0"
/local/domain/0/backend/vif/3/0/mac = "00:21:f6:00:00:00"
/local/domain/0/backend/vif/3/0/online = "1"
/local/domain/0/backend/vif/3/0/frontend-id = "3"
/local/domain/0/backend/vif/3/0/feature-sg = "1"
/local/domain/0/backend/vif/3/0/feature-gso-tcpv4 = "1"
/local/domain/0/backend/vif/3/0/feature-rx-copy = "1"
/local/domain/0/backend/vif/3/0/feature-rx-flip = "0"
/local/domain/0/backend/vif/3/0/hotplug-status = "connected"

You can also find the VNC address and port associated with your guest:

xenstore-ls -f /local/domain/0/backend/vfb/3/0

/local/domain/0/backend/vfb/3/0/vncunused = "1"
/local/domain/0/backend/vfb/3/0/domain = "0004fb000006000042165a8746757cac"
/local/domain/0/backend/vfb/3/0/vnc = "1"
/local/domain/0/backend/vfb/3/0/uuid = "2d259c00-dcf8-bfec-0290-0a056e9a3769"
/local/domain/0/backend/vfb/3/0/vnclisten = "127.0.0.1"
/local/domain/0/backend/vfb/3/0/frontend = "/local/domain/3/device/vfb/0"
/local/domain/0/backend/vfb/3/0/state = "4"
/local/domain/0/backend/vfb/3/0/keymap = "fr"
/local/domain/0/backend/vfb/3/0/online = "1"
/local/domain/0/backend/vfb/3/0/frontend-id = "3"
/local/domain/0/backend/vfb/3/0/xauthority = "/root/.Xauthority"
/local/domain/0/backend/vfb/3/0/feature-resize = "1"
/local/domain/0/backend/vfb/3/0/hotplug-status = "connected"
/local/domain/0/backend/vfb/3/0/location = "127.0.0.1:5900"
/local/domain/0/backend/vfb/3/0/request-update = "1"

What’s next...

Using Xenstore tools from guests is very convenient if you use Oracle VM with databases on a SAN and want to perform snapshots as sources for backups. This is because Oracle VM dom0 name their devices with Linux device-mapper and use the “scsi_id -g -u -s /block/” command behind the scene. Those same serial numbers can also be discovered in storage arrays to control you are not missing any LUN for your snapshot. Obviously you could also query Oracle VM manager but that’s not always as obvious since you cannot even be sure from your DomU name. 

One more word for those who are using Netapp Data Ontap... Once you have the scsi serial number, you can check for the LUN with the “lun serial -x /vol/name/lun” command. It allows to perfectly match LUN and filesystems or ASM diskgroups, the same way NetApp SnapDrive would do on a Linux physical host! Hopefully, Netapp will code this integration for us soon in their tool to ease our workload.

Custom DTrace Probes for Perl on Oracle Linux 6

Chris Andrews (@chrisandrews) has developed the libusdt library a few years back so that dynamic language developers can publish custom DTrace providers easily. He has also developed the associated modules for Ruby, Perl and Node.js... If you are dealing with one of these languages, publishing a DTrace provider usually takes no more than 5 lines.

Last September, Chris has adapted his library so that it works with Oracle Linux 6 DTrace and the Unbreakable Enterprise Kernel 3. This post explains how to use Perl Devel::DTrace::Provider module in this context.

Installing the Devel::DTrace::Provider Perl Module

The 1.11 version of Devel::DTrace::Provider doesn’t come with the latest libusdt. We can assume, next release will do! In the meantime, if you install the module on Perl, it fails. In order to workaround that issue, install it from the source and change the libusdt library with the latest one... As a prerequisites, make sure you have an Oracle Linux 6 running DTrace and you have downloaded the DTrace utilities and the associated headers from your Oracle Premier Support. You should be able to run the script below as root:

modprobe dtrace
modprobe profile
modprobe sdt
modprobe systrace
modprobe fasttrap
chmod 666 /dev/dtrace/helper

Install the following Perl Module that are required by Devel::DTrace::Provider:

yum -y install perl-Data-OptList perl-Params-Util \
     perl-Sub-Install perl-JSON perl-Sub-Exporter

Download the module gzip tarball. The 1.11 release can be downloaded from here. Download the libusdt library from its GitHub.com page. You can download it as a zip here. Once you have the 2 distributions on your filesystem, decompress them and make sure libusdt is part of the Perl Module like below:

tar -zxvf Devel-DTrace-Provider-1.11.tar.gz
cd Devel-DTrace-Provider-1.11
rm -rf libusdt
unzip ../master.zip
mv libusdt-master libusdt

Build and install the Perl module:

perl Makefile.PL
make
sudo make install

You can simply test the module like in the example below:

sudo make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/builder.t ...... ok   
t/imported.t ..... ok   
t/imported2.t .... ok   
t/json.t ......... ok   
t/noargs.t ....... ok   
t/provider.t ..... ok    
t/unavailable.t .. ok   
All tests successful.
Files=7, Tests=43,  0 wallclock secs ( 0.04 usr  0.01 sys +  0.25 cusr  0.04 csys =  0.34 CPU)
Result: PASS

Include a Custom DTrace Provider in your Perl script

Once the module installed, you can use it from any of your Perl scripts. Create a file name helloworld.pl that declares and enable a probe for a new provider named myprovider:

use Devel::DTrace::Provider;

my $provider = Devel::DTrace::Provider->new('myprovider', 'myapp');
my $probe = $provider->add_probe('start', 'monitor', ['string']);
$provider->enable;

$probe->fire('HelloWorld');

The script below will be able to intercept the probe and display its parameter:

sudo dtrace -qZn 'myprovider*::: { printf("%s:%s:%s:%s (%s)\n", probeprov, probemod, probefunc, probename, copyinstr(arg0)) }'

Run your Perl script:

perl helloworld.pl

It displays the probe as well as value sent as an output parameter:

myprovider10894:myapp:monitor:start (HelloWorld)

You can now enjoy extending your application with a custom DTrace provider on Linux too...