If you’re interested in setting up your own Oracle infrastructure, this documentation series is for you!
In this series I will describe what I did and how I did it, along with some useful tips and tricks to help get things done.
This is for educational purposes only, but much of what is presented readily translates to real world production systems.
Before we get into the details, let’s cover some basic terms of reference in the form of 3 questions.
Q1. What do I mean by an Oracle infrastructure?
Well, aside from knowing and understanding the Oracle DBA role in a general sense, for me there are 9 Oracle technologies/configurations which Senior Oracle DBAs like myself should aspire to know and know well. They cover the essentials of the operating system, administration, backup/restore, High Availability (HA), Disaster Recovery (DR), data replication, networking, storage, virtualization and of course, the database itself. Those 9 technologies/configurations are:
- Oracle VM
- Data Guard
- Oracle Linux
- Oracle Database
- Recovery Manager
- Grid Infrastructure
- Real Application Clusters
- Oracle Enterprise Manager
In this series I will cover most if not all of these from an installation, setup and configuration point of view. For some of them, I’ll also perform some functionality testing to make sure they do what they’re supposed to do. This will be useful when writing test plans for the real world.
Q2. Why bother?
Why bother doing anything? For me, being an Oracle DBA is not just my job. It’s what I do. I strive to be the best DBA I can be and that means research and practice. Some people, particularly IT Managers with those elusive training budgets, seem to think DBAs acquire skills and knowledge through osmosis. It’s true that DBAs do have super powers, but osmosis is not one of them! Unfortunately, understanding something and getting good at it takes practice and Oracle is no different. This project serves as my way of researching and practicing some of the most important Oracle technologies and configurations. It has already helped me pass the Upgrade to Oracle Database 12c certification exam. I may even push on and take the Oracle Database 12c RAC and Grid Infrastructure Expert Certification next. My point is, be your own expert. Don’t rely upon or hang on every word of the self-appointed ‘experts’ and ‘know-it-alls’ of the Oracle industry. As Nike say, “Just do it”!
Q3. Why do it this way?
There are cheaper and easier ways to achieve the same things I set out to do. Things like Oracle VM templates and Oracle VM Virtualbox provide a quicker point-and-click way to get something done fairly rapidly. If speed is your thing, those resources are great. If understanding is your thing, then you’ll need to do things the old fashioned way by getting your hands dirty. You’ll emerge the other side knowing and understanding way more than if you simply used the short cuts first. Once you actually understand what is going on under the covers, by all means transition to using those short cuts. One of my goals was to understand these technologies better and you can only really do that effectively by actually doing it. Not by reading about it. Not by attending presentations. Not by taking short cuts and not through osmosis!
Here’s a summary of the hardware I used for this project. Some of this hardware I already had and some I had to buy. You won’t need to use exactly what I used and you may find cheaper alternatives which will work just as well. Other items you may choose not to use at all. Your choice. With that said, I think it’s often interesting to know exactly what someone else used and more importantly, why they used it.
|Link To Component||Condition||Approximate Cost|
|Oracle VM Server||Used||$1,500|
|Oracle Database 12c & Oracle VM Manager Server||New||$260|
|Shared Storage Server||New||$220|
|Gigabit Network Switch #1||New||$70|
|Gigabit Network Switch #2||Used||$20|
|4 Port USB/PS2 KVM||Used||$30|
|USB Mouse with PS/2 adapter||Used||$5|
|Cat 6 Network Patch Cables||New||$20|
Product : HP ProLiant DL380 G7
CPU : Dual 2.66 GHz Quad Core Xeon E5640
RAM : 72 GB
Networking : HP NC382i Dual Port Gb Server Adapters (4 Ethernet ports total)
Storage : HP Smart Array P410i RAID Controller, 2 x 72 GB + 6 x 300 GB SAS 10K HDs (8 HDs total)
This is the heart of the system, so it was important to get this right. When I first started working with Oracle Database 12c, I quickly realized it’s not available for 32 bit Linux. That rendered several of my older PCs redundant. My other PCs which could run 64 bit software did not fare much better. Let’s just say, a 2.66 GHz Pentium D CPU with 3 GB RAM and a 7.2K SATA desktop drive doesn’t get the job done. At least not very quickly. For example, I ran the DBUA to upgrade an 11.2 database to 12.1 and it took over 6 hours to complete. Who has that kind of time? Not me!
I could have bought new motherboards, CPUs and memory for these aging PCs and rebuilt my physical Oracle Database 11g RAC system with Oracle Database 12c. However, that wouldn’t have given me the opportunity to work with Oracle VM again. Plus I’d still have way too many PCs at home which all need cooling, all need a UPS outlet and all need to be stored somewhere. I also considered buying a customized super PC with lots of drives, lots of memory and lots of CPU. Something capable of running up to the 8 Linux VMs I had in mind for this project. However, that option quickly became very expensive. Some quotes I got were over $3,000!
Then I had the brilliant idea of tapping into the refurbished server market. Ever wondered what happens to servers when they go out of warranty and are replaced by a newer generation of server? No, neither did I, but it turns out companies buy them, refurbish them and sell them on. Who knew? Makes sense since solid state technology doesn’t wear out that quickly and there’s basically nothing wrong with the hardware other than the OEM won’t warranty it. The parts which do wear out, the hard drives, are easily replaced. HP branded hard drives can be purchased from a variety of sources including eBay and Amazon. The 72 GB drives I use cost between $10 and $20 per drive. Not bad. The 300 GB drives are slightly more expensive at between $25 and $40. Still, not bad. Note, if you intend to configure more than one LUN using the internal drives and trust me you will, make sure you buy additional memory for the HP P410i RAID controller. Without its own memory, the P410i only supports one LUN.
My research turned up a fantastic company called Orange Computers. Using their website’s system builder feature, I was able to put together a customized specification for an HP ProLiant DL380 G7 for around $1,500. That’s still a lot of money, but consider what you’re getting. Server class hardware, multiple built in redundancies, designed to run 24x7x365 and guaranteed compatibility. This last point is extremely important. Oracle VM and Oracle Linux isn’t guaranteed to support every single combination of CPU, motherboard, NIC, storage array, etc. If you’re going to sink some serious cash into a system to run Oracle VM and Oracle Linux, wouldn’t you want to know for sure it’ll work rather than just hoping it will?
Oracle has an Oracle Linux and Oracle VM Hardware Certification List. It’ll tell you if your name brand system will fly. For me, the HP ProLiant DL380 G7 checked all the right boxes. Orange Computers can also sell you an extended warranty and have their own in house technical support. They are super easy to deal with and go the extra mile to ensure you get what you need. I highly recommend them.
Product : Customized PC
Motherboard : Gigabyte GA-H97M-D3H LGA1150 mATX
RAM : 2 x 8 GB Crucial Ballistix Sport DDR3-1600 CL9 Dual Channel (16 GB total)
Networking : Intel PWLA8391GT PRO/1000 GT PCI Network Adapter (2 NICs total)
I gutted one of my older PCs and bought a new motherboard, CPU, RAM and an SSD. The CPU is pretty decent for what I need it for and the motherboard provides hardware RAID configuration options.The 16 GB of RAM is more than what is required for Oracle VM Manager, but this server will also double as a regular database server when the Oracle VM Server is not in use. Since I had plenty of spare HDs, I used the SSD drive as the boot/OS drive and the 2 HDs configured as a RAID 1 array. I’ve had good performance and reliability from the Samsung 840 and now the 850 EVO series of SSDs. I got them from Amazon for $49.99. They seem to have lightening sales on things like this every now and then. Look out for them. I’ve read about Linux compatibility issues with some PCI NICs, but Intel seem to be well supported, so I stick with them. I got the rest of the components from my local Micro Center. Their customer service has improved to the point of actually being useful. I always like to my purchases verified for compatibility. Newegg.com can do this too, but they want to charge you for the advice. Seriously! Note, you will most likely need a 2.5” to 3.5” mounting bracket for the SSD.
Product : Customized PC
Motherboard : Gigabyte GA-H97M-D3H LGA1150 mATX
RAM : 2 x 4 GB Crucial Ballistix Sport DDR3-1600 CL9 Dual Channel (8 GB total)
Networking : Intel PWLA8391GT PRO/1000 GT PCI Network Adapter (2 NICs total)
I gutted another one of my older PCs and bought a new motherboard, CPU, RAM and two SSDs. The CPU is pretty decent for what I need it for and the motherboard provides hardware RAID configuration options. I already had the 2 Western Digital HDs, so configured them as a RAID 1 array. These were the basis of the shared storage for my previous Oracle Database 11g RAC setup and were purchased at a time before SSDs were affordable. Speed is important for these drives, hence the 10K rpm specification. For your own setup, disk capacity is probably not the issue but speed will be. I used one of the SSDs for the boot/OS drive and the other is used for database storage. I did want to try an SSD RAID 1 array, but when I went back to Amazon to snag more Samsung 850 EVO SSDs for $49.99 a piece, they’d sold out! I guess plenty of other people realized what a steal that was. Lesson learned – buy more next time you see them heavily discounted. They really are excellent drives. In truth, this system needs little memory and I’ve run it previously with 4 GB and it worked fine. It has 8 GB now because I initially bought a 1 x 4GB DIMM, not 2 x 2 GB DIMMs. Duh! It was less hassle just to buy an additional 4 GB DIMM. For best performance, DIMMs should be used in pairs of equal size.
I already had this switch. It is used for public and VM management traffic. You can still buy this new for around $70, almost half what I paid for it just a couple of years ago. Sixteen ports is overkill for this project. An 8 port switch will suffice and will typically be about half the cost. D-Link make a decent looking 8 port switch which I don’t have as does TRENDnet which I do have. It has a metal case too. Nice.
I already had this switch. It will be used for the shared storage traffic. This is an old switch, but is still going strong. The 8 ports are more than enough and you can pick up a used one for around $20. If you’re going to buy new switches for this project, you may as well get a couple of the newer D-Link 8 port switches and save yourself between $20 and $30 on my estimate for network switches.
I already had this KVM. It is used to access the console of the HP DL380, the Oracle VM Manager computer and the shared storage filer computer. It’s an old KVM which you can buy used for around $30. I bought an updated version of this KVM a couple of years ago, but it’s not as well behaved as this one. For this project you will need to get into the BIOS of all 3 computers. To do that you typically need to press a specific key as the computer boots up. While connected to a computer’s console via the newer KVM, it routinely ignored my key stroke during boot up. Tres annoying. The older KVM didn’t have that problem so I stuck with that one. Your mileage may vary.
Product : ViewSonic VA503b
I already had this screen. It doesn’t really matter what screen you use as long as it has a VGA connection and plays nicely with your KVM. It doesn’t have to be that fancy either since you won’t be spending that much time working with this screen. It makes me cringe to think I paid $150 for this screen back in 2006. I found a refurbished one on eBay for $20. It is a nice screen though. ViewSonic make decent gear.
Product : Gateway KB0532
The important point about this keyboard is it has a PS/2 style connector which plugs directly into the back of the KVM. That’s important because it guarantees your boot up key stroke will be recognized when you’re trying to get into the computer BIOS.
USB keyboard key strokes can be ignored if the USB drivers are not loaded early enough in the boot up sequence. That can prevent you from getting into the BIOS and will ultimately lead to expletives and threats of physical violence against an inanimate object. Any PC keyboard with a PS/2 style connector should do the trick, but you can pick one of these up pretty cheaply.
Product : HP USB Mouse P/N: 505062-001 Rev.A
I already had this mouse left over from my HP desktop. It is the cheapest and nastiest mouse ever. The sort of thing you might expect to find in a Christmas cracker, but hey it works.
I have this connected to a USB port on the front of the KVM. I have a USB/PS2 adapter for it, but when I use that to connect it to the KVM, it doesn’t work. Weird. Literally any USB mouse should work.
Product : Tripp Lite SMART1500SLT 1500VA 900W
I already had this UPS. It provides 8 outlets all of which are protected and have battery backup. I use it for the HP DL380 and the network switches. I am a little paranoid about noise, heat and dirty power. This unit scores high on only two of those since it is a little noisy due to the rear panel fan. It is also quite heavy. That said, all 8 outlets provide the same functionality so you don’t have to choose between just surge protection or surge and battery protection. Personally I wouldn’t buy another one. Instead I’d buy another CyberPower.
I bought this additional UPS because I didn’t want to overload the Tripp Lite unit. I really like these CyberPower units and have several of them. They are much smaller and lighter than the Tripp Lite. They also clean the power, are completely silent and have a small LCD screen which displays the voltage and battery load. Only half of the 10 outlets provide battery backup as well as surge protection, but that’s enough for the OVM Manager and shared storage filer computers plus the KVM and LCD screen.
I already had plenty of patch cables and a total of 6 are needed for this system. These come in packs of 5 and are very reasonably priced. You may as well get 2 packs of different lengths. That way you can choose the appropriate length for each connection and have some cables left over which you’re bound to use at some point.
I get this stuff from The Container Store and it is totally awesome. I already had most of it, but bought an additional wire shelf so I could position the HP DL380 at the right height for the box fan. The HP DL380 is obviously a rack mount server and since I don’t have a purpose built server rack, these shelves are a perfect alternative.
Product : Lasko 20” 3 Speed Box Fan
I already had this fan. Did I mention my paranoia about heating and cooling? You can often pick up these types of fan from your local grocery store or favorite DIY outlet for very little money. They’re not silent even on the lowest setting, but they do provide great air flow over quite a large area. They’re also incredibly cheap to run. Depending upon your system’s location and proximity to air conditioning, you may not even need it.
So there you have it, all the hardware you’ll need to build your own Oracle infrastructure.
As I mentioned earlier, you won’t necessarily have to use all the equipment I used as specified. Cheaper and reduced specification options are available, but hopefully this gives you some idea of what you’ll need to make everything work successfully.
It was a lot of fun putting this equipment together. Click on the thumbnail right to see what it all looks like when assembled. Nice!
In Part 2, we’ll cover all the software you’ll need to build your own Oracle infrastructure. Stay tuned for updates.
If you have any comments or questions about this post, please use the Contact form here.